Jump to content

How to upload heavy data into Processwire?


Vineet Sawant
 Share

Recommended Posts

Hi,

I'm trying to import some heavy data into Processwire, but I'm not sure what would be the best way to do it.

Usually I use CSV to Pages plugin, but this time the data is too heavy(~40k rows with 10+ columns of excel sheet), thus this plugin can't help.

I also tried Tasker plugin but I can't seem to go through the setup itself, it requires some template setup but I'm totally clueless about how to do it, so that plugin is not of any use either.

I wanted to know from you guys how you do it and in future what would be the best way to migrate thousands of rows of data in to PW.

 

Thanks.

 

 

Link to comment
Share on other sites

I also would recommend to write your own shell script and bootstrap ProcessWire.

You could use for example the PHP League CSV composer package and write your own import script where you save the CSV entries as pages via the API. ?

It is not that hard and you can import large data this way. If you want to, I could post an example.

Regards, Andreas

Link to comment
Share on other sites

Here is a script I made for the import of thousands of customers.

You have to save this as shell script (f. e. sync-customers.php.sh), make the script executable and execute it via command line (./sync-customers.php.sh).

Spoiler

#!/usr/bin/env php
<?php
namespace {
	// Bootstrap ProcessWire
	include("./../../index.php");
}

namespace ProcessWire {

	echo "Synchronisation started...\n";

	// Source: http://csv.thephpleague.com/
	use League\Csv\Reader;

	$csv = Reader::createFromPath("./../assets/csv/customer.csv", "r");
	$csv->setDelimiter(";");
	$csv->setHeaderOffset(0);

	// $header = $csv->getHeader();
	$records = $csv->getRecords();

	// var_dump($header);
	// var_dump($records);

	/*
	 * Save records in new array
	 */
	$recordsArr = array();

	foreach ($records as $record) {

		// Save columns in variables
		$supplierID = 				$record["SupplierID"];
		$customername1 = 			$record["Customername1"];
		$street = 					$record["Street"];
		$postcode = 				$record["Postcode"];
		$city = 					$record["City"];
		$country = 					$record["Country"];
		/*
		$customername2 = 			$record["Customername2"];
		$email1 = 					$record["Email1"];
		$email2 = 					$record["Email2"];
		$additionalInformation = 	$record["AdditionalInformation"];
		$fieldworker = 				$record["Fieldworker"];
		$indoorservice = 			$record["Indoorservice"];
		$webseite = 				$record["IF:Webseite"];
		$verband = 					$record["IF:Verband"];
		$segment = 					$record["IF:Segment"];
		$unternehmenskette = 		$record["IF:Unternehmenskette"];
		*/

		$recordsArr[] = array(
			"supplierID" => 	$supplierID,
			"customername1" => 	$customername1,
			"street" => 		$street,
			"postcode" => 		$postcode,
			"city" => 			$city,
			"country" => 		$country
		);

	}

	// Remove duplicates
	$recordsArr = array_map("unserialize", array_unique(array_map("serialize", $recordsArr)));

	// var_dump($recordsArr);
	// echo count($recordsArr) . ".\n";

	// Get customers
	$customersPage = pages()->get("template=customers");
	$customers = pages()->find("parent=$customersPage, template=customer");

	/*
	 * Delete customers
	 */
	/*
	foreach ($customers as $customer) {

		$log->save("customers", "Customer " . $customer->title . " deleted.");
		echo("Customer " . $customer->title . " deleted.\n");

		pages()->delete($customer);

	}
	*/

	/*
	 * Create or update customers
	 */
	foreach ($recordsArr as $r => $record) {

		// Save columns in variables
		$supplierID = 				$record["supplierID"];
		$customername1 = 			$record["customername1"];
		$street = 					$record["street"];
		$postcode = 				$record["postcode"];
		$city = 					$record["city"];
		$country = 					$record["country"];

        $customersPage = pages()->get("template=customers");

		// Create customer
		if (!$customers->has("title=$supplierID")) {

			$customer = new Page();
			$customer->parent = $customersPage;
			$customer->template = "customer";
			$customer->title = $supplierID;

			$customer->of(false);

			$customer->save();

			$customer->set("customer_name", $customername1);
			$customer->set("customer_postal_code", $postcode);
			$customer->set("customer_city", $city);

			// Create distribution country if it doesnt exist
			$distributionCountry = pages()->get("title=$country, template=distribution-country");

			if (!$distributionCountry->id) {

				$distributionCountry = new Page();
				$distributionCountry->parent = pages()->get("template=distribution-countries");
				$distributionCountry->template = "distribution-country";
				$distributionCountry->title = $country;

				$distributionCountry->of(false);

				$distributionCountry->save();

				$log->save("distribution-countries", "Distribution country " . $distributionCountry->title . " created.");
				echo("Distribution country " . $distributionCountry->title . " created.\n");

			}

			$customer->set("customer_distribution_country", $country);

			$customer->save();

			$log->save("customers", "Customer " . $customer->title . " created.");
			echo("Customer " . $customer->title . " created.\n");

		// Update customer
		} else {

			// Get customer
			$customer = $customers->get("parent=$customersPage, title=$supplierID, template=customer");

			if (($customer->customer_name !== $customername1) ||
				($customer->customer_postal_code !== $postcode) ||
				($customer->customer_city !== $city) ||
				((string)$customer->customer_distribution_country->title !== $country)) {



				$customer->of(false);

				$customer->set("customer_name", $customername1);
				$customer->set("customer_postal_code", $postcode);
				$customer->set("customer_city", $city);
				$customer->set("customer_distribution_country", $country);

				$customer->save();

				$log->save("customers", "Customer " . $customer->title . " updated.");
				echo("Customer " . $customer->title . " updated.\n");

			}

		}

	}

	/*
	 * Delete leftover customers
	 */
	$savedCustomersArr = array();
	$customersArr = array();

	foreach ($customers as $customer) {

		$savedCustomersArr[] = $customer->title->getLanguageValue("default");

	}

	foreach ($records as $record) {

		// Save columns in variables
		$supplierID = $record["SupplierID"];

		$customersArr[] = $supplierID;

	}

	$deletedCustomersArr = array_diff($savedCustomersArr, $customersArr);
	$deletedCustomersArr = array_unique($deletedCustomersArr);

	// var_dump($savedCustomersArr);
	// var_dump($customersArr);
	// var_dump($deletedCustomersArr);

	foreach ($deletedCustomersArr as $deletedCustomer) {

		$customer = pages()->findOne("parent=$customersPage, title=$deletedCustomer, template=customer");

		$log->save("customers", "Customer " . $deletedCustomer . " deleted.");
		echo("Customer " . $deletedCustomer . " deleted.\n");

		pages()->delete($customer);

	}

	echo "Synchronisation finished...\n";

}

 

 

  • Like 4
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...