How to upload heavy data into Processwire?

[Vineet Sawant 76]

Hi,

I'm trying to import some heavy data into Processwire, but I'm not sure what would be the best way to do it.

Usually I use CSV to Pages plugin, but this time the data is too heavy(~40k rows with 10+ columns of excel sheet), thus this plugin can't help.

I also tried Tasker plugin but I can't seem to go through the setup itself, it requires some template setup but I'm totally clueless about how to do it, so that plugin is not of any use either.

I wanted to know from you guys how you do it and in future what would be the best way to migrate thousands of rows of data in to PW.

 

Thanks.

 

 

[flydev 👊🏻 1,861]

Check this recent thread, you will find the answer :

 

 

 

 

 

[AndZyk 706]

I also would recommend to write your own shell script and bootstrap ProcessWire.

You could use for example the PHP League CSV composer package and write your own import script where you save the CSV entries as pages via the API. 😉

It is not that hard and you can import large data this way. If you want to, I could post an example.

Regards, Andreas

[Vineet Sawant 76]

Thank you both.

@AndZyk It'd be very kind of you if you can post an example.

Thanks a ton 🙂

[AndZyk 706]

Here is a script I made for the import of thousands of customers.

You have to save this as shell script (f. e. sync-customers.php.sh), make the script executable and execute it via command line (./sync-customers.php.sh).

Spoiler

#!/usr/bin/env php
<?php
namespace {
	// Bootstrap ProcessWire
	include("./../../index.php");
}

namespace ProcessWire {

	echo "Synchronisation started...\n";

	// Source: http://csv.thephpleague.com/
	use League\Csv\Reader;

	$csv = Reader::createFromPath("./../assets/csv/customer.csv", "r");
	$csv->setDelimiter(";");
	$csv->setHeaderOffset(0);

	// $header = $csv->getHeader();
	$records = $csv->getRecords();

	// var_dump($header);
	// var_dump($records);

	/*
	 * Save records in new array
	 */
	$recordsArr = array();

	foreach ($records as $record) {

		// Save columns in variables
		$supplierID = 				$record["SupplierID"];
		$customername1 = 			$record["Customername1"];
		$street = 					$record["Street"];
		$postcode = 				$record["Postcode"];
		$city = 					$record["City"];
		$country = 					$record["Country"];
		/*
		$customername2 = 			$record["Customername2"];
		$email1 = 					$record["Email1"];
		$email2 = 					$record["Email2"];
		$additionalInformation = 	$record["AdditionalInformation"];
		$fieldworker = 				$record["Fieldworker"];
		$indoorservice = 			$record["Indoorservice"];
		$webseite = 				$record["IF:Webseite"];
		$verband = 					$record["IF:Verband"];
		$segment = 					$record["IF:Segment"];
		$unternehmenskette = 		$record["IF:Unternehmenskette"];
		*/

		$recordsArr[] = array(
			"supplierID" => 	$supplierID,
			"customername1" => 	$customername1,
			"street" => 		$street,
			"postcode" => 		$postcode,
			"city" => 			$city,
			"country" => 		$country
		);

	}

	// Remove duplicates
	$recordsArr = array_map("unserialize", array_unique(array_map("serialize", $recordsArr)));

	// var_dump($recordsArr);
	// echo count($recordsArr) . ".\n";

	// Get customers
	$customersPage = pages()->get("template=customers");
	$customers = pages()->find("parent=$customersPage, template=customer");

	/*
	 * Delete customers
	 */
	/*
	foreach ($customers as $customer) {

		$log->save("customers", "Customer " . $customer->title . " deleted.");
		echo("Customer " . $customer->title . " deleted.\n");

		pages()->delete($customer);

	}
	*/

	/*
	 * Create or update customers
	 */
	foreach ($recordsArr as $r => $record) {

		// Save columns in variables
		$supplierID = 				$record["supplierID"];
		$customername1 = 			$record["customername1"];
		$street = 					$record["street"];
		$postcode = 				$record["postcode"];
		$city = 					$record["city"];
		$country = 					$record["country"];

        $customersPage = pages()->get("template=customers");

		// Create customer
		if (!$customers->has("title=$supplierID")) {

			$customer = new Page();
			$customer->parent = $customersPage;
			$customer->template = "customer";
			$customer->title = $supplierID;

			$customer->of(false);

			$customer->save();

			$customer->set("customer_name", $customername1);
			$customer->set("customer_postal_code", $postcode);
			$customer->set("customer_city", $city);

			// Create distribution country if it doesnt exist
			$distributionCountry = pages()->get("title=$country, template=distribution-country");

			if (!$distributionCountry->id) {

				$distributionCountry = new Page();
				$distributionCountry->parent = pages()->get("template=distribution-countries");
				$distributionCountry->template = "distribution-country";
				$distributionCountry->title = $country;

				$distributionCountry->of(false);

				$distributionCountry->save();

				$log->save("distribution-countries", "Distribution country " . $distributionCountry->title . " created.");
				echo("Distribution country " . $distributionCountry->title . " created.\n");

			}

			$customer->set("customer_distribution_country", $country);

			$customer->save();

			$log->save("customers", "Customer " . $customer->title . " created.");
			echo("Customer " . $customer->title . " created.\n");

		// Update customer
		} else {

			// Get customer
			$customer = $customers->get("parent=$customersPage, title=$supplierID, template=customer");

			if (($customer->customer_name !== $customername1) ||
				($customer->customer_postal_code !== $postcode) ||
				($customer->customer_city !== $city) ||
				((string)$customer->customer_distribution_country->title !== $country)) {



				$customer->of(false);

				$customer->set("customer_name", $customername1);
				$customer->set("customer_postal_code", $postcode);
				$customer->set("customer_city", $city);
				$customer->set("customer_distribution_country", $country);

				$customer->save();

				$log->save("customers", "Customer " . $customer->title . " updated.");
				echo("Customer " . $customer->title . " updated.\n");

			}

		}

	}

	/*
	 * Delete leftover customers
	 */
	$savedCustomersArr = array();
	$customersArr = array();

	foreach ($customers as $customer) {

		$savedCustomersArr[] = $customer->title->getLanguageValue("default");

	}

	foreach ($records as $record) {

		// Save columns in variables
		$supplierID = $record["SupplierID"];

		$customersArr[] = $supplierID;

	}

	$deletedCustomersArr = array_diff($savedCustomersArr, $customersArr);
	$deletedCustomersArr = array_unique($deletedCustomersArr);

	// var_dump($savedCustomersArr);
	// var_dump($customersArr);
	// var_dump($deletedCustomersArr);

	foreach ($deletedCustomersArr as $deletedCustomer) {

		$customer = pages()->findOne("parent=$customersPage, title=$deletedCustomer, template=customer");

		$log->save("customers", "Customer " . $deletedCustomer . " deleted.");
		echo("Customer " . $deletedCustomer . " deleted.\n");

		pages()->delete($customer);

	}

	echo "Synchronisation finished...\n";

}