Upload images with curl

[JimmyGulp]

Hi folks, I'm trying to work out how I can upload an image to a page using curl on the command line.

I'm attempting to migrate a system from a database (that needs decomissioning) to PW, and I've done a fair job of making something that will import data from the CSV exports using some perl to export the data, fix it up and then curl to import into the pages already created.

A lot of these pages have an image(s) associated with them, and I'm struggling to find a good answer on the best way to import the 4500 images. Currently I'm using curl to import the text data to the pages by sending POST requests to the PW edit page (they're all template based), but that doesn't seem to be the instant solution I was hoping for! 

I've used Chrome's inspector to try and understand how the AJAX upload works and see if I can use that some way but I've gotten a little lost in that.

As a final thought, if I can't upload directly in this fashion, if I were to host the images elsewhere, is there a way to have the template check for the local version of a file, if it's missing, use some hidden field (I could populate that with a URL per image) to extract an array of URLs, and fetch them so they'd all end up locally?

Thanks for any direction you can point me in!

[BitPoet]

I usually do stuff like that on a local copy of the target system where I can forego all the curl stuff and run a local PHP import script instead that uses PW's api. Then, when all that is done, I upload everything to the server.

[JimmyGulp]

Ah, good thinking, I'll have a look down that route and see where it leads me ?

[millipedia]

When we've needed to import images from an old CMS (looking at you WordPress) to a new PW installation then I've just used the API to add an image to a field by passing the URL of the image.

I normally just run a PHP script that looks something like this

<?php namespace ProcessWire;

/**
 * 
 * Import data from a CSV
 * 
 */

// Boostrap PW. 
// this example lives in an 'import' folder
include("../index.php");

echo 'in import <br><br>';

// read in data from your CSV file in the same folder
while ($row = $files->getCSV('your_export.csv')) {

  	// in this case we already had pages created in PW which we mapped to the old system
 	// using a field 'article_key' so we're using that here to find the right page.
  	// You can do something similar or you could create a new page here.
	$article_key=$row['article_key'];
    $p = $pages->get("template=article,article_key={$article_key}");

    if ($p->id) { // check we've got a page
      
      	echo '<br> Got page ID: ' . $p->id;

        // Check if this CSV row has a field containing our image path
        if ($row['article_pic'] !== '') {
			
          // in this case it was the path to a url on the old site (which obvs
          // still needs to be accessible online).
          $post_image = 'https://our-old-site.co.uk/uploads/' . $row['article_pic'];

          $p->setOutputFormatting(false);
            $p->featured_image->removeAll(); // remove any old version of the image if you want
            $p->featured_image->add($post_image); // add our new image
          $p->save();

          echo ' Added: ' . $row['article_pic'] . PHP_EOL;

        } else {
            echo ' No image';
        }
    }
}

This code is just cobbled together from things I've done in the past so you'll need to fix it your needs.

If you have 4,500 images then you'll want to run the import in batches rather than trying to do them all at once.

 

 

[JimmyGulp]

Thanks Millipedia, that looks to be a great way to do the import, thank you for the example, I really appreciate it!

[Jan Romero]

If you actually want to upload an image using curl, you can do this:

curl "https://example.com/my/page/upload" -F mypic=@"C:\temp\guy.brush"

-F tells curl to make a multipart/form-data request (the full switch is --form). “mypic” is the name of the POST parameter, so that’s what we’ll be looking for on our server.

ProcessWire code:

<?php namespace ProcessWire;

//return a bad status if anything weird happens
http_response_code(500);

//let’s pretend the POST request goes to the page to which we want to add the image
//here you would check if the request is allowed etc.
$page->of(false);

//use ProcessWire’s tempDir feature to generate a temporary directory which will receive the image
//the temporary directory will be destroyed after one hour
$upload_dir = files()->tempDir('', 3600);

//here’s the name “mypic” from the curl command again.
$u = new WireUpload('mypic');
//here you could put some more settings for WireUpload, such as setMaxFileSize() or setOverwrite()
$u->setMaxFiles(1);
$u->setDestinationPath($upload_dir);
$u->setValidExtensions([ 'jpg', 'jpeg', 'png', 'gif', 'webp', 'brush' ]);

//read the file from the request into $upload_dir and receive an array of filenames
$uploadedFiles = $u->execute();

//handle errors
$uploadErrors = $u->getErrors();
if ($uploadErrors)
    die(var_dump($uploadErrors));

//the image is still in the temp dir. If everything went well,
//we now need to move it to our page and save the page.
$path = $upload_dir . $uploadedFiles[0];
$pageimage = $page->images->add($path)->last();

//delete the file from the temporary directory
unlink($path);

$page->save();

http_response_code(200);
die('wow thanks for the pic, it’s very nice'); //this will show up in the curl command line

 

[JimmyGulp]

Many thanks again all for your direction and code snippets. Used @millipedia's code and expanded that, which also allowed me to import the other useful data in the data export to do a nice big single import (about 10minutes to do the whole thing on a local copy, the remote one via AWS is taking a little longer), and a heck of a lot faster than using curl and fishing out the TOKEN bits to work around the CSRF protection!

Now just to put something together that'll do the bits this script missed (pages hadn't been created at the time, etc etc), but I've got a good head start on where I was!