Import External Images

[Pretobrazza]

@Macrura 

Hi Marc,

I tried a different script as hereunder.  I tried save "body" as well as "textarea" (the field where the links are) but to no avail. The output gives me 3420 lines in the style of:

Saving child page: Akkerzieken in de Zwinstreek
Saving child page: 't Heistenaartje - Ode aan de zeedijk
Saving child page: 't Heistenaartje - Jacobus Landuyt
Saving child page: 't Heistenaartje - Mijn eerste vaart
etc etc

The script is:

$p = $pages->get(2356);
try {
    foreach($p->children as $c) {
        echo "Saving child page: " . $c->title . "<br>";
        $c->of(false);
        $c->save("textarea");
        if($c->isChanged()) {
            echo "Child page " . $c->title . " was saved.";
        }
    }
    echo "Children saved successfully!";
} catch (Exception $e) {
    echo $e->getMessage();
}
 

Kind regards,
Bernard

 

[Pretobrazza]

@Macrura 
Hi Marc,

Did you have time to think about a solution to import the pictures in bulk? If you want to, I can always test new ideas you might have. I'm more than willing to change the script in your module to test if it would work. You know, I've got a gut feeling that it has got to do with the hook that your module is only supposed to kick in after  $c->save("textarea");

Kind regards,
Bernard

[Macrura]

Hi @Pretobrazza - generally this always works, i haven't run into any issues, and it should still be working on many current sites. 

Have you confirmed that it's basically down to the hook not running? If you are using Tracy Debugger you can go into the module and check by putting a bd('test) into the method that is hooked on save (importExternalImages($event));

But if you are running your own script, you can also just use the module's API, by doing this:

$p = $pages->get(2356);
$m = $modules->get('ImportExternalImages');
try {
    foreach($p->children as $c) $m->importImages($c);
} catch (Exception $e) {
    echo $e->getMessage();
}

[Pretobrazza]

 

Hi @Macrura, 

I ran the script and I got this message:  Unknown image size '60%' (not in $config->imageSizes)

In 2020, I got the same problem. (Posted April 15, 2020).
Also, now, when I manually save an article which has pictures with a %,  the right or left alignment don't get taken into account.
At least things are getting more clear, not so? 
 

[Macrura]

@Pretobrazza -

thanks, so i found the issue regarding your error; there is some redundancy of code in this module between the api and textformatter which should be fixed at some point by creating another method for strictly processing images. In the meantime I added the check to prevent it processing the width attr if there is a percent, when using the module's api.

I don't know if this will fix the issue of the images not importing though, probably worth a try and see if you hit any other errors.

 

[Pretobrazza]

@Macrura, Indeed it's not always obvious to be in pace when the elements around are changing. I'm glad to have run into a problem and that the cause has been found. If things go wrong at my next trial, don't worry as I already deleted the content twice an re-imported it with ImportPagesCSV. 

So, what would you like me to try to import images in bulk? 

[Macrura]

@Pretobrazza i think you probably need to try one more time using the module api and see if it works; if that doesn't work then maybe you can post some of the markup that has the image references and i can try and test to see if it is caused by the attributes..

[Pretobrazza]

@Macrura 
Hello Marc,
Really, you're a champion ??

Anyone from a Joomla/Seblod content (and maybe others too) wishing to import content in bulk and being worried about the images, this is a lifesaver! 

This morning I updated PW to 3.0.212 and  ImportExternalImages to 2.0.14. In the script to import the pictures after I imported the articles and all their fields I needed with ImportPagesCSV,  I used the script hereunder (subject to be improved - Noobs will love you for this ? Maybe with a (total) img import counter as well at the end of the ride??) to monitor the process: 

<?
$p = $pages->get(2356);
$m = $modules->get('ImportExternalImages');
try {
    foreach($p->children as $c) {
	$c->of(false); 
		echo "Saving child page: " . $c->title . "<br>";
	$m->importImages($c);
	$c->save();
        if($c->isChanged()) {
	echo "Child page " . $c->title . " was saved.";}} // didn't get the echo's though
} catch (Exception $e) {
    echo $e->getMessage();
}
?>

At one point during impot, in one of the articles, the img had a typo mistake of  width="25ù" instead of width="25%" (Azerty keyboard key) and the import halted. When correcting the typo error in the article of which the title was reported thanks to the script, by refreshing the 'import' page the script went happily on importing pictures. At one point there was a timeout, but who cares ... I just refreshed  page again and the module went on doing what it was supposed to do. 

Upon inspection, I couldn't help to be astonished as even the alignment of the pictures as in the original articles was maintained. Really: Hats off! ?

Again, congratulations! Let this be a lifesaver to many people wishing to join the PW family!

Kind regards,
Bernard    

(temp overview https://kusttram.com/1325pw/overzicht-heemkundige-teksten/Pagina134 of a site in development without layout and temp pages - I will install dynamic search from @kongondo and from there seblod might be in the rear view mirror for many coming from the Joomla/Seblod setup - Here is the joomla/seblod combi I'm going to migrate from: https://www.zwinstreek.eu/geschiedenis/heemkundige-kringen/zoeken-in-publicaties?search=pdf&task=search  )

 

 

 

[Macrura]

I think if you want to use the isChanged() method, you first have to enable change tracking:

https://processwire.com/api/ref/wire/track-changes/

https://processwire.com/api/ref/wire/is-changed/

Sanitization added on all of the img attrs is something that would improve the module, thanks for the report...I'll try and add that soon.

In terms of timeouts, when you're doing your own PHP import scripts, you usually need to come up with a way to do it in batches, and use some condition to skip already processed pages, like if count($page->images), which would presumably be 0 if the page wasn't processed yet.

I seem to recall using this on at least 1 site that had content imported from Joomla...

[gebeer]

7 hours ago, Pretobrazza said:

and from there seblod might be in the rear view mirror

I moved from Joomla/Seblod to ProcessWire about 10 years ago. It was the best decision I ever made in my developer life. Happy coding with PW :-)

[Pretobrazza]

12 hours ago, gebeer said:

I moved from Joomla/Seblod to ProcessWire about 10 years ago. It was the best decision I ever made in my developer life. Happy coding with PW ?

@gebeerFor sure I remember you from those days. That's when I jumped over too. For sure I didn't make many sites (one for a soccer club and a shop) but I used PW especially at the time for a setup of touchscreens in a museum I work for. Nearly each room in that museum had a different screen layout being powered by one PW. Even though I'm not a good coder, I even managed to make streetview to look into the right direction in each of the POI's (from a setting in admin). The poi's on the left of the map are more documented but this only shows the power of PW and what it can do: https://durvenleven.be/pw/handelszaken/  In another room there was something like this:   https://durvenleven.be/pw/personen/

(truly sorry that I'm out of the tread but I couldn't help it)
     

[Matthieu]

Hello,

I'm trying to use your module, but I think I miss something...

I import first some pages after if I save manually each one all is fine but if I try to bulk save nothing append.

Third case, when I use the api way, the url of image in textarea are well updated but the "images" field stay empty.

Any idea how to clean this situation? Thanks

[Macrura]

On 8/30/2023 at 1:09 PM, Matthieu said:

Hello,

I'm trying to use your module, but I think I miss something...
I import first some pages after if I save manually each one all is fine but if I try to bulk save nothing append.

Third case, when I use the api way, the url of image in textarea are well updated but the "images" field stay empty.
Any idea how to clean this situation? Thanks

@Matthieu

Thanks for posting this question, and sorry for the delay – the module runs off of a ProcessWire hook, which is the "save" hook, and it only works when you are in the "ProcessPageEdit" process.

But, we already provided an api for this module, all you have to do is init the module and then use the "importImages()" method and it will work.

Sample Code:

$m = $modules->get("ImportExternalImages");
$p = $pages->get(/path/to/page); // or page ID
$m->importImages($p);

This also assumes you already setup the images and textarea field preferences in the module. After running that api call, the module will search the specified textarea field, analyze it for image URLS and attempt to import all of the found images into the specified image field.