Jump to content

[WIP] External storage for images and files (AWS, Dropbox etc.)


Sergio
 Share

Recommended Posts

Hi everyone,

Yesterday I began working on a module to create a filesystem abstraction for files and images using the Flysytem library (http://flysystem.thephpleague.com), out of the necessity of having the images copied (and probably linked) on Amazon S3 and other places like Dropbox. There two reasons why I decided to tackle this:

1 - When I work on the project in my machine, I need a way to automatically sync the images from the server and/or the external storage.

When an image is added on the server and the database is imported on my local env, PW shows a blank thumbnail of it. The idea for the module if to check if the page image has a width == 0 and if it exists on the server, add it to the local filesystem.

2 - In the past, I had to move a large website to a different server in a hurry and a lot of images were left behind (it was a mess). So I'm planning for a possible future worst-case scenario of the server exploding 🙂 

The code I quickly wrote is below (please bear with me that it's pretty raw at the moment). One thing I had to figure it out is why PW fires the Pageimage::size hook wherever a page is loaded in the admin, even though the thumbnails are already created. I was planning to save the image variations on S3 as well. Can someone clarify?

I know that @LostKobrakai was working on a similar project, and so I would like to ask him and everyone else if you think I (and who may help) should evolve this idea into a full-featured module where the user can select which server (adapter) to use (AWS, Digital Ocean spaces, Dropbox etc.)

<?php namespace ProcessWire;

use Aws\S3\S3Client;
use League\Flysystem\AwsS3v3\AwsS3Adapter;
use League\Flysystem\Filesystem;
use League\Flysystem\Adapter\Local;

use Spatie\Dropbox\Client;
use Spatie\FlysystemDropbox\DropboxAdapter;

class ProcessFly extends WireData implements Module, ConfigurableModule {

	public static function getModuleInfo() {

		return array(
			'title' => 'ProcessWire Flysystem Integration', 
			'version' => 001, 
            'summary' => 'Synchronize all the page assets uploaded through PW to a specified bucket in Amazon S3 and...',
            'author' => 'Sérgio Jardim',
			'singular' => true, 
            'autoload' => true, 
            'icon' => 'image'
			);
	}

    public function init() {

        $this->client = S3Client::factory([
            'credentials' => [
                'key'    => '',
                'secret' => '',
            ],
            'region' => 'sa-east-1',
            'version' => 'latest',
        ]);
        
        $this->bucket_name = "";
        $this->dir_name = "images";
        $this->s3_adapter = new AwsS3Adapter($this->client, $this->bucket_name, $this->dir_name);

        $this->s3_filesystem = new Filesystem($this->s3_adapter);


        // DROPBOX

        $this->authorizationToken = "";
        $this->dropbox_client = new Client($this->authorizationToken);
        $this->adapter_dropbox = new DropboxAdapter($this->dropbox_client);
        $this->dropbox_filesystem = new Filesystem($this->adapter_dropbox);

        $this->addHookAfter('Pages::saved', $this, 'checkImageS3');	// Download images that are not in local filesystem for whatever reason but are available in the remove one.

        $this->addHookAfter('InputfieldFile::fileAdded', $this, 'uploadImageS3');	// Fired when a file/image is added to a page in the admin

   
		
        // $this->addHookAfter('Pageimage::size', $this, 'uploadImageS3'); // Fired when a file is resized via API. Note: this hook is also called when the page is loaded in Admin. Need to know why.
                
        $this->addHookAfter('Pageimage::url', $this, 'redirectImageURL'); //Replace image URL for the S3 path
        
     	
    }

    public function redirectImageURL($event){
		if($event->page->template == 'admin') return;
		else
			$event->return = "https://s3-sa-east-1.amazonaws.com/[bucket name]/images/" .  $event->object->page . "/" . $event->object->name;
    }
    

    // UPLOAD

    public function uploadImageS3($event){

		if(count($event->return)) { //if it is a image resize event get image variation data
			$file = $event->return; 
		} else {
			$file = $event->arguments(0); 	
		}
		$filename = $file->name;
		$filename_original = $file->url;
        
        $pathToFile = $file->page . "/" . $filename;

        $system_path = $file->filename();
       
        
        try{
            
            $file_on_s3 = $this->s3_filesystem->has($pathToFile); //check if file exists on AWS
            if(!$file_on_s3) {
                $contents = file_get_contents($system_path);
                $this->s3_filesystem->put($pathToFile, $contents, array(
                    'visibility' => 'public',
                )); //upload file with the same folder structure as in assets/files: page_id/file_name

                //Also add a copy to Dropbox (OPTIONAL)
                $this->dropbox_filesystem->put($pathToFile, $contents);
            }
            
		} 
		catch (Exception $e){
			throw new WireException("Error: Image not Added to S3: {$e->getMessage()}");  
        }
      
		
    }

    public function checkImageS3($event){
        $page = $event->arguments(0);
        
		if(count($page->images)) { 
			foreach($page->images as $file) {
                if($file->width == 0) {
                    return $this->downloadImageS3($page, $file);
                };
            }
        }
		
    }

    public function downloadImageS3($page, $file){

        
        $pathToFile = $page->id . "/" . $file->name;
       
            
        $file_on_s3 = $this->s3_filesystem->has($pathToFile); //check if file exists on AWS
        if($file_on_s3) {
            $image = "https://s3-sa-east-1.amazonaws.com/[bucket name]/images/" . $pathToFile;
            // $page->images->remove($image);
            $page->images->add($image);
            $page->save();
        } else {
            throw new WireException("Error: Image not found on S3: {$file->name}");  
        }
       
		
    }

}

 

  • Like 7
Link to comment
Share on other sites

13 hours ago, Sergio said:

One thing I had to figure it out is why PW fires the Pageimage::size hook wherever a page is loaded in the admin, even though the thumbnails are already created. I was planning to save the image variations on S3 as well. Can someone clarify?

If I'm not totally wrong, Pageimage->size (->width, ->height) checks if the image-variation is available, and / or if there are options present that force a recreation. The creation of variations is done in image rendering engines, that will be started from PageImage->size only if there is no variation available, or if it needs to be recreated, (forced by global or individual options).

There must be a flag availabe for pageimages called something like isModified. Maybe this can help? If you need further support on PageImage and getting out informations or tweak its behave to suite your needs, please tell me and I will dig into it. 🙂

  • Like 1
Link to comment
Share on other sites

9 hours ago, horst said:

There must be a flag availabe for pageimages called something like isModified. Maybe this can help? If you need further support on PageImage and getting out informations or tweak its behave to suite your needs, please tell me and I will dig into it.

I will check on that flag, @horst ! Thank you for clarifying and to offer your help. 😃

  • Like 1
Link to comment
Share on other sites

  • 1 year later...
 Share

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      Select Images
      An inputfield that allows the visual selection and sorting of images, intended for use with the FieldtypeDynamicOptions module. Together these modules can be used to create a kind of "image reference" field.

      Integration with FieldtypeDynamicOptions
      InputfieldSelectImages was developed to be used together with FieldtypeDynamicOptions (v0.1.3 or newer):
      Create a Dynamic Options field. Choose "Select Images" as the "Inputfield type". Select Images appears in the "Multiple item selection" category but you can set "Maximum number of items" to 1 if you want to use Select Images for single image selections. Define selectable options for the field via a FieldtypeDynamicOptions::getSelectableOptions hook. See some examples below. FieldtypeDynamicOptions is recommended but is not a strict requirement for installing InputfieldSelectImages in case you want to use an alternative way to store the field data.
      Selection of Pageimages
      In this example the field allows selection of Pageimages that are in the "images" field of the home page.
      The field will store URLs to the Pageimages so it works as a kind of "image reference" field. You can use the "Format as Pagefile/Pageimage object(s)" option for the Dynamic Options field to have the formatted value of the field be automatically converted from the stored Pageimage URLs to Pageimage objects.
      $wire->addHookAfter('FieldtypeDynamicOptions::getSelectableOptions', function(HookEvent $event) { // The page being edited $page = $event->arguments(0); // The Dynamic Options field $field = $event->arguments(1); // For a field named "select_images" if($field->name === 'select_images') { $options = []; // Get Pageimages within the "images" field on the home page foreach($event->wire()->pages(1)->images as $image) { // Add an option for each Pageimage // When the key is a Pageimage URL the inputfield will automatically create a thumbnail // In this example the label includes the basename and the filesize /** @var Pageimage $image */ $options[$image->url] = "{$image->basename}<br>{$image->filesizeStr}"; } $event->return = $options; } }); Selection of image files not associated with a Page
      When not working with Pageimages you must add a "data-thumb" attribute for each selectable option which contains a URL to a thumbnail/image.
      In this example the field allows selection of image files in a "/pics/" folder which is in the site root.
      $wire->addHookAfter('FieldtypeDynamicOptions::getSelectableOptions', function(HookEvent $event) { // The page being edited $page = $event->arguments(0); // The Dynamic Options field $field = $event->arguments(1); // For a field named "select_images" if($field->name === 'select_images') { $options = []; // Get files that are in the /pics/ folder $root = $event->wire()->config->paths->root; $path = $root . 'pics/'; $files = $event->wire()->files->find($path); // Add an option for each file foreach($files as $file) { $basename = str_replace($path, '', $file); $url = str_replace($root, '/', $file); // The value must be an array with the following structure... $options[$url] = [ // The label for the image 'label' => $basename, 'attributes' => [ // An image URL in the "data-thumb" attribute 'data-thumb' => $url, ], ]; } $event->return = $options; } }); The field values don't have to be image URLs
      The values stored by the Dynamic Options field don't have to be image URLs. For example, you could use the images to represent different layout options for a page, or to represent widgets that will be inserted on the page.
      Also, you can use external URLs for the thumbnails. In the example below the options "calm" and "crazy" are represented by thumbnails from placecage.com.
      $wire->addHookAfter('FieldtypeDynamicOptions::getSelectableOptions', function(HookEvent $event) { // The page being edited $page = $event->arguments(0); // The Dynamic Options field $field = $event->arguments(1); // For a field named "calm_or_crazy" if($field->name === 'calm_or_crazy') { $options = []; // Add options that are illustrated with thumbnails from placecage.com $options['calm'] = [ // The label for the option 'label' => 'Nicolas Cage is a calm man', 'attributes' => [ // An image URL in the "data-thumb" attribute 'data-thumb' => 'https://www.placecage.com/260/260', ] ]; $options['crazy'] = [ // The label for the option 'label' => 'Nicolas Cage is a crazy man', 'attributes' => [ // An image URL in the "data-thumb" attribute 'data-thumb' => 'https://www.placecage.com/c/260/260', ] ]; $event->return = $options; } }); Field configuration
      You can define labels for the button, notices, etc, that are used within the inputfield if the defaults don't suit.

       
      https://github.com/Toutouwai/InputfieldSelectImages
      https://processwire.com/modules/inputfield-select-images/
    • By Robin S
      Process Images
      A basic, proof-of-concept Textformatter module for ProcessWire. When the Textformatter is applied to a rich text field it uses Simple HTML DOM to find <img> tags in the field value and passes each img node through a hookable TextformatterProcessImages::processImg() method.
      This is a very simple module that doesn't have any configurable settings and doesn't do anything to the field value unless you hook the TextformatterProcessImages::processImg() method.
      Hook example
      When added to /site/ready.php the hook below will replace any Pageimages in a rich text field with a 250px square variation and wrap the <img> tag in a link to the original full-size image.
      For help with Simple HTML DOM refer to its documentation.
      $wire->addHookAfter('TextformatterProcessImages::processImg', function(HookEvent $event) { // The Simple HTML DOM node for the <img> tag /** @var \simple_html_dom_node $img */ $img = $event->arguments(0); // The Pageimage in the <img> src, if any (will be null for external images) /** @var Pageimage $pageimage */ $pageimage = $event->arguments(1); // The Page object in case you need it /** @var Page $page */ $page = $event->arguments(2); // The Field object in case you need it /** @var Field $field */ $field = $event->arguments(3); // Only for images that have a src corresponding to a PW Pageimage if($pageimage) { // Set the src to a 250x250 variation $img->src = $pageimage->size(250,250)->url; // Wrap the img in a lightbox link to the original $img->outertext = "<a class='lightboxclass' href='{$pageimage->url}'>{$img->outertext}</a>"; } });  
      GitHub: https://github.com/Toutouwai/TextformatterProcessImages
      Modules directory: https://processwire.com/modules/textformatter-process-images/
    • By spercy16
      These issues should be fairly easy for any intermediate to advanced ProcessWire developer to answer. I'm new to PHP and relatively new to ProcessWire and just need a bit of help. What I'm trying to do is bring in a couple of cards from my Projects page to display on my home page. I finally got the code right to bring in the cards but right now they're using my original images instead of my resized "variations". So firstly, I would like to know how to reference the variations of my images instead of using the original. Secondly, I need to grab only four of the cards from the Project page and not import in all ten. It should be just two small changes to my code to do these things (I would imagine). Here is the code I currently have for that section:
      <?php // https://processwire.com/api/arrays/ // check if the array of images has items if (count($pages->get("/projects/")->images)) : // get array of images from the field $images = $pages->get("/projects/")->images; $count = 0; // iterate over each one foreach ($images as $image) : $count++; $sectionText = $pages->get("/projects/")->get("paragraph_$count"); $img = $image; $buttonCode = $pages->get("/projects/")->get("url_$count"); ?> <span id="card<?php echo $count?>" class="card"> <img class="cardThumb" src="<?php echo $img->url; ?>" alt="<?php echo $image->description; ?> Thumbnail" /> <div class="cardBody"> <div class="cardText"> <h2><?php echo $img->description; ?></h2> <?php echo $sectionText; ?> </div> <div class="primaryBtn"> <a href="https://www.paypal.com/donate?hosted_button_id= <?php echo $buttonCode; ?> &source=url"> <button> <i class="fas fa-donate"></i> Donate </button> </a> </div> </div> </span> <?php endforeach; endif; ?> Thanks in advance for any help!
    • By skeltern
      WebP image support is great and works fine. But once created I've issues to get rid of all API generated WebP variations.
      The backend image field variations "Delete" works and I can remove all variations JPEG plus WebP. Image list is clean but all WebP API variations are still stored in file system (for instance files/12345/84.900x675.webp etc). I can only use ImageSizer with temp 'force' option to request fresh WebP variations or have to delete WebP files from folders. No other way so far. Tested with 2 sites and latest master PW 3.0.165.
      Is there somewhere a "magic button" or config/setup thing to solve my sticky WebP issue?
    • By psy
      I've searched, and maybe missed, the solution. I have a 'normal' images field and uploading images with a file size greater than 10Kb is fine. Any image size smaller results in the never-ending spinner and no upload.
      No min/max width/height set on image uploads in admin, ie just the defaults.
      Any ideas on how to fix?
      Using:
      PW: 3.0.175
      PHP: 7.3
      Marking it as "Resolved" rather than "Solved" as it auto-magically fixed itself. No idea whether PW, PHP, or just an internet hiccough... All good now 🤞
×
×
  • Create New...