[WIP] External storage for images and files (AWS, Dropbox etc.)

[Sergio]

Hi everyone,

Yesterday I began working on a module to create a filesystem abstraction for files and images using the Flysytem library (http://flysystem.thephpleague.com), out of the necessity of having the images copied (and probably linked) on Amazon S3 and other places like Dropbox. There two reasons why I decided to tackle this:

1 - When I work on the project on my machine, I need a way to automatically sync the images from the server and/or the external storage.

When an image is added on the server and the database is imported on my local env, PW shows a blank thumbnail of it. The idea for the module if to check if the page image has a width == 0 and if it exists on the server, add it to the local filesystem.

2 - In the past, I had to move a large website to a different server in a hurry and a lot of images were left behind (it was a mess). So I'm planning for a possible future worst-case scenario of the server exploding ? 

The code I quickly wrote is below (please bear with me that it's pretty raw at the moment). One thing I had to figure it out is why PW fires the Pageimage::size hook wherever a page is loaded in the admin, even though the thumbnails are already created. I was planning to save the image variations on S3 as well. Can someone clarify?

I know that @LostKobrakai was working on a similar project, and so I would like to ask him and everyone else if you think I (and who may help) should evolve this idea into a full-featured module where the user can select which server (adapter) to use (AWS, Digital Ocean spaces, Dropbox etc.)

<?php namespace ProcessWire;

use Aws\S3\S3Client;
use League\Flysystem\AwsS3v3\AwsS3Adapter;
use League\Flysystem\Filesystem;
use League\Flysystem\Adapter\Local;

use Spatie\Dropbox\Client;
use Spatie\FlysystemDropbox\DropboxAdapter;

class ProcessFly extends WireData implements Module, ConfigurableModule {

	public static function getModuleInfo() {

		return array(
			'title' => 'ProcessWire Flysystem Integration', 
			'version' => 001, 
            'summary' => 'Synchronize all the page assets uploaded through PW to a specified bucket in Amazon S3 and...',
            'author' => 'Sérgio Jardim',
			'singular' => true, 
            'autoload' => true, 
            'icon' => 'image'
			);
	}

    public function init() {

        $this->client = S3Client::factory([
            'credentials' => [
                'key'    => '',
                'secret' => '',
            ],
            'region' => 'sa-east-1',
            'version' => 'latest',
        ]);
        
        $this->bucket_name = "";
        $this->dir_name = "images";
        $this->s3_adapter = new AwsS3Adapter($this->client, $this->bucket_name, $this->dir_name);

        $this->s3_filesystem = new Filesystem($this->s3_adapter);


        // DROPBOX

        $this->authorizationToken = "";
        $this->dropbox_client = new Client($this->authorizationToken);
        $this->adapter_dropbox = new DropboxAdapter($this->dropbox_client);
        $this->dropbox_filesystem = new Filesystem($this->adapter_dropbox);

        $this->addHookAfter('Pages::saved', $this, 'checkImageS3');	// Download images that are not in local filesystem for whatever reason but are available in the remove one.

        $this->addHookAfter('InputfieldFile::fileAdded', $this, 'uploadImageS3');	// Fired when a file/image is added to a page in the admin

   
		
        // $this->addHookAfter('Pageimage::size', $this, 'uploadImageS3'); // Fired when a file is resized via API. Note: this hook is also called when the page is loaded in Admin. Need to know why.
                
        $this->addHookAfter('Pageimage::url', $this, 'redirectImageURL'); //Replace image URL for the S3 path
        
     	
    }

    public function redirectImageURL($event){
		if($event->page->template == 'admin') return;
		else
			$event->return = "https://s3-sa-east-1.amazonaws.com/[bucket name]/images/" .  $event->object->page . "/" . $event->object->name;
    }
    

    // UPLOAD

    public function uploadImageS3($event){

		if(count($event->return)) { //if it is a image resize event get image variation data
			$file = $event->return; 
		} else {
			$file = $event->arguments(0); 	
		}
		$filename = $file->name;
		$filename_original = $file->url;
        
        $pathToFile = $file->page . "/" . $filename;

        $system_path = $file->filename();
       
        
        try{
            
            $file_on_s3 = $this->s3_filesystem->has($pathToFile); //check if file exists on AWS
            if(!$file_on_s3) {
                $contents = file_get_contents($system_path);
                $this->s3_filesystem->put($pathToFile, $contents, array(
                    'visibility' => 'public',
                )); //upload file with the same folder structure as in assets/files: page_id/file_name

                //Also add a copy to Dropbox (OPTIONAL)
                $this->dropbox_filesystem->put($pathToFile, $contents);
            }
            
		} 
		catch (Exception $e){
			throw new WireException("Error: Image not Added to S3: {$e->getMessage()}");  
        }
      
		
    }

    public function checkImageS3($event){
        $page = $event->arguments(0);
        
		if(count($page->images)) { 
			foreach($page->images as $file) {
                if($file->width == 0) {
                    return $this->downloadImageS3($page, $file);
                };
            }
        }
		
    }

    public function downloadImageS3($page, $file){

        
        $pathToFile = $page->id . "/" . $file->name;
       
            
        $file_on_s3 = $this->s3_filesystem->has($pathToFile); //check if file exists on AWS
        if($file_on_s3) {
            $image = "https://s3-sa-east-1.amazonaws.com/[bucket name]/images/" . $pathToFile;
            // $page->images->remove($image);
            $page->images->add($image);
            $page->save();
        } else {
            throw new WireException("Error: Image not found on S3: {$file->name}");  
        }
       
		
    }

}

 

[horst]

13 hours ago, Sergio said:

One thing I had to figure it out is why PW fires the Pageimage::size hook wherever a page is loaded in the admin, even though the thumbnails are already created. I was planning to save the image variations on S3 as well. Can someone clarify?

If I'm not totally wrong, Pageimage->size (->width, ->height) checks if the image-variation is available, and / or if there are options present that force a recreation. The creation of variations is done in image rendering engines, that will be started from PageImage->size only if there is no variation available, or if it needs to be recreated, (forced by global or individual options).

There must be a flag availabe for pageimages called something like isModified. Maybe this can help? If you need further support on PageImage and getting out informations or tweak its behave to suite your needs, please tell me and I will dig into it. ?

[Sergio]

9 hours ago, horst said:

There must be a flag availabe for pageimages called something like isModified. Maybe this can help? If you need further support on PageImage and getting out informations or tweak its behave to suite your needs, please tell me and I will dig into it.

I will check on that flag, @horst ! Thank you for clarifying and to offer your help. ?

[Youbility]

@Sergio Coming across this, did you ever have the chance to finish the FlySystem adapter? Would be incredibly useful to be able to read from and write to cloud storage.

[Sergio]

Hi @Youbility, yes I did. It's working on production at https://www.brightline.org/

Here's a slightly different version of that one: https://gist.github.com/sjardim/d74fae71b5bfe6a44ab88efc9aaa5279

Hope that helps!

[joer80]

Do the files stay on the server or go to cloud instead?

[Sergio]

On 7/25/2023 at 4:22 AM, joer80 said:

Do the files stay on the server or go to cloud instead?

Both. ? In the end we didn't need to link to the copied files on S3, kept the local ones. S3 works as backup only in this case.