Save Files in Amazon S3 / Cloud?

[vanderbreye]

Hej!

Has anyone ever tried to save & load the files in / from the S3 instead / beside of the local filesystem ?

by using http://framework.zend.com/manual/de/zend.cloud.html it is very esay to implement to save/load:

http://blog.ebene7.com/2011/01/21/amazon-s3-mit-php-stream-wrapper-verwenden/

- but will PW work with it?

By module? Or even better: a plugin that automatically saves/serves any files handled by the backend..

We will now try to build this, but we'll also happy about any thoughts about it!

[Marty Walker]

Hi vanderbreye,

I'd love to see something like this is PW. I've used one or two add-ons for ExpressionEngine that do similar. Are you, by chance, planning any features that allow users to only have access to the files they've uploaded?

Regards

Marty

[Michael Murphy]

Hi vanderbreye,

I like the idea - I've always wanted the ability to use dropbox for site assets, as it would be a much smoother workflow especially when working with others (eg. use a shared folder and eliminate the need to use the web interface to update files).

@diogo mentioned this interesting service a while ago which might be worth investigating

https://www.filepicker.io/

not sure if it works both ways - I think it just gives you a simple interface to the files which would then be uploaded to your server

[diogo]

@diogio mentioned this interesting service a while ago which might be worth investigating

@diogo

[vanderbreye]

Ok, it is simple to implement the S3 directly in the core upload

BUT: how should i begin to write the module, which binds on the uploadprocess?

It is important to have file+tmp name of all uploded files in it. is there a hook i can use?

a) hook in the uploadprocess and

c) give a global option where to place the files: local / AWS / or both.

B) hook in the output of the images+thumbs (based on options location)

d) hook in the backend thumbnail view to show (based on options location)

[ryan]

Something that is storing images exclusively on another server probably needs to be a different animal from the existing image/file fieldtypes. Rather than trying to hook into functionality there, I would go and create a new fieldtype so you can start fresh. On the other hand, if the goal is to just keep a copy of any files used at the other server, and then replace references to them at runtime (like when output formatting is on or something) then I think that would be more of a scenario where hooks would be useful.

[MatthewSchenker]

Greetings,

Sorry for resurrecting an old discussion.  But it is still very relevant!

I've been looking lately at solutions for setting up automated backups to my Amazon S3 service.

Does anyone know what kind of code we could use in the PocessWire API to make this work?

Thanks,

Matthew

[arjen]

What kind of backup are you thinking of? If it's the whole package you can write a bash script which tars and gzips the entire folder your ProcessWire site is in and a dump of the database - both with nice filenames. Next the scripts connects to the S3 server and uploads both the files. In that case you wouldn't need to touch the API.

[MatthewSchenker]

Hi Arjen,

I was hoping (maybe) to create something that my clients can initiate as an option.  I already am creating control panels for a lot of other functions and was thinking this would be part of it.

Thanks,

Matthew

[Macrura]

I've been using this script off CodeCanyon (http://codecanyon.net/item/smartbackup-an-intuitive-backup-manager/3102970) then i create a custom admin page with an iframe to the interface (which is installed in a subfolder)....

the script itself can backup to ftp, sftp and dropbox;  i have it backing up the site folder and the database; it zips those and sends to remote ftp. it runs on a cron so can backup at any interval, and can keep x # of backups, also notifies you by email...

[arjen]

You could initiate a script I was talking about from a custom admin page. What kind of functionality are you exactly talking about? One click backup from the ProcessWire backend? Also restore?

[MatthewSchenker]

Greetings,

In this situation, I completed a site for a client, and I am neither hosting the site nor offering service on the site. The client is on his own! I mentioned that he should NOT trust the host to do backups. Then he asked me if I could build some way to make backups easier. So I hunted through the API thinking, "the building blocks I need for a PW backup script must be here somewhere -- I can always find the bits I need for any solution."

I am thinking that "backing up" is really no different than creating a profile. So maybe initiating something like this would work: http://processwire.com/talk/topic/530-profile-export-module-also-upgrade-pw-20-to-21/page-2

If there was a situation where the site needed to be restored, the client would call me to take care of it.

Thanks,

Matthew

[arjen]

Why shouldn't he trust the hosting provider? Also I would point out that the "site profile exporter was originally designed to serve as an upgrade tool from PW 2.0 to 2.1". Personally I would write a small script which zips and writes a filedump and mysqldump to the S3 server.

[MatthewSchenker]

Greetings,

Thanks everyone for these ideas. As always with ProcessWire, there are many ways to accomplish what needs to be done.

In my search for an AWS solution, I came across this code for Laravel. I think it can either be adapted to ProcessWire, perhaps as a module:

https://github.com/aws/aws-sdk-php-laravel

Thanks,

Matthew

[arjen]

Please let us know how you decided to approach this project

[dfunk006]

Hi,

Has anyone been able to save files on Amazon S3 instead of the local filesystem using File/Image Fieldtypes or creating a new Fieldtype?

[Raymond Geerts]

Currently working on a big project where there will be a lot of user content (photo's). As soon as i finished the modifcation of the Frontend User Profiles module with build-in friends system, that is something i too have to figure out. Instead of Amazone S3 i was looking in to Google Cloud Storage. One reason is they just lowered their prices and i personly like their API. Been playing around with their YouTube v2 API before and since a while with the YouTube v3 API. But also their FreeBase API.

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

One of our other big website (which still runs on MODx) has abot 40 GB of image data. The plan is to migrate this website some day to ProcessWire too.

So for this a 'special' fieldtype might solve it for storing images, instead having the ability at core level would be a dream come true

[dfunk006]

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

Yes, a solution like this will surely be helpful for sites with large data or sites that are hosted on a distributed server architecture.

If anyone is interested in developing a FieldType/Module for this, please let me know either via PM or by replying to this thread: https://processwire.com/talk/topic/6036-custom-fieldtype-for-amazon-s3/ and i'll be happily willing to sponsor it! 

[DaveP]

Amazon have their own S3 PHP SDK - https://github.com/aws/aws-sdk-php/tree/master/src/Aws/S3

[zyON]

I really need a solution for this and starting to look into this topic as we use Amazon Cloudfront CDN services to serve heavy media content like video and photos.

I've found this script: http://timkay.com/aws/ that may can work on shared hosted accounts (I think it only uses core Perl libraries) so maybe this could be used with some kind of post hook event on the file uploads via PW? 

[zyON]

I've created a module that works with S3 and Cloudfront. See this topic.