Jump to content

Save Files in Amazon S3 / Cloud?


vanderbreye
 Share

Recommended Posts

Hej!

Has anyone ever tried to save & load the files in / from the S3 instead / beside of the local filesystem ?

by using http://framework.zend.com/manual/de/zend.cloud.html it is very esay to implement to save/load:

http://blog.ebene7.com/2011/01/21/amazon-s3-mit-php-stream-wrapper-verwenden/

- but will PW work with it?

By module? Or even better: a plugin that automatically saves/serves any files handled by the backend..

We will now try to build this, but we'll also happy about any thoughts about it!

  • Like 4
Link to comment
Share on other sites

Hi vanderbreye,

I'd love to see something like this is PW. I've used one or two add-ons for ExpressionEngine that do similar. Are you, by chance, planning any features that allow users to only have access to the files they've uploaded?

Regards

Marty

  • Like 1
Link to comment
Share on other sites

Hi vanderbreye,

I like the idea - I've always wanted the ability to use dropbox for site assets, as it would be a much smoother workflow especially when working with others (eg. use a shared folder and eliminate the need to use the web interface to update files).

@diogo mentioned this interesting service a while ago which might be worth investigating

https://www.filepicker.io/

not sure if it works both ways - I think it just gives you a simple interface to the files which would then be uploaded to your server

  • Like 1
Link to comment
Share on other sites

Ok, it is simple to implement the S3 directly in the core upload ;)

BUT: how should i begin to write the module, which binds on the uploadprocess?

It is important to have file+tmp name of all uploded files in it. is there a hook i can use?

a) hook in the uploadprocess and

c) give a global option where to place the files: local / AWS / or both.

B) hook in the output of the images+thumbs (based on options location)

d) hook in the backend thumbnail view to show (based on options location)

Link to comment
Share on other sites

Something that is storing images exclusively on another server probably needs to be a different animal from the existing image/file fieldtypes. Rather than trying to hook into functionality there, I would go and create a new fieldtype so you can start fresh. On the other hand, if the goal is to just keep a copy of any files used at the other server, and then replace references to them at runtime (like when output formatting is on or something) then I think that would be more of a scenario where hooks would be useful.

Link to comment
Share on other sites

  • 1 year later...

Greetings,

Sorry for resurrecting an old discussion.  But it is still very relevant!

I've been looking lately at solutions for setting up automated backups to my Amazon S3 service.

Does anyone know what kind of code we could use in the PocessWire API to make this work?

Thanks,

Matthew

Link to comment
Share on other sites

What kind of backup are you thinking of? If it's the whole package you can write a bash script which tars and gzips the entire folder your ProcessWire site is in and a dump of the database - both with nice filenames. Next the scripts connects to the S3 server and uploads both the files. In that case you wouldn't need to touch the API.

  • Like 1
Link to comment
Share on other sites

I've been using this script off CodeCanyon (http://codecanyon.net/item/smartbackup-an-intuitive-backup-manager/3102970) then i create a custom admin page with an iframe to the interface (which is installed in a subfolder)....

the script itself can backup to ftp, sftp and dropbox;  i have it backing up the site folder and the database; it zips those and sends to remote ftp. it runs on a cron so can backup at any interval, and can keep x # of backups, also notifies you by email...

Link to comment
Share on other sites

You could initiate a script I was talking about from a custom admin page. What kind of functionality are you exactly talking about? One click backup from the ProcessWire backend? Also restore?

Link to comment
Share on other sites

Greetings,

In this situation, I completed a site for a client, and I am neither hosting the site nor offering service on the site. The client is on his own! I mentioned that he should NOT trust the host to do backups. Then he asked me if I could build some way to make backups easier. So I hunted through the API thinking, "the building blocks I need for a PW backup script must be here somewhere -- I can always find the bits I need for any solution."

I am thinking that "backing up" is really no different than creating a profile. So maybe initiating something like this would work: http://processwire.com/talk/topic/530-profile-export-module-also-upgrade-pw-20-to-21/page-2

If there was a situation where the site needed to be restored, the client would call me to take care of it.

Thanks,

Matthew

Link to comment
Share on other sites

Greetings,

Thanks everyone for these ideas. As always with ProcessWire, there are many ways to accomplish what needs to be done.

In my search for an AWS solution, I came across this code for Laravel. I think it can either be adapted to ProcessWire, perhaps as a module:

https://github.com/aws/aws-sdk-php-laravel

Thanks,

Matthew

  • Like 1
Link to comment
Share on other sites

  • 7 months later...

Currently working on a big project where there will be a lot of user content (photo's). As soon as i finished the modifcation of the Frontend User Profiles module with build-in friends system, that is something i too have to figure out. Instead of Amazone S3 i was looking in to Google Cloud Storage. One reason is they just lowered their prices and i personly like their API. Been playing around with their YouTube v2 API before and since a while with the YouTube v3 API. But also their FreeBase API.

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

One of our other big website (which still runs on MODx) has abot 40 GB of image data. The plan is to migrate this website some day to ProcessWire too.

So for this a 'special' fieldtype might solve it for storing images, instead having the ability at core level would be a dream come true :)

  • Like 2
Link to comment
Share on other sites

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

Yes, a solution like this will surely be helpful for sites with large data or sites that are hosted on a distributed server architecture.

If anyone is interested in developing a FieldType/Module for this, please let me know either via PM or by replying to this thread: https://processwire.com/talk/topic/6036-custom-fieldtype-for-amazon-s3/ and i'll be happily willing to sponsor it!  :)

  • Like 2
Link to comment
Share on other sites

  • 1 month later...

I really need a solution for this and starting to look into this topic as we use Amazon Cloudfront CDN services to serve heavy media content like video and photos.

I've found this script: http://timkay.com/aws/ that may can work on shared hosted accounts (I think it only uses core Perl libraries) so maybe this could be used with some kind of post hook event on the file uploads via PW? 

Link to comment
Share on other sites

  • 1 month later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...