vanderbreye Posted June 1, 2012 Share Posted June 1, 2012 Hej! Has anyone ever tried to save & load the files in / from the S3 instead / beside of the local filesystem ? by using http://framework.zend.com/manual/de/zend.cloud.html it is very esay to implement to save/load: http://blog.ebene7.com/2011/01/21/amazon-s3-mit-php-stream-wrapper-verwenden/ - but will PW work with it? By module? Or even better: a plugin that automatically saves/serves any files handled by the backend.. We will now try to build this, but we'll also happy about any thoughts about it! 4 Link to comment Share on other sites More sharing options...
Marty Walker Posted June 4, 2012 Share Posted June 4, 2012 Hi vanderbreye, I'd love to see something like this is PW. I've used one or two add-ons for ExpressionEngine that do similar. Are you, by chance, planning any features that allow users to only have access to the files they've uploaded? Regards Marty 1 Link to comment Share on other sites More sharing options...
Michael Murphy Posted June 4, 2012 Share Posted June 4, 2012 Hi vanderbreye, I like the idea - I've always wanted the ability to use dropbox for site assets, as it would be a much smoother workflow especially when working with others (eg. use a shared folder and eliminate the need to use the web interface to update files). @diogo mentioned this interesting service a while ago which might be worth investigating https://www.filepicker.io/ not sure if it works both ways - I think it just gives you a simple interface to the files which would then be uploaded to your server 1 Link to comment Share on other sites More sharing options...
diogo Posted June 4, 2012 Share Posted June 4, 2012 @diogio mentioned this interesting service a while ago which might be worth investigating @diogo Link to comment Share on other sites More sharing options...
vanderbreye Posted June 5, 2012 Author Share Posted June 5, 2012 Ok, it is simple to implement the S3 directly in the core upload BUT: how should i begin to write the module, which binds on the uploadprocess? It is important to have file+tmp name of all uploded files in it. is there a hook i can use? a) hook in the uploadprocess and c) give a global option where to place the files: local / AWS / or both. B) hook in the output of the images+thumbs (based on options location) d) hook in the backend thumbnail view to show (based on options location) Link to comment Share on other sites More sharing options...
ryan Posted June 6, 2012 Share Posted June 6, 2012 Something that is storing images exclusively on another server probably needs to be a different animal from the existing image/file fieldtypes. Rather than trying to hook into functionality there, I would go and create a new fieldtype so you can start fresh. On the other hand, if the goal is to just keep a copy of any files used at the other server, and then replace references to them at runtime (like when output formatting is on or something) then I think that would be more of a scenario where hooks would be useful. Link to comment Share on other sites More sharing options...
MatthewSchenker Posted August 12, 2013 Share Posted August 12, 2013 Greetings, Sorry for resurrecting an old discussion. But it is still very relevant! I've been looking lately at solutions for setting up automated backups to my Amazon S3 service. Does anyone know what kind of code we could use in the PocessWire API to make this work? Thanks, Matthew Link to comment Share on other sites More sharing options...
arjen Posted August 12, 2013 Share Posted August 12, 2013 What kind of backup are you thinking of? If it's the whole package you can write a bash script which tars and gzips the entire folder your ProcessWire site is in and a dump of the database - both with nice filenames. Next the scripts connects to the S3 server and uploads both the files. In that case you wouldn't need to touch the API. 1 Link to comment Share on other sites More sharing options...
MatthewSchenker Posted August 12, 2013 Share Posted August 12, 2013 Hi Arjen, I was hoping (maybe) to create something that my clients can initiate as an option. I already am creating control panels for a lot of other functions and was thinking this would be part of it. Thanks, Matthew Link to comment Share on other sites More sharing options...
Macrura Posted August 12, 2013 Share Posted August 12, 2013 I've been using this script off CodeCanyon (http://codecanyon.net/item/smartbackup-an-intuitive-backup-manager/3102970) then i create a custom admin page with an iframe to the interface (which is installed in a subfolder).... the script itself can backup to ftp, sftp and dropbox; i have it backing up the site folder and the database; it zips those and sends to remote ftp. it runs on a cron so can backup at any interval, and can keep x # of backups, also notifies you by email... Link to comment Share on other sites More sharing options...
arjen Posted August 13, 2013 Share Posted August 13, 2013 You could initiate a script I was talking about from a custom admin page. What kind of functionality are you exactly talking about? One click backup from the ProcessWire backend? Also restore? Link to comment Share on other sites More sharing options...
MatthewSchenker Posted August 13, 2013 Share Posted August 13, 2013 Greetings, In this situation, I completed a site for a client, and I am neither hosting the site nor offering service on the site. The client is on his own! I mentioned that he should NOT trust the host to do backups. Then he asked me if I could build some way to make backups easier. So I hunted through the API thinking, "the building blocks I need for a PW backup script must be here somewhere -- I can always find the bits I need for any solution." I am thinking that "backing up" is really no different than creating a profile. So maybe initiating something like this would work: http://processwire.com/talk/topic/530-profile-export-module-also-upgrade-pw-20-to-21/page-2 If there was a situation where the site needed to be restored, the client would call me to take care of it. Thanks, Matthew Link to comment Share on other sites More sharing options...
arjen Posted August 13, 2013 Share Posted August 13, 2013 Why shouldn't he trust the hosting provider? Also I would point out that the "site profile exporter was originally designed to serve as an upgrade tool from PW 2.0 to 2.1". Personally I would write a small script which zips and writes a filedump and mysqldump to the S3 server. Link to comment Share on other sites More sharing options...
MatthewSchenker Posted August 17, 2013 Share Posted August 17, 2013 Greetings, Thanks everyone for these ideas. As always with ProcessWire, there are many ways to accomplish what needs to be done. In my search for an AWS solution, I came across this code for Laravel. I think it can either be adapted to ProcessWire, perhaps as a module: https://github.com/aws/aws-sdk-php-laravel Thanks, Matthew 1 Link to comment Share on other sites More sharing options...
arjen Posted August 18, 2013 Share Posted August 18, 2013 Please let us know how you decided to approach this project Link to comment Share on other sites More sharing options...
dfunk006 Posted April 4, 2014 Share Posted April 4, 2014 Hi, Has anyone been able to save files on Amazon S3 instead of the local filesystem using File/Image Fieldtypes or creating a new Fieldtype? Link to comment Share on other sites More sharing options...
Raymond Geerts Posted April 4, 2014 Share Posted April 4, 2014 Currently working on a big project where there will be a lot of user content (photo's). As soon as i finished the modifcation of the Frontend User Profiles module with build-in friends system, that is something i too have to figure out. Instead of Amazone S3 i was looking in to Google Cloud Storage. One reason is they just lowered their prices and i personly like their API. Been playing around with their YouTube v2 API before and since a while with the YouTube v3 API. But also their FreeBase API. MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too. I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data. One of our other big website (which still runs on MODx) has abot 40 GB of image data. The plan is to migrate this website some day to ProcessWire too. So for this a 'special' fieldtype might solve it for storing images, instead having the ability at core level would be a dream come true 2 Link to comment Share on other sites More sharing options...
dfunk006 Posted April 4, 2014 Share Posted April 4, 2014 MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too. I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data. Yes, a solution like this will surely be helpful for sites with large data or sites that are hosted on a distributed server architecture. If anyone is interested in developing a FieldType/Module for this, please let me know either via PM or by replying to this thread: https://processwire.com/talk/topic/6036-custom-fieldtype-for-amazon-s3/ and i'll be happily willing to sponsor it! 2 Link to comment Share on other sites More sharing options...
DaveP Posted April 4, 2014 Share Posted April 4, 2014 Amazon have their own S3 PHP SDK - https://github.com/aws/aws-sdk-php/tree/master/src/Aws/S3 2 Link to comment Share on other sites More sharing options...
zyON Posted May 14, 2014 Share Posted May 14, 2014 I really need a solution for this and starting to look into this topic as we use Amazon Cloudfront CDN services to serve heavy media content like video and photos. I've found this script: http://timkay.com/aws/ that may can work on shared hosted accounts (I think it only uses core Perl libraries) so maybe this could be used with some kind of post hook event on the file uploads via PW? Link to comment Share on other sites More sharing options...
zyON Posted June 19, 2014 Share Posted June 19, 2014 I've created a module that works with S3 and Cloudfront. See this topic. 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now