Jump to content
vanderbreye

Save Files in Amazon S3 / Cloud?

Recommended Posts

Hej!

Has anyone ever tried to save & load the files in / from the S3 instead / beside of the local filesystem ?

by using http://framework.zend.com/manual/de/zend.cloud.html it is very esay to implement to save/load:

http://blog.ebene7.com/2011/01/21/amazon-s3-mit-php-stream-wrapper-verwenden/

- but will PW work with it?

By module? Or even better: a plugin that automatically saves/serves any files handled by the backend..

We will now try to build this, but we'll also happy about any thoughts about it!

  • Like 4

Share this post


Link to post
Share on other sites

Hi vanderbreye,

I'd love to see something like this is PW. I've used one or two add-ons for ExpressionEngine that do similar. Are you, by chance, planning any features that allow users to only have access to the files they've uploaded?

Regards

Marty

  • Like 1

Share this post


Link to post
Share on other sites

Hi vanderbreye,

I like the idea - I've always wanted the ability to use dropbox for site assets, as it would be a much smoother workflow especially when working with others (eg. use a shared folder and eliminate the need to use the web interface to update files).

@diogo mentioned this interesting service a while ago which might be worth investigating

https://www.filepicker.io/

not sure if it works both ways - I think it just gives you a simple interface to the files which would then be uploaded to your server

  • Like 1

Share this post


Link to post
Share on other sites
@diogio mentioned this interesting service a while ago which might be worth investigating

@diogo ;)

Share this post


Link to post
Share on other sites

Ok, it is simple to implement the S3 directly in the core upload ;)

BUT: how should i begin to write the module, which binds on the uploadprocess?

It is important to have file+tmp name of all uploded files in it. is there a hook i can use?

a) hook in the uploadprocess and

c) give a global option where to place the files: local / AWS / or both.

B) hook in the output of the images+thumbs (based on options location)

d) hook in the backend thumbnail view to show (based on options location)

Share this post


Link to post
Share on other sites

Something that is storing images exclusively on another server probably needs to be a different animal from the existing image/file fieldtypes. Rather than trying to hook into functionality there, I would go and create a new fieldtype so you can start fresh. On the other hand, if the goal is to just keep a copy of any files used at the other server, and then replace references to them at runtime (like when output formatting is on or something) then I think that would be more of a scenario where hooks would be useful.

Share this post


Link to post
Share on other sites

Greetings,

Sorry for resurrecting an old discussion.  But it is still very relevant!

I've been looking lately at solutions for setting up automated backups to my Amazon S3 service.

Does anyone know what kind of code we could use in the PocessWire API to make this work?

Thanks,

Matthew

Share this post


Link to post
Share on other sites

What kind of backup are you thinking of? If it's the whole package you can write a bash script which tars and gzips the entire folder your ProcessWire site is in and a dump of the database - both with nice filenames. Next the scripts connects to the S3 server and uploads both the files. In that case you wouldn't need to touch the API.

  • Like 1

Share this post


Link to post
Share on other sites

Hi Arjen,

I was hoping (maybe) to create something that my clients can initiate as an option.  I already am creating control panels for a lot of other functions and was thinking this would be part of it.

Thanks,

Matthew

Share this post


Link to post
Share on other sites

I've been using this script off CodeCanyon (http://codecanyon.net/item/smartbackup-an-intuitive-backup-manager/3102970) then i create a custom admin page with an iframe to the interface (which is installed in a subfolder)....

the script itself can backup to ftp, sftp and dropbox;  i have it backing up the site folder and the database; it zips those and sends to remote ftp. it runs on a cron so can backup at any interval, and can keep x # of backups, also notifies you by email...

Share this post


Link to post
Share on other sites

You could initiate a script I was talking about from a custom admin page. What kind of functionality are you exactly talking about? One click backup from the ProcessWire backend? Also restore?

Share this post


Link to post
Share on other sites

Greetings,

In this situation, I completed a site for a client, and I am neither hosting the site nor offering service on the site. The client is on his own! I mentioned that he should NOT trust the host to do backups. Then he asked me if I could build some way to make backups easier. So I hunted through the API thinking, "the building blocks I need for a PW backup script must be here somewhere -- I can always find the bits I need for any solution."

I am thinking that "backing up" is really no different than creating a profile. So maybe initiating something like this would work: http://processwire.com/talk/topic/530-profile-export-module-also-upgrade-pw-20-to-21/page-2

If there was a situation where the site needed to be restored, the client would call me to take care of it.

Thanks,

Matthew

Share this post


Link to post
Share on other sites

Greetings,

Thanks everyone for these ideas. As always with ProcessWire, there are many ways to accomplish what needs to be done.

In my search for an AWS solution, I came across this code for Laravel. I think it can either be adapted to ProcessWire, perhaps as a module:

https://github.com/aws/aws-sdk-php-laravel

Thanks,

Matthew

  • Like 1

Share this post


Link to post
Share on other sites

Please let us know how you decided to approach this project :)

Share this post


Link to post
Share on other sites

Hi,

Has anyone been able to save files on Amazon S3 instead of the local filesystem using File/Image Fieldtypes or creating a new Fieldtype?

Share this post


Link to post
Share on other sites

Currently working on a big project where there will be a lot of user content (photo's). As soon as i finished the modifcation of the Frontend User Profiles module with build-in friends system, that is something i too have to figure out. Instead of Amazone S3 i was looking in to Google Cloud Storage. One reason is they just lowered their prices and i personly like their API. Been playing around with their YouTube v2 API before and since a while with the YouTube v3 API. But also their FreeBase API.

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

One of our other big website (which still runs on MODx) has abot 40 GB of image data. The plan is to migrate this website some day to ProcessWire too.

So for this a 'special' fieldtype might solve it for storing images, instead having the ability at core level would be a dream come true :)

  • Like 2

Share this post


Link to post
Share on other sites

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

Yes, a solution like this will surely be helpful for sites with large data or sites that are hosted on a distributed server architecture.

If anyone is interested in developing a FieldType/Module for this, please let me know either via PM or by replying to this thread: https://processwire.com/talk/topic/6036-custom-fieldtype-for-amazon-s3/ and i'll be happily willing to sponsor it!  :)

  • Like 2

Share this post


Link to post
Share on other sites

I really need a solution for this and starting to look into this topic as we use Amazon Cloudfront CDN services to serve heavy media content like video and photos.

I've found this script: http://timkay.com/aws/ that may can work on shared hosted accounts (I think it only uses core Perl libraries) so maybe this could be used with some kind of post hook event on the file uploads via PW? 

Share this post


Link to post
Share on other sites

I've created a module that works with S3 and Cloudfront. See this topic.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By BitPoet
      Here's a small new module that started as a spinoff of a memcache proof-of-concept. Cache your strings and partials in-memory using Redis as a backend.
      CacheRedis
      All you need is a running Redis database. The module supports connection through regular TCP, over TLS and via unix domain sockets. Host and port are configurable, and authentication is supported too. Here's a screenshot of the module configuration options:

      I'll not go into too many details about the usage, you can see everything explained in the README on GitHub, and just highlight the most important points.
      When the module is active, you'll have a wired $redis variable, which means you can reach the module as $redis, wire("redis") or within Wire classes / modules as $this->redis.
      CacheRedis is strongly influenced by WireCache, and its usage is (hopefully) straight forward.
      // Store an entry in the cache under the given key name with the given value: $redis->store("cached_value_number_one", $expire, $value); // Retrieve a value from the cache $value = $redis->fetch($key); // Delete a cache entry $redis->delete("my_other_cached_value"); // Clear the whole cache $redis->flush(); // Retrieve a value from the cache, and if not found, create it through the given function // and store it with a lifetime of 5 minutes (300 seconds) $value = $redis->fetch($key, 300, function() use($page) { return "Page last changed on " . strftime('%m/%d/%Y %H:%M', $page->modified | $page->created); }); // Render a file using wireRenderFile and store it in the cache. // We'll pass a selector as the expiry value so this cache gets // emptied every time a page matching the selector is saved. $news = $redis->renderFile("partials/news.php", 'template=blog-post', ["page" => $page]); The module is still very crude work in progress, but I hope some of you feel daring, try it out and let me know in case anything breaks.
      Have fun!
    • By joshua
      This module is (yet another) way for implementing a cookie management solution.
      Of course there are several other possibilities:
      - https://processwire.com/talk/topic/22920-klaro-cookie-consent-manager/
      - https://github.com/webmanufaktur/CookieManagementBanner
      - https://github.com/johannesdachsel/cookiemonster
      - https://www.oiljs.org/
      - ... and so on ...
      In this module you can configure which kind of cookie categories you want to manage:

      You can also enable the support for respecting the Do-Not-Track (DNT) header to don't annoy users, who already decided for all their browsing experience.
      Currently there are four possible cookie groups:
      - Necessary (always enabled)
      - Statistics
      - Marketing
      - External Media
      All groups can be renamed, so feel free to use other cookie group names. I just haven't found a way to implement a "repeater like" field as configurable module field ...
      When you want to load specific scripts ( like Google Analytics, Google Maps, ...) only after the user's content to this specific category of cookies, just use the following script syntax:
      <script type="optin" data-type="text/javascript" data-category="statistics" data-src="/path/to/your/statistic/script.js"></script> <script type="optin" data-type="text/javascript" data-category="marketing" data-src="/path/to/your/mareketing/script.js"></script> <script type="optin" data-type="text/javascript" data-category="external_media" data-src="/path/to/your/external-media/script.js"></script> <script type="optin" data-type="text/javascript" data-category="marketing">console.log("Inline scripts are also working!");</script> The type has to be "optin" to get recognized by PrivacyWire, the data-attributes are giving hints, how the script shall be loaded, if the data-category is within the cookie consents of the user. These scripts are loaded asynchronously after the user made the decision.
      If you want to give the users the possibility to change their consent, you can use the following Textformatter:
      [[privacywire-choose-cookies]] It's planned to add also other Textformatters to opt-out of specific cookie groups or delete the whole consent cookie.
      You can also add a custom link to output the banner again with a link / button with following class:
      <a href="#" class="privacywire-show-options">Show Cookie Options</a> <button class="privacywire-show-options">Show Cookie Options</button> This module is still in development, but we already use it on several production websites.
      You find it here: PrivacyWire Git Repo
      Download as .zip
      I would love to hear your feedback 🙂
      CHANGELOG
      0.0.5 Multi-language support included completely (also in TextFormatter). Added possibility to async load other assets (e.g. <img type="optin" data-category="marketing" data-src="https://via.placeholder.com/300x300">) 0.0.4 Added possibility to add an imprint link to the banner 0.0.3 Multi-language support for module config (still in development) 0.0.2 First release 0.0.1 Early development
    • By MoritzLost
      This is a new module that provides a simple solution to clearing all your cache layers at once, and an extensible interface to perform various cache-related actions.
      The simple motivation behind this module was that I was tired of manually clearing caches in several places after deploying a change on a live site. The basic purpose of this module is a simple Clear all caches link in the Setup menu which clears out all caches, no matter where they hide. You can customize what exactly the module does through it's configuration menu:
      Expire or delete all cache entries in the database, or selectively clear caches by namespace ($cache API) Clear the the template render cache. Clear out specific folders inside your site's cache directory (/site/assets/cache) Refresh version strings for static assets to bust client-side browser caches (this requires some setup, see the full documentation for details). This is the basic function of the module. However, you can also add different cache management action through the API and execute them through the module's interface. For this advanced usage, the module provides:
      An interface to see all available cache actions and execute them. A system log and logging output on the module page to see verify what the module is doing. A CacheControlTools class with utility functions to clear out different caches. An API to add cache actions, execute them programmatically and even modify the default action. Permission management, allowing you granular control over which user roles can execute which actions. The complete documentation can be found in the module's README.
      Beta release
      Note that I consider this a Beta release. Since the module is relatively aggressive in deleting some caches, I would advise you to install in on a test environment before using it on a live site.
      Let me know if you're getting any errors, have trouble using the module or if you have suggestions for improvement!
      In particular, can someone let me know if this module causes any problems with the ProCache module? I don't own or use it, so I can't check. As far as I can tell, ProCache uses a folder inside the cache directory to cache static pages, so my module should be able to clear the ProCache site cache as well, I'd appreciate it if someone can test that for me.
      Future plans
      If there is some interest in this, I plan to expand this to a more general cache management solution. I particular, I would like to add additional cache actions. Some ideas that came to mind:
      Warming up the template render cache for publicly accessible pages. Removing all active user sessions. Let me know if you have more suggestions!
      Links
      https://github.com/MoritzLost/ProcessCacheControl ProcessCacheControl in the Module directory

    • By David Karich
      Admin Page Tree Multiple Sorting
      ClassName: ProcessPageListMultipleSorting
      Extend the ordinary sort of children of a template in the admin page tree with multiple properties. For each template, you can define your own rule. Write each template (template-name) in a row, followed by a colon and then the additional field names for sorting.
      Example: All children of the template "blog" to be sorted in descending order according to the date of creation, then descending by modification date, and then by title. Type:
      blog: -created, -modified, title  Installation
      Copy the files for this module to /site/modules/ProcessPageListMultipleSorting/ In admin: Modules > Check for new modules. Install Module "Admin Page Tree Multible Sorting". Alternative in ProcessWire 2.4+
      Login to ProcessWire backend and go to Modules Click tab "New" and enter Module Class Name: "ProcessPageListMultipleSorting" Click "Download and Install"   Compatibility   I have currently tested the module only under PW 2.6+, but think that it works on older versions too. Maybe someone can give a feedback.     Download   PW-Repo: http://modules.processwire.com/modules/process-page-list-multiple-sorting/ GitHub: https://github.com/FlipZoomMedia/Processwire-ProcessPageListMultipleSorting     I hope someone can use the module. Have fun and best regards, David
×
×
  • Create New...