Jump to content
vanderbreye

Save Files in Amazon S3 / Cloud?

Recommended Posts

Hej!

Has anyone ever tried to save & load the files in / from the S3 instead / beside of the local filesystem ?

by using http://framework.zend.com/manual/de/zend.cloud.html it is very esay to implement to save/load:

http://blog.ebene7.com/2011/01/21/amazon-s3-mit-php-stream-wrapper-verwenden/

- but will PW work with it?

By module? Or even better: a plugin that automatically saves/serves any files handled by the backend..

We will now try to build this, but we'll also happy about any thoughts about it!

  • Like 4

Share this post


Link to post
Share on other sites

Hi vanderbreye,

I'd love to see something like this is PW. I've used one or two add-ons for ExpressionEngine that do similar. Are you, by chance, planning any features that allow users to only have access to the files they've uploaded?

Regards

Marty

  • Like 1

Share this post


Link to post
Share on other sites

Hi vanderbreye,

I like the idea - I've always wanted the ability to use dropbox for site assets, as it would be a much smoother workflow especially when working with others (eg. use a shared folder and eliminate the need to use the web interface to update files).

@diogo mentioned this interesting service a while ago which might be worth investigating

https://www.filepicker.io/

not sure if it works both ways - I think it just gives you a simple interface to the files which would then be uploaded to your server

  • Like 1

Share this post


Link to post
Share on other sites
@diogio mentioned this interesting service a while ago which might be worth investigating

@diogo ;)

Share this post


Link to post
Share on other sites

Ok, it is simple to implement the S3 directly in the core upload ;)

BUT: how should i begin to write the module, which binds on the uploadprocess?

It is important to have file+tmp name of all uploded files in it. is there a hook i can use?

a) hook in the uploadprocess and

c) give a global option where to place the files: local / AWS / or both.

B) hook in the output of the images+thumbs (based on options location)

d) hook in the backend thumbnail view to show (based on options location)

Share this post


Link to post
Share on other sites

Something that is storing images exclusively on another server probably needs to be a different animal from the existing image/file fieldtypes. Rather than trying to hook into functionality there, I would go and create a new fieldtype so you can start fresh. On the other hand, if the goal is to just keep a copy of any files used at the other server, and then replace references to them at runtime (like when output formatting is on or something) then I think that would be more of a scenario where hooks would be useful.

Share this post


Link to post
Share on other sites

Greetings,

Sorry for resurrecting an old discussion.  But it is still very relevant!

I've been looking lately at solutions for setting up automated backups to my Amazon S3 service.

Does anyone know what kind of code we could use in the PocessWire API to make this work?

Thanks,

Matthew

Share this post


Link to post
Share on other sites

What kind of backup are you thinking of? If it's the whole package you can write a bash script which tars and gzips the entire folder your ProcessWire site is in and a dump of the database - both with nice filenames. Next the scripts connects to the S3 server and uploads both the files. In that case you wouldn't need to touch the API.

  • Like 1

Share this post


Link to post
Share on other sites

Hi Arjen,

I was hoping (maybe) to create something that my clients can initiate as an option.  I already am creating control panels for a lot of other functions and was thinking this would be part of it.

Thanks,

Matthew

Share this post


Link to post
Share on other sites

I've been using this script off CodeCanyon (http://codecanyon.net/item/smartbackup-an-intuitive-backup-manager/3102970) then i create a custom admin page with an iframe to the interface (which is installed in a subfolder)....

the script itself can backup to ftp, sftp and dropbox;  i have it backing up the site folder and the database; it zips those and sends to remote ftp. it runs on a cron so can backup at any interval, and can keep x # of backups, also notifies you by email...

Share this post


Link to post
Share on other sites

You could initiate a script I was talking about from a custom admin page. What kind of functionality are you exactly talking about? One click backup from the ProcessWire backend? Also restore?

Share this post


Link to post
Share on other sites

Greetings,

In this situation, I completed a site for a client, and I am neither hosting the site nor offering service on the site. The client is on his own! I mentioned that he should NOT trust the host to do backups. Then he asked me if I could build some way to make backups easier. So I hunted through the API thinking, "the building blocks I need for a PW backup script must be here somewhere -- I can always find the bits I need for any solution."

I am thinking that "backing up" is really no different than creating a profile. So maybe initiating something like this would work: http://processwire.com/talk/topic/530-profile-export-module-also-upgrade-pw-20-to-21/page-2

If there was a situation where the site needed to be restored, the client would call me to take care of it.

Thanks,

Matthew

Share this post


Link to post
Share on other sites

Greetings,

Thanks everyone for these ideas. As always with ProcessWire, there are many ways to accomplish what needs to be done.

In my search for an AWS solution, I came across this code for Laravel. I think it can either be adapted to ProcessWire, perhaps as a module:

https://github.com/aws/aws-sdk-php-laravel

Thanks,

Matthew

  • Like 1

Share this post


Link to post
Share on other sites

Please let us know how you decided to approach this project :)

Share this post


Link to post
Share on other sites

Hi,

Has anyone been able to save files on Amazon S3 instead of the local filesystem using File/Image Fieldtypes or creating a new Fieldtype?

Share this post


Link to post
Share on other sites

Currently working on a big project where there will be a lot of user content (photo's). As soon as i finished the modifcation of the Frontend User Profiles module with build-in friends system, that is something i too have to figure out. Instead of Amazone S3 i was looking in to Google Cloud Storage. One reason is they just lowered their prices and i personly like their API. Been playing around with their YouTube v2 API before and since a while with the YouTube v3 API. But also their FreeBase API.

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

One of our other big website (which still runs on MODx) has abot 40 GB of image data. The plan is to migrate this website some day to ProcessWire too.

So for this a 'special' fieldtype might solve it for storing images, instead having the ability at core level would be a dream come true :)

  • Like 2

Share this post


Link to post
Share on other sites

MODx Revolution has a nice solution for this where you can setup data providers. Like the default "localhost" there is one for S3 storage too.

I'm hoping ProcessWire will have such a solution someday too where you can install data providers (S3, Google Cloud, other CDN) and choose where certain data is going to be stored. Specialy with huge sites having the choice for something else then localhost to store the data. CDN or Cloud Storage have nicer pricing then regular hosting providers when it comes to storage huge amounts of data.

Yes, a solution like this will surely be helpful for sites with large data or sites that are hosted on a distributed server architecture.

If anyone is interested in developing a FieldType/Module for this, please let me know either via PM or by replying to this thread: https://processwire.com/talk/topic/6036-custom-fieldtype-for-amazon-s3/ and i'll be happily willing to sponsor it!  :)

  • Like 2

Share this post


Link to post
Share on other sites

I really need a solution for this and starting to look into this topic as we use Amazon Cloudfront CDN services to serve heavy media content like video and photos.

I've found this script: http://timkay.com/aws/ that may can work on shared hosted accounts (I think it only uses core Perl libraries) so maybe this could be used with some kind of post hook event on the file uploads via PW? 

Share this post


Link to post
Share on other sites

I've created a module that works with S3 and Cloudfront. See this topic.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Paul Greinke
      Hi there. I wrote a custom module for one of my projects. In fact I maybe want to use my module in other projects too. In order to be variable and customizable  I need to implement some custom hooks into my module. So I can afterwards hook into the my functions in order to modify them to match the needs of the new project.
      I tried simply defining functions with the '__' prefix. But that did not work. I'm imagining something like the following:
      <?php class MyClass { public function ___someFunction() { // Do something } } // ready.php $this->addHookBefore('MyClass::someFunction', function($event) { // some customization }); Is there a way to accomplish that? 
    • By adrian
      Hi everyone,
      Here's a new module that I have been meaning to build for a long time.
      http://modules.processwire.com/modules/process-admin-actions/
      https://github.com/adrianbj/ProcessAdminActions
       
      What does it do?
      Do you have a bunch of admin snippets laying around, or do you recreate from them from scratch every time you need them, or do you try to find where you saw them in the forums, or on the ProcessWire Recipes site?
      Admin Actions lets you quickly create actions in the admin that you can use over and over and even make available to your site editors (permissions for each action are assigned to roles separately so you have full control over who has access to which actions).
       
      Included Actions
      It comes bundled with several actions and I will be adding more over time (and hopefully I'll get some PRs from you guys too). You can browse and sort and if you have @tpr's Admin on Steroid's datatables filter feature, you can even filter based on the content of all columns. 

      The headliner action included with the module is: PageTable To RepeaterMatrix which fully converts an existing (and populated) PageTable field to either a Repeater or RepeaterMatrix field. This is a huge timesaver if you have an existing site that makes heavy use of PageTable fields and you would like to give the clients the improved interface of RepeaterMatrix.
      Copy Content To Other Field
      This action copies the content from one field to another field on all pages that use the selected template.
      Copy Field Content To Other Page
      Copies the content from a field on one page to the same field on another page.
      Copy Repeater Items To Other Page
      Add the items from a Repeater field on one page to the same field on another page.
      Copy Table Field Rows To Other Page
      Add the rows from a Table field on one page to the same field on another page.
      Create Users Batcher
      Allows you to batch create users. This module requires the Email New User module and it should be configured to generate a password automatically.
      Delete Unused Fields
      Deletes fields that are not used by any templates.
      Delete Unused Templates
      Deletes templates that are not used by any pages.
      Email Batcher
      Lets you email multiple addresses at once.
      Field Set Or Search And Replace
      Set field values, or search and replace text in field values from a filtered selection of pages and fields.
      FTP Files to Page
      Add files/images from a folder to a selected page.
      Page Active Languages Batcher
      Lets you enable or disable active status of multiple languages on multiple pages at once.
      Page Manipulator
      Uses an InputfieldSelector to query pages and then allows batch actions on the matched pages.
      Page Table To Repeater Matrix
      Fully converts an existing (and populated) PageTable field to either a Repeater or RepeaterMatrix field.
      Template Fields Batcher
      Lets you add or remove multiple fields from multiple templates at once.
      Template Roles Batcher
      Lets you add or remove access permissions, for multiple roles and multiple templates at once.
      User Roles Permissions Batcher
      Lets you add or remove permissions for multiple roles, or roles for multiple users at once.
       
      Creating a New Action
      If you create a new action that you think others would find useful, please add it to the actions subfolder of this module and submit a PR. If you think it is only useful for you, place it in /site/templates/AdminActions/ so that it doesn't get lost on module updates.
      A new action file can be as simple as this:
      class UnpublishAboutPage extends ProcessAdminActions { protected function executeAction() { $p = $this->pages->get('/about/'); $p->addStatus(Page::statusUnpublished); $p->save(); return true; } } Each action:
      class must extend "ProcessAdminActions" and the filename must match the class name and end in ".action.php" like: UnpublishAboutPage.action.php the action method must be: executeAction() As you can see there are only a few lines needed to wrap the actual API call, so it's really worth the small extra effort to make an action.
      Obviously that example action is not very useful. Here is another more useful one that is included with the module. It includes $description, $notes, and $author variables which are used in the module table selector interface. It also makes use of the defineOptions() method which builds the input fields used to gather the required options before running the action.
      class DeleteUnusedFields extends ProcessAdminActions { protected $description = 'Deletes fields that are not used by any templates.'; protected $notes = 'Shows a list of unused fields with checkboxes to select those to delete.'; protected $author = 'Adrian Jones'; protected $authorLinks = array( 'pwforum' => '985-adrian', 'pwdirectory' => 'adrian-jones', 'github' => 'adrianbj', ); protected function defineOptions() { $fieldOptions = array(); foreach($this->fields as $field) { if ($field->flags & Field::flagSystem || $field->flags & Field::flagPermanent) continue; if(count($field->getFieldgroups()) === 0) $fieldOptions[$field->id] = $field->label ? $field->label . ' (' . $field->name . ')' : $field->name; } return array( array( 'name' => 'fields', 'label' => 'Fields', 'description' => 'Select the fields you want to delete', 'notes' => 'Note that all fields listed are not used by any templates and should therefore be safe to delete', 'type' => 'checkboxes', 'options' => $fieldOptions, 'required' => true ) ); } protected function executeAction($options) { $count = 0; foreach($options['fields'] as $field) { $f = $this->fields->get($field); $this->fields->delete($f); $count++; } $this->successMessage = $count . ' field' . _n('', 's', $count) . ' ' . _n('was', 'were', $count) . ' successfully deleted'; return true; } }  
      This defineOptions() method builds input fields that look like this:

      Finally we use $options array in the executeAction() method to get the values entered into those options fields to run the API script to remove the checked fields.
      There is one additional method that I didn't outline called: checkRequirements() - you can see it in action in the PageTableToRepeaterMatrix action. You can use this to prevent the action from running if certain requirements are not met.
      At the end of the executeAction() method you can populate $this->successMessage, or $this->failureMessage which will be returned after the action has finished.
       
      Populating options via URL parameters
      You can also populate the option parameters via URL parameters. You should split multiple values with a “|” character.
      You can either just pre-populate options:
      http://mysite.dev/processwire/setup/admin-actions/options?action=TemplateFieldsBatcher&templates=29|56&fields=219&addOrRemove=add
      or you can execute immediately:
      http://mysite.dev/processwire/setup/admin-actions/execute?action=TemplateFieldsBatcher&templates=29|56&fields=219&addOrRemove=add

      Note the “options” vs “execute” as the last path before the parameters.
       
      Automatic Backup / Restore
      Before any action is executed, a full database backup is automatically made. You have a few options to run a restore if needed:
      Follow the Restore link that is presented after an action completes Use the "Restore" submenu: Setup > Admin Actions > Restore Move the restoredb.php file from the /site/assets/cache/AdminActions/ folder to the root of your site and load in the browser Manually restore using the AdminActionsBackup.sql file in the /site/assets/cache/AdminActions/ folder I think all these features make it very easy to create custom admin data manipulation methods that can be shared with others and executed using a simple interface without needing to build a full Process Module custom interface from scratch. I also hope it will reduce the barriers for new ProcessWire users to create custom admin functionality.
      Please let me know what you think, especially if you have ideas for improving the interface, or the way actions are defined.
       
       
    • By joshua
      ---
      Module Directory: https://modules.processwire.com/modules/privacy-wire/
      Github: https://github.com/blaueQuelle/privacywire/
      Packagist:https://packagist.org/packages/blauequelle/privacywire
      Module Class Name: PrivacyWire
      Changelog: https://github.com/blaueQuelle/privacywire/blob/master/Changelog.md
      ---
      This module is (yet another) way for implementing a cookie management solution.
      Of course there are several other possibilities:
      - https://processwire.com/talk/topic/22920-klaro-cookie-consent-manager/
      - https://github.com/webmanufaktur/CookieManagementBanner
      - https://github.com/johannesdachsel/cookiemonster
      - https://www.oiljs.org/
      - ... and so on ...
      In this module you can configure which kind of cookie categories you want to manage:

      You can also enable the support for respecting the Do-Not-Track (DNT) header to don't annoy users, who already decided for all their browsing experience.
      Currently there are four possible cookie groups:
      - Necessary (always enabled)
      - Functional
      - Statistics
      - Marketing
      - External Media
      All groups can be renamed, so feel free to use other cookie group names. I just haven't found a way to implement a "repeater like" field as configurable module field ...
      When you want to load specific scripts ( like Google Analytics, Google Maps, ...) only after the user's content to this specific category of cookies, just use the following script syntax:
      <script type="text/plain" data-type="text/javascript" data-category="statistics" data-src="/path/to/your/statistic/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="marketing" data-src="/path/to/your/mareketing/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="external_media" data-src="/path/to/your/external-media/script.js"></script> <script type="text/plain" data-type="text/javascript" data-category="marketing">console.log("Inline scripts are also working!");</script> The data-attributes (data-type and data-category) are required to get recognized by PrivacyWire. the data-attributes are giving hints, how the script shall be loaded, if the data-category is within the cookie consents of the user. These scripts are loaded asynchronously after the user made the decision.
      If you want to give the users the possibility to change their consent, you can use the following Textformatter:
      [[privacywire-choose-cookies]] It's planned to add also other Textformatters to opt-out of specific cookie groups or delete the whole consent cookie.
      You can also add a custom link to output the banner again with a link / button with following class:
      <a href="#" class="privacywire-show-options">Show Cookie Options</a> <button class="privacywire-show-options">Show Cookie Options</button>  
      I would love to hear your feedback 🙂
      CHANGELOG
      You can find the always up-to-date changelog file here.
    • By joshua
      As we often use Matomo (former known as Piwik) instead of Google Analytics we wanted to embed Matomo not only in the template code but also via the ProcessWire backend.
      That's why I developed a tiny module for the implementation.
      The module provides the possibility to connect to an existing Matomo installation with the classical site tracking and also via the Matomo Tag Manager.
      If you have also PrivacyWire installed, you can tell MatomoWire to only load the script, if the user has accepted cookies via PrivacyWire.
      To offer an Opt-Out solution you can choose between the simple Opt-Out iFrame, delivered by your Matomo installation, or a button to choose cookies via PrivacyWire.
      You'll find the module both in the module directory and via github:
      ProcessWire Module Directory MatomoWire @ GitHub MatomoWire @ Packagist ->installable via composer require blauequelle/matomowire I'm looking forward to hear your feedback!


    • By Robin S
      If your module has a lot of config fields you might want to divide them into groups inside a tabbed interface. Here is a demonstration module showing how this can be done.
      https://github.com/Toutouwai/ModuleConfigTabs

      Thanks to @kixe for providing my starting point in this forum topic.
×
×
  • Create New...