Jump to content
zyON

Module: Amazon S3 / Cloudfront

Recommended Posts

Hi, 

After the move to ProcessWire on one of our major websites, I had to build myself a module/plugin so I could use the Amazon S3 / CloudFront infrastructure.

The module/plugin uses Amazon S3 PHP SDK and the idea was to provide a clean way to upload/backup files to S3 and distribute the assets via Cloudfront. 

The current version of the module/plugin will copy the page files to Amazon S3 so you can then serve the via Cloudfront automatically. Note that the files are still also copied to the server where your PW installation resides.

There's an option to backup the deleted files to another folder on S3 because when you delete a file on PW via the admin, the file is also deleted from it's folder on S3.

The module also supports Apeisa's Thumbnail module so it also stores the thumbnails on S3. Note that the native size() method to create image thumbnails on S3 is currently not supported.

As it is, the module will only upload new assets, so beware that if you already have pages created with assets you'll definitely have errors, so my advice is to test this with a blank installation of PW.

If anyone wants to test it and contribute to it, I think this provides a good proof of concept for a functionality that is requested by several users (me included).

Please note that I'm not a PHP developer, my skills (very) limited and I'm aware that it can be improved and I'm open to suggestions, feedback and most of all, collaborators.

The module is hosted on github here and it's available in the ProcessWire modules directory.

Nelson Mendes
Edited by zyON
  • Like 22

Share this post


Link to post
Share on other sites

I've updated the module so now it's possible to set a custom time in seconds to control the cache in the browser and in CloudFront. Previously it used the Expires Header but it's advisable to user the Cache-Control directive instead. 

  • Like 1

Share this post


Link to post
Share on other sites

There's a new version available. Now supporting file versioning to handle the caching issues of CloudFront. 

There's an option on the module configuration that will automatically rename the uploaded files by inserting a timestamp in the file name. This way all the files have unique names and that will make it easier to replace old files already cached by CloudFront (something that can be a huge PITA as many CloudFront, or other CDN systems know well).

Screen-Shot-2014-06-30-at-11.59.26-am.pn

  • Like 7

Share this post


Link to post
Share on other sites

New version available

The module now has the ability to update the filename when a new thumbnail is created via the thumbnails plugin. 

This is important if you use Amazon CloudFront option to serve the files and want to show the changes you've made to the thumbnails immediately (without waiting for the cache to expire).

Because when you create a new image crop it's filename doesn't change, I had to force this change in both the thumbnails files and the source file. 

  • Like 5

Share this post


Link to post
Share on other sites

Is it possible to serve files from S3  and delete them from the asses folder?

Great module zyON.

Share this post


Link to post
Share on other sites

Manol, the plugin doesn't handle the deletion of the files in the assets folder. If the plugin is active, the files will be delivered through S3 without relying on the local ones , so theoretically you could manually delete them from you local assets folder but I think of it as a backup that you can quickly start using just by deactivating the plugin.

EDIT: Sorry, I forgot to mention (too much work) that in the admin, the local files are used. So it's definitely not a good Idea.!

  • Like 1

Share this post


Link to post
Share on other sites

Great module. It's possible to use the module working just with API ? I mean, creating pages with images and files not from the admin but via code.

Share this post


Link to post
Share on other sites

Great module. It's possible to use the module working just with API ? I mean, creating pages with images and files not from the admin but via code.

Thanks. In the current version of the module it will only work with assets uploaded via the admin. 

That's something in the roadmap for me but I don't have a date I can share right now because I'm really lacking the time at this moment.

  • Like 1

Share this post


Link to post
Share on other sites

To use amazon cloud front, does the file have to be in s3?  Or is it possible to have cloud front work with local processwire files?

Great plugin!  I am bookmarking it.  Was just curious how amazon works!

Thanks!

Share this post


Link to post
Share on other sites

Great module!

I'm creating a film database site with private content that authenticated users can download.

I've used your plugin and configured it to work with signed URLs so users can't distribute the download link.

Is signed urls the best method to ensure that downloads are only available for authenticated users?

is it possible to have a setting within a file upload field to flag whether to upload the file to S3/cloudfront or store it locally on the PW server?

Jonathan

Share this post


Link to post
Share on other sites

Just today tested the module on a pw 2.5.22 site and apart form the figuring out how to set stuff up with s3 bucket, policy and cloudfront distribution, all worked like a charm

Thanks lots zyON!

  • Like 2

Share this post


Link to post
Share on other sites

A few things have changed regarding amazon s3 and cloudfront, I have made a checklist that takes you step by step through setting it all up, works like a charm, posting here in case I forget the steps myself for next site... :undecided:

1 - Set up a bucket for this site on amazon s3

2 - make sure it is on us standard

3 - go to Account => security credentials => get started with IAM users

4 - create new user

5 - download user security credentials

6 - select user and go to Inline Policy

7 - Create User Policy

8 - {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1425901261000",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::name-of-bucket/*"
            ]
        }
    ]
}

9 - go to PW admin and install the amazons3cloudfront module

10 - enter the AWS acces key and secret key for the user you have created

11 - enter the name of the bucket you have created

12 - test the module to see if images are uploaded to the bucket

13 - go to a page in PW and drag & drop an image

14 - refresh the page on the amazon s3 console and you will see the image there in the file and page folder

15 - go to amazon aws console and go to cloudfront

16 - Create Distribution

17 - Web distribution => Get Started

18 - Origin Domain Name => from dropdown choose the bucket you want

19 - Restrict Bucket Access - NO

20 - Create Distribution

21 - Allow the distributiion time to deploy and once status is deployed, it can be used

22 - copy domain name of distribution for entry in PW Admin of module
 

That's it, test and enjoy!

  • Like 1

Share this post


Link to post
Share on other sites

Hotlinking Cloudfront solution:

s3 has bucket policy that you can instruct to only allow images get commands from certain domains and or IP's. But cloudfront does not have this, so this only prtects from direct linking to your s3 files. You can apply some level of defensive tactics to limit hotlinking to your cloudfront distributions.

Here's what I figured out:

1 - On cloudfront create a second distribution that you connect to the same bucket that you are already using, wait for it to fully deploy, then:

2 - Check if ithe files can be served from the new cloudfront distribution, then in PW amazons3/loudfront module, change domain name of cloudfront distribution to new distribution, check if all works and disable the old distribution, it can be deleted it you like (I would if I know it is being used for hotlinking).

This does not stop hotlinking right away, but allows you to stop images from being served from your cloudfront distribution as soon as you switch.

The hotlinking sites will now no longer have images showing up and will have to go to your site and get the new locations.

At least it is an easy defense technique to protect against paying for traffic for images not served via your site. Not fool proof, but it slows things down and makes it harder for leaches...

BTW, it is wise to apply a cache-control http header to the s3 bucket and get the distribution to use this, so your images are served with cache control headers, saves you from unnecessary traffic costs on your s3 account.

Would be nice to see this module working together with the minimize solution module. That would really make things easy!

  • Like 1

Share this post


Link to post
Share on other sites

To use amazon cloud front, does the file have to be in s3?  Or is it possible to have cloud front work with local processwire files?

Great plugin!  I am bookmarking it.  Was just curious how amazon works!

Thanks!

CloudFront uses S3. There's no other way.

Share this post


Link to post
Share on other sites

I've not tried yet to block other sites from hotlinking my CloudFront content. 

You can also check to apply a cache control http header on the module itself. It will apply it to every file uploaded via the module. 

Hotlinking Cloudfront solution:

s3 has bucket policy that you can instruct to only allow images get commands from certain domains and or IP's. But cloudfront does not have this, so this only prtects from direct linking to your s3 files. You can apply some level of defensive tactics to limit hotlinking to your cloudfront distributions.

Here's what I figured out:

1 - On cloudfront create a second distribution that you connect to the same bucket that you are already using, wait for it to fully deploy, then:

2 - Check if ithe files can be served from the new cloudfront distribution, then in PW amazons3/loudfront module, change domain name of cloudfront distribution to new distribution, check if all works and disable the old distribution, it can be deleted it you like (I would if I know it is being used for hotlinking).

This does not stop hotlinking right away, but allows you to stop images from being served from your cloudfront distribution as soon as you switch.

The hotlinking sites will now no longer have images showing up and will have to go to your site and get the new locations.

At least it is an easy defense technique to protect against paying for traffic for images not served via your site. Not fool proof, but it slows things down and makes it harder for leaches...

BTW, it is wise to apply a cache-control http header to the s3 bucket and get the distribution to use this, so your images are served with cache control headers, saves you from unnecessary traffic costs on your s3 account.

Would be nice to see this module working together with the minimize solution module. That would really make things easy!

Share this post


Link to post
Share on other sites

To work on PW 3+, you need to change two lines on AmazonS3Cloudfront.module:

Line 1 - Add PW namespace:

<?php namespace ProcessWire;

Line 21 - Add a slash before the class name. This references it on global namespace, not on PW's:

use \Aws\S3\S3Client;
  • Like 2

Share this post


Link to post
Share on other sites

Hi @zyON, I just submitted a pull request:

---

Now when a image is resized via API, upload this size variation too to S3.
I also changed line 21 to make it run on PW 3+

---

Could you kindly review it? :)

  • Like 1

Share this post


Link to post
Share on other sites

Hi nmendes, awesome module, don't know if you're still maintaining it. I think a couple improvements could be done:

  1. Would be the possibility to mimic the pw structure under a folder within a bucket that way many different webpages could use the same bucket ( because the number of buckets per user is limited)
  2. That one would be the best if you could. Not mimic pw structure but use S3 as the place to save the files, because Gb price on a server is far more expensive than using S3 (used by dropbox, spotify and other big companies), if you've a page with many videos, audios...

Thanks and congratulations.

Share this post


Link to post
Share on other sites

1. it is possible. Are you imagining something like "example.com/2060/name-of-file"? Where 2060 is the page id.

2. Didn't get exactly what you want in this one. You mean bypass completely the local filesystem and only store the files on S3?  

Share this post


Link to post
Share on other sites

I think people have looked at it and found the processwire admin hard codes local folders so it would be hard to have a website only use a remote server for files without modifications to processwire.  (Only save files in s3)

Share this post


Link to post
Share on other sites
On 4/1/2016 at 9:34 PM, Sérgio Jardim said:

Hi @zyON, I just submitted a pull request:

---

Now when a image is resized via API, upload this size variation too to S3.
I also changed line 21 to make it run on PW 3+

---

Could you kindly review it? :)

Hi @Sérgio Jardim

I'm looking for this functionality, too. Did you actually get this working, where image resizes are uploaded to S3? It seems the module is perhaps no longer being developed, but it still seems to work in PW 3 - I'd like to enhance at least my copy of it. 

Thanks

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Mike Rockett
      Jumplinks for ProcessWire
      Release: 1.5.56
      Composer: rockett/jumplinks
      Jumplinks is an enhanced version of the original ProcessRedirects by Antti Peisa.
      The Process module manages your permanent and temporary redirects (we'll call these "jumplinks" from now on, unless in reference to redirects from another module), useful for when you're migrating over to ProcessWire from another system/platform. Each jumplink supports wildcards, shortening the time needed to create them.
      Unlike similar modules for other platforms, wildcards in Jumplinks are much easier to work with, as Regular Expressions are not fully exposed. Instead, parameters wrapped in curly braces are used - these are described in the documentation.
      Under Development: 2.0, to be powered by FastRoute
      As of version 1.5.0, Jumplinks requires at least ProcessWire 2.6.1 to run.
      View on GitLab
      Download via the Modules Directory
      Read the docs
      Features
      The most prominent features include:
      Basic jumplinks (from one fixed route to another) Parameter-based wildcards with "Smart" equivalents Mapping Collections (for converting ID-based routes to their named-equivalents without the need to create multiple jumplinks) Destination Selectors (for finding and redirecting to pages containing legacy location information) Timed Activation (activate and/or deactivate jumplinks at specific times) 404-Monitor (for creating jumplinks based on 404 hits) Additionally, the following features may come in handy:
      Stale jumplink management Legacy domain support for slow migrations An importer (from CSV or ProcessRedirects) Feedback & Feature Requests
      I’d love to know what you think of this module. Please provide some feedback on the module as a whole, or even regarding smaller things that make it whole. Also, please feel free to submit feature requests and their use-cases.
      Note: Features requested so far have been added to the to-do list, and will be added to 2.0, and not the current dev/master branches.
      Open Source

      Jumplinks is an open-source project, and is free to use. In fact, Jumplinks will always be open-source, and will always remain free to use. Forever. If you would like to support the development of Jumplinks, please consider making a small donation via PayPal.
      Enjoy! :)
    • By BitPoet
      As threatened in Ryan's announcement for 3.0.139, I built a little module for sliding toggles as a replacement for checkboxes. Styling of the input is CSS3 only (with all the usual caveats about older browsers), no JS necessary, and may still be a bit "rough around the edges", so to speak, since I didn't have much time for testing on different devices or brushing things up enough so I'd feel comfortable pushing it to the module directory. But here's the link to the GitHub repo for now:
      InputfieldSlideToggle
      Fieldtype and Inputfield that implements smartphone-style toggles as replacement for checkbox inputs. The visualization is CSS-only, no additional JS necessary.
      Status
      Still very alpha, use with caution!
      Features / Field Settings
      Size
      You can render the toggles in four different sizes: small, medium, large and extra large.
      Off Color
      Currently, "unchecked" toggles can be displayed either in grey (default) or red.
      On Color
      "Checked" toggles can be rendered in one of these colors: blue (default), black, green, grey, orange or red.
      Screenshots

      Some examples with checkbox label


      View all Size and Color Combinations
      Small toggles Medium toggles Big toggles Extra big toggles  









    • By Orkun
      Hi Guys
      I needed to add extended functionalities for the InputfieldDatetime Module (module is from processwire version 2.7.3) because of a Request of Customer.
      So I duplicated the module and placed it under /site/modules/.
      I have added 3 new Settings to the InputfieldDatetime Module.
      1. Day Restriction - Restrict different days based on weekdays selection (e.g. saturday, sunday) - WORKING

       
      2. Time Slots - Define Time slots based on custom Integer Value (max is 60 for 1 hour) - WORKING

       
      3. Time Range Rules per Weekday - Define a minTime and MaxTime per Weekday (e.g. Opening Hours of a Restaurant) - NOT WORKING PROPERLY

       
      The Problem
      Time Slots and Day Restriction working fine so far. But the Time Range Rules per Weekday doesn't work right.
      What should happen is, that when you click on a date, it should update the minTime and maxTime of the Time Select.
      But the change on the select only happens if you select a date 2 times or when you select a date 1 time and then close the datepicker and reopen it again.
      The time select doesn't get change when you select a date 1 time and don't close the picker.
      Here is the whole extended InputfieldDatetime Module.
      The Files that I have changed:
      InputfieldDatetime.module InputfieldDatetime.js jquery-ui-timepicker-addon.js (https://trentrichardson.com/examples/timepicker/) - updated it to the newest version, because minTime and maxTime Option was only available in the new version  
      Thats the Part of the JS that is not working correctly:
      if(datetimerules && datetimerules.length){ options.onSelect = function(date, inst) { var day = $(this).datetimepicker("getDate").getDay(); day = day.toString(); var mintime = $(this).attr('data-weekday'+day+'-mintime'); var maxtime = $(this).attr('data-weekday'+day+'-maxtime'); console.log("weekday: "+day); console.log("minTime: "+mintime); console.log("maxTime: "+maxtime); var optionsAll = $(this).datetimepicker( "option", "all" ); optionsAll.minTime = mintime; optionsAll.maxTime = maxtime; $(this).datetimepicker('destroy'); $(this).datetimepicker(optionsAll); $(this).datetimepicker('refresh'); //$.datepicker._selectDate($(this).attr("id"),date); //$.datepicker._base_getDateDatepicker(); // var inst = $.datepicker._getInst($(this)); // $.datepicker._updateDatepicker(inst); /*$(this).datetimepicker('destroy'); InputfieldDatetimeDatepicker($(this), mintime, maxtime); $(this).datetimepicker('refresh'); */ // $(this).datetimepicker('option', {minTime: mintime, maxTime: maxtime}); } } Can you have a look and find out what the Problem is?
      InputfieldDatetime.zip
       
      Kind Regards
      Orkun
    • By teppo
      This module tracks changes, additions, removals etc. of public (as in "not under admin") pages of your site. Like it's name says, it doesn't attempt to be a version control system or anything like that - just a log of what's happened.
      At the moment it's still a work in progress and will most likely be a victim of many ruthless this-won't-work-let's-try-that-instead cycles, but I believe I've nailed basic functionality well enough to post it here.. so, once again, I'll be happy to hear any comments you folks can provide
      https://modules.processwire.com/modules/process-changelog/
      https://github.com/teppokoivula/ProcessChangelog
      How does it work?
      Exactly like it's (sort of) predecessor, Process Changelog actually consists of two modules: Process Changelog and Process Changelog Hooks. Hooks module exists only to serve main module by hooking into various functions within Pages class, collecting data of performed operations, refining it and keeping up a log of events in it's own custom database table (process_changelog.) Visible part is managed by Process Changelog, which provides users a (relatively) pretty view of the contents of said log table.
      How do you use it?
      When installed this module adds new page called Changelog under Admin > Setup which provides you with a table view of collected data and basic filtering tools See attached screenshots to get a general idea about what that page should look like after a while.
      For detailed installation instructions etc. see README.md.
       


    • By Gadgetto
      Status update links (inside this thread) for SnipWire development will be always posted here:
      2019-08-08
      2019-06-15
      2019-06-02
      2019-05-25
      If you are interested, you can test the current state of development:
      https://github.com/gadgetto/SnipWire
      Please note that the software is not yet intended for use in a production system (alpha version).
      If you like, you can also submit feature requests and suggestions for improvement. I also accept pull requests.
      ---- INITIAL POST FROM 2019-05-25 ----
      I wanted to let you know that I am currently working on a new ProcessWire module that fully integrates the Snipcart Shopping Cart System into ProcessWire. (this is a customer project, so I had to postpone the development of my other module GroupMailer).
      The new module SnipWire offers full integration of the Snipcart Shopping Cart System into ProcessWire.
      Here are some highlights:
      simple setup with (optional) pre-installed templates, product fields, sample products (quasi a complete shop system to get started immediately) store dashboard with all data from the snipcart system (no change to the snipcart dashboard itself required) Integrated REST API for controlling and querying snipcart data webhooks to trigger events from Snipcart (new order, new customer, etc.) multi currency support self-defined/configurable tax rates etc. Development is already well advanced and I plan to release the module in the next 2-3 months.
      I'm not sure yet if this will be a "Pro" module or if it will be made available for free.
      I would be grateful for suggestions and hints!
      (please have a look at the screenshots to get an idea what I'm talking about)
       




×
×
  • Create New...