dfunk006

Custom Fieldtype for Amazon S3

Recommended Posts

Hi,

Would anyone be interested in creating a custom fieldtype where images are uploaded to an Amazon S3 bucket instead of the /site/assets/ folder?

Thanks 

  • Like 3

Share this post


Link to post
Share on other sites

I need this also.  Backblaze b2 would be great even as it is even cheaper on both bandwith and storage space.

Share this post


Link to post
Share on other sites

Hey @joer80,
I've been developing a module that mirrors every (or ones you specify) upload to S3 bucket. I've started with S3 and Cloudfront support and I'm planning to support other CDNs in the future. Backblaze seems like a great provider to include as well. Once it is finished, it will be a near complete CDN solution.

Right now basic uploads are working fine, but it's not ready for production yet. However, I hope to complete and release it in a few weeks. Is there any feature that you think would be nice?

Some features I'm planning to include before releasing:

  • Full Amazon S3 support
  • Origin pull support
  • Process module/page for managing buckets and objects, such as clearing buckets, force uploading files & directories, maintenance mode etc.
  • Configuration page
  • Caching uploaded file paths in DB to keep track of uploaded files and fallback to local to local urls to prevent 404 errors.
  • Textformatter module for rewriting local urls to CDN urls. Optional custom domain support etc.
  • Form integration with AJAX for quick upload/upload all/force upload/delete/delete all etc buttons for file/image fields while editing a page/image.
  • Hooks for custom file uploads
  • Extendable/hookable API for new services

Future considerations

  • Other providers. Backblaze, MaxCDN etc.
  • Wireshell integration, (not sure how extensible WS is, but it'd be nice to have)
  • CSS rewrites? (for imports, images etc)

 

Edited by abdus
Added plans
  • Like 3

Share this post


Link to post
Share on other sites

With a custom field type you are not required to store the file locally.  You can choose to have it only on s3 or b2.  Is that the case with your module?  Or do you need to keep it both places?

With so many vps being small ssd, it would be great to offline files to cloud. 

Share this post


Link to post
Share on other sites
29 minutes ago, joer80 said:

Or do you need to keep it both places?

There's no need. I'll provide an option to remove local copy right away on successful upload, and a maintenance function to remove mirrored files from local drive as well.

The module doesn't even need to be a custom fieldtype, it hooks into built-in modules to get file paths then upload to cloud.

  • Like 3

Share this post


Link to post
Share on other sites

If that is the case, I would like to test it out as soon as it is ready!  That would be great!

Share this post


Link to post
Share on other sites
On 5/1/2017 at 1:06 PM, abdus said:

Hey @joer80,
I've been developing a module that mirrors every (or ones you specify) upload to S3 bucket. I've started with S3 and Cloudfront support and I'm planning to support other CDNs in the future. Backblaze seems like a great provider to include as well. Once it is finished, it will be a near complete CDN solution.

Right now basic uploads are working fine, but it's not ready for production yet. However, I hope to complete and release it in a few weeks. Is there any feature that you think would be nice?

Some features I'm planning to include before releasing:

  • Full Amazon S3 support
  • Origin pull support
  • Process module/page for managing buckets and objects, such as clearing buckets, force uploading files & directories, maintenance mode etc.
  • Configuration page
  • Caching uploaded file paths in DB to keep track of uploaded files and fallback to local to local urls to prevent 404 errors.
  • Textformatter module for rewriting local urls to CDN urls. Optional custom domain support etc.
  • Form integration with AJAX for quick upload/upload all/force upload/delete/delete all etc buttons for file/image fields while editing a page/image.
  • Hooks for custom file uploads
  • Extendable/hookable API for new services

Future considerations

  • Other providers. Backblaze, MaxCDN etc.
  • Wireshell integration, (not sure how extensible WS is, but it'd be nice to have)
  • CSS rewrites? (for imports, images etc)

 

How is your project coming along?

Share this post


Link to post
Share on other sites

Hey @joer80,

I've been dealing with finals/graduation/masters applications lately and cannot spend as much time coding as I would like. Because of this I haven't been active in the forum for the last two weeks as well.
However, hopefully tomorrow, after submitting a project which filled my last three weeks, I'll be finished with school (for now) and be able to go back on coding. Once I get a usable beta going, I'll open a topic for the updates and feedback.

Thanks a lot for your interest.
Abdus.

  • Like 3

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By taoguang
      By default,  pw upload image or video to local ecs space, How to change to Amazon S3 space?
    • By _NameLess_
      Hello,
      as an example imagine this structure:
      Templates:
          CarManufacturerTemplate
              Name: String
          CarTemplate
              Name: String
              Owner: String
              CarManufacturer: Page(where Template == CarManufacturerTemplate)
      Pages:
          CarManufacturers
              Manufacturer 1 [Template = CarManufacturerTemplate]
                  Name = "Toyota"
              Manufacturer 2 [Template = CarManufacturerTemplate]
                  Name = "Ford"
          Cars
              Car 1 [Template = CarTemplate]
                  Name = "Car 1"
                  Owner = "Steve"
                  CarManufacturer = Manufacturer 1 (Shown as "Toyota")
              Car 2 [Template = CarTemplate]
                  Name = "Car 2"
                  Owner = "Bob"
                  CarManufacturer = Manufacturer 1 (Shown as "Toyota")
              Car 3 [Template = CarTemplate]
                  Name = "Car 3"
                  Owner = "Jack"
                  CarManufacturer = Manufacturer 2 (Shown as "Ford")
      So there are multiple manufacturers, multiple cars, and every car has a reference to its manufacturer.
      On the page mysite/cars/car-1/ I want to be able to present the data in a manufacturer-specific way but without doing something like this:
      <?php
      if ($page->CarManufacturer->Name == "Toyota") {
        //Show Toyota specific formatting
      } else if ($page->CarManufacturer->Name == "Ford") {
        //Show Ford specific formatting
      } ... ?>
      Instead I want to be able to give every child-page of the CarManufacturers-page some code which takes the required data as parameters and creates the formatted output on its own. This way, whenever I add a new manufacturer, I don't have to go through all the places which output manufacturer-specific data and add another if-statement. Instead I want to be able to write code on a per-manufacturer basis.
      A possible way of doing this would be a FieldType which allows me to enter PHP code in the backend.
      I'd add another field to the CarManufacturer-template:
      Templates:
          CarManufacturerTemplate
              Name: String
              CustomFunctions: MyCustomTypeWhichExecutesPHP
      And  the backend page for Manufacturer 1 (Toyota) looks like this:
      |-------------
      | Name:
      | |--------
      | | Toyota
      | |--------
      | CustomFunctions:
      | |------------------------------------------------------
      | | public function OutputCarData($SomeCarData) {
      | |   //Show Toyota specific formatting
      | | }
      | |------------------------------------------------------
      And in the template-code of CarTemplate I can just do this:
      <?php
      $SomeCarData = $page->whatever;
      $page->CarManufacturer->CustomFunctions->OutputCarData($SomeCarData);
      ?>
      And I don't need to change that code at all when I add another car manufacturer. I just type in another function OutputCarData which does the formatting appropriately.
      I hope there is a simpler way for this than creating a custom FieldType. If you have any further questions feel free to ask and I will try to answer them or provide more details in what I want to achieve.
      Many thanks,
      _NameLess_