Jump to content
fbg13

FieldtypeFileS3 - store files on AWS S3

Recommended Posts

FieldtypeFileS3

https://github.com/f-b-g-m/FieldtypeFileS3

The module extends the default FieldtypeFile and InputfieldFile modules and adds few extra methods.

For the most part it behaves just like the default files modules, the biggest difference is how you get the file's url. Instead of using $page->fieldname->eq(0)->url you use $page->fieldname->eq(0)->s3url(). Files are not stored locally, they are deleted when the page is saved, if page saving is ommited the file remains on the local server until the page is saved. Another difference is the file size, the default module get the file size directly from the local file, while here it's stored in the database.

There is an option to store the files locally, its intented in case one wants to stop using S3 and change back to local storage. What it does is it changes the s3url() method to serve files from local server instead of S3, disables uploading to S3 and disables local file deletion on page save. It does not tranfer files from S3 to local server, that can be done with the aws-cli's through the sync function. Files stored on S3 have the same structure as they would have on the local server.

-------------------------------------------------------- --------------------------------------------------------

Been struggling with this for quite a while, but i think i finally managed to make it work/behave the way i wanted.


All feedback is welcome!

  • Like 14

Share this post


Link to post
Share on other sites

Hi @fbg13, Thank you for your module!

I am in the process of learning AWS (EC2, RDS, S3) services and moving my first PW project to 'the cloud'.
As a part of all this, I have tested your module, and I did run into an issue: 

  1. The upload seemingly never finishes (ie. the progress bar does not finish). It does not display any warnings, but stays stuck at 100% transfer.
  2. Upon reloading the page I get a PHP-error: Warning: filemtime(): stat failed for /var/www/html/site/assets/files/12978/fullsizeoutput_2aa6.jpeg in /var/www/html/wire/core/Pagefile.php on line 389
  3. Even so, the file did transfer correctly and is present in the S3 bucket
  4. It is also present in the database and on the page
  5. In the database I see that the modified and created fields of the file are showing incorrect timestamps: 1970-01-01 01:00:00

I have configured the InputfieldFileS3 module with the correct Key, Secret and Bucket name.
The ACL is set to public-read, the region is set to the same my VPC is located,  SSL is on, Use my domain & Store files locally are off. 

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

 

Share this post


Link to post
Share on other sites

@eelkenet I just tested it (windows 10 and ubuntu 17.04) and had no problems on php 5.6 an up.

On versions < php 5.6 i got the never ending spinner but not the warning.

What php version are you using?

Share this post


Link to post
Share on other sites

Hi, I've tested the field on another environment and there it worked fine using the same S3 credentials and settings. 
So, there must be some significant difference between the two causing this. 

Where it does not work: 

The working environment:

  • ProcessWire 3.0.36
  • PHP 5.6.27 on a shared hosting environment (Debian GNU/Linux 7)

As the domains are for staging/development I can send you a private message with links to the phpinfo output , perhaps there is some PHP module I forgot to install? 

Also, I checked the Chrome console after uploading, which shows why the spinner never finishes: the server error above is also showing up in the JSON that InputfieldFileS3.js tries to parse:

[{"error":false,"message":"Added file: img_8737.jpg","file":"\/site\/assets\/files\/12978\/img_8737.jpg","size":1391384,"markup":"<li id='file_72fa2d34b55cc08ec37b4551217bf582' class='InputfieldFileItem ui-widget ui-widget-content'><p class='InputfieldFileInfo InputfieldItemHeader ui-state-default ui-widget-header'><i class='fa fa-file-image-o fa-fw HideIfEmpty'><\/i><a class='InputfieldFileName' title='img_8737.jpg' target='_blank' href='https:\/\/s3-eu-central-1.amazonaws.com\/foamtalentcall-1\/12978\/img_8737.jpg'>img_8737.jpg<\/a> <span class='InputfieldFileStats'>1,359&nbsp;kB<\/span> <label class='InputfieldFileDelete'><input type='checkbox' name='delete_tests3_72fa2d34b55cc08ec37b4551217bf582' value='1' title='Delete' \/><i class='fa fa-fw fa-trash'><\/i><\/label><\/p><div class='InputfieldFileData description ui-widget-content'><div class='InputfieldFileDescription'><label for='description_tests3_72fa2d34b55cc08ec37b4551217bf582' class='detail'>Description<\/label><input type='text' name='description_tests3_72fa2d34b55cc08ec37b4551217bf582' id='description_tests3_72fa2d34b55cc08ec37b4551217bf582' value='' \/><\/div><input class='InputfieldFileSort' type='text' name='sort_tests3_72fa2d34b55cc08ec37b4551217bf582' value='0' \/><\/div><\/li>","replace":false,"overwrite":0}]<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />
<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />

I checked the Pagefile.php code and it seems that there is no default escape in the switch statement which tries to get information from the uploaded file.

Share this post


Link to post
Share on other sites

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

To be honest I don't know much about AWS and the module is very basic.

Share this post


Link to post
Share on other sites
13 minutes ago, fbg13 said:

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Yes, both the files and images fields work fine. Also resizing images through the API works, so it is not a rights issue I would say. 

Did you test it yourself with the latest version of PW? Maybe something changed between 3.0.36 and 3.0.62.

 

50 minutes ago, fbg13 said:

To be honest I don't know much about AWS and the module is very basic.

Yeah I'm also just starting out, but for this problem I think you should consider the EC2-instance just as a regular old Virtual Private Server.  

Share this post


Link to post
Share on other sites

I tried on PW 3.0.62 and 3.0.65.

1 hour ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

Did you try this?

19 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

Share this post


Link to post
Share on other sites

 

2 hours ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

I tried. While the front-end is still 'loading', and before reloading the page manually, I checked both the EC2 server and the S3 bucket.
The file is not available anywhere on the (EC2) server, but it did in fact transfer correctly to the S3 storage. 

The same warning as before also gets logged to error_log:

PHP Warning:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/hamburg-11.jpg in /var/www/html/wire/core/Pagefile.php on line 389, referer: http://ec2-52-57-230-10.eu-central-1.compute.amazonaws.com/admin/page/edit/?id=12978&s=1&c=1

 

2 hours ago, fbg13 said:

I tried on PW 3.0.62 and 3.0.65.

Maybe your error reporting does not show warnings? And do your entries in the database show the correct timestamps?

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally. 

 

2 hours ago, fbg13 said:
  22 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

No errors this way, nothing new gets added to Php's error_log.

Share this post


Link to post
Share on other sites
12 minutes ago, eelkenet said:

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally.

The file should be there though, as it is only removed when the page is saved. I have no idea why it's deleted.

Edit InputfieldFileS3.module file (line 83) and add

if(file_exists($pagefile->filename)){
	$this->wire("log")->error("file exists");
}

after

if(!$this->localStorage) {
	$this->uploadFileToS3($pagefile, $this->input->get->id);
}

and check the log. You can add it before too.

Share this post


Link to post
Share on other sites

@eelkenet Add it on the FieldtypeFileS3 file too on line 85, just before

if($field->fileSchema & self::fileSchemaDate) {
  $item['modified'] = date('Y-m-d H:i:s', $pagefile->modified);
  $item['created'] = date('Y-m-d H:i:s', $pagefile->created);
}

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

I'll try to find another way to handle this.

Share this post


Link to post
Share on other sites
1 hour ago, fbg13 said:

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

Hi @fbg13, I'm not sure what you want me to do here, can you rephrase? Thanks!

Share this post


Link to post
Share on other sites

@fbg13 

1 hour ago, fbg13 said:

Add it on the FieldtypeFileS3 file too on line 85

I did, with an else statement to make sure it would log anything. At this point the file does NOT exist! 
I added a check for the modified date on InputfieldFileS3.module (line 83) too, and that is valid too! (The exact time of uploading). 

Share this post


Link to post
Share on other sites

@eelkenet Somehow the file gets deleted before the page is saved.

Do you have any other modules installed? Did you change any of the field's default settings?

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

You can comment them out then you know for sure, search for unlink on InputfieldFileS3.module/FieldfieldFileS3.module.

Share this post


Link to post
Share on other sites
49 minutes ago, fbg13 said:

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

@fbg13 I logged all the unlink calls that I could find. This unlink gets executed in my case: https://github.com/fbg13/FieldtypeFileS3/blob/master/InputfieldFileS3.module#L229

In addition I also called error_log(), and this happens about 1 ms before the original error gets thrown. 

Share this post


Link to post
Share on other sites

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

Share this post


Link to post
Share on other sites
2 minutes ago, fbg13 said:

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

I did not save the page. The upload is 'still processing', ie. stuck at 100%. 

Share this post


Link to post
Share on other sites

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. :-[

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

  • Like 1

Share this post


Link to post
Share on other sites
50 minutes ago, fbg13 said:

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. :-[

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

@fbg13 Yes, that fixes it! Congratulations, no errors, and the file is on the S3 :) :frantics:

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By d'Hinnisdaël
      Format Datetime fields as Carbon instances.
      You can find the latest release and the complete readme on Github.
      Installation
      composer require daun/datetime-carbon-format Usage
      All Datetime fields will now be formatted as Carbon instances instead of strings. Some examples:
      // $page->date is a Datetime field // Output format: j/n/Y echo $page->date; // 20/10/2020 echo $page->date->add('7 days'); // 27/10/2020 echo $page->date->format('l, F j'); // Monday, October 20 echo $page->date->year; // 2020 echo $page->date->diffForHumans(); // 28 minutes ago Frontend only
      The ProcessWire admin seems to expect datetime fields to be strings. This module will only return Carbon instances on frontend page views.
      Date output format
      When casting a Carbon instance to a string (usually when outputting the field in a template), the field's date output format will be respected.
      Links
      GitHub • Readme • Carbon docs
       
       
      PS. I remember reading about a Carbon module in a recent newsletter, but couldn't find it anywhere. Was that you, @bernhard?
    • By MoritzLost
      TrelloWire
      This is a module that allows you to automatically create Trello cards for ProcessWire pages and update them when the pages are updated. This allows you to setup connected workflows. Card properties and change handling behaviour can be customized through the extensive module configuration. Every action the module performs is hookable, so you can modify when and how cards are created as much as you need to. The module also contains an API-component that makes it easy to make requests to the Trello API and build your own connected ProcessWire-Trello workflows.
      Warning: This module requires ProcessWire 3.0.167 which is above the current stable master release at the moment.
      Features
      All the things the module can do for you without any custom code: Create a new card on Trello whenever a page is added or published (you can select applicable templates). Configure the target board, target list, name and description for new cards. Add default labels and checklists to the card. Update the card whenever the page is updated (optional). When the status of the card changes (published / unpublished, hidden / unhidden, trashed / restored or deleted), move the card to a different list or archive or delete it (configurable). You can extend this through hooks in many ways: Modifiy when and how cards are created. Modify the card properties (Target board & list, title, description, et c.) before they are sent to Trello. Create your own workflows by utilizing an API helper class with many convenient utility methods to access the Trello API directly. Feedback & Future Plans
      Let me know what you think! In particular:
      If you find any bugs report them here or on Github, I'll try to fix them. This module was born out of a use-case for a client project where we manage new form submissions through Trello. I'm not sure how many use-cases there are for this module. If you do use it, tell me about it! The Trello API is pretty extensive, I'll try to add some more helper methods to the TrelloWireApi class (let me know if you need anything in particular). I'll think about how the module can support different workflows that include Twig – talk to me if you have a use-case! Next steps could be a dashboard to manage pages that are connected to a Trello card, or a new section in the settings tab to manage the Trello connection. But it depends on whether there is any interest in this 🙂 Links
      Repository on Github Complete module documentation (getting started, configuration & API documentation) TrelloWire in the modules directory Module configuration

    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate multible repeater items from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Configurable dialogs for copy and paste Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Attention, please note!
      Nested repeaters cannot be supported technically. Therefore a check is made to prevent this. However, a nested repeater can only be detected if the field name ends for example with "_repeater1234". For example, if your MatrixRepeater field is named like this: "content_repeater" or "content_repeater123", this field is identified as nested and the module does not load. In version 2.0.1 the identification has been changed so that a field ending with the name repeater is only detected as nested if at least a two-digit number sequence follows. But to avoid this problem completely, make sure that your repeater matrix field does NOT end with the name "repeater".
      Changelog
       
      2.0.1
      Bug fix: Thanks to @ngrmm I could discover a bug which causes that the module cannot be loaded if the MatrixRepeater field ends with the name "repeater". The code was adjusted and information about the problem was provided 2.0.0
      Feature: Copy multiple items at once! The fundament for copying multiple items was created by @Autofahrn - THX! Feature: Optionally you can disable the copy and/or paste dialog Bug fix: A fix suggestion when additional and normal repeater fields are present was contributed by @joshua - THX! 1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module (Version 2.0.1)
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: https://modules.processwire.com/modules/inputfield-repeater-matrix-duplicate/
      > Old stable version (1.0.4): https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate/releases/tag/1.0.4
    • By picarica
      so i am trying to put CustomHooksForVariations.module, a custom module, i am placing it into site/modules direcotry yet my modules page in admin panel gives me errors
      so this is the screen show when i refresh modules, i dont know why the shole hook is written on top of the page :||

      and this next image is when i try to install it, i saw that it is not defiuned modules.php but it shouldnt need to be ?, any ways i dont want to edit site's core just to make one moulde work there has to be a way

    • By Robin S
      A new module that hasn't had a lot of testing yet. Please do your own testing before deploying on any production website.
      Custom Paths
      Allows any page to have a custom path/URL.
      Note: Custom Paths is incompatible with the core LanguageSupportPageNames module. I have no experience working with LanguageSupportPageNames or multi-language sites in general so I'm not in a position to work out if a fix is possible. If anyone with multi-language experience can contribute a fix it would be much appreciated!
      Screenshot

      Usage
      The module creates a field named custom_path on install. Add the custom_path field to the template of any page you want to set a custom path for. Whatever path is entered into this field determines the path and URL of the page ($page->path and $page->url). Page numbers and URL segments are supported if these are enabled for the template, and previous custom paths are managed by PagePathHistory if that module is installed.
      The custom_path field appears on the Settings tab in Page Edit by default but there is an option in the module configuration to disable this if you want to position the field among the other template fields.
      If the custom_path field is populated for a page it should be a path that is relative to the site root and that starts with a forward slash. The module prevents the same custom path being set for more than one page.
      The custom_path value takes precedence over any ProcessWire path. You can even override the Home page by setting a custom path of "/" for a page.
      It is highly recommended to set access controls on the custom_path field so that only privileged roles can edit it: superuser-only is recommended.
      It is up to the user to set and maintain suitable custom paths for any pages where the module is in use. Make sure your custom paths are compatible with ProcessWire's $config and .htaccess settings, and if you are basing the custom path on the names of parent pages you will probably want to have a strategy for updating custom paths if parent pages are renamed or moved.
      Example hooks to Pages::saveReady
      You might want to use a Pages::saveReady hook to automatically set the custom path for some pages. Below are a couple of examples.
      1. In this example the start of the custom path is fixed but the end of the path will update dynamically according to the name of the page:
      $pages->addHookAfter('saveReady', function(HookEvent $event) { $page = $event->arguments(0); if($page->template == 'my_template') { $page->custom_path = "/some-custom/path-segments/$page->name/"; } }); 2. The Custom Paths module adds a new Page::realPath method/property that can be used to get the "real" ProcessWire path to a page that might have a custom path set. In this example the custom path for news items is derived from the real ProcessWire path but a parent named "news-items" is removed:
      $pages->addHookAfter('saveReady', function(HookEvent $event) { $page = $event->arguments(0); if($page->template == 'news_item') { $page->custom_path = str_replace('/news-items/', '/', $page->realPath); } }); Caveats
      The custom paths will be used automatically for links created in CKEditor fields, but if you have the "link abstraction" option enabled for CKEditor fields (Details > Markup/HTML (Content Type) > HTML Options) then you will see notices from MarkupQA warning you that it is unable to resolve the links.
      Installation
      Install the Custom Paths module.
      Uninstallation
      The custom_path field is not automatically deleted when the module is uninstalled. You can delete it manually if the field is no longer needed.
       
      https://github.com/Toutouwai/CustomPaths
      https://modules.processwire.com/modules/custom-paths/
×
×
  • Create New...