Jump to content

FieldtypeFileS3 - store files on AWS S3


fbg13
 Share

Recommended Posts

FieldtypeFileS3

https://github.com/f-b-g-m/FieldtypeFileS3

The module extends the default FieldtypeFile and InputfieldFile modules and adds few extra methods.

For the most part it behaves just like the default files modules, the biggest difference is how you get the file's url. Instead of using $page->fieldname->eq(0)->url you use $page->fieldname->eq(0)->s3url(). Files are not stored locally, they are deleted when the page is saved, if page saving is ommited the file remains on the local server until the page is saved. Another difference is the file size, the default module get the file size directly from the local file, while here it's stored in the database.

There is an option to store the files locally, its intented in case one wants to stop using S3 and change back to local storage. What it does is it changes the s3url() method to serve files from local server instead of S3, disables uploading to S3 and disables local file deletion on page save. It does not tranfer files from S3 to local server, that can be done with the aws-cli's through the sync function. Files stored on S3 have the same structure as they would have on the local server.

-------------------------------------------------------- --------------------------------------------------------

Been struggling with this for quite a while, but i think i finally managed to make it work/behave the way i wanted.


All feedback is welcome!

  • Like 15
Link to comment
Share on other sites

  • 5 months later...

Hi @fbg13, Thank you for your module!

I am in the process of learning AWS (EC2, RDS, S3) services and moving my first PW project to 'the cloud'.
As a part of all this, I have tested your module, and I did run into an issue: 

  1. The upload seemingly never finishes (ie. the progress bar does not finish). It does not display any warnings, but stays stuck at 100% transfer.
  2. Upon reloading the page I get a PHP-error: Warning: filemtime(): stat failed for /var/www/html/site/assets/files/12978/fullsizeoutput_2aa6.jpeg in /var/www/html/wire/core/Pagefile.php on line 389
  3. Even so, the file did transfer correctly and is present in the S3 bucket
  4. It is also present in the database and on the page
  5. In the database I see that the modified and created fields of the file are showing incorrect timestamps: 1970-01-01 01:00:00

I have configured the InputfieldFileS3 module with the correct Key, Secret and Bucket name.
The ACL is set to public-read, the region is set to the same my VPC is located,  SSL is on, Use my domain & Store files locally are off. 

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

 

Link to comment
Share on other sites

Hi, I've tested the field on another environment and there it worked fine using the same S3 credentials and settings. 
So, there must be some significant difference between the two causing this. 

Where it does not work: 

The working environment:

  • ProcessWire 3.0.36
  • PHP 5.6.27 on a shared hosting environment (Debian GNU/Linux 7)

As the domains are for staging/development I can send you a private message with links to the phpinfo output , perhaps there is some PHP module I forgot to install? 

Also, I checked the Chrome console after uploading, which shows why the spinner never finishes: the server error above is also showing up in the JSON that InputfieldFileS3.js tries to parse:

[{"error":false,"message":"Added file: img_8737.jpg","file":"\/site\/assets\/files\/12978\/img_8737.jpg","size":1391384,"markup":"<li id='file_72fa2d34b55cc08ec37b4551217bf582' class='InputfieldFileItem ui-widget ui-widget-content'><p class='InputfieldFileInfo InputfieldItemHeader ui-state-default ui-widget-header'><i class='fa fa-file-image-o fa-fw HideIfEmpty'><\/i><a class='InputfieldFileName' title='img_8737.jpg' target='_blank' href='https:\/\/s3-eu-central-1.amazonaws.com\/foamtalentcall-1\/12978\/img_8737.jpg'>img_8737.jpg<\/a> <span class='InputfieldFileStats'>1,359&nbsp;kB<\/span> <label class='InputfieldFileDelete'><input type='checkbox' name='delete_tests3_72fa2d34b55cc08ec37b4551217bf582' value='1' title='Delete' \/><i class='fa fa-fw fa-trash'><\/i><\/label><\/p><div class='InputfieldFileData description ui-widget-content'><div class='InputfieldFileDescription'><label for='description_tests3_72fa2d34b55cc08ec37b4551217bf582' class='detail'>Description<\/label><input type='text' name='description_tests3_72fa2d34b55cc08ec37b4551217bf582' id='description_tests3_72fa2d34b55cc08ec37b4551217bf582' value='' \/><\/div><input class='InputfieldFileSort' type='text' name='sort_tests3_72fa2d34b55cc08ec37b4551217bf582' value='0' \/><\/div><\/li>","replace":false,"overwrite":0}]<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />
<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />

I checked the Pagefile.php code and it seems that there is no default escape in the switch statement which tries to get information from the uploaded file.

Link to comment
Share on other sites

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

To be honest I don't know much about AWS and the module is very basic.

Link to comment
Share on other sites

13 minutes ago, fbg13 said:

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Yes, both the files and images fields work fine. Also resizing images through the API works, so it is not a rights issue I would say. 

Did you test it yourself with the latest version of PW? Maybe something changed between 3.0.36 and 3.0.62.

 

50 minutes ago, fbg13 said:

To be honest I don't know much about AWS and the module is very basic.

Yeah I'm also just starting out, but for this problem I think you should consider the EC2-instance just as a regular old Virtual Private Server.  

Link to comment
Share on other sites

I tried on PW 3.0.62 and 3.0.65.

1 hour ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

Did you try this?

19 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

Link to comment
Share on other sites

 

2 hours ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

I tried. While the front-end is still 'loading', and before reloading the page manually, I checked both the EC2 server and the S3 bucket.
The file is not available anywhere on the (EC2) server, but it did in fact transfer correctly to the S3 storage. 

The same warning as before also gets logged to error_log:

PHP Warning:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/hamburg-11.jpg in /var/www/html/wire/core/Pagefile.php on line 389, referer: http://ec2-52-57-230-10.eu-central-1.compute.amazonaws.com/admin/page/edit/?id=12978&s=1&c=1

 

2 hours ago, fbg13 said:

I tried on PW 3.0.62 and 3.0.65.

Maybe your error reporting does not show warnings? And do your entries in the database show the correct timestamps?

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally. 

 

2 hours ago, fbg13 said:
  22 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

No errors this way, nothing new gets added to Php's error_log.

Link to comment
Share on other sites

12 minutes ago, eelkenet said:

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally.

The file should be there though, as it is only removed when the page is saved. I have no idea why it's deleted.

Edit InputfieldFileS3.module file (line 83) and add

if(file_exists($pagefile->filename)){
	$this->wire("log")->error("file exists");
}

after

if(!$this->localStorage) {
	$this->uploadFileToS3($pagefile, $this->input->get->id);
}

and check the log. You can add it before too.

Link to comment
Share on other sites

@eelkenet Add it on the FieldtypeFileS3 file too on line 85, just before

if($field->fileSchema & self::fileSchemaDate) {
  $item['modified'] = date('Y-m-d H:i:s', $pagefile->modified);
  $item['created'] = date('Y-m-d H:i:s', $pagefile->created);
}

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

I'll try to find another way to handle this.

Link to comment
Share on other sites

@fbg13 

1 hour ago, fbg13 said:

Add it on the FieldtypeFileS3 file too on line 85

I did, with an else statement to make sure it would log anything. At this point the file does NOT exist! 
I added a check for the modified date on InputfieldFileS3.module (line 83) too, and that is valid too! (The exact time of uploading). 

Link to comment
Share on other sites

@eelkenet Somehow the file gets deleted before the page is saved.

Do you have any other modules installed? Did you change any of the field's default settings?

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

You can comment them out then you know for sure, search for unlink on InputfieldFileS3.module/FieldfieldFileS3.module.

Link to comment
Share on other sites

49 minutes ago, fbg13 said:

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

@fbg13 I logged all the unlink calls that I could find. This unlink gets executed in my case: https://github.com/fbg13/FieldtypeFileS3/blob/master/InputfieldFileS3.module#L229

In addition I also called error_log(), and this happens about 1 ms before the original error gets thrown. 

Link to comment
Share on other sites

2 minutes ago, fbg13 said:

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

I did not save the page. The upload is 'still processing', ie. stuck at 100%. 

Link to comment
Share on other sites

50 minutes ago, fbg13 said:

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. :-[

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

@fbg13 Yes, that fixes it! Congratulations, no errors, and the file is on the S3 :) :frantics:

Link to comment
Share on other sites

  • 4 years later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Similar Content

    • By MarkE
      This fieldtype and inputfield bundle was built for storing measurement values within a field, rendering them in a variety of formats and converting them to other units or otherwise modifying them via the API.
      The API consists of a number of predefined functions, some of which include...
      render() for rendering the measurement object, valueAs() for converting the value to another unit value, convertTo() for converting the whole measurement object to different units, and add() and subtract() for for modifying the stored value by the value (converted as required) in another measurement. In the admin the inputfield includes a checkbox (which can be optionally disabled) for converting values on page save. For an example if a value was typed in as centimeters, the unit was changed to metres, and the page saved with this checkbox selected, said value would be automatically converted so that e.g. 170 cm becomes 1.7 m.

      A simple length field using Fieldtype Measurement and Inputfield Measurement.
      Combination units (e.g. feet and inches) are also supported.
      Please note that this module is 'proof of concept' at the moment - there are limited units available and quite a lot of code tidying to do. More units will be added shortly.
      See the GitHub at https://github.com/MetaTunes/FieldtypeMeasurement for full details and updates.
    • By tcnet
      File Manager for ProcessWire is a module to manager files and folders from the CMS backend. It supports creating, deleting, renaming, packing, unpacking, uploading, downloading and editing of files and folders. The integrated code editor ACE supports highlighting of all common programming languages.
      https://github.com/techcnet/ProcessFileManager

      Warning
      This module is probably the most powerful module. You might destroy your processwire installation if you don't exactly know what you doing. Be careful and use it at your own risk!
      ACE code editor
      This module uses ACE code editor available from: https://github.com/ajaxorg/ace

      Dragscroll
      This module uses the JavaScript dragscroll available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability to drag the table horizontally with the mouse pointer.
      PHP File Manager
      This module uses a modified version of PHP File Manager available from: https://github.com/alexantr/filemanager
       
    • By tcnet
      This module implements the website live chat service from tawk.to. Actually the module doesn't have to do much. It just need to inserted a few lines of JavaScript just before the closing body tag </body> on each side. However, the module offers additional options to display the widget only on certain pages.
      Create an account
      Visit https://www.tawk.to and create an account. It's free! At some point you will reach a page where you can copy the required JavaScript-code.

      Open the module settings and paste the JavaScript-code into the field as shown below. Click "Submit" and that's all.

      Open the module settings
      The settings for this module are located int the menu Modules=>Configure=>LiveChatTawkTo.

       
    • By tcnet
      Session Viewer is a module for ProcessWire to list session files and display session data. This module is helpful to display the session data of a specific session or to kick out a logged in user by simply delete his session file. After installation the module is available in the Setup menu.

      The following conditions must be met for the module to work properly:
      Session files
      Session data must be stored in session files, which is the default way in ProcessWire. Sessions stored in the database are not supported by this module. The path to the directory where the session files are stored must be declared in the ProcessWire configuration which is by default: site/assets/sessions.
      Serialize handler
      In order to transform session data easier back to a PHP array, the session data is stored serialized. PHP offers a way to declare a custom serialize handler. This module supports only the default serialize handlers: php, php_binary and php_serialize. WDDX was dropped in PHP 7.4.0 and is therefore not supported by this module as well as any other custom serialize handler. Which serialize handler is actually used you can find out in the module configuration which is available under Modules=>Configure=>SessionViewer.

      Session data
      The session data can be displayed in two different ways. PHP's default output for arrays print_r() or by default for this module nice_r() offered on github: https://github.com/uuf6429/nice_r. There is a setting in the module configuration if someone prefers print_r(). Apart from the better handling and overview of the folded session data the output of nice_r() looks indeed nicer.

      Links
      ProcessWire module directory
      github.com
    • By Robin S
      Repeater Easy Sort
      Adds a compact "easy-sort" mode to Repeater and Repeater Matrix, making those fields easier to sort when there are a large number of items.
      The module also enhances Repeater Matrix by allowing a colour to be set for each matrix type. This colour is used in the item headers and in the "add new" links, to help visually distinguish different matrix types in the inputfield.
      Screencasts
      A Repeater field

      A Repeater Matrix field with custom header colours

      Easy-sort mode
      Each Repeater/Matrix item gets an double-arrow icon in the item header. Click this icon to enter easy-sort mode.
      While in easy-sort mode:
      The items will reduce in width so that more items can be shown on the screen at once. The minimum width is configurable in the field settings. Any items that were in an open state are collapsed, but when you exit easy-sort mode the previously open items will be reopened. You can drag an item left/right/up/down to sort it within the items. The item that you clicked the icon for is shown with a black background. This makes it easier to find the item you want to move in easy-sort mode. You can click an item header to open the item. An "Exit easy-sort mode" button appears at the bottom of the inputfield. Configuration
      In the field settings for Repeater and Repeater Matrix fields you can define a minimum width in pixels for items in easy-sort mode. While in easy-sort mode the items will be sized to neatly fill the available width on any screen size but will never be narrower than the width you set here.
      In the field settings for Repeater Matrix you can define a custom header colour for each matrix type using an HTML "color" type input. The default colour for this type of input is black, so when black is selected in the input it means that no custom colour will be applied to the header.
      Exclusions
      The easy-sort mode is only possible on Repeater/Matrix fields that do not use the "item depth" option.
       
      https://github.com/Toutouwai/RepeaterEasySort
      https://processwire.com/modules/repeater-easy-sort/
×
×
  • Create New...