Jump to content

FieldtypeFileS3 - store files on AWS S3


fbg13
 Share

Recommended Posts

FieldtypeFileS3

https://github.com/f-b-g-m/FieldtypeFileS3

The module extends the default FieldtypeFile and InputfieldFile modules and adds few extra methods.

For the most part it behaves just like the default files modules, the biggest difference is how you get the file's url. Instead of using $page->fieldname->eq(0)->url you use $page->fieldname->eq(0)->s3url(). Files are not stored locally, they are deleted when the page is saved, if page saving is ommited the file remains on the local server until the page is saved. Another difference is the file size, the default module get the file size directly from the local file, while here it's stored in the database.

There is an option to store the files locally, its intented in case one wants to stop using S3 and change back to local storage. What it does is it changes the s3url() method to serve files from local server instead of S3, disables uploading to S3 and disables local file deletion on page save. It does not tranfer files from S3 to local server, that can be done with the aws-cli's through the sync function. Files stored on S3 have the same structure as they would have on the local server.

-------------------------------------------------------- --------------------------------------------------------

Been struggling with this for quite a while, but i think i finally managed to make it work/behave the way i wanted.


All feedback is welcome!

  • Like 15
Link to comment
Share on other sites

  • 5 months later...

Hi @fbg13, Thank you for your module!

I am in the process of learning AWS (EC2, RDS, S3) services and moving my first PW project to 'the cloud'.
As a part of all this, I have tested your module, and I did run into an issue: 

  1. The upload seemingly never finishes (ie. the progress bar does not finish). It does not display any warnings, but stays stuck at 100% transfer.
  2. Upon reloading the page I get a PHP-error: Warning: filemtime(): stat failed for /var/www/html/site/assets/files/12978/fullsizeoutput_2aa6.jpeg in /var/www/html/wire/core/Pagefile.php on line 389
  3. Even so, the file did transfer correctly and is present in the S3 bucket
  4. It is also present in the database and on the page
  5. In the database I see that the modified and created fields of the file are showing incorrect timestamps: 1970-01-01 01:00:00

I have configured the InputfieldFileS3 module with the correct Key, Secret and Bucket name.
The ACL is set to public-read, the region is set to the same my VPC is located,  SSL is on, Use my domain & Store files locally are off. 

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

 

Link to comment
Share on other sites

Hi, I've tested the field on another environment and there it worked fine using the same S3 credentials and settings. 
So, there must be some significant difference between the two causing this. 

Where it does not work: 

The working environment:

  • ProcessWire 3.0.36
  • PHP 5.6.27 on a shared hosting environment (Debian GNU/Linux 7)

As the domains are for staging/development I can send you a private message with links to the phpinfo output , perhaps there is some PHP module I forgot to install? 

Also, I checked the Chrome console after uploading, which shows why the spinner never finishes: the server error above is also showing up in the JSON that InputfieldFileS3.js tries to parse:

[{"error":false,"message":"Added file: img_8737.jpg","file":"\/site\/assets\/files\/12978\/img_8737.jpg","size":1391384,"markup":"<li id='file_72fa2d34b55cc08ec37b4551217bf582' class='InputfieldFileItem ui-widget ui-widget-content'><p class='InputfieldFileInfo InputfieldItemHeader ui-state-default ui-widget-header'><i class='fa fa-file-image-o fa-fw HideIfEmpty'><\/i><a class='InputfieldFileName' title='img_8737.jpg' target='_blank' href='https:\/\/s3-eu-central-1.amazonaws.com\/foamtalentcall-1\/12978\/img_8737.jpg'>img_8737.jpg<\/a> <span class='InputfieldFileStats'>1,359&nbsp;kB<\/span> <label class='InputfieldFileDelete'><input type='checkbox' name='delete_tests3_72fa2d34b55cc08ec37b4551217bf582' value='1' title='Delete' \/><i class='fa fa-fw fa-trash'><\/i><\/label><\/p><div class='InputfieldFileData description ui-widget-content'><div class='InputfieldFileDescription'><label for='description_tests3_72fa2d34b55cc08ec37b4551217bf582' class='detail'>Description<\/label><input type='text' name='description_tests3_72fa2d34b55cc08ec37b4551217bf582' id='description_tests3_72fa2d34b55cc08ec37b4551217bf582' value='' \/><\/div><input class='InputfieldFileSort' type='text' name='sort_tests3_72fa2d34b55cc08ec37b4551217bf582' value='0' \/><\/div><\/li>","replace":false,"overwrite":0}]<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />
<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />

I checked the Pagefile.php code and it seems that there is no default escape in the switch statement which tries to get information from the uploaded file.

Link to comment
Share on other sites

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

To be honest I don't know much about AWS and the module is very basic.

Link to comment
Share on other sites

13 minutes ago, fbg13 said:

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Yes, both the files and images fields work fine. Also resizing images through the API works, so it is not a rights issue I would say. 

Did you test it yourself with the latest version of PW? Maybe something changed between 3.0.36 and 3.0.62.

 

50 minutes ago, fbg13 said:

To be honest I don't know much about AWS and the module is very basic.

Yeah I'm also just starting out, but for this problem I think you should consider the EC2-instance just as a regular old Virtual Private Server.  

Link to comment
Share on other sites

I tried on PW 3.0.62 and 3.0.65.

1 hour ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

Did you try this?

19 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

Link to comment
Share on other sites

 

2 hours ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

I tried. While the front-end is still 'loading', and before reloading the page manually, I checked both the EC2 server and the S3 bucket.
The file is not available anywhere on the (EC2) server, but it did in fact transfer correctly to the S3 storage. 

The same warning as before also gets logged to error_log:

PHP Warning:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/hamburg-11.jpg in /var/www/html/wire/core/Pagefile.php on line 389, referer: http://ec2-52-57-230-10.eu-central-1.compute.amazonaws.com/admin/page/edit/?id=12978&s=1&c=1

 

2 hours ago, fbg13 said:

I tried on PW 3.0.62 and 3.0.65.

Maybe your error reporting does not show warnings? And do your entries in the database show the correct timestamps?

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally. 

 

2 hours ago, fbg13 said:
  22 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

No errors this way, nothing new gets added to Php's error_log.

Link to comment
Share on other sites

12 minutes ago, eelkenet said:

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally.

The file should be there though, as it is only removed when the page is saved. I have no idea why it's deleted.

Edit InputfieldFileS3.module file (line 83) and add

if(file_exists($pagefile->filename)){
	$this->wire("log")->error("file exists");
}

after

if(!$this->localStorage) {
	$this->uploadFileToS3($pagefile, $this->input->get->id);
}

and check the log. You can add it before too.

Link to comment
Share on other sites

@eelkenet Add it on the FieldtypeFileS3 file too on line 85, just before

if($field->fileSchema & self::fileSchemaDate) {
  $item['modified'] = date('Y-m-d H:i:s', $pagefile->modified);
  $item['created'] = date('Y-m-d H:i:s', $pagefile->created);
}

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

I'll try to find another way to handle this.

Link to comment
Share on other sites

@fbg13 

1 hour ago, fbg13 said:

Add it on the FieldtypeFileS3 file too on line 85

I did, with an else statement to make sure it would log anything. At this point the file does NOT exist! 
I added a check for the modified date on InputfieldFileS3.module (line 83) too, and that is valid too! (The exact time of uploading). 

Link to comment
Share on other sites

@eelkenet Somehow the file gets deleted before the page is saved.

Do you have any other modules installed? Did you change any of the field's default settings?

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

You can comment them out then you know for sure, search for unlink on InputfieldFileS3.module/FieldfieldFileS3.module.

Link to comment
Share on other sites

49 minutes ago, fbg13 said:

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

@fbg13 I logged all the unlink calls that I could find. This unlink gets executed in my case: https://github.com/fbg13/FieldtypeFileS3/blob/master/InputfieldFileS3.module#L229

In addition I also called error_log(), and this happens about 1 ms before the original error gets thrown. 

Link to comment
Share on other sites

2 minutes ago, fbg13 said:

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

I did not save the page. The upload is 'still processing', ie. stuck at 100%. 

Link to comment
Share on other sites

50 minutes ago, fbg13 said:

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. :-[

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

@fbg13 Yes, that fixes it! Congratulations, no errors, and the file is on the S3 :) :frantics:

Link to comment
Share on other sites

  • 4 years later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Similar Content

    • By monollonom
      PageMjmlToHtml
      Github: https://github.com/romaincazier/PageMjmlToHtml
      Modules directory: https://processwire.com/modules/page-mjml-to-html/
      A module allowing you to write your Processwire template using MJML and get a converted HTML output using MJML API.
      This is considered to be in alpha and as such needs some testing before being used in production!

      About
      Created by Mailjet, MJML is a markup language making it a breeze to create newsletters displayed consistently across all email clients.
      Write your template using MJML combined with Processwire’s API and this module will automatically convert your code into a working newsletter thanks to their free-to-use Rest API.
      Prerequisite
      For this module to work you will need to get an API key and paste it in the module’s configuration.
      Usage
      Once your credentials are validated, select the template(s) in which you’re using the MJML syntax, save and go visualize your page(s) to see if everything’s good. You will either get error/warning messages or your email properly formatted and ready-to-go.
      From there you can copy/paste the raw generated code in an external mailing service or distribute your newsletter using ProMailer.
      Features
      The MJML output is cached to avoid repetitive API calls Not cached if there are errors/warnings Cleared if the page is saved Cleared if the template file has been modified A simple (dumb?) code viewer highlights lines with errors/warnings A button is added to quickly copy the raw code of the generated newsletter Not added if the page is rendered outside of a PageView Only visible to users with the page’s edit permission A shortcut is also added under “View” in the edit page to open the raw code in a new tab Multi-languages support
      Notes
      The code viewer is only shown to superusers. If there’s an error the page will display:
      Only its title for guests Its title and a message inviting to contact the administrator for editors If you are using the markup regions output strategy, it might be best to not append files to preserve your MJML markup before calling the MJML API. This option is available in the module’s settings.
    • By Marco Ro
      Hi guys!
      I'm a bit anxious because this is the first module I present! (beta modulo) But I will finally be able to share something with the community too! :)
      This is a BETA version of the PayPal payment system called: PayPal Commerce Platform.
      It is an advanced system (Business Pro account is needed) that brings various benefits in terms of fees and above all integrates direct payment with credit/debit cards. 
      The module integrates with Padloper 0.0.2, which is the current installation I'm using.
      This system integrates the classic PayPal buy button, the alternative or local payment method and the new payment system: credit/debit cards that doesn't go through the PayPal account. It is a Stripe-style payment, it connects directly with the bank and integrates 3D security validation.
      I say that it is a BETA because this module currently only works with Sandbox account, to put it live you need to change API url manually (manually for the moment).
      Because this module is not ready for live:
      I would like to have your opinion on how I built the module (is the first one I do). I don't want to share something that is not fish but I need a comparison with someone more experienced than me, for be sure that this is the best way to code the module.
      If you want to try this I created a git, you will find all the instructions for installation and correct operation. (Git has a MIT licensed)
      https://github.com/MarcooRo/processwire-PayPal-Commerce-Platform I hope I did something that you guys can like :)
    • By monollonom
      (once again I was surprised to see a work of mine pop up in the newsletter, this time without even listing the module on PW modules website 😅. Thx @teppo !)
      FieldtypeQRCode
      Github: https://github.com/romaincazier/FieldtypeQRCode
      Modules directory: https://processwire.com/modules/fieldtype-qrcode/
      A simple fieldtype generating a QR Code from the public URL of the page, and more.
      Using the PHP library QR Code Generator by Kazuhiko Arase.

      Options
      In the field’s Details tab you can change between .gif or .svg formats. If you select .svg you will have the option to directly output the markup instead of a base64 image. SVG is the default.
      You can also change what is used to generate the QR code and even have several sources. The accepted sources (separated by a comma) are: httpUrl, editUrl, or the name of any text/URL/file/image field.
      If LanguageSupport is installed the compatible sources (httpUrl, text field, ...) will return as many QR codes as there are languages. Note however that when outputting on the front-end, only the languages visible to the user will be generated.
      Formatting
      Unformatted value
      When using $page->getUnformatted("qrcode_field") it returns an array with the following structure:
      [ [ "label" => string, // label used in the admin "qr" => string, // the qrcode image "source" => string, // the source, as defined in the configuration "text" => string // and the text used to generate the qrcode ], ... ] Formatted value
      The formatted value is an <img>/<svg> (or several right next to each other). There is no other markup.
      Should you need the same markup as in the admin you could use:
      $field = $fields->get("qrcode_field"); $field->type->markupValue($page, $field, $page->getUnformatted("qrcode_field")); But it’s a bit cumbersome, plus you need to import the FieldtypeQRCode's css/js. Best is to make your own markup using the unformatted value.
      Static QR code generator
      You can call FieldtypeQRCode::generateQRCode to generate any QR code you want. Its arguments are:
      string $text bool $svg Generate the QR code as svg instead of gif ? (default=true) bool $markup If svg, output its markup instead of a base64 ? (default=false) Hooks
      Please have a look at the source code for more details about the hookable functions.
      Examples
      $wire->addHookAfter("FieldtypeQRCode::getQRText", function($event) { $page = $event->arguments("page"); $event->return = $page->title; // or could be: $event->return = "Your custom text"; }) $wire->addHookAfter("FieldtypeQRCode::generateQRCodes", function($event) { $qrcodes = $event->return; // keep everything except the QR codes generated from editUrl foreach($qrcodes as $key => &$qrcode) { if($qrcode["source"] === "editUrl") { unset($qrcodes[$key]); } } unset($qrcode); $event->return = $qrcodes; })
    • By Sebi
      AppApiFile adds the /file endpoint to the AppApi routes definition. Makes it possible to query files via the api. 
      This module relies on the base module AppApi, which must be installed before AppApiFile can do its work.
      Features
      You can access all files that are uploaded at any ProcessWire page. Call api/file/route/in/pagetree?file=test.jpg to access a page via its route in the page tree. Alternatively you can call api/file/4242?file=test.jpg (e.g.,) to access a page by its id. The module will make sure that the page is accessible by the active user.
      The GET-param "file" defines the basename of the file which you want to get.
      The following GET-params (optional) can be used to manipulate an image:
      width height maxwidth maxheight cropX cropY Use GET-Param format=base64 to receive the file in base64 format.
    • By MarkE
      This fieldtype and inputfield bundle was built for storing measurement values within a field, rendering them in a variety of formats and converting them to other units or otherwise modifying them via the API.
      The API consists of a number of predefined functions, some of which include...
      render() for rendering the measurement object, valueAs() for converting the value to another unit value, convertTo() for converting the whole measurement object to different units, and add() and subtract() for for modifying the stored value by the value (converted as required) in another measurement. In the admin the inputfield includes a checkbox (which can be optionally disabled) for converting values on page save. For an example if a value was typed in as centimeters, the unit was changed to metres, and the page saved with this checkbox selected, said value would be automatically converted so that e.g. 170 cm becomes 1.7 m.

      A simple length field using Fieldtype Measurement and Inputfield Measurement.
      Combination units (e.g. feet and inches) are also supported.
      Please note that this module is 'proof of concept' at the moment - there are limited units available and quite a lot of code tidying to do. More units will be added shortly.
      See the GitHub at https://github.com/MetaTunes/FieldtypeMeasurement for full details and updates.
×
×
  • Create New...