module FieldtypeFileS3 - store files on AWS S3

[fbg13]

FieldtypeFileS3

https://github.com/f-b-g-m/FieldtypeFileS3

The module extends the default FieldtypeFile and InputfieldFile modules and adds few extra methods.

For the most part it behaves just like the default files modules, the biggest difference is how you get the file's url. Instead of using $page->fieldname->eq(0)->url you use $page->fieldname->eq(0)->s3url(). Files are not stored locally, they are deleted when the page is saved, if page saving is ommited the file remains on the local server until the page is saved. Another difference is the file size, the default module get the file size directly from the local file, while here it's stored in the database.

There is an option to store the files locally, its intented in case one wants to stop using S3 and change back to local storage. What it does is it changes the s3url() method to serve files from local server instead of S3, disables uploading to S3 and disables local file deletion on page save. It does not tranfer files from S3 to local server, that can be done with the aws-cli's through the sync function. Files stored on S3 have the same structure as they would have on the local server.

-------------------------------------------------------- --------------------------------------------------------

Been struggling with this for quite a while, but i think i finally managed to make it work/behave the way i wanted.

All feedback is welcome!

[eelkenet]

Hi @fbg13, Thank you for your module!

I am in the process of learning AWS (EC2, RDS, S3) services and moving my first PW project to 'the cloud'.
As a part of all this, I have tested your module, and I did run into an issue: 

1. The upload seemingly never finishes (ie. the progress bar does not finish). It does not display any warnings, but stays stuck at 100% transfer.
2. Upon reloading the page I get a PHP-error: Warning: filemtime(): stat failed for /var/www/html/site/assets/files/12978/fullsizeoutput_2aa6.jpeg in /var/www/html/wire/core/Pagefile.php on line 389
3. Even so, the file did transfer correctly and is present in the S3 bucket
4. It is also present in the database and on the page
5. In the database I see that the modified and created fields of the file are showing incorrect timestamps: 1970-01-01 01:00:00

I have configured the InputfieldFileS3 module with the correct Key, Secret and Bucket name.
The ACL is set to public-read, the region is set to the same my VPC is located,  SSL is on, Use my domain & Store files locally are off. 

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

 

[fbg13]

@eelkenet I just tested it (windows 10 and ubuntu 17.04) and had no problems on php 5.6 an up.

On versions < php 5.6 i got the never ending spinner but not the warning.

What php version are you using?

[eelkenet]

Hi, I've tested the field on another environment and there it worked fine using the same S3 credentials and settings. 
So, there must be some significant difference between the two causing this. 

Where it does not work: 

• ProcessWire 3.0.62
• PHP 7.0.16 on an Amazon EC2 Linux instance (Amazon Linux AMI release 2017.03)
• I installed LAMP using these instructions: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html, with MySQL running on a RDS instance. 

The working environment:

• ProcessWire 3.0.36
• PHP 5.6.27 on a shared hosting environment (Debian GNU/Linux 7)

As the domains are for staging/development I can send you a private message with links to the phpinfo output , perhaps there is some PHP module I forgot to install? 

Also, I checked the Chrome console after uploading, which shows why the spinner never finishes: the server error above is also showing up in the JSON that InputfieldFileS3.js tries to parse:

[{"error":false,"message":"Added file: img_8737.jpg","file":"\/site\/assets\/files\/12978\/img_8737.jpg","size":1391384,"markup":"<li id='file_72fa2d34b55cc08ec37b4551217bf582' class='InputfieldFileItem ui-widget ui-widget-content'><p class='InputfieldFileInfo InputfieldItemHeader ui-state-default ui-widget-header'><i class='fa fa-file-image-o fa-fw HideIfEmpty'><\/i><a class='InputfieldFileName' title='img_8737.jpg' target='_blank' href='https:\/\/s3-eu-central-1.amazonaws.com\/foamtalentcall-1\/12978\/img_8737.jpg'>img_8737.jpg<\/a> <span class='InputfieldFileStats'>1,359&nbsp;kB<\/span> <label class='InputfieldFileDelete'><input type='checkbox' name='delete_tests3_72fa2d34b55cc08ec37b4551217bf582' value='1' title='Delete' \/><i class='fa fa-fw fa-trash'><\/i><\/label><\/p><div class='InputfieldFileData description ui-widget-content'><div class='InputfieldFileDescription'><label for='description_tests3_72fa2d34b55cc08ec37b4551217bf582' class='detail'>Description<\/label><input type='text' name='description_tests3_72fa2d34b55cc08ec37b4551217bf582' id='description_tests3_72fa2d34b55cc08ec37b4551217bf582' value='' \/><\/div><input class='InputfieldFileSort' type='text' name='sort_tests3_72fa2d34b55cc08ec37b4551217bf582' value='0' \/><\/div><\/li>","replace":false,"overwrite":0}]<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />
<br />
<b>Warning</b>:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/img_8737.jpg in <b>/var/www/html/wire/core/Pagefile.php</b> on line <b>389</b><br />

I checked the Pagefile.php code and it seems that there is no default escape in the switch statement which tries to get information from the uploaded file.

[fbg13]

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

To be honest I don't know much about AWS and the module is very basic.

[eelkenet]

13 minutes ago, fbg13 said:

@eelkenet Does the normal file field work on the Amazon EC2 Linux instance? The module uploads the file to the server, then uploads it to S3. The file stays on the server until the page is saved, so I don't know why filemtime() throws that warning.

Yes, both the files and images fields work fine. Also resizing images through the API works, so it is not a rights issue I would say. 

Did you test it yourself with the latest version of PW? Maybe something changed between 3.0.36 and 3.0.62.

 

50 minutes ago, fbg13 said:

To be honest I don't know much about AWS and the module is very basic.

Yeah I'm also just starting out, but for this problem I think you should consider the EC2-instance just as a regular old Virtual Private Server.  

[fbg13]

I tried on PW 3.0.62 and 3.0.65.

1 hour ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

Did you try this?

19 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

[eelkenet]

 

2 hours ago, fbg13 said:

Try to upload another file and while it's "loading" check to see if the file is available on the server (Amazon EC2 Linux instance).

I tried. While the front-end is still 'loading', and before reloading the page manually, I checked both the EC2 server and the S3 bucket.
The file is not available anywhere on the (EC2) server, but it did in fact transfer correctly to the S3 storage. 

The same warning as before also gets logged to error_log:

PHP Warning:  filemtime(): stat failed for /var/www/html/site/assets/files/12978/hamburg-11.jpg in /var/www/html/wire/core/Pagefile.php on line 389, referer: http://ec2-52-57-230-10.eu-central-1.compute.amazonaws.com/admin/page/edit/?id=12978&s=1&c=1

 

2 hours ago, fbg13 said:

I tried on PW 3.0.62 and 3.0.65.

Maybe your error reporting does not show warnings? And do your entries in the database show the correct timestamps?

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally. 

 

2 hours ago, fbg13 said:

  22 hours ago, eelkenet said:

Turning Store files locally on skips the entire S3 upload it seems, so no errors but also no new files in S3. 

Did you get any errors when doing it this way? Also take a look at the server's error log.

No errors this way, nothing new gets added to Php's error_log.

[fbg13]

12 minutes ago, eelkenet said:

So, what seems to happen is that retrieving info about the file is the problem, but only after the file has transferred to S3. 
I think this might be caused by the fact that Pagefile.php does not account for missing files, and tries to get info from something that does not exist locally.

The file should be there though, as it is only removed when the page is saved. I have no idea why it's deleted.

Edit InputfieldFileS3.module file (line 83) and add

if(file_exists($pagefile->filename)){
	$this->wire("log")->error("file exists");
}

after

if(!$this->localStorage) {
	$this->uploadFileToS3($pagefile, $this->input->get->id);
}

and check the log. You can add it before too.

[eelkenet]

@fbg13 This logs that the file exists at this point. 

[fbg13]

@eelkenet Add it on the FieldtypeFileS3 file too on line 85, just before

if($field->fileSchema & self::fileSchemaDate) {
  $item['modified'] = date('Y-m-d H:i:s', $pagefile->modified);
  $item['created'] = date('Y-m-d H:i:s', $pagefile->created);
}

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

I'll try to find another way to handle this.

[eelkenet]

1 hour ago, fbg13 said:

Also replace the file_exists check with $pagefile->modified on the InputfieldFileS3.module file.

Hi @fbg13, I'm not sure what you want me to do here, can you rephrase? Thanks!

[eelkenet]

@fbg13 

1 hour ago, fbg13 said:

Add it on the FieldtypeFileS3 file too on line 85

I did, with an else statement to make sure it would log anything. At this point the file does NOT exist! 
I added a check for the modified date on InputfieldFileS3.module (line 83) too, and that is valid too! (The exact time of uploading). 

[fbg13]

@eelkenet Somehow the file gets deleted before the page is saved.

Do you have any other modules installed? Did you change any of the field's default settings?

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

You can comment them out then you know for sure, search for unlink on InputfieldFileS3.module/FieldfieldFileS3.module.

[eelkenet]

49 minutes ago, fbg13 said:

I don't think my module deletes the file, there are 3 calls to the unlink() function and none should delete it under normal circumstances.

@fbg13 I logged all the unlink calls that I could find. This unlink gets executed in my case: https://github.com/fbg13/FieldtypeFileS3/blob/master/InputfieldFileS3.module#L229

In addition I also called error_log(), and this happens about 1 ms before the original error gets thrown. 

[fbg13]

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

[eelkenet]

2 minutes ago, fbg13 said:

Did you save the page or just selected a file? If you saved the page it's normal for that one to run.

I did not save the page. The upload is 'still processing', ie. stuck at 100%. 

[fbg13]

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. 

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

[eelkenet]

50 minutes ago, fbg13 said:

@eelkenet Hmm, the deletion does take place before the db save and runs on ajax too. 

It's weird that I get no errors and the time is correct too.

Changing if(!$this->localStorage) to if(!$this->localStorage && !$this->isAjax) should fix it.

@fbg13 Yes, that fixes it! Congratulations, no errors, and the file is on the S3  

[Zeka]

@fbg13 The link is dead, where I can download your module? @eelkenet maybe you have an copy? 

[fbg13]

https://github.com/elabx/FieldtypeFileS3