Processwire with AWS S3 and s3fs

[Karl_T]

Short question: is it possible that Processwire uses AWS S3 with s3fs as a remote file system(mount to asset folder)? Please advise anything that have to be take care of.

 

Background:

I am currently trying to make Processwire running inside AWS Beanstalk as I want to take advantages of the auto scaling function that my client wanted.

I have found a discussion here: 

 

By reading this and the link inside, I realize that to use the auto scaling I need to configure my web server to be stateless. So, I was looking for a method that can serve the purpose and then I found s3fs.  s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem, quoted from their github. I guess that mounting s3 bucket using s3fs to the asset folder should be the right thing to do.

My site needs many image upload as it is an e-commerce site using padloper while the admin always using the local file system for images. I have thought of using modules like AmazonS3Cloudfront or FieldtypeFileS3. It seems like those modules are not support my use case. s3fs suits better and more simple. One of my concerns is that I am not sure can the URL of the images can be generated by the default API like $image->url correctly. 

Before implementing this, I would like to ask the advise from anyone having implemented this with processwire as I am new to AWS. Is it possible? Any better alternatives?

I think if not implementing auto scaling, it is also good to separate asset away in some cases, like reducing requests. Thank you for your reading. 

[Sergio]

Never done it myself, but I know it's possible to decouple the image processing and keep the goodies on an API for manipulation, similar to PW's.

You can take a look at Cloudinary feature to upload an image by its URL. Also, there's IMGIX .

[Sergio]

For documents such as PDFs, you can use the S3 options you already talked about to store them there, because you'll only need the file URL after all.

Remember to config Processwire to store the session on the database.

[Karl_T]

Thanks for your information. I want to keep everything managed inside PW admin though as what my client need is a simple and native drag and drop image upload. 

9 hours ago, Sérgio said:

Remember to config Processwire to store the session on the database.

Yes thanks for your reminder, I am planning to store the session on Redis for better performance. 

[LostKobrakai]

ProcessWire's image sizer engines can only work with files via the filesystem, so you'd need to mount S3 into it. Just keep in mind that this would slow down any image processing (download upfront and upload afterwards) and it means that all files need to be go through your mountpoint as files and images are in the same folders. 

Edit: And you should really evaluate to which degree your page does need image processing. If you can avoid it, you could just extend FieldtypeFile, which is way easier to work with than when you need the image sizer to work as well. 

[Karl_T]

After some time, I have tried using s3fs and another claimed faster replacement called goofys to mount s3 bucket to the asset folder. Processwire can work with s3fs, and has no problem with the image sizer fortunately, with the initial loading time increase from 100-200ms to 300-400ms, which is not quite acceptable. I guess the main reason for this is because the cache file is located inside the asset folder. And then I tried goofys. However, it takes over seconds to load the page. I don't know why it is the case. If anybody have experience with s3fs and goofys please share with me. Hopefully I can share my experience later on this topic.

Edit: With debug mode off, goofys greatly improve the loading speed from 1-5s to 300-400ms

[BitPoet]

1 hour ago, Karl_T said:

I guess the main reason for this is because the cache file is located inside the asset folder

Have you tried linking s3fs to assets/files instead of the whole assets folder, just to see what difference that makes?

Also, I could imagine that it's not only the cache folder that may slow things down at some point but also session and log, both of which could benefit from something with less round trips (and that handles concurrent writes better) than s3fs. Sessions could (and should) thus probably be switched to SessionHandlerDB.

I think goofys is at a disadvantage since it doesn't have local caching. The scenarios where it overtakes s3fs (e.g. creating many files at once) fall short when s3fs can use its cache (that's why the benchmarks on the goofys page only compare uncached operations).

[Karl_T]

Thanks for your suggestion. It is really helpful. I have my debug mode enabled and only cache the assets/files folder now. The loading speed is 250-350ms. I am happy now! 

I guess the filecompiler in cache folder is the issue. Mounting the cache folder also break the whole processwire system some time showing syntax error with broken file, which is a disaster.

However, it is still a huge(100%) slow down compared to not using such mounting file system, which may result in a much lower concurrent request cap. I want to try the elastic file system, the native cloud shared file system, but it serves limited region right now.

[joer80]

Have you tried using EFS?  https://aws.amazon.com/efs/?nc2=h_m1

Turning on cloud front is also an option to speed things up.

[Karl_T]

I want to try but unfortunately AWS do not provide this service in my region. I think it should be the best option regardless of the cost. 

I have not try cloud front yet but would like to use if that can speed the site up a lot. The first response time is really not good now, around 300ms while network latency is around 140ms.

[joer80]

Yeah, from what I understand, Any number of different ec2 servers could all run the processwire files and serve users with apache or nginx but connect to one file system for assets and one database.  (Aurora and EFS are both fully managed and fault tolerant so ec2 would be stateless.)

100 gigs of EFS is about $30 per month with unlimited requests and bandwith, so no additional charges.