Jump to content

Easier HA by storing images in db?


joer80
 Share

Recommended Posts

I was thinking about some of the ways to provide a low cost high availability service in my hosting environment for my clients. 

I already use RDS by amazon, so it automatically replicates my database over 2 availability zones and does fail over.  This seems to take care of the database layer, but my files are spread out/out of sync over the 2 servers home folders.  There is still the need to sync the asset files.

Would it be a good way to solve this by simply using an image field that stores the user uploaded images in the database instead of in the assets folder?  I would guess I would need to make a new image field type for this.

For performance, I would think you would want it to cache to the local disks asset folder on the first load so it doesn't cause extra db strain.  It would only pull the blob one time.  

It sounds like a nightmare trying to rsync a number of home directories and only use that source when the copy is full.

Maybe the way to do this would be to make a hook on page save instead of adding a new field type.  ie. On page save, add images to the database so I can rebuild if missing.

Let me know your thoughts!

Thanks!

Link to comment
Share on other sites

Another option might be to make a hook that uploads the files to s3.  That should already be HA.  I think we have a plugin for this already, but it may not work as if you fire up a new web server, the file will not be in the local install but it will be on s3.  It will need to know how to pull it back down if missing.

This also adds quite a bit of hassle in needing to setup s3 buckets and IAM roles for every website.

Link to comment
Share on other sites

I was just typing that I thought an object storage like S3 would be better geared towards replicating that kind of content, as I can say from experience that blob-heavy databases can be a real PITA. They always grow far faster than expected, and the number of queries rises what feels like exponentially. The necessary backup space and -time quadruples in the blink of an eye, and the "quick restore" suddenly takes an hour.

I guess the setup for new sites could be scripted leveraging S3 and IAM apis. Create user, create access key, assign role, create bucket and assign policy are all available through the REST APIs. Then all that is left is to deploy the access key to PW's config or module settings and all replicas should know it too.

Link to comment
Share on other sites

23 minutes ago, BitPoet said:

I was just typing that I thought an object storage like S3 would be better geared towards replicating that kind of content, as I can say from experience that blob-heavy databases can be a real PITA. They always grow far faster than expected, and the number of queries rises what feels like exponentially. The necessary backup space and -time quadruples in the blink of an eye, and the "quick restore" suddenly takes an hour.

I guess the setup for new sites could be scripted leveraging S3 and IAM apis. Create user, create access key, assign role, create bucket and assign policy are all available through the REST APIs. Then all that is left is to deploy the access key to PW's config or module settings and all replicas should know it too.

Scripting via the rest apis could be very handy and make it worth while.  

One pro also is also the file storage is cheaper.  Around half the price per gig for storage.

Doing something like aurora over a regular mysql could be different though.  Not only is it fully managed and replicated but the storage system is also distributed in a way that may take a lot of the pain points you mentioned away.  

Docs say you do not need to assign a storage number and you are only charged for what you use and it can grow up to 64 TB without rebooting.

It also has up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across three Availability Zones (AZs) by default.

It says it keeps 6 copies at all times and instance failover typically takes less than 30 seconds.

Link to comment
Share on other sites

I've a prove of concept S3 filesystem module based on flysystem and the core inputfield file running in a project of mine. I'm not comfortable to put it out in the public, but there are ways to make it work. Personally I'd try to avoid mounting remote filesystems, as there's more networking involved: client <-> server <-> S3. I'd rather work with direct s3 urls for the frontend.

  • Like 2
Link to comment
Share on other sites

7 minutes ago, LostKobrakai said:

I've a prove of concept S3 filesystem module based on flysystem and the core inputfield file running in a project of mine. I'm not comfortable to put it out in the public, but there are ways to make it work. Personally I'd try to avoid mounting remote filesystems, as there's more networking involved: client <-> server <-> S3. I'd rather work with direct s3 urls for the frontend.

That is a good point.  

We probably need to do this through an s3 files and s3 images field type. 

If the project used sessions stored in the database and only these fields for files and images, I think all user uploaded assets would be the same on all web servers.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...