ryan Posted April 28, 2017 Share Posted April 28, 2017 This week we’ve got lots of details on our successful migration to Amazon AWS services for the processwire.com network of sites. We've also got updates on the current core and ProDrafts status. https://processwire.com/blog/posts/amazon-aws-now-powering-processwire.com-sites/ 15 Link to comment Share on other sites More sharing options...
Mike Rockett Posted April 29, 2017 Share Posted April 29, 2017 Great stuff, Ryan! In SA, I've actually picked up a considerable speed improvement, especially on the forums. Not sure if that difference would affect you or Europe as much as it does here. Very snappy. 1 Link to comment Share on other sites More sharing options...
joer80 Posted May 4, 2017 Share Posted May 4, 2017 Are the assets folders located and synced on all ec2 instances? Or hosted on a common file system all ec2 instances can access at the same time? How are the various EC2 instances mirrored? Did you need to install a script to have this work? 1 Link to comment Share on other sites More sharing options...
LostKobrakai Posted May 4, 2017 Share Posted May 4, 2017 The first question to ask would probably be if it's even hosted on multiple instances. 1 Link to comment Share on other sites More sharing options...
joer80 Posted May 4, 2017 Share Posted May 4, 2017 28 minutes ago, LostKobrakai said: The first question to ask would probably be if it's even hosted on multiple instances. The blog post says: Quote using a t2 medium instance for the web servers. Since servers is plural, I take it that there are more than one? Link to comment Share on other sites More sharing options...
joer80 Posted July 14, 2017 Share Posted July 14, 2017 I know it mentions there is a client going with a load balanced solution. In his case, how does he keep the assets folder synced across all of the ec2 instances? Link to comment Share on other sites More sharing options...
ryan Posted July 14, 2017 Author Share Posted July 14, 2017 We aren't using a load balancer for this processwire.com server (just a single Amazon instance). But for the sites where we are (and running PW), there is a shell script that syncs /site/assets/ and /site/templates/, across the instances. I don't know exactly how it works, but can find out more. I know it runs every 30 minutes (cron job) and whenever a change is detected. Note that it also specifically excludes /site/assets/cache/ and /site/assets/sessions/, as those vary between the instances. Also important, is that when it comes to using the PW admin, we setup a separate hostname for that purpose (admin.domain.com) that always hits the same instance (which we call the "A" instance). This is the instance that the shell script considers the master when it comes to syncing files. That way, any changes that occur to the synced files always originate on the A instance. There is a setting in AWS that lets you lock a hostname to a specific load balanced EC2 instance. Then we keep any editing locked to that hostname by adding this to /site/templates/admin.php: if($config->httpHost != 'admin.domain.com') { $session->redirect('https://admin.domain.com/processwire/'); } And we keep non-authenticated requests out of the admin hostname by having this in the /site/templates/_init.php file ($config->prependTemplatefile setting): if($config->httpHost == 'admin.domain.com' && !$user->isLoggedin()) { $session->redirect('https://www.domain.com' . $page->url); } This ensures search engines don't get in and start indexing another copy of the site on the admin hostname. 4 Link to comment Share on other sites More sharing options...
joer80 Posted July 14, 2017 Share Posted July 14, 2017 Thank you for the details! Very interesting! I thought about doing an rsync script approach like that for my setup, and having a master of sorts that sends files to the other instances, but I was thinking as you add ec2 instances, they could be a part of the load balanced pool before they have all of the files locally which could cause missing images on pages. It could be 15-30 minutes before the transfer starts, and that could take a bit if you have a lot of files, and if your master goes down, you can't admin without selecting a new master. I was thinking of going another route and using the Elastic File System service and having all of the EC2 instances mount it on boot and use it to store the assets folder. Sessions can be served out of RDS. That way any ec2 can be a master and can upload files to the network share and they are ready as soon as they boot. 1 Link to comment Share on other sites More sharing options...
joer80 Posted July 14, 2017 Share Posted July 14, 2017 Here is an example config file for how to tell elastic beanstalk to mount a shared file system on the ec2 instances on boot. I am going to use this to give it a try I think. You just create this file in your project folder before you upload it. https://github.com/awslabs/eb-php-wordpress/blob/master/.ebextensions/efs-mount.config Assuming I get something like /pw-assets/ mounted in my web server, is it hard to tell PW to use that location for the assets folder? Or do I need to do have the boot image make a symbolic link that connects the 2? Link to comment Share on other sites More sharing options...
joer80 Posted July 18, 2017 Share Posted July 18, 2017 I ended up getting this working, but instead of doing the assets folder I did the whole sites folder. That way I could install a module and it go to all http servers. I also added my ec2-user to the webapp group so I could sftp into the share and edit files. 1 Link to comment Share on other sites More sharing options...
LostKobrakai Posted July 18, 2017 Share Posted July 18, 2017 Isn't this just shifting the bottleneck from the single server's resources to the speed of accessing the single shared filesystem? I mean you've still a single server being hit for all your requests. Link to comment Share on other sites More sharing options...
joer80 Posted July 18, 2017 Share Posted July 18, 2017 No, both the rds and the efs are managed clusters. Link to comment Share on other sites More sharing options...
joer80 Posted July 18, 2017 Share Posted July 18, 2017 Limits page says efs has a default throughput of 3 Gigabytes per second each and you can request more. You can mount 10 and files are redundant across multiple Azs/datacenters. It can go to petabyte levels. Link to comment Share on other sites More sharing options...
joer80 Posted July 18, 2017 Share Posted July 18, 2017 If you need more default throughput you could always mount user assets, modules and templates separately instead of the whole sites folder. I just did it for convenience. Local storage on each ec2 machine would be a bit quicker but you have to sync them somehow. Using their deploy tools is one way you can do it and push out updates without something like rsync. The wire files don't change that much so I don't mind doing a deploy for those. I just wanted sftp access to the templates to be able to make changes instantly as needed, instead of a versioned setup since I am the only dev and its faster for me. I guess you could always do a hybrid approach and do a deploy for template changes and efs for user assets. Another thing to consider is if you turn on their cdn to serve static assets, I would matter even less. Your EFS is just the authority/pull point. Link to comment Share on other sites More sharing options...
mrjasongorman Posted July 19, 2017 Share Posted July 19, 2017 I tried to find a thread that spoke about this, but couldn't find one, so i thought this thread mentioning the processwire.com site itself was a good place. I've noticed that processwire.com has a wildcard SSL certificate, however it doesn't automatically redirect to HTTPS or use HSTS header to inform the browser to prefer HTTPS connections. Just wondered if there was a reason for this? To clarify this only seems to happen on the main site, not the forum. going to http://processwire.com 2 Link to comment Share on other sites More sharing options...
AndZyk Posted July 19, 2017 Share Posted July 19, 2017 There is a thread by you, were this was mentioned before. 2 Link to comment Share on other sites More sharing options...
mrjasongorman Posted July 19, 2017 Share Posted July 19, 2017 haha, so there is, 10 months is a long time in software development, and clearly a long time for my memory 1 Link to comment Share on other sites More sharing options...
gentitope Posted April 18, 2018 Share Posted April 18, 2018 (edited) @joer80 pls can you share more on how you manage to get efs working. I have this scenario that i mounted the efs on ec2 instance and the symlinked to files directory in pw assets directory. Then i moved my pw files image directory to the efs mounted directory. But each time i try to access the image i have the error that file not found. my symlink is intact with correct permission and user. Thanks Edited April 18, 2018 by LostKobrakai remove bold Link to comment Share on other sites More sharing options...
joer80 Posted April 19, 2018 Share Posted April 19, 2018 15 hours ago, gentitope said: @joer80 pls can you share more on how you manage to get efs working. I have this scenario that i mounted the efs on ec2 instance and the symlinked to files directory in pw assets directory. Then i moved my pw files image directory to the efs mounted directory. But each time i try to access the image i have the error that file not found. my symlink is intact with correct permission and user. Thanks My EFS setup is no longer running. I decided to take a more traditional approach that lets me login via ssh and make direct changes to the server. Sorry I could not be of more assistance! Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now