Better apache settings? Running on bluehost. (new title)

[MuchDev]

I would really like to see if there was a way to enable multithreading in Processwire. Is there any sort of functionality that I can enable that would allow image re-sizing to split itself across multiple cores? Resizing large lists of big images can really put some serious stress on my server which causes the whole site to wait until re-sizing is done. This has been fine for my site while it is in development mode, but as I near launch I am seeing that the image re-sizer could be a bit more efficient.

Upon loading a list of 100 images to be re-sized into their derivatives I will see my processor usage peg at 50% while my second core sits idle at around .1%. I originally thought I would get around this by adding a couple more cores for the live site, but now I am realizing that this wouldn't actually fix anything. Is this something I should just give up on and not have my server do so much work, or am I able to set a magic config flag that will solve all of my problems

Thanks !

[teppo]

I'm not aware of any magic config flags that would solve this, sorry. Actually, I'm not even entirely sure if this is something that PW alone can handle -- not much of an expert when it comes to stuff like multithreading, but I'd say that it probably depends a lot on how your Apache is configured, which MPM it's using (prefork, worker, etc.) and so on. Sure, I've seen tricks like sleeping a few (micro)seconds between tasks to allow other processes get resources too, but that's just about it

Before getting too worried, though, you might want to consider if there really are going to be hundreds of operations on huge images all the time after after your site has gone live. For most sites this won't be the case, and in a situation like you mentioned above (one core with 50% load, another sitting idle), you've still got plenty of resources to serve regular visitors.

In any case we've had no issues with image resizing on our production sites, ever, so I wouldn't worry about this too much.

One thing you could try if you're really worried about images in particular would be Imagick. From what I've heard, Imagemagick (which Imagick uses behind the scenes, unless I'm somehow mistaken) seems to be in some cases faster than GD. This thread would probably be a good place to start, if you're looking for a PW solution (not entirely sure about the state of that project, though).

Hope this helps a bit.

[MuchDev]

Hey thanks for that info. It's helping me do a little bit more background reading and I'm learning quite a bit about apache. I think I have the server set up pretty much what I should. I am using worker, mpm prefork and cgi (not fast-cgi due to 500 errors). So I guess I may have some other issue.

The reason why I am posting all this is the whole site grinds to a halt when I'm running big operations. For instance, the first time I load a page the server has to generate all of it's needed images, usually 2 or 3 derivatives depending on the templates layout. This operation takes roughly 20-30 sec. During this time the website becomes completely inaccessible. 

So right now I am running a script that loops all of my site's images to make sure that I have preview images for my search. As I write this the script has been running for about 45 minutes. That I am OK with, I don't care if an operation like this takes a long time. The problem is, the rest of the site is completely unresponsive .

So here is the question I guess 

Is this normal or should I investigate something?

[teppo]

In a bit of a hurry right now, but if I'm reading you right and resizing couple of images takes 20-30 seconds, that's definitely not normal. Of course source images can make a huge difference, but still that sounds really slow. Unless you're running a really, really low-end server, perhaps, which doesn't seem too likely judging from your previous post.

(Note: most of the time there's no need to store images larger than, say, 1280..1600px. Of course this depends on your needs, but usually I find it more sensible to resize huge images to a slightly more reasonable size when they're first uploaded. This is where max image size settings in image fields come in handy.)

I'm actually a bit confused about your setup. Worker and prefork are different MPM modules, so which one is it that you're running? Also, you've mentioned you're using CGI; that seems like a bit of a weird choice. CGI is supposed to be very slow compared to running PHP as a module (mod_php). FastCGI is apparently faster than CGI, but to my best knowledge mod_php is still going to be faster (and from what I've seen, FastCGI seems to cause weird issues here and there -- though don't take my word on it!)

So, as a first step, you might want to take a closer look at your Apache configuration

[MuchDev]

Sorry for giving you confusing information, but I really appreciate all of your effort in helping me figure out what the heck is going on. So the server I'm on is a vps with a 2 core opteron and 2gb of ram, but I feel that the way that bluehost has it configured is leaving things a bit on the slow side. When I originally set things up I had all sorts of errors and timeouts. I managed to get this all to go away by setting my server from fcgi to cgi. I have tweaked some memory settings, and have my allowable php memory cranked up to 256mb. This is working, but man this sucker is slow.  I contacted my hosting provider and it looks as if they do not have mod_php as an option. I am told however that " there may be a way to manually add it via ssh" though. Here is my current settings:

apache2_ul_54 (1) Profile

Apache 2.2

• Asis (More information)
• Auth Digest (More information)
• AuthnAlias (More information)
• AuthnDBD (More information)
• AuthnDBM (More information)
• AuthnDefault (More information)
• AuthzDBM (More information)
• Cache (More information)
• DAVFs (More information)
• DAVLock (More information)
• Dav (More information)
• Deflate (More information)
• Disk Cache (More information)
• Env (More information)
• Expires (More information)
• File Cache (More information)
• Fileprotect
• Headers (More information)
• Log Forensic (More information)
• MPM Worker (More information)
• Mime Magic (More information)
• Mod FastCGI v2.3.9 (More information)
• Mod SuPHP 0.7.2 (More information)
• Proxy (More information)
• Speling (More information)
• UniqueId (More information)
• Version (More information)

PHP 5.4

• Bcmath (More information)
• CGI (More information)
• Calendar (More information)
• CurlSSL (More information)
• Exif (More information)
• FTP (More information)
• GD (More information)
• Gettext (More information)
• Iconv (More information)
• Imap (More information)
• MailHeaders (More information)
• Mbregex
• Mbstring (More information)
• Mcrypt (More information)
• Mhash (More information)
• Mime Magic (More information)
• MySQL "Improved" extension. (More information)
• Mysql (More information)
• Mysql of the system (More information)
• Openssl (More information)
• PDO (More information)
• PDO MySQL (More information)
• POSIX (More information)
• Pear (More information)
• Pspell (More information)
• SOAP (More information)
• Sockets (More information)
• TTF (FreeType) (More information)
• Tidy (More information)
• Wddx (More information)
• XSL (More information)
• XmlRPC (More information)
• Zip (More information)
• Zlib (More information)

Additional OptMods

• IonCube Loader 4.6.1 for PHP (More information)
• Mod Security 2.8.0 (More information)

There are on the other hand other options that I can investigate as it will allow me to do a custom build. Do you happen to have a good guide with some pertinent information about how I should configure this thing so that it is best suited for processwire?

[MuchDev]

So I can now verify that all images are limited to around 1200px width. I am trying to test out fcgi again and am getting timeouts before the page can finish loading even when setting the timeout to an absurdly large number and setting the set_time_limit() method inside of the re-sizing loop. From my research I see that using php as a module is actually extremely insecure and prone to problems. Is this something that you agree with?

I really want to figure out what is going on with this. I am totally wanting to get this fixed.

[teppo]

From my research I see that using php as a module is actually extremely insecure and prone to problems. Is this something that you agree with?

From what I can tell, mod_php is really less secure in some ways than FastCGI (or SuPHP, for that matter) , thought it's (partly) a question of configuration and environment:

Main issue is apparently that mod_php runs as the Apache user (in Ubuntu that would typically be www-data), which means that you'll have to give this user r/w access to various files and folders on the disk. Some articles make this sound worse than it really is, though; you most definitely don't need to give it full access to all your files. Still, this can be a problem, especially in shared hosting environments. Also, unless you set up separate Apaches for each hosted site, if one of those sites has (critical) security issues, it could potentially compromise all other sites too.

To answer your question: when you're in control of the whole server and can set up permissions as you wish, I don't see this as a huge problem -- or, rather, I see it as a reasonable compromise. PHP as a module happens to be extremely fast, especially compared to something like CGI. You do want to be careful about what you install and run (and how you setup permissions on your server), but on the other hand, that's common sense anyway.

FastCGI needs more memory than mod_php, but provides more security too. CGI is by default both insecure and slow, but can be configured to be somewhat more secure. SuPHP is somewhere in the middle, more secure than mod_php but also slower than FastCGI or mod_php (though SuPHP memory consumption seems reasonable; this seems to be something of a problem for FastCGI).

Looks like your best bet would be either FastCGI (for security) or mod_php (for speed); latter requires care when setting up your environment, while former will apparently be slower no matter what you do. Decisions, decisions

(Note: I'm not an expert on any of this. I've always used mod_php for various reasons, and am pretty much used to any insecurities it brings about. Right now I'm looking into FastCGI myself, and I know that others have made PW run with it, but this isn't very high on my list to be honest.)

[MuchDev]

OK! I spent some time and got dso(which is the same as mod_php right?) loaded on my server and it feels great! Transfers seem peppier and operations are completing faster. 

I am not too worried about the security of the site from a hacker standpoint as I am planning on getting the client on a regular offsite and duplicated local backup solution. I will be storing no client data locally so a data breach is pretty much a non-issue, but I would like to take some precautions to make sure that malicious code is not run on the site. What I am focused on is the site's speed, and have gone to pretty great lengths to try and pre-process as many images as possible yet because of this I end up maxing out the one processor whenever I batch images. 

As far as speed goes, when you do an operation like re-sizing a page of images or any other heavy working script does your site still allow other operations to complete? Whenever I do anything super heavy my site just waits. How would I go about investigating this issue?

[MuchDev]

In a bit of a hurry right now, but if I'm reading you right and resizing couple of images takes 20-30 seconds, that's definitely not normal. Of course source images can make a huge difference, but still that sounds really slow. Unless you're running a really, really low-end server, perhaps, which doesn't seem too likely judging from your previous post.

So I just re read your post and now I see where the confusion is from. So what I tried to say was for a page of 100 items which each have a thumbnail and a large version, derivatives can take around 20-30 sec. While the server performs this, the whole site waits for it to complete. I feel like the site should be able to prioritize operations like this and still allow page request to the rest of the site. 

[LostKobrakai]

The problem with image resizing is the fact that it's php. In php every line of code gets executed after the one before is finished. The same thing happens on resizing the image. The template generations is halted as long as the new picture isn't ready. Possible workarounds would be:

Using the thumbnails module (or it's new reinvention). This lets the thumbnails be generated on upload.

Using a autoload module, to generate thumbnails every 15 minutes or so.

Each of this versions require you to have predefined sizes of thumbnails.

To be fair, on a regularly visited site, this shouldn't be a problem, as each thumbnail has to be generated only once. So only the first visitor to see a new image(s) should get the loading time, the second one just gets the thumbnail. So as long as you don't upload 100s of images before someone visits the site, it should do the job quite well out of the box.

[pjg]

Wearing my non-pw hat for a second it is not uncommon for multimedia heavy sites (in my case large amounts of mapping) to be pre-seeded, usually using an automated routine written in an automation package (free ones are available and reasonably easy to use straight in the browser). This process in our case resizes, creates pyramids and initiates the cache. While this is overkill for 100 images and I'd agree with the solutions given by the previous poster, pre-seeding in this fashion is certainly an option for sites using vast amounts of multimedia.

[SteveB]

I've run into long waits for thumbnails too on a site where some frontend pages have up to a couple hundred thumbnails from rather large source files. I'm also using tiny thumbs in the backend page lists. In normal operation with template caching and routine editing it's fine but during setup it was insane.

I wrote a script which uses the API to see which variations are not made yet and make them. Before turning it loose you can use assess method to just get a report. Returns a handy array to use in other processing scripts. It's alpha, not gift wrapped and polished, but I can make it available somewhere if anybody wants it. Example of usage below.

This puzzles me:

While my script is building images all other PW activity pretty much grinds to a halt (that's not the puzzling part) but other pages on the site using PHP/MySQL independent of PW continue to work just fine. They use a different database so maybe some db lock/transaction thing is holding things up?

$fab = new imagePrefab();
$fab->background = 1;		//don't make browser wait
$fab->verbose = 1;	//Yes we want some feedback...
$fab->setLog();		//...in a log file                                   

$ok = $fab->setBuilderCallback('buildImageMyWay');	//optional callback function
if(!$ok)die(" Callback function not found at ".__LINE__);

$data = $fab->build('my-template', 'image', array(array(48,48),array(110,110),array(800,0)));