php segfault help

[Ben]

I'm not sure there's anything that can be done as the segfault is likely a PHP bug - but there are a handful of libraries that can leak memory (e.g. Simple HTML DOM Parser) so I figured it wouldn't hurt to ask. The backstory is this, basically I'm in the process of selecting a CMS - looking at WP, EE and ProcessWire. ProcessWire seems to be the brightest bulb of the lot, but I'm getting a nasty error when using ApacheBench to load test. Neither EE or WP cause the fault under the same load, so I don't beleive it's a bad extension and/or configuration. The issue initially showed up with a development site, but I've since set up a vanilla install of the skyscraper demo to reduce the potential of having introduced the problem with my own code.

The system is WAMP: Apache 2.2.22, PHP 5.4.0 (also tried 5.2 and 5.3 with same outcome).

The load test is an Apache Bench where 127.0.0.9 is the skyscraper site, with 10 threads and 1000 total requests:

C:\>ab -n 1000 -c 10 http://127.0.0.9/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.9 (be patient)
apr_socket_recv: An existing connection was forcibly closed by the remote host.   (730054)
Total of 3 requests completed

There is no error in the apache logs, but in windows event viewer the cause is clear:

Faulting application name: httpd.exe, version: 2.2.22.0, time stamp: 0x4f6715c9
Faulting module name: php5ts.dll, version: 5.4.0.0, time stamp: 0x4f4e7f05
Exception code: 0xc0000005
Fault offset: 0x0004c633
Faulting process id: 0xf98
Faulting application start time: 0x01cd084ebe38d790
Faulting application path: C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin\httpd.exe
Faulting module path: C:\php\php5ts.dll
Report Id: 0a4e4bf8-7442-11e1-95ce-14feb5ecd87c

Outwardly, this returns a blank (white screen) - I was able to mash F5 refresh enough times to get it to show up in the browser. Without running ab with concurrent threads, everything is fine. Setup is pretty much a stock install (php + gd2, mysqli, mbstring), and I've not encounted stability issues prior. Apache doesn't appear to be the issue because the fault can occur in less than 20 requests, though usually many more are handled.

If you're still reading this, thanks for sticking with... My question. If you were to divide and conquer in order to discover what is behind it, how would you go about doing so?

Thanks.

[netcarver]

Hi Ben,

Welcome to the PW forum & thanks for posting this. I'm new to ProcessWire myself but want to ask if this problem shows up if you run ap with just a single thread?

[Ben]

Thanks Netcarver. With one thread it runs the course without fail. It's definitely seems to be related to concurrency. I found a mysqli bug on php.net that I almost thought was it, but after updating to php trunk it's still there. I'll let you know if I figure anything else out.

[netcarver]

Ok, out of interest, what worker module is your Apache install using?

---

Ben,

sorry but a follow-up Q: is your apache2 executable compiled with multi-threaded support by any chance?

On linux I can get at this with

apache2ctl -V

but don't know how on Windows .

[Ben]

It's the same, only using httpd.exe, "httpd -V"

It is multi-threaded, I think. Here's the info:

Server version: Apache/2.2.22 (Win32)

Server built: Mar 19 2012 12:20:03

Server's Module Magic Number: 20051115:30

Server loaded: APR 1.4.6, APR-Util 1.4.1

Compiled using: APR 1.4.6, APR-Util 1.4.1

Architecture: 32-bit

Server MPM: WinNT

threaded: yes (fixed thread count)

forked: no

Server compiled with....

-D APACHE_MPM_DIR="server/mpm/winnt"

-D APR_HAS_SENDFILE

-D APR_HAS_MMAP

-D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)

-D APR_HAS_OTHER_CHILD

-D AP_HAVE_RELIABLE_PIPED_LOGS

-D DYNAMIC_MODULE_LIMIT=128

-D HTTPD_ROOT="/apache"

-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

-D DEFAULT_ERRORLOG="logs/error.log"

-D AP_TYPES_CONFIG_FILE="conf/mime.types"

-D SERVER_CONFIG_FILE="conf/httpd.conf"

[netcarver]

Hi Ben,

Thanks for posting that. Back to your original question re: slicing and dicing to find out what's going on. Some thoughts, incomplete and in no particular order...

• If you have access to a Linux box I'd try running your ap stress test there to see if it is isolated to Windows.
  
• If I only had a Windows box, I'd now try verifying that each of the bundled php modules (especially MySQLi) is also compiled thread-safe.
  
• Can you identify any components that PW is exercising that the other CMS systems aren't (ie. are they using the MySQL module instead of PWs MySQLi?)
  

Although it's from a while back, there is a report here of a similar looking issue.

[netcarver]

Ben,

I doubt this will help you directly, but just bookmarking this post as it has an interesting use of ProcMon to track down the same exception code in php5ts.dll.

Edited to add: Thinking aloud from this idea. Perhaps PHP is writing to an error log file and there is some contention over that?

Edited March 22, 2012 by netcarver

[ryan]

Welcome to the forums Ben! I would suggest not using the skyscrapers site profile because that will only work with PW 2.0 and that version is now very out of date and not a good one to use for benchmarking. Though I don't think that has anything to do with the error you are experiencing, but want to make sure you are using an up-to-date version. I do plan to update the skyscrapers profile to work with PW 2.2+, but haven't yet had the time to do it. So for now, I would suggest using PW 2.2 and the basic site profile that it comes with. I've been building a list of people waiting for the PW 2.2 compatible skyscrapers profile and happy to send it to you when it's ready--just let me know.

Netcarver: thanks for your great responses here. I'm really impressed with your knowledge on this stuff.

[Ben]

Netcarver, I think you are on to something with testing on LAMP - I'll setup another test today with muti-process apache to try and rule out PHP/extension thread-saftey issues and report back.

Ryan, I'll use the base install theme for future testing, always a good idea to keep it as simple as possible. Oh, and thanks for all the great work - though I've only been working with it for the past 3 days, ProcessWire has the best feel out of all the CMS options I've tried. Coming from a Django background, I appreciate the focus on framework over extraneous features. It strikes a really good balance.

Thanks guys!

[Ben]

Netcarver, genius.

Ok, so using LAMP with the prefork mpm - I'm seeing rock solid stability. First with 1000 requests using 10 workers (ab -n 1000 -c 10), then with a barrage of 120 concurrent requests (ab -n 120 -c 120). I can only guess that there's some php interaction with the worker mpm that is at fault. Searching about, I'm seeing fuzzy references to php limitations using the worker mpm, but no definitive list of off-limit extensions or programming patterns. I might dig a little deeper, but seeing as LAMP deployment is an option it's no showstopper.. Thanks again.

[netcarver]

Hi Ben,

Glad to hear that the LAMP stack worked out for you. BTW, which distro did you use for that?

FWIW, the official PHP docs do not recommend using threaded MPM & Apache2.

[Ben]

Sorry for the delay (vacation). The distro was an Amazon Linux AMI on EC2, which from my understanding is a forked version of CentOS.