Sleep Is For The Weak

As I talked about in a previous post, PHP is an interpreted language.  Each page load requires the code to be read, interpreted, and compiled on the fly each time a browser makes a request.  I’m running nginx as my base web server which doesn’t natively support PHP so I have to use FastCGI.  In the previous post, if you could have seen the full output of top you would have seen four PHP-CGI processes adding up to that 75% CPU load.

Just like the vBulletin board tuning I did on Apache earlier I opted to install APC OpCode Cache on my server.  This gives the advantage of pre-compiling the PHP into OpCode and then calling that from cache rather than recompiling every time a page loads.

Installing APC on FreeBSD is a simple task.  I used

[lars@insa]$ sudo portinstall -Rr www/pecl-apc

to install the FreeBSD Port for APC and added the following to my /usr/local/etc/php.ini file

extension=apc.so

apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 32
apc.filters=wp-cache-config

As I mentioned in previous posts I intend to benchmark my server configuration each step of the way.  Below are the benchmarks run from my home against the remote server.

longhammer$ ab -c 25 -t 500 http://insa.w9zeb.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking insa.w9zeb.org (be patient)
Finished 4578 requests

Server Software:        nginx/0.6.36
Server Hostname:        insa.w9zeb.org
Server Port:            80

Document Path:          /
Document Length:        37071 bytes

Concurrency Level:      25
Time taken for tests:   500.001 seconds
Complete requests:      4578
Failed requests:        5
(Connect: 5, Receive: 0, Length: 0, Exceptions: 0)
Write errors:           0
Total transferred:      170893494 bytes
HTML transferred:       169938150 bytes
Requests per second:    9.16 [#/sec] (mean)
Time per request:       2730.456 [ms] (mean)
Time per request:       109.218 [ms] (mean, across all concurrent requests)
Transfer rate:          333.78 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:       47  736 5350.5    143   67352
Processing:   731 1915 673.9   1764   11314
Waiting:      106  285 162.9    239    2093
Total:        850 2651 5429.3   1925   71230

last pid:   903;  load averages:  2.33,  1.79,  0.91    up 0+00:10:56  20:40:35

43 processes:  5 running, 38 sleeping

CPU: 56.6% user,  0.0% nice, 11.7% system,  0.6% interrupt, 31.2% idle

Mem: 52M Active, 18M Inact, 31M Wired, 64K Cache, 20M Buf, 892M Free

Swap: 1920M Total, 1920M Free

Completed Requests:  3542 vs 4578 a 29.25% improvement

Requests Per Second: 7.08 vs 9.16 a 29.38% improvement

Time Per Request:  141.169 vs 109.218 a 22.63% improvement

Transfer Rate: 257.99 KB/s vs 333.78 KB/s which is a 24.73% improvement.

CPU Utilization: 75.2% vs 56.6% which is a 24.73% improvement

Thanks to: LiquidX for the image

I’m in the process of moving http://w9zeb.org to a new server.  I’m moving out of a shared environment managed by H-Sphere, hosted on CentOS 5.2 and onto a FreeBSD 7.1-Release Virtual Machine hosted on VMWare ESXi.  It’s not that I was unhappy with my previous host, but rather my interest in tinkering with all of the pieces that deliver a website.  I wanted to be able to tune MySQL, PHP, the file system, etc. and benchmark the effects of those changes without worrying about breaking the other 100 or so sites hosted under H-Sphere.

The technologies that I am planning on using for this site initially are as follows:

• FreeBSD 7.1-Release
• nginx 0.6.36
• varnish 2.0.4
• MySQL 5.1.33
• PHP 5.2.9
• FastCGI

I am intending to benchmark the performance of the server at each step along the way.  Below is the first benchmark which is on a raw, virtually untuned server.

longhammer$ ab -c 25 -t 500 http://insa.w9zeb.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking insa.w9zeb.org (be patient)
Finished 3542 requests

Server Software:        nginx/0.6.36
Server Hostname:        insa.w9zeb.org
Server Port:            80

Document Path:          /
Document Length:        37071 bytes

Concurrency Level:      25
Time taken for tests:   500.021 seconds
Complete requests:      3542
Failed requests:        5
(Connect: 5, Receive: 0, Length: 0, Exceptions: 0)
Write errors:           0
Total transferred:      132094346 bytes
HTML transferred:       131356986 bytes
Requests per second:    7.08 [#/sec] (mean)
Time per request:       3529.228 [ms] (mean)
Time per request:       141.169 [ms] (mean, across all concurrent requests)
Transfer rate:          257.99 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:       40  445 4386.8     48   67208
Processing:   651 2982 579.3   2959    8071
Waiting:      342 2173 442.8   2228    3694
Total:        694 3427 4468.5   3015   73897

During this base benchmark top reported the following load on the server

last pid:  9201;  load averages:  4.21,  3.84,  2.55    up 0+04:10:32  20:21:32
42 processes:  5 running, 37 sleeping
CPU: 75.2% user,  0.0% nice, 23.1% system,  1.7% interrupt,  0.0% idle
Mem: 66M Active, 33M Inact, 81M Wired, 1200K Cache, 92M Buf, 811M Free
Swap: 1920M Total, 1920M Free

As you can see the CPU is being hammered and our load averages are > 2.5 during the test.  I know that with some tuning we can improve these numbers.  Be sure to check back for followups on the performance tuning I am doing on the new server.

Thanks to: paradoxperfect for the image

In part 2 of tuning a 1&1 VPS to improve vBulletin I installed an Op-Code Cache utility. Because PHP is an interpreted language using simply mod_php requires the code to be read, interpreted, and compiled on the fly each time a page loads. This doesn’t take very long on modern computers. However the more traffic your site gets obviously the more even milliseconds add up.

I looked at several Op-Code Cache utilities, and after some discussion with another tech decided not to use eAccelerator but rather APC. In part because of the simplicity of installation, and because it’s going to be included as a core part of PHP 6. I would like to thank @floris for this post on installing APC on CentOS

Below are benchmarks run using Apache Bench: ingo # ab -c 100 -t 5000 website.com/forums

APC disabled Performance:

Concurrency Level: 100
Time taken for tests: 53.368813 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50000
Total transferred: 28500000 bytes
HTML transferred: 15900000 bytes
Requests per second: 936.88 [#/sec] (mean)
Time per request: 106.738 [ms] (mean)
Time per request: 1.067 [ms] (mean, across all concurrent requests)
Transfer rate: 521.50 [Kbytes/sec] received

APC Enabled Performance:

Concurrency Level: 100
Time taken for tests: 33.140479 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50000
Total transferred: 28500000 bytes
HTML transferred: 15900000 bytes
Requests per second: 1508.73 [#/sec] (mean)
Time per request: 66.281 [ms] (mean)
Time per request: 0.663 [ms] (mean, across all concurrent requests)
Transfer rate: 839.82 [Kbytes/sec] received

As you can see our requests per second went up by 61% which is a pretty big deal. Also our mean time per request dropped. All in all for a fairly simple installation this was a worth while improvement.

In Part 3, we’ll discuss some extremely minor MySQL tuning I did which made quite possibly the biggest difference of them all as far as performance.

Thanks to: i_am_sam_lee for the image