Sleep Is For The Weak

In this fifth and final installment I would like to discuss using Varnish to further improve the performance of your website.  Varnish is a state-of-the-art, high-performance HTTP accelerator.  What this actually means when you remove the buzz words and replace them with English is that Varnish is a reverse caching web proxy.  When we think about caching web proxies we usually think of something that sits on a corporate network that caches incoming web content so that sites that see a lot of traffic have most, if not all, of their content stored locally.  A reverse proxy works exactly the same, but not.  When we run dynamic websites like WordPress, Joomla, Plone, or Drupal, each time a page is loaded the PHP script has to pull data from the database, render a static .html page and serve that static page to your browser.  Earlier we talked about a plugin for WordPress called WP-SuperCache which does cache these pre-rendered files and serves them from the file system.  Not all content managers have a plugin like WP-SuperCache.  Also WP-SuperCache still has to check the database to determine if the content it has cached is fresh or if it needs to be updated on every page load.  Finally WP-SuperCache must run on the server that is running WordPress.

A typical dynamic web server will look something like this:

Basic Dynamic Server Configuration

Enter Varnish.  Varnish, like the caching web proxy at your office, is a proxy.  It sits between my nginx web server and your browser.  The difference is, while your proxy at work is configured to cache the content of many websites and serve that cached content to users within your office, Varnish is configured to cache the content of only one website.  Varnish serves that cached content to everyone who attempts to visit the site.

A typical dynamic web server with Varnish will look something like this:

Cache Enabled Basic Server Configuration

Varnish can also be made aware of more than one backend server.  Which means I could have a single internet facing varnish server, and two or more load balanced web servers behind it.

Two options for how this might be configured are laid out here:

A Load Balanced configuration

Scaled, high performance, load balanced configuration

Varnish is also able to add, remove, modify, or otherwise mangle the headers passed from the server to your browser.  You can strip cookies, add content expires, and a whole host of other things.  Things like expires headers can be used to make the client cache content such as images, style sheets, and javascript in their browser on their local machine.  That is the fastest way to serve content as it eliminates the network completely.  Varnish really is a fantastic product for accelerating your dynamic website.

Because my connection at home is nowhere near fast enough to really benchmark my new server, especially with Varnish in place, I decided to run ApacheBench locally.  To give you something to compare against I ran it against my single backend server, as well as against the Varnish front end.

Completed Requests:  17316 vs 50000 a 65.368% improvement

Requests Per Second: 34.63 vs 219.305 a 83.99% improvement

Time Per Request:  28.88 vs 4.39 a 22.63% improvement

Continuing my saga of trying to extract every ounce of performance out of my new hosting environment, it’s time to move on to caching.  Up until this point every time a page is loaded the PHP scripts have to query the database for the appropriate records, take the information from the database, and render the page into static HTML that your browser can render.  Computers are fast so this doesn’t take very long, as evidenced by previous benchmarks.  However you’ll notice that even with the APC OpCode Cache in place we’re still hammering the CPU for 50% of its capacity during the benchmarking.  I know this can be improved!

Enter the plugin for WordPress called WP-SuperCache.  Now because I’m running the nginx web server and not Apache I will be unable to use the “super” portion of WP-SuperCache which works with the Apache Module Mod_Rewrite.  However I can still use a static cache of pre-rendered .html to improve site performance.  Installation of WP-SuperCache is pretty straightforward.  You can either upload the folder to your /wp-content/plugins/ folder, or you can use the FTP plugin system within your WordPress Dashboard.  I opted for the manual method as I was able to just wget the file from the server which has a lot more bandwidth than I have at home.

Once WP-SuperCache is installed you have to enable it through the dashboard.  After the plugin is enabled you can adjust its settings.  Because of my selection in webservers I opted to set the plugin to Half-On, Don’t Cache pages for logged in users, and Cache Rebuild.  I then benchmarked the site again from my home computer.

longhammer$ ab -c 25 -t 500 -r http://insa.w9zeb.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking insa.w9zeb.org (be patient)
Finished 4667 requests

Server Software:        nginx/0.6.36
Server Hostname:        insa.w9zeb.org
Server Port:            80

Document Path:          /
Document Length:        37195 bytes

Concurrency Level:      25
Time taken for tests:   500.041 seconds
Complete requests:      4667
Failed requests:        8
(Connect: 8, Receive: 0, Length: 0, Exceptions: 0)
Write errors:           0
Total transferred:      174921511 bytes
HTML transferred:       173825455 bytes
Requests per second:    9.33 [#/sec] (mean)
Time per request:       2678.598 [ms] (mean)
Time per request:       107.144 [ms] (mean, across all concurrent requests)
Transfer rate:          341.62 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:       40  785 5663.6    149   67346
Processing:   451 1753 675.3   1605    7765
Waiting:       49  175 126.6    156    1983
Total:        553 2538 5737.4   1767   70928

During the benchmark top reported the following server load

last pid:  7037;  load averages:  0.00,  0.12,  0.20    up 0+00:48:03  21:17:42
42 processes:  1 running, 41 sleeping
CPU:  0.2% user,  0.0% nice,  1.1% system,  0.2% interrupt, 98.5% idle
Mem: 53M Active, 33M Inact, 57M Wired, 1444K Cache, 55M Buf, 848M Free
Swap: 1920M Total, 1920M Free

I’d like to highlight a few of the important numbers in comparison to the APC benchmark.

Completed Requests:  4578 vs 4667 a 1.94% improvement

Requests Per Second: 9.16 vs 9.33 a 1.86% improvement

Time Per Request:  109.218 vs 107.144 a 1.90% improvement

Transfer Rate: 333.78 KB/s vs 341.62 KB/s which is a 2.35% improvement.

CPU Utilization: 56.6% vs 0.2% which is a 99.65% improvement

Thanks to: Bolt of Blue for the image

In part 2 of tuning a 1&1 VPS to improve vBulletin I installed an Op-Code Cache utility. Because PHP is an interpreted language using simply mod_php requires the code to be read, interpreted, and compiled on the fly each time a page loads. This doesn’t take very long on modern computers. However the more traffic your site gets obviously the more even milliseconds add up.

I looked at several Op-Code Cache utilities, and after some discussion with another tech decided not to use eAccelerator but rather APC. In part because of the simplicity of installation, and because it’s going to be included as a core part of PHP 6. I would like to thank @floris for this post on installing APC on CentOS

Below are benchmarks run using Apache Bench: ingo # ab -c 100 -t 5000 website.com/forums

APC disabled Performance:

Concurrency Level: 100
Time taken for tests: 53.368813 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50000
Total transferred: 28500000 bytes
HTML transferred: 15900000 bytes
Requests per second: 936.88 [#/sec] (mean)
Time per request: 106.738 [ms] (mean)
Time per request: 1.067 [ms] (mean, across all concurrent requests)
Transfer rate: 521.50 [Kbytes/sec] received

APC Enabled Performance:

Concurrency Level: 100
Time taken for tests: 33.140479 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Non-2xx responses: 50000
Total transferred: 28500000 bytes
HTML transferred: 15900000 bytes
Requests per second: 1508.73 [#/sec] (mean)
Time per request: 66.281 [ms] (mean)
Time per request: 0.663 [ms] (mean, across all concurrent requests)
Transfer rate: 839.82 [Kbytes/sec] received

As you can see our requests per second went up by 61% which is a pretty big deal. Also our mean time per request dropped. All in all for a fairly simple installation this was a worth while improvement.

In Part 3, we’ll discuss some extremely minor MySQL tuning I did which made quite possibly the biggest difference of them all as far as performance.

Thanks to: i_am_sam_lee for the image

A friend of mine runs a moderate sized vBulletin web forum. By moderate sized I mean in 12 months of being online he has 2,851 members, 18,257 threads, 206,904 posts, and an average number of online users that floats between 50 in the middle of the night, and 200 during peak hours, transmitting a daily average of 7gb of data.

About six months ago another forum member and I helped move the site off of “A Small Orange” shared hosting and on to a 1&1 Virtual Private Server. The performance boost was immediately apparent. Of course like any web forum we grew from just over 1000 members to 2000 members in 4 months, two months later another 800 members. Performance is still better than it was on ASO, but it was starting to suffer.

I fired up YSlow and determined that there were a fair number of efficiencies that could be gained by tuning some settings in Apache. Specifically by using mod_deflate & expires headers. The settings I added to /etc/httpd/conf/httpd.conf are as follows.

#Mod_Deflate Rules
# Netscape 4.x or IE 5.5/6.0
BrowserMatch ^Mozilla/4 no-gzip
# IE 5.5 and IE 6.0 have bugs! Ignore them until IE 7.0+
BrowserMatch \bMSIE\s7 !no-gzip
# IE 6.0 after SP2 has no gzip bugs!
BrowserMatch \bMSIE.*SV !no-gzip
# Sometimes Opera pretends to be IE with “Mozila/4.0″
BrowserMatch \bOpera !no-gzip

AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/atom_xml
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/x-httpd-php
AddOutputFilterByType DEFLATE application/x-httpd-fastphp
AddOutputFilterByType DEFLATE application/x-httpd-eruby
AddOutputFilterByType DEFLATE text/html
Header append Vary User-Agent

These settings alone lowered the daily average from 7gb down to 3gb! Mod_Deflate uses gzip to compress data by filter type. Older browsers aren’t able to read gzipped data. The first few rules at the top actually tell apache not to use gzip if the browser matches.

Next we turn on Header Expires. These add a header for all jpg, jpeg, gif, js, css, and png files that say the content expires and should be refreshed in April of 2010. This will hopefully tell browsers like IE, Firefox, and Safari to cache the files in their local store. Things like logos, navigation images, etc. don’t change often. Downloading them every time a page loads is wasteful.

<FilesMatch “(jpg|jpeg|gif|js|css|png)”>

Header set Expires “Thu, 15 Apr 2010 20:00:00 GMT”

</FilesMatch>

These are just two of the steps I’ve taken to improve performance of this vBulletin site. I will detail some of the additional changes in upcoming posts. If you have any questions feel free to comment. If you have suggestions on how I could improve these settings even further I’d love to hear from you as well.

Thanks to: Leff for the image

About three months ago I started a new job where I was issued a MacBook computer. At that time I wasn’t sure I liked the machine. It did most of what I wanted it to, but had a number of quirks I found rather annoying. I’ve continued using the MacBook, and have come to appreciate some of the software that’s available for OS X. Applications like TextMate, Cornerstone, and CocoaMySQL are fantastic. I’m sure there are similar applications for Windows, and other Unix Like operating systems (FreeBSD, Linux). However the integration between these applications and the operating system really is very good.

The default Apple Terminal application is also very good. Most terminals are suitable for my needs, however the keyboard shortcuts are great. Copy/Paste “Just Works”, opening new terminal windows are just a Command+n away. All in all a solid terminal, with a lot of features, that works with a minimum of annoyances.

Applications like QuickSilver make using the computer even faster. Ctrl+Space itu Enter launches iTunes. Ctrl+Space file Enter launches filezilla. Ctrl+Space text Enter launces textedit. really a simple function to add, and I’ve been told QuickSilver does a lot more than function as an application launcher, but this feature alone is enough to make me happy.

Considering the slow hard drive installed in the MacBook I’m surprised at how snappy the system is overall at most tasks. If I continue editing video from conferences we attend I am likely to have a 320gb 7200rpm drive installed. In part because 1 hour of video is 12gb, and in part because drive speed does matter while editing videos.

I wouldn’t say the MacBook is a superior computer to a commodity PC with a Unix Like operating system installed, and a fair amount of system customization to fit the users individual needs. But I will say that out of the box, with a minimal amount of configuration Apple has provided a pretty fantastic system.

Thanks to: The_Jorr for the image