Boosting performance using an IIS web garden

If you've been involved with any sort of serious web hosting environments the term "Web Farm" will be familiar to you. This is where multiple computing environments are brought together to act as one to bolster computing power. IIS 6 brought a new feature to the table called a "Web Garden" which is a similar concept of colating computer power except this time using software on a single hardware node. A Web Garden can only be used when IIS is running in worker process isolation mode (default), and is configured on a per application pool level. The idea behind configuring an application pool to use multiple worker processes (i.e a Web Garden) is to improve scalibility and reliability, however in practice this has not always been the case with single CPU/Core systems as these worker processes are all competing for the same CPU time. However since multi processor and multi core CPU systems have become the rule rather than the exception for low/medium spec servers over the last few years the advantage of using a a Web Garden have started to make a lot more sense. Just recently while posting on the IIS.net forums I came across a situation where a systems adminstrator was at wits end because he had a server with 8 CPU's running a busy PHP/MySQL based forum which was giving terrible performance to users despite the fact that the CPU utilisation of the system was only sitting at about 15%. As it turns out this environment was a prime candidate for using a Web Garden which is exactly what he did.

Under a default installation of IIS 6 only a single worker process (w3wp.exe in your process list) is spawned to service each application pool. This is how the server in question was originally configured, and this was the observation made of the system;

Under normal load we have about 300 users browsing our boards. Under 'peak' load, we can only handle about 500 users. Our user base would probably allow us to get 2000 or more concurrent users at a time, but our server is holding us back. Under peak load, The IIS process is using about 15% of CPU, and about 450,000 MB of RAM. There is always 1GB or more of free memory available. Disk I/O is fine per Performance Manager. My MySql database is also fine. During peak load, the site becomes unresponsive. Clicks take two minutes or more.

Obviously not ideal, and likewise not your run of the mill performance issue as there were still a massive amount of system resources left. After some general troubleshooting to ensure basic PHP and MySQL configuration directives and performance values where as they should be it became obvious that the bottleneck was the web environment. The next move was to create a web garden which had immediate results;

I upped our worker threads for our message boards from 1 to 8 (we have 8 procs). Bingo! Finally the memory and CPU's start to actually get used. My queries per second in mySQL went from a prior max of 20/sec to 60+/sec. Up your worker processes from the default to get more out of your PHP/IIS solutions!

So creating the web garden did the trick, and the server has now able to utilize a lot more of the hardware resources. How far could it go though?


Shortly after implementing this change, we set our record for most active concurrent user's - which is 991. With this load we still had memory and CPU available. Page response was instantaneous. Throughput in mySQL has since reached a max of 140 queries/second! The prior high was about 20. We feel we could currently handle 1500 or more concurrent sessions, with the ability to scale more if we just add some RAM.

So in this case going from an IIS 6 application pool using just the one worker process to a web garden using 8 worker processes (one worker process per physical CPU/core) tripled the concurrent session capacity and slashed response times. Obviously not everyone is going to have a server with 8 CPU's, but with dual and quad core systems now commonplace there is still significant performance potential in configuring your IIS 6 server application pools to use a web garden to maximize the use of computing power.

Average rating
(14 votes)

Comments

Anonymous's picture

Mysteries of web garden perf improvements..

Pro: Web gardens increase concurrency N times.
Con: This may increase memory usage N times. If this results in page file thrashing, the box slows right down. (perf test perf improvements before deployment :-))

Anecdote: Seeing increased throughput from web gardening can be a sign of:
- long running synchronous acivity (e.g. slow db, slow web service)
- bad locking (e.g. global lock held for majority of request duration)
This should be understood. Otherwise future web garden growth may not be as fruitful.

rickj

Brashquido's picture

Hi Rick, Race conditions can

Hi Rick,

Race conditions can be another undesirable effect of web gardens which can lead to instablity. Should be noted that the server referenced in this article was configured to use PHP in ISAPI mode rather than FastCGI/CGI. In this case DB had been ruled out, which more or less leaves the IIS/PHP stack or the code itself as the bottleneck. Profiling the PHP scripts with something like xDebug would be a good step in this case to establish a finer grained understanding of where any potential performance issues are.
----------------
Dominic Ryan
3 x Microsoft IIS MVP, MCSE, MCSA
IIS Aid owner/webmaster

SteveF's picture

I looked at Web Gardens on

I looked at Web Gardens on another application written in ASP and using sessions. I was advised to not use Web Gardens because a request could be picked up by a worker process other than the one that set the session state and would not know or be able to access the user's session state.

Dominic,
If this is true on ASP, could it also be true on PHP Sessions?

Brashquido's picture

Hi Steve, It really depends

Hi Steve,

It really depends on the individual web application. If the sessions your application uses are created in the w3wp.exe process then you shouldn't use a web garden as the sessions will not be consistent across you garden. However, I'd be a bit suprised to see any maintained classic ASP applications still using instrinsic ASP sessions these days as by default IIS 6 recycles worker processes anyway, which would have a similar effect on inproc sessions. As for PHP you shouldn't generally have any issues as PHP as session data is saved externally as a file (specified by the session.save_path PHP.ini directive) and most PHP apps even have custom session handlers that use a DB to store session data.

Anonymous's picture

Awesome

I applied this to the live server - it was running two dual cores but with 80% cpu time to spare (users were getting poor asp.net performance). It also had 3Gb ram spare (out of 8). Now, it's using more resorces and the system is much more responsive for all the concurrent users. Now we can support more concurrent users, it will help in the marketing of the product to future clients, where we would not have approaced them in the past knowing we could not have them!!

Brashquido's picture

Nice to hear the performance

Nice to hear the performance improvements! Be sure to keep a close eye on things as discussed above using a web garden should not be considered a set and forget configuration.
----------------
Dominic Ryan
4 x Microsoft IIS MVP, MCSE, MCSA
IIS Aid owner/webmaster

goose2000's picture

Hi, I'm building a new IIS

Hi,

I'm building a new IIS server and was considering using the garden. The machine will be a PHP MySQL application server, no ASP or .NET stuff. It has 4 cores.

From notes posted here, it sounds pretty comfortable (considering the caching issue & PHP apps) decision to make. The machine will host 1 Drupal and 2 Moodle sites. Traffic is expected to be considerable. I am also running memcached daemon on two external machines; how could this play out? I think it should be ok.

Brashquido's picture

Don't discount FastCGI

Keep in mind why running PHP ISAPI as a web garden can bring such good performance results. It's basically a concurrency issue where the server load is spread across multiple w3wp.exe processes. FastCGI to a degree works in the same manner, however instead of there being multiple w3wp.exe processes, there is multiple php-cgi.exe processes with one w3wp.exe process controlling them. Same deal, multiple php-cgi.exe processes increases the concurrency. Another thing to consider is that as of PHP 5.3 the PHP ISAPI module will be dropped from the php.net distribution. If I were you I'd configure your server to use both FastCGI and ISAPI, and then run some stress tests to see which is going to the the best for you in regards to speed, scalability and reliability.

----------------
Dominic Ryan
5 x Microsoft IIS MVP, MCSE, MCSA
IIS Aid owner/webmaster

goose2000's picture

Thanks for the response. I

Thanks for the response. I am reconsidering a re-do of the system now. I read the article "Difference between PHP thread safe and non thread safe binaries" - and then with the news that we will not have an ISAPI module in the Windows distro, it is all leaning the other way now.

So, I think I will undo my install and then use the IIS Aid package for FastCGI

Great site!

Brashquido's picture

ISAPI Vs FastCGI

I think it is still worth testing both ISAPI and FastCGI. It really depends on where the bottle neck is, and if you read the forum thread linked in the above article you'll see that the poster was having throughput issues with the worker process itself. This was why using a web garden was so effective in their case. The PHP ISAPI module will be dropped from the php.net distribution, however the source will still be there so you can compile your own if need be. Moving all IIS/PHP users from ISAPI to FastCGI is like getting everyone to upgrade from XP to Vista. It certainly isn't going to happen overnight.

----------------
Dominic Ryan
5 x Microsoft IIS MVP, MCSE, MCSA
IIS Aid owner/webmaster