If you are hosting your web application in Windows Azure, here are some tips regarding monitoring your servers and application during a LoadStorm test that were provided to me from a Microsoft software engineer:

Sharing with you the steps for performance testing/bottleneck identification. Attaching all the requisite documents and counters.

Explanation of perfmon counters - http://msdn.microsoft.com/en-us/library/aa905154(SQL.80).aspx

Other tools we use:

• Logman (built into system) – to control performance counters
• Logparser – when you want to evaluate http://logparserplus.com/
• DMVs - http://gallery.technet.microsoft.com/ScriptCenter/en-us/

Ways to Use Perfmon Counters Config File via Command

1. One on SQL Server
   • Config file for perfmon counters – attached
   • Command to run from admin command prompt in beginning – (remember to modify timings/location of perflogs/location of config file) or look up Logman syntax
   • Logman create counter SQLPerf -f bin -b 24/01/2011 13:36:00 -E 24/01/2011 13:50:00 -si 05 -v mmddhhmm -o "E:\perflogs\SQLPerf" -cf "E:\perflogs\AllSQLCounters-minus-cpu-disk-memory-details.Config"
   • For eg above command creates a DataCollector on Windows 2008 machine which uses a config file stored in certain location and outputs the log in binary format to certain location.
2. One on web server
   • Config file for perfmon counters – attached
   • Command to run from admin command prompt in beginning – (remember to modify timings/location of perflogs/location of config file) or look up Logman syntax
   • Logman create counter ASPPerf -f bin -b 24/01/2011 13:36:00 -E 24/01/2011 13:50:00 -si 05 -v mmddhhmm -o "E:\perflogs\ASPPerf" -cf "E:\perflogs\ASP-NET-perfcounters-trim.config"
3. One on SQL Server
   • logman start SQLPerf
4. One on Web Server
   • logman start ASPPerf - after some 20-30 minutes (idea is to ensure good amount of data is collected and lot of normal/peak/average load is collected)
5. One on SQL Server
   • logman stop SQLPerf
6. One on Web Server
   • Logman stop ASPPerf

Web Role and Worker Role

To find issues in webrole/worker role, we enable perfcounters and collect them in Azure storage account.

1. You can do remote enabling on do it on role.start calm - please see: http://msdn.microsoft.com/en-us/library/hh411521.aspx
2. Please watch this Channel 9 MSDN video about monitoring applications in Azure

The way it works is very simple, you enable a collection of perf counters, you scheduled tx to wad storage. And then you use a tool like Cerebrata like a performance viewer. Or you can enable remote desktop and look at log (not preferred due to ephemeral nature of roles).

You can also get crash logs, IIS logs, and event logs in same way.

Once you run tests with perf counters enabled, you will be better shape to identify bottlenecks. We shared with you perf counter list earlier,can share again.

When you work with SQL Azure, we need to use DMV as we discussed before -http://msdn.microsoft.com/en-us/library/ff394114.aspx They will give us idea about issues.

Suggestions for Testing Cycles

My suggestion is to plan your performance testing. Then sign up for a free subscription. Build test scenarios to act like real users.

Run a test for 25 or less users. Once you have the monitoring working well, run bigger tests.

Start from 50 users across all activities and go up to 5000 users so that you have an idea about at what point you need to provision more web roles or go for federation in SQL Azure.

Don't be in a big hurry with jumping straight to large numbers of users. Run a test and review the results - both from a user perspective (RPS, response times, errors) AND from the server perspective (perf counters).

Practice correlating all the data. How many concurrent users is pushing the server to a limit somewhere? How many requests per second? When we start to see errors returned to the client, are we almost out of memory? Are we getting CPU bound? Do we have enough database connections?

Performance optimization is difficult. Be scientific and don't try to change too many variables at once. Change something and then test again to see the effect. Be patient and persistent.