I saw a tweet today that referenced web performance improvement, and that linked me to an interesting post on Developer.com. Performance Improvement: Understanding by Robert Bogue is a good article that provides an overview of web performance measurement and what common problems lead to poor performance in web apps. Although his perspective and examples are Microsoft-centric, the concepts are generally applicable to any architecture.

Performance is not one of his favorite topics because, "it requires a ton of knowledge about how systems work, and either a ton of guesswork or some very detailed work with load testing."

Robert addresses the FOUR PRIMARY BOTTLENECKS as:

• CPU
• Memory
• Disk
• Network (or Communications)

He believes CPU issues are the easiest to find and memory are the hardest.

CPU bottlenecks are easy because the Windows shows you the CPU utilization; he says, "To get statistics on CPU utilization use Performance Monitor and in the Processor object include Percent (%) Processor Time for each CPU."

Memory bottlenecks are difficult, "because there aren't good indicators for memory." The Memory objects' pages/sec counts the times that requests couldn't be satisfied from physical memory, thus requiring disk access. One suggestion he has is to cut back on virtual memory used by minimizing the paging file.

Disk usage is easy, albeit tedious, to measure. His performance focus is on "how many IO operations you can get from the drive." There are several factors involved in drive performance such as rotational speed, partitioning, interface type, track seek time, and array standard. Useful metrics found in the Physical Disk object are: Avg. Disk Queue Length, Avg. Disk sec/Read, and Avg. Disk sec/Write.
These will tell you how busy the drive is and how long it takes to read/write information from the drives.

Network bottlenecks can form between the servers in your architecture (web/database/application) or between your app and the users. NIC stats are notoriously inaccurate, and if you trust the numbers, the Network Interface object has counters for Bytes Received/sec and Bytes Sent/sec that can be monitored to see if the NIC is giving you the throughput you expect. If the NIC is slower than the server apps are trying to communicate, then the Output Queue Length counter will exceed 10. Robert suggests you eliminate network bottlenecks by using a faster and/or aggregated network interfaces. Link Aggregation Control Protocol (LACP) allows you to create a logical grouping of multiple network interfaces that function as one and improve throughput performance.

He goes on to define load testing, and I didn't find anything new or very useful to pass along to you. That said, my favorite suggestion he makes about web performance is, "there's little point in evaluating the disk performance numbers until you've resolved any memory issues because in low memory situations the disks are used as virtual memory." My days of actually getting my hands dirty in page faults and page swapping go back to VMS on a VAX. I guess the principles still apply, and it quite logical to avoid paging altogether if you are tuning for the best system performance.