Joe Emison is the VP of Technology at BuildFax. He is also the Chief Systems Architect at BUILDERadius. A world-class multi-tasker, Joe lives in Ashville, North Carolina.
Joe has experience with load and performance testing, and we are grateful he invested his time to share some good thoughts with us about a topic we are excited about.
On an interesting note, Joe is also a JD graduate from Yale Law School. How many alpha geeks do you know that went to law school? It’s a rare breed that only the folks at LoadStorm can hunt down. 😉
Let’s start the interview.
Please share one tip or best practice that is important to you regarding performance (or load) testing or engineering?
Before you start, you need to understand how everything works. Down to this level of detail:
- How many queries does your application make to the database for different activities?
- How many connections should your application make to the database server?
- How much memory should your application be using?
Otherwise, you won’t know how to use the results of load testing.
What is your technical background?
I started programming in Turbo Pascal in 1994. I wrote door games for BBSs, and the most popular one was LORDnet: a nationally distributed game in pre-Internet times. Over the years, I won the International Weird Software Contest with my entry The Mutant Chicken Races. Also I had the distinction to write the first Windows-iPod application called EphPod.
I found my calling with data mining and processing when I co-founded BuildFax. I realized that no one had figured out how to take data that was massively distributed (thousands of end points), and was in thousands of different (and often semi-structured) formats, and pull them together into valuable data product. I created the Pragmatic Extract-Transform-and-Load platform to handle this problem, and BuildFax became (and is still) the only national source for local building permit data.
BuildFax is the first and only data set that reaches inside of a property to reveal hidden property details. Don’t you want to know everything you can about a house before you buy it? Well, now you can because Joe can put all of the history at your fingertips. Smart.
Joe is also a published author on other blogs. For example, his article The Next Generation of AVMs in LifeValuation magazine shares some great information about Automated Valuation Models. Plenty geeky for the real estate crowd. Almost geeky enough for us.
How do you see development or testing evolving over the past few years? the next few years?
I think it’s hard to prognosticate on development and testing in general, but as far as load testing goes, I see a steady move toward low-cost on-demand load testing. I just can’t see these typical “pay $10K+ and wait weeks to test” being the future of load testing. On-demand stress test tools are growing. Load testing services are going to need to have web-based control panels that allow launching arbitrary numbers of servers at very low cost. I was recently quoted by a vendor more than $1/user/minute for an automatically launched load test! That won’t be a viable business in model in five years, any more than charging 25 cents/minute for domestic long distance is today.
Is there anything commonly overlooked in web application testing?
What is overlooked too often by many non-former-programmers who oversee web application development is building in the time to refactor code and generally keep your application code in good and maintainable shape.
Do you prefer using requests per second (RPS), concurrent users, or some other metric to define load?
I prefer RPS to concurrent users because I view concurrent users as less of a performance metric than a marketing metric. 100 “simultaneous users” are not going to produce anywhere near 100 simultaneous requests. More importantly, how many requests those “simultaneous users” are going to request is going to be application and user-specific.
Do you feel like performance testing is an accepted critical part of the development life cycle?
I don’t, but I also don’t think that it should be an accepted critical part of the development life cycle. A lot of development is thrown away, and a lot more is targeted at such a low volume of traffic–and/or low development costs–that performance testing is unnecessary.
How does cloud computing affect the future of automated testing?
Automated testing providers who don’t use cloud computing (meaning on-demand, automated-launch virtual machines) won’t be able to compete on price.
What resources do you typically remove/restrict from a system to perform a stress test?
Because we’re in the cloud (AWS), we launch a separate set of servers just for stress testing. We are then able to test exactly what the production environment has. We consider stress testing pushing the traffic until the system fails. Some people intentionally break a part of their network to see what happens, but we don’t do that.
A subsection of our usability testing relates to response time, but beyond that, I do not see anything in performance testing that addresses the core elements of usability testing. Moreover, I don’t know any experts in either who would be as good at the other.
We know they are very different, but are there any ways they are alike?
Methodologically, both require a lot of knowledge up front, testing, and experimenting. Beyond that, one is more objective (load/performance testing), and they require different skills.
What is your favorite web application or testing book that you own? That you wish you owned?
Continuous Integration: Improving Software Quality and Reducing Risk by Paul M. Duvall, Steve Matyas, and Andrew Glover.
That wraps it up for this interview. Once again, thanks to Joe for sharing his experience in performance testing with us! You can follow him on Twitter as @BuildFax. Tweet him a thanks for his thoughts. They also have a page on Facebook, and I’ll bet the would like it if you like it.