Is Premature Optimization Really Evil?

Photo Courtesy of: rosendahl

My first initial answer is … it depends. (Isn’t that the case for everything?) 🙂

Today I had conversation with a colleague and we were discussing how application developers don’t plan ahead.

This eventually brought up the question/topic:

“Is premature optimization pure evil?”

My colleague stated that all optimization should be done up front, while I on the other hand, disagree. In my opinion all development should be done first and then lets figure out where we might have some problems, with some exceptions (which we both agreed upon).

So, what did we agree upon?

Lets look at a classic example:

Developer Joe and his team were asked to create an order management system that could handle up to 10,000 orders a day. This doesn’t sound like that big of an issue, and Joe and his team also agreed. It shouldn’t be a big issue. A fast database, a fast web server, and a few third party controls to interact with. No problem! Heck, this is pretty straightforward they thought. In reality, the requirements were very straight forward, no hidden features, no scope creep, nothing like that. Heck, it was a dream project come true. With that said – development ensued, and two months later Joe’s team delivered the product and everyone was excited to see the product run from end to end. (By the way, this was not done TDD or with any Agile Methods – which IMO would have caught any of our upcoming issues, if the tests were written correctly. But then again “If ‘if’ was a fifth then we’d all be drunk”).

Joe’s team fired up the application for a demo and ran some orders through it and it worked great. They even ran 10 orders through the system in one minute (~14,400 Orders a day). They had gone above and beyond the call of duty (original requirement being 10,000 orders a day)! They were due for a raise! Life couldn’t be better! The system was released and the company started to use it. Orders were flowing through the system without failure and the business was happy.

Until the company decided to sell an exclusive hot item two weeks later …

Then SHTF… All of a sudden, customers were experiencing time outs,  orders were being duplicated, and the customer service line was being rung off the hook. Essentially, all hell broke loose. After looking into the system, the developers noticed that the system had only accepted 6,000 orders. What gives? This system should handle 14,400 according to their tests. *everyone is scratching their head at this point*

Scalability is Relative

Scalability in this case, is relative. While looking at the stats, Joe’s team realized that those 6000 orders came in in about 1.5 hours. This was the first 1.5 hours that this product was available. At the end of the day the company eventually sold under 10,000 units, so why did this issue happen?

Lets take a look at some stats:

	Orders Per Minute	Time Between Orders (seconds)
Original Spec (10,000 Per day)	~7	8.64
Current State (6000 in 90 Minutes)	~67	0.9

Orders Per Minute Equation: Orders / Minutes          Minutes: (HoursInDay * MinutesInHour)

Orders Per Second Equation: Seconds / Orders       Seconds: (HoursInDay * MinutesInHour * SecondsInMinute)

 

Do we see the problem here?

Again, scalability is relative. In this case, analysis and optimization was not done ahead of the time to anticipate the flood of orders at a peak period. The spec was straight forward:  The system is to be designed to handle an order rate of 10,000 orders a day. The system was designed exactly to the spec, and in this case, the system was designed incorrectly.

Now, I have a confession… 6 years ago, I was Joe. Yes, I made this very mistake. I was a hero for a couple of weeks, then SHTF. 🙂 Live and learn. Live and learn.

When We Agreed

My colleague and I both agreed that in certain instances premature optimization should be done, such as this case. The developers should optimize the order process to handle issues at peak times. Perhaps implementing a order service, based on MQ, or implementing a service bus for this type of high-availability process would be the best. In this case, we agree. All of this analysis and optimization should be done up front because this type of thing WILL be a problem at any peak utilization times. We both agree that neglecting to analyze the problem domain can and will lead to performance problems down the road. Plain and simple.

When We Disagreed

My colleague and I still disagree on the statement of “all code should be optimized up front”. In my opinion this is a BDUF approach that I tend to shy away from. I don’t necessary mean that no design should happen up front, but I do think it should be limited. In an agile environment things tend to change, and we need to be able to adapt quickly. Wasting months in front of a whiteboard, with of reams of documentation, and endless meetings only slows the process down if something has to change down the road. Re-updating project baselines, and adjusting documentation and so forth is a huge hassle. I would like to state that there is a place for this type of development though. Some companies do require BDUF approaches because of huge regulatory compliance issues, so sometimes there is not much you can do about it. However process driven approaches such as ITIL and Agile can work together if done properly. With that said, I tend to shy away from BDUF approaches because of a simple example that is also exemplified by Jeff Atwood in a previous post of his (and I agree with him and the many other experts in the field). Jeff adds a third rule to M.A. Jacksons Optimization Rules that I like agree with the most:

And I would add a third: don’t optimize work that doesn’t have to be done.  – Jeff Atwood

This can be demonstrated as a sort routine. Lets say you have a menu that will be populated via user selections and the maximum number of values you can have in the menu is 100. You can either implement IComparable and utilize the List<T>.Sort() method to sort the class (which utilizes the Quicksort Algorithm) which would be SUPER easy but may not be the “Fastest” method of sorting, or you could write an extension method to create your own sort routine to sort the 100 items. At this point we’re splitting hairs. A millisecond, or even 10 milliseconds, wont matter at this point in UI response. We’re only sorting a menu that will never grow beyond 100 items. At this point, pre-mature optimization is evil. There is NO need to rewrite something that that does the job well enough. Can we do it better? Yes! Is there a need to? NO. We wont get any value out of it. The time spend on development is wasted compared to the benefit to the application.

On the other hand, if the menu had no upper bound limit, then we would want to implement the best sort routine we could because some routines are exponentially faster than others. In this case we would want to optimize early because the possibility of a bottleneck can and might occur in certain situations.

It all boils down to: “It Depends”

At the end of the day, the true answer to “Is Premature optimization evil?” is “It Depends”.

It depends on the problem. It depends on the architecture and design. It depends on analysis of the domain. It depends on the factors which “could” come into play.

Finally, what it really boils down to is: It depends on MONEY. We can optimize all day and gobs of the companies money in development/optimization costs. This is why deep analysis into the problems need to take place. As architects we need to define the business process that could impact our architecture design and implementation and development of code. At some point we need the business to interject into this analysis and optimization party to say “Listen, I’m willing to deal with X, but I’m not willing to deal with Y”. At that point, our jobs as Architects is to define how much X costs compared to how much Y costs. Only then will the business truly know what should be done juxtaposed with what should not be done.

To sum it up in my own personal opinion, premature optimization can be evil. We, as architects and developers need to think about the problem at hand before we write a line of code. As I stated above, there is no need for a full 90 page document on the process, but at least whiteboard with your colleagues. Are you a solo developer and need someone to bounce ideas off of? Join your local user group, join some groups online, visit the MSDN architecture forum and ask questions.