(Below is from a tweet/X thread I wrote last year)
#Performance testing is important. At the same time, we don’t want to the performance tuning too early, or set an unrealistic performance goal. A better way in my opinion is to tune/optimize the app (or web services) performance as it scales up (getting more users). In my career, I observed quite a few examples of “performance tuning too early“, “unrealistic performance goal” and “unintentionally impacted production systems during perf testing” (exchange server, for example). #AppPerformance #PerformanceTesting #PerformanceTuning
For the second category, unrealistic performance goal is one reason the Ascension One Nurse app wasn’t successful – (below link is no longer available, unfortunately) https://ncoe-static-public-site.pub.cloud-03.pcf.ascension.org (the iOS app is no longer in the App Store either). I was briefly involved in the development more than 2 years ago, fill in as architect. But the new dev manager and his boss were obsessed with some unrealistic performance matrix before the launch. Guess what? It never got launched completely.
Category 3, once when doing perf testing we overloaded the exchange servers as we forgot in our code we sent email first. The exchange server admin had to intervene manually to make sure the company wide email system is working properly. We should use the exchange server testing environment instead of the production environment.
Btw, I believe the company likely spent millions of dollars on the app development – mostly thrown away for now as the app is no longer in the #AppStore. Some of the backend web services could be used in other scenarios. For that matter, if app code is reused then it’s not thrown away.
This project was one of the reasons I left that place to pursue other opportunities. I feel the management on my work place is mostly spinning on the wheels: and try to scape goat the vendors for any mistakes. #officePolitics || A funny part of all this is: I believe the project eventually got cancelled after they spent all the money. Note from the vendor (the contracting company who provided dev/qa etc.) point of view, an open ended project (bill by hour) is the way to go. But in real world, even an executive (more on LinkedIn) sponsored project usually has a budget. I don’t think the unrealistic performance testing alone sunk the project. But it did contribute to the ultimate failure or put the nail on the coffin (if I may).
Also, this is not the 1st time I worked on an executive’s “pet project”. I recall long time ago, when I was working for my 1st US employer, I worked on a project like this. The main difference though, is we did deliver something. I think one way to counter this “never going to production” tendency is to follow Josh Long.
Interestingly enough, in another place and another project, another executive sponsored application re-write project (MyMercy) failed too – that one failed early and failed fast, so in a way that’s a good thing.
There are other failed projects that I was a part of it: maybe someday I will write more about them :-(. Remember the saying we all learned more from failures than from success. I guess ideally it someone else’s failure. Hopefully someday I will write more on the topic in terms of failed projects: why they fail. Statistics shows IT projects have a high failure rate (personally I recall this EDS/Navy project), especially the rewrite ones (refer to Reddit – Why exactly does most software rewrite fail?)
A related observations over my years working at corporate is: what if we especially the higher-ups treat companies’ money and resources more like her/his own (assume he/she is thrift just like Warren Buffett 🙂 I have seen many managers treat companies’ money as if he/she won the jackpot.
(06-05-2024, new) This is certainly not the only place I joined the effort or was dragged into premature performance tuning. I think I got my share of this at my current job too “performance tuning too early“. I just wrote on LinkedIn below (sorry a bit repetitive from my earlier story above): Got to love this pre-mature performance tuning thing. I do have a story on #performanceTesting: once, at a place I worked, we ran the performance testing but we didn’t pay attention that our testing email system was actually on the production environment, hundreds of thousands emails were generated and dragged down the exchanger server 🙁
Signs I am getting old (getting more and more repetitive. Maybe time for me to find a new job or work for myself 🙂
Success Outcomes
Last but not least, I did have some successful stories to tell on performance tuning: I recall in year 2006, I did quite bit on work on this for a CAD translator; in year 2013/2014, I worked on an interesting problem at Mercy (modeling the Salesforce hierarchy inside Java); then in year 2017/2018, while at Mastercard, I participated in some performance tuning work for web services which happened to have 9 fold increase of usage and our goal is 100 transactions per second. Then at Ascension, we were scaling up the Covid Screen and Go app quickly (the email outage issue happened because of we were not careful), and also at one time, we noticed the auto scaler is not working properly. Or in other words, the load balancer was sending traffic to unhealthy nodes, or instances, and we were able to correct the health-check and fix the root cause of the issue.
Some of the tools I used: Dynatrace