The road to becoming a CIO is usually through application development, and my background is no exception. Before my time at Nebraska State, I started working in operations. Then I moved to Infrastructure, where I spent the majority of my founding years. Wherever my journey has taken me, whether in private or public organizations, customers share a common assumption about their struggling or newly implemented systems, devoting more computing power to a problem will solve it.
This band-aid treatment, of course, only masks the problem. Over time, the band-aid always falls off, and the situation becomes more costly and difficult to resolve the next time it arises.
I’ve read other articles that say it makes sense to buy the hardware. The logic is economically based that “developers are expensive and hardware is cheap”. Hardware cost is only a fraction of the overall cost when factoring in supporting infrastructure, storage, backups, and hardware support costs. Looking at modern hardware specs, it’s tempting to go all-in. However, if you do, you run the risk of robbing the development team to pay the infrastructure and operations teams.
New hardware provides immediate scalability, but while scaling your infrastructure is fast, scaling your support staff isn’t so easy. The right support team will be knowledgeable in specific areas and have the required training on your system, but the additional support costs will quickly make this move more expensive. Throwing hardware at a performance issue is no substitute for developing the right team to design your product in the first place.
Next, apply the “technical debt”. A poorly designed application acquires technical debt, such as credit interest. Here’s how it goes: The development team releases a poorly designed app because the programmers took shortcuts to meet deadlines and/or budget constraints. The developers didn’t have enough time to fix minor performance issues. Over time, the team fails to resolve them, although they intend to provide fixes in a future release. Ultimately, minor flaws become expensive problems and no amount of hardware can get you out of the hole. The compounded problem is your technical debt.
Technical debt has other causes, but I often see it in situations where developers are under pressure to build additional functionality into an application in a concise amount of time. In these situations, developers find that the requirements lack definition or that the customers left something out during the requirements gathering phase.
New hardware offers immediate scalability, but if scaling your infrastructure is fast, scaling your support staff isn’t so easy
I believe any development team can help customers minimize or eliminate their technical debt by following three simple rules:
1. Resist the temptation to call a project complete. When the team finds minor coding issues, don’t ignore them or hide them with extra processing power. Take the time to tackle the code before it becomes unmanageable. The earliest would be best.
2. Pinpoint precisely where performance issues are and clean only identified areas for immediate impact. Then rinse and repeat. In my experience, the problem is often inefficient interaction with a database or a poorly optimized database; nested loops launching queries that launch other questions; excessive table sweeps; etc
3. Changes you make should ultimately result in lower rates. The customer should see this as a long-term win. It makes no sense to spend expensive development time with very little return on investment. Make sure there are significant performance gains before you start. Never optimize beyond your stated performance goals. Know when it’s time to stop and move on.
Along the way, a technology leader must educate decision makers to invest their resources appropriately and exercise good judgment in the best interests of their organization. As technology leaders, we can hold our product owners accountable by educating them on the true cost of hardware. Where I am in my journey, I intend to help the State of Nebraska make decisions that reduce material processing costs, infrastructure costs, and therefore the total cost possession for state taxpayers.