Or, the hidden costs of not maintaining software.
After 34 years in IT, I have worked at many companies and on many systems. The one question I hear a lot is, “How often should we upgrade our software and systems to the latest version?” The answer is simple: Don’t wait to maintain mission-critical software and systems.
I recently completed a project that took more than seven months, with two full-time contractors, part-time database programmers and several testers. Our mission was to upgrade the client’s N-tier computer systems to current versions of the existing software on upgraded Windows Server OS virtual machines and on an upgraded database server. The last upgrade was in 2014 on many of the servers, so all the production software versions were old or deprecated. We let the client know we had compatibility and security concerns for its software and systems.
First Things First: Server and Software Inventory
We began by taking inventory of all servers to be upgraded, determining the software version needed, as well as any problems the systems would have during the upgrade. We then requested new virtual machines for each server from the client’s IT department. We also recorded all software versions and plug-ins to establish what needed upgrading.
To upgrade the systems, we built a parallel system using new servers to mimic the production servers, which we later rolled over to over a period of months. The biggest issue we faced was creating firewall holes and new DNS names to communicate between the layers and then make sure all of the new software worked together and that external clients could access it.
Before upgrading the production systems, we upgraded all the client’s development and user acceptance testing (UAT) servers to ensure that the upgrade would work in our environment.
Upgrading the Database
We started by upgrading MS SQL Server 2008 R2 to 2017. Thanks to Microsoft’s upgrading functionality, this was relatively uncomplicated even though we had many tables, stored procedures, and applications to test. The biggest issue was the time it took to regression test the entire system, which had more than 2,100 stored procedures, 1,600 tables, and 285 views. Our upgrades demonstrated minor bugs that had to be fixed; the system then had to be retested repeatedly. Kudos to Microsoft for its awesome database migration process as it worked so well.
And on to the Middle Tier and Web Applications
We then upgraded our middle tier servers running IIS and SQL Server Reporting Services (SSRS). The laborious process – the client had several hundred SSRS reports – included having to open each report solution, convert each of these reports to SSRS 2016 in each environment, run the report, and compare results from the old report to the upgraded one. We then rebuilt and deployed the reports to the new middle-tier server and retested. Along with these changes, our middle-tier server service architecture code required numerous upgrades to the Kendo, .Net Framework, and NuGet packages.
Several web and support applications needed to be upgraded. The applications are mostly model-view-view-model web design, and that code required numerous upgrades to Kendo, jQuery, .Net Framework, and NuGet packages. As we upgraded these packages, they each had their own requirements for them to work properly. We also found issues in the Kendo and jQuery upgrades, which resulted in the website interfaces not functioning as expected. These had to be fixed for the system to run correctly.
The True Cost of Upgrading
After climbing over mountains of code and documenting many systems, we arrived at our destination—but at what cost? Although it is difficult to account for the exact cost for the upgrade, I estimate that we had more than 2,000 developer hours, 300 DBA hours, and 200 tester hours. This equates to a six-figure sum for this upgrade. The most important issue wasn’t the cost of the upgrade but the loss of the resources that were focused on upgrade work instead of developing new software or enhancing existing software to help the client meet their business objectives.
The client is now dedicating time in its monthly sprint cycle to regularly upgrade its systems. These minor updates every few months will increase efficiency and avoid the need to perform a huge upgrade after several years of getting “out of sync” with software versions. Because the upgrades are small, focused upgrades will be minimal if bugs are found and will be easier to fix. This should also increase system security as newer versions of software will have the latest security updates. All in all, it’s better to upgrade incrementally than all at once.