I'm adaptable. I like change and challenge. But lately, I feel as if I'm traveling through hyperspace in Han Solo's Millennium Falcon.

The list of IT advances over the last decade reads like a "good news/bad news" joke: Centrally managed, mainframe-based CICS green screens have mostly gone the way of the dinosaurs, but no one hacked green screens. When control was centralized, people using customizable workstations weren't calling help desks complaining they couldn't see the sign-on screen (after they'd zoomed the screen's resolution past any reasonable bounds).

Beyond the glass house changes, we're also dealing with wireless computing, with handheld devices for functions that once required hard connections, and with other "innovations du jour."

Consider e-mail. Two years ago, George Washington University students couldn't walk past a kiosk without checking their accounts. Now e-mail is passé, replaced by IM, text messaging, Facebook, and blogging for peer-to-peer, minute-by-minute communicating. When something new and cool appears, students have it in hand before our IT staff even knows it exists. And universities can't ban devices. We can only list them as "non-supported."

Managing complicated environments

The environments we manage these days are letting us serve users in ways that once were impossible. However, the many new IT products and utilities necessary for providing this expanded service have created an extremely complicated environment for us to manage.

At GW, we run the SunGard Banner Higher Education ERP suite and Oracle Financials ERP. We are also deploying the EMC Documentum® suite enterprise-wide. Merely scheduling upgrades and testing is complicated, given our university's business cycles for recruitment, admissions, financial aid, registration, housing, etc., and our fiscal-year operations for federal reporting and so on. Our latest analysis revealed the ideal upgrade window is the fifth Thursday in March of leap years.

That's for the systems that users see. Databases, servers, operating systems, utility systems, the SAN, and backup software have their own refresh cycles. GW maintains test and development environments for every production server and application; it manages dual, mutually supporting data centers. Still, finding the least onerous time to implement a component is difficult.

Also, even as IT is supposed to be moving to service-oriented architectures, end-users' workloads for understanding new software functions and our testing and configuration of those functions requires more commitment from everyone. (I'm still waiting for the first department head to beg me to stop making his/her life easier.) Some IT industry leaders, however, are developing plug-and-play components to help us make infrastructure changes on the fly, without requiring risky, unsustainable levels of end-user involvement.

Stressing collaborative testing

To implement ERP-caliber releases and upgrades, we developed (and are refining) a process that stresses collaborative testing between users and IT staff.

Our IT team spends the first third of a project analyzing the release's content and earmarking critical changes and enhancements. They prepare for comprehensive testing in myriad ways, including arranging for servers and databases, testing desktop clients, analyzing security, and preparing core scripts for end-user testing. Every hour spent preparing pays off in more comprehensive analysis and less impact on campus operations.

Next, the IT staff trains "power users" from critical offices on the new features and teaches them to lead end-user training. The power users apply IT's test scripts, their knowledge of their operations, and their understanding of the previous version of the software to analyze and confirm the accuracy of the "new and improved" functionality.

We log every anomaly. Some are real problems; some are misunderstandings; some are artifacts of test scripts. Generally, anomalies stem from reports that refer to changed database elements, unfamiliar screen interfaces or navigation steps, or edits that are catching previously accepted things. Our programmers work with the ERP vendor to investigate all issues. They are typically fixed quickly and sent back to users for confirmation.

In the final stage, chief user representatives sign off on the upgrade, and a C-level executive committee confirms the "go" decision. We wrap up any final technical items and migrate the software to production. The cutover plan is begun (it's extremely detailed, often by sub-hour increments), and the system goes live at a specific date and time.

Such a process is necessary due to GW's thousands of users, the limited windows for making large system changes, and the huge investment of time and energy required to implement them. For less-intensive database upgrades or security patches, we use a scaled-down version of the process to confirm that the system is operating as it had previously, albeit more efficiently or safely.

Attention to detail has allowed us to upgrade several large ERP-class systems without ever having to roll back to a previous version due to unanticipated issues. Each upgrade plan, however, does include a contingency date in case of last-minute surprises. Establishing this date (usually falling no more than two weeks later) reduces the psychological pressure to "just do it" regardless of risk. And it gives decision makers a logical, practical alternative. Taking the pressure off leads to a more reasoned risk analysis.

Bill Gates reportedly said, "If GM had kept up with technology like the computer industry has, we would all be driving $25 cars that got 1,000 mpg." But if the auto industry released changes as the IT industry does, we'd have our vehicles in the shop in January for new tires, in February for new airbags, and in March for newly developed ceramic brakes.

As an industry, our IT environments will never grow less complicated. But as we make changes, we can lessen the end-user impact. All of us want to serve our end users more efficiently and effectively. We need to concentrate on reducing the unintended adverse impacts of generally positive changes.