We demonstrated to campus that organic and non-mandated partnerships are not only possible but can be mutually beneficial and build long-term trust.
Recently the Office of the Chief Information Officer and The Grainger College of Engineering partnered in an IT Operational Excellence initiative to combine previously separate environments for virtualization, storage, and backup to provide a more reliable, scalable central offering. The result benefits both units and serves as an example of how the campuswide Operational Excellence mission is strengthening the university.
When the Operational Excellence initiative began in May 2021 with the goal of reimagining how administrative areas function at all levels, Darius Summerville, assistant director for Infrastructure Services at Grainger Engineering IT Shared Services had already been brainstorming ways to improve the college’s services.
“An idea had been swirling around my head: could we move our virtual environment at the Grainger College of Engineering to a self-service model? After attending Dell’s annual conference and speaking with peers and other vendors, a path forward on this was identified. However, I knew I could not do it alone,” Summerville said.
Grainger Engineering IT had been expending financial, administrative and human capital running very standard IT services for virtual machines, data storage, and data backup for only their college.
With an idea, Summerville reached out to Brad Mahaffey, director of enterprise infrastructure for Technology Services, the IT service arm of the Office of the Chief Information Officer, to see whether something could be done together.
At that time, Tech Services had been experiencing high input costs and stagnated growth for these services, resulting in high costs for customers who used them. As a result, most units with the means and ability to run them on their own began to do so.
“[Brad was] interested in the stability and efficiencies we could build by combining our environments. Then, we just had to socialize it to our teams and others,” Summerville said.
The timing couldn’t have been better. “We struck while the campus had started taking a deep look into operational excellence and reducing duplicative services. Our teams both had these conversations multiple times over the years, but this time felt different,” Summerville said.
Once Mahaffey and Summerville had permission from their respective departments to move forward, they hit the ground running. Summerville said that making a project team was easy, since Frank Penrose, lead systems engineer for Grainger Engineering IT, Joel Franzen, lead infrastructure engineer for Tech Services and he had all worked together at Grainger Engineering IT previously and were familiar with the services. Working alongside Mahaffey, Summerville, Penrose and Franzen on the project were also Engineering IT’s Casey Coughlen, senior Linux system administrator and Paul Taraszka, site reliability engineer.
In total, we migrated at least 368 virtual machines, 566.61 terabytes of used storage, 1,254 virtualized central processing units and 2.87 terabytes of computer memory (vRAM).
“There was trust, respect and a mutual energy around getting things done that impacted the quality of life for all IT professionals. These relationships and that energy is what propelled this project forward!” Summerville explained.
Summerville, who worked on the project’s administrative needs alongside Mahaffey, explained that the rest of the project team were the ones who did a lot of the work. “They were up at 5 a.m. most days moving virtual machines or working to get the Tech Services environment ready.”
“In total, we migrated at least 368 virtual machines, 566.61 terabytes of used storage, 1,254 virtualized central processing units and 2.87 terabytes of computer memory (vRAM). There is a lot of work behind these migrations,” said Frank Penrose, project team member and lead systems engineer for Grainger Engineering IT. “Before the project, this service was in only one site, and now we are in two. In addition, backups of physical machines is now a legitimate part of the service whereas before, it was sort of a bolt-on.”
Summerville explained more about the work that needed to be done to make this collaboration happen. “There was coordination of downtime with other teams, technical clean up, and answering questions of customers and others that this project team handled with great tenacity and efficiency.”
Together, the team executed the project efficiently with the least amount of disruption to users. Both units surpassed an at-scale threshold for these services to create a central offering that could leverage its size for reliability, scalability and lowered costs. They built a more robust environment and created an easy avenue for any other partners in the future. In result, the campus service moved beyond a single administrator to a team of three that could work on the service and respond to incidents.
Both Tech Services and Grainger Engineering IT benefitted from this partnership. Grainger Engineering IT removed the technical and administrative overhead previously required to run the services just for their college. Tech Services created a central service offering with enough scale to lower the service costs and offer a viable alternative for other IT units to potentially benefit from as well. Tech Services was also able to rearchitect old and costly technical models in favor of lower cost and easily expandable technical models. Together, Grainger Engineering IT and Tech Services also partnered on staffing this new service.
“We hopefully have put this service in a position to be attractive to others. We’re open for business!” Summerville said.
This central service offering is looking for more units to partner with. For more information about the service or partnering with the service, reach out to virtualhosting@illinois.edu or learn more about virtual hosting.
Mahaffey expanded on the benefits of this partnership. “We demonstrated to campus that organic and non-mandated partnerships are not only possible but can be mutually beneficial and build long-term trust. It’s a win for both sides, from an operational excellence perspective. Both sides can now start focusing on service improvements now that the major issues of the base services are resolved,” he said.
The strong collaboration and support initiated by Summerville and Mahaffey played a crucial role in the project’s success.
“[Brad] and I were not strangers! This was a great opportunity to demonstrate what great relationships and shared visions can achieve here (at Illinois),” Summerville said.
The server virtualization project is one of several IT Operational Excellence initiatives underway. To learn more, visit the IT Operational Excellence page on the Office of the CIO website.