Refining Disaster Recovery in a Challenging Environment
A clever use of software-defined network virtualization took Mohave Community College from a crippling 72-hour outage to a 45-minute (and still dropping) recovery time.
Category: IT Infrastructure & Systems
Institution: Mohave Community College
Project: Disaster Recovery
Project lead: Mark Van Pelt, chief information officer
Tech lineup: CDW-G, Veeam, VMware
Project team members Brian Massey (left) and Joshua Walters (right) (Photo courtesy of Mohave Community College)
Nestled in the northwest corner of Arizona, Mohave County is the fifth-largest county in the contiguous United States. That makes for some serious challenges for the 16-person IT team at Mohave Community College, which has campuses far-flung in Kingman, Lake Havasu City, Bullhead City and Colorado City. "My closest campus is 45 minutes away and my farthest is four-and-a-half hours away, and I have to drive through two other states to get there," said Mark Van Pelt, who has been MCC's chief information officer for three years.
Mark Van Pelt
Van Pelt said he took the position well knowing that the county's vast size and the college's limited resources would bring unique challenges, especially in terms of network infrastructure and disaster recovery. For instance, fiber is unavailable in Mojave County, and much of the connectivity comes from low-speed microwave connections.
"I came in as the seventh CIO in a five-year period," he added. "When I arrived, we were averaging about two outages per week in terms of major systems. I wouldn't say the infrastructure was collapsing, but we were certainly challenged. The youngest equipment on campus was about the same age as most of our students. In terms of networking equipment and most server infrastructure, I knew I had a big job ahead of me when I got here."
Mohave Community College's Lake Havasu City campus (Photo courtesy of Mohave Community College)
Then three months after his arrival in 2016, disaster struck. A systems engineer made a mistake that ended up corrupting the domain databases, and there were no good backups. "The same person who corrupted them was supposed to be backing them up," Van Pelt said. "One night he was supposed to be doing maintenance. The next morning we came in and nobody could log in or see anything." That was a Wednesday. For the next three days, the college was completely down. "I was still getting a handle on what we were or were not doing correctly, but that led to three sleepless nights for our group. Or when we were sleeping, we were sleeping at our desks. We would run a script, knowing it would take an hour, put our heads down and set an alarm to wake up when the script finishes."
The recovery from that outage led Van Pelt and his team on a journey to find a more reliable, yet affordable, disaster recovery solution. Two MCC IT members, Joshua Walters, VMware administrator/team lead, and Brian Massey, network administrator, worked with CDW-G and VMware to implement a solution based on NSX, VMware's software-defined network virtualization and security platform.