Campus Experiences System Failures, Outages

University suffers four unrelated technical snafus

John Lambert The Santa ClaraFebruary 16, 2017

Recent outages within Santa Clara communication networks have left students and faculty searching for answers

The four separate network issues all occurred during the month of January. The string of technical difficulties barred students from accessing Camino and administrators from receiving inbound calls. Santa Clara’s network also experienced a distributed denial of service attack as well as firewall instability issues.

According to university administrators, although similar in nature and consequence, the issues were unrelated.

“We had four (outages) happen to us, all of them unrelated,” said Chief Information Officer and Vice Provost for Information Services Bob Owen. “I’ve been doing this for thirty years (and) I’ve never seen any of these all hit within a month’s period of time.”

On Jan. 3, the first connectivity issue occurred with the university’s recently acquired Cisco firewalls. The day after Santa Clara began implementing the higher-end security system, Owen and his team found a series of bugs and defects within the firewalls’ application to the Santa Clara network.

After a series of meetings and phone calls with high-ranking Cisco officials, patches were applied and system configurations tweaks were completed on Jan. 12. This ensured full firewall stability, according to Owen

Although the firewalls are currently running effectively, an alarming total of nine instances of interrupted network connection were discovered within the month.

On Jan. 10, the university experienced a distributed denial-of-service (DDoS) attack, causing fluctuating network connectivity for almost two business days. Although this time period coincided with the firewall incidents, school officials determined by end of day on the Jan. 10 that the two occurrences were unrelated.

Upon determining that it was someone within the Santa Clara network sending an outward DDoS attack, rather than school firewalls coming under fire themselves, administrators terminated the connected account. They then manipulated the Cisco firewall settings and restored the internet connection to full capability.

“There was a computer inside of us directing an attack that was outside of our network,” Owen said. “We were just the vehicle.”

The third technical difficulty occurred on Jan. 16 when the university’s AT&T telephone service was accidently terminated, causing the school to lose all inbound calling capability. Because students rarely use university telephones, few were directly affected by the loss of connection. However, administrators who rely on telephone service were greatly affected that day and until the following morning.

Unlike the first two attacks which resulted from implementation and protection issues, Owen said the AT&T outage can be best explained by incompetence. After accidentally disconnecting Santa Clara’s telephone service, AT&T was allegedly unable to find any personnel worldwide to address the school’s complications.

“I told AT&T, frankly if you want our business and you want to continue to have our business, you better come to me … on what you’re going to do to make it right,” Owen said.

Frustrated by the phone company, Owen said he plans to see offers from AT&T’s competitor, Verizon, to weigh whether or not a telephone carrier switch is warranted.

“(AT&T) could be the absolute cheapest and come to us with almost zero cost, (but) it doesn’t matter. What we’re after is reliability,” said Rian Aldridge, director of cyber-infrastructure technologies.

The final infrastructure challenge came on Jan. 23 when students lost access to Okta sign-on systems, preventing them from accessing Gmail and Canvas. For an entire day, students could not access anything from Google Docs to submission portals on Camino.

“I think it’s essential to have access to Camino, especially when we have such a hectic schedule,” said sophomore Gen Kimura. “If we need to finish a paper last minute we need to be able to submit it.”

These complications occurred from the school’s simultaneous reliance on new Okta servers and certain data sources from Novell, the school’s former applications program. According to Owen, a full transition to Okta is expected to occur by June 30.

Though the school was aware that Novell’s antiquated systems remained on Santa Clara’s servers, the likelihood of any interference between the two systems was a surprise to administrators.

“We are now paying the price for having continued to use a technology system, Novell, that went out of use 15 years ago,” Owen said.

Although the string of technical difficulties was discouraging, Owen said the university continues its commitment to quality connectivity. Despite the network and connectivity failures, Owen said that the university uses some of the best available resources on the market

“It’s all about what kind of service levels we provide, not what’s the cheapest, not what’s the easiest,” Aldridge said. “It’s definitely about reliability.”

Contact John Lambert at jvlambert@scu.edu or call (408) 554-4852.