A technology issue with King’s College single sign-on services prevented various students, faculty and staff from logging into online university systems for three days between Wednesday November 9 and Saturday November 12.
Login errors have caused disruptions in regular procedures, particularly when a student or faculty/staff member logged out of their school account and were unable to log back in. The outage primarily affected messaging services, but other third-party apps that require a TKC login such as Student Portal and Schoology were also impacted.
“Anyone who logged out of their email on Wednesday and tried to log back in continued to receive this error message,” said Bracey Fuenzalida, director of information technology (IT) at King’s College. “They kept getting a message saying, ‘The service is currently unavailable. Please check with your administrator.’”
“I got here a few minutes before 9 a.m. on Wednesday morning,” said Cole Smith, administrator of Educational Ventures for King’s College. “I logged into my computer as usual and went to Google Chrome, and there’s usually my saved profile with all my regular tabs. I clicked on a few of those and nothing happened. was loading for me. It was asking me to re-enter my info like my username and password. And then the screen basically wouldn’t change… So I went and talked to Bracey about it for a bit and we reset the password multiple times, changed quite a few things, but couldn’t find anything to fix it.
The source of the problem ended up being a switched setting in the school Active Directory and Federated Services single sign-on system.
“Something has been switched to a completely different property,” Fuenzalida said. “It was in a place we never touch, so we know it wasn’t human error.”
When asked how the error happened, Fuenzalida said the IT team “still don’t know how it was changed… When a parameter changes like it did, we check our processes created to see if we may have inadvertently created a rule that changed a property of something. Nothing we had in place changed that setting or its properties. We are still doing all sorts of testing at this time to see what could have caused this change, their goal is to proactively fix the system so that this error does not occur again.
Fuenzalida also said the problem was definitely not caused by a hacker. “We have six different layers to detect hacking,” he said. “If there’s an intrusion, one or more of these layers let us know, and we had none of that. We also had no data loss, security breach or information theft.
Fuenzalida and the other two members of the school’s IT department worked around the clock during the shutdown, contacting Google and Microsoft customer service for help.
“Google told us right away, ‘Not our problem,'” Fuenzalida said. “I called Microsoft Office 365 three times, and each time we spent two hours checking everything.”
Fuenzalida sent three school-wide emails with updates regarding the issue throughout the week. The first email, sent Wednesday at 10:30 a.m., informed the school of a “Google Workspace disruption.” The second email, sent the following day at 5 p.m., asked all students, staff and faculty to avoid disconnecting from any school system until the issue was resolved.
“Honestly, I realize now that it didn’t affect me much because I don’t do all my homework on Google Docs,” said Paige Hagy, City Editor for the Empire State Tribune. “I keep it on my computer with Microsoft Word, but I know for people who store all their homework there, it was a huge inconvenience for them.”
“The day before Interregnum I heard other students were having trouble and tried to log in,” said Evan Louey-Dacus, a senior at House of CS Lewis. “I use my email to apply for a job, and all my college papers are in my Google King account. One of my essays was actually late because of the issue.
“At one point I was pulling my hair out,” Fuenzalida said. “We are looking, we are looking, we are looking. Finally, we just said, “Let’s start, one by one, by looking at the account credentials that allow the transmission of information from our server here to the server in the cloud and see what happens.” We kept trying and trying until we finally found the problem. It is the properties of a particular account that have been modified.
After reversing the setting, the team used test accounts to make sure the login system was working properly again. At 2:12 a.m. on Saturday morning, Fuenzalida sent a third email to inform the school of the news that the connection interruption had been corrected.
“I woke up on Saturday morning and was relieved,” Fuenzalida said. “All credit goes to the two guys I have on my team. Those guys were relentless in their pursuit of what they were looking for…One of the guys is Marat. I affectionately call him Drago from Rocky, because he’s from russia and he’s a genius he does things with systems and computers that i wish i could do in my dreams i also have richard christensen who runs the help desk and he’s also good than anything. He has a keen sense of cyber security and is able to understand how things work. When the three of us get close, it bodes very well.
An email posted on Sunday, November 13 announced that due to email access disruptions, the start of course registration for the Spring 2023 semester would be postponed from November 14 to November 16.
“Before registration opened, we wanted students to be able to get answers to any registration-related questions and for staff to be able to remove any registration holds that might otherwise have been lifted had our systems been fully operational. “, Dr Matthew Parks, Acting Provost of King’s College, said in a statement. “We made the decision at the end of the day on Friday.”
Due to regular communications from the IT team and anticipation of potential problems, the connection interruption did not have a significant impact on the execution of Interregnum from November 11 to 12.
“It wasn’t too difficult,” said Kaylee Vroon, co-chair of the Interregnum committee. “We switched to everyone’s personal emails for the day until this was fixed. The Interregnum email stayed connected the whole time, so we had access to what we needed. We had plans in place if the houses couldn’t submit because of this, but in the end we didn’t need them.
“We had a guy on hand here for Interregnum,” Fuenzalida explained, referring to Christensen. “He was a judge and he also watched our system. We had no glitch. (The connection problems during the obligatory showing of the film were completely unrelated to the interruption of the connection to the server.)
Although the background operations of Interregnum went smoothly, some students still had to work around connection issues to enter the competition. “I lost access to my email just before Interregnum started,” said Rachel Wells, chamberlain of the Household of Queen Elizabeth I. In fact, I had to borrow the computer from a friend and her email to complete our submission. »
Classes were also able to take place normally on Wednesday and Thursday. “One or two of the teachers said they had recorded their presentation for the class on USB sticks,” Fuenzalida said. “I think what happened was because I communicated clearly and effectively with everyone ahead of time…people were able to plan ahead.”
“We almost take it for granted,” Fuenzalida said, referring to the computer systems the school runs on, such as telephone, email and internet service. “When everything is working normally, nobody says a word. It’s only when there’s a disturbance that people notice. But that’s the nature of the profession we’ve chosen, as they say in that famous movie.” He pointed to a picture of the Godfather on the wall behind his desk. “This is the company we chose.”
Melinda Huspen is the campus editor of the Empire State Tribune. She is a student at King’s College in Journalism, Culture and Society.