A Facilities Overview with Patrick Kazeze
The Facilities Team Takes on Many Roles
Our Facilities team is unique in that they are jack-of-all-trades and cover many disciplines. They are customer facing with exceptional customer service skills. They also are involved in the project management cycle, and to some degree project managers in their own right. They work with almost all of the different sections that make up ICN from Finance to Engineering to Business Services to Operations.
When a customer site is conceptualized the Facilities team is involved from start to finish; from the planning stages until the site becomes operational. Individuals within this team oversee operations, and if a sunset of a site occurs, they ensure that the site is taken down and ICN property is removed, leaving as little trace as possible that an ICN presence was there.
The team provides oversight of our network vendor, which operates our Network Operations Center (NOC). They oversee the planned network maintenance, monthly testing of backup generators, and HVAC at over 1500 sites. This group also operates and manages local and long-distance calling features for our voice customers.
In addition, the team performs monthly audits of the work completed by our network vendor (FNS), to ensure that Service Level Agreements (SLA) are met along with other metrics. In essence, the team is the quality assurance piece that validates that the work is completed in accordance with ICN standards.
TAC and Supporting our Customers
The TAC team (Technical Assistance Center) is responsible for the structured cabling side of the house, for both copper and fiber. What exactly does this mean? Simply put, they handle the infrastructure once it is fed inside a building to a Point-of-Presence (POP). TAC pulls the cabling through walls, up between floors, via baskets trays they build or conduit runs. The cabling is neatly organized and identified to the FOTS rooms (fiber-optic transmission system), where the team patches it to the requested location, and then neatly terminates the cable in the room for the customer. The TAC also pulls fiber and copper in data rooms and patch equipment using jumpers to complete circuits. This cabling provides connectivity for equipment located all around the Capitol Complex (and statewide), which provides the backbone for customers’ phones and computers.
In addition, the TAC is also responsible for installing FirstNet equipment at Public Safety Answering Points (PSAPs) across the State for the Iowa Department of Homeland Security and Emergency Management. This mission-critical service provides our public safety users with a backup circuit to communicate on, in the event of an unplanned Network issue. The backup and dedicated FirstNet connection was especially important during the Derecho storm, which damaged several commercial cell towers.
A Typical Day for the TAC team…Expect the Unexpected
They approach a typical day fully aware it is in their best interest to expect the unexpected. At the end of each day, they have a 'game plan' etched in pencil, which consists of known projects that are scheduled to take place the following business day. Everything may change overnight, and it sometimes does. Fortunately, much like the entire organization, we clearly understand our schedules must remain fluid, as we are constantly adapting to an ever-changing technical environment, and the opportunities it presents each day.
We are very fortunate to have a customer-base that can be flexible as well. An example is, our TAC team may have scheduled an all-day fiber installation for multiple agencies in one of the Capitol Complex buildings. The fiber installation is scheduled with our customer well in advance, but without warning, the TAC team could be assigned support tickets (incident requests) for a customer’s wireless access point that isn't working 30 miles away, desktop telephone devices may not be working for another customer in another State building, and a wireless analog phone may be offline at another customer location. When these opportunities arise, our TAC team quickly prioritize the tasks at hand, and shift their focus to the high priority incident requests. When acceptable, they negotiate new anticipated completion dates with our site contacts for the projects currently underway and forge ahead accordingly.
Our Network Operations Center – First Line of Support
The first line of support when a customer calls is to our Iowa-based Network Operations Center (NOC). The NOC personnel are customer-facing and receive all help desk issues. They have to be smart about many facets of applications and able to communicate effectively to the customer when solving problems or providing assistance.
Fielding a Myriad of Requests
There is no typical day per se for this team as they provide customer support 24 hours a day, 7 days a week. The NOC is built around crisis management and immediate responses. Customers don't typically call the NOC to see how their day is going. It is almost always a problem, and the requests can come in by phone or by email.
There can be any myriad of requests from a past phone issue to a complex service outage. The NOC will take in all the requests for support and triage them, then manage the workflow by directing the work to the respective areas. They are able to resolve most of the issues for most customers within the NOC. They keep track of all problems, keep the customer informed on the support progress, and provide feedback as they receive it. The team at the NOC are quite literally the hub of the customer facing side of ICN, not much happens that doesn't flow through the NOC at some point.
Automation Increases Efficiencies
The team adapted to change as ICN rolled out ServiceNow in early 2018, which replaced the previous ticketing system called ServiceDesk. With that change, the Facilities group automated a good number of processes. As time marches on we will continue to automate more and more. We are always looking for opportunities to maximize efficiencies, and automation has also provided better oversight of our assets.
The number of tickets ICN receives can vary from month-to-month. During our slower months, we receive around 300 tickets per month and during our busy months it is around 600. The average falls somewhere between 450 - 475 tickets a month.
Closing Thoughts about the Network
The ICN is a unique organization in so many ways. Our robust Network provides services to K-12 schools, libraries, universities and colleges, hospitals and clinics, state and federal government, and public safety entities.
ICN's customers can rest assured and focus on their specialties knowing that they have a solid fiber optic infrastructure to support their operations. As part of that supporting infrastructure we are adding to our services portfolio by virtualizing our DDoS Mitigation and Firewall services, along with adding peering connections and gateways to cloud services. As we look to the future, we will continue to evaluate the needs of our customers and look for services and solutions that satisfy those needs.