Iowa Telecommunications & Technology Commission May 2020 Minutes
Iowa Communications Network Staff
Call to Order:
Introduction of Krista Wenzel:
The Commission welcomed Krista Wenzel from Waverly, IA. She has worked with municipals for the last seven years, completing financials, regulatory, and operational support. She has also provided regulatory support to the local telecoms within the Midwest.
Approval of the May 21, 2020 Meeting Minutes:
Finance Update – Mike Cruise
May Fiscal Year (FY) 2020 Budget-to-Actual Report
Monthly Revenues and Expenses:
The month of May resulted in an operating margin of -$136,113.74. This is favorable to the budget of -$229,408 by $93,294.26. This is the tenth month in a row that we have been favorable to budget. We did see increases in Internet/Ethernet bandwidth and voice usage due to COVID-19 related requests. Outside Plant (OSP) expenses increased as many projects were booked in the month of May.
Year-to-Date Revenues and Expenses:
The May year-to-date operating margin was -$1,084,728.62, which is favorable to the budgeted margin of -$2,075,465 by $990,736.38.
Comparison to FY 2019:
The operating margin for FY 2020 is favorable to the May FY 2019 operating margin by $275,218.16. This can be attributed to an increase in revenue of $271,381.57.
Year-to-Date Comparison to FY 2019:
The May year-to-date margin is $893,475.79 favorable to the FY 2019 margin which was -$1,978,204.41. There were no major issues in May.
Business Services Update – Randy Goddard
Iowa Homeland Security & Emergency Management (HSEMD) and 911 Project:
Wireline and Hosted Public Safety Answering Point (PSAP) Equipment:
- Wireline Aggregation – ICN is working on the migration plan with CenturyLink, public-safety answering points (PSAPs), and other HSEMD contractors. Testing was done to ensure the Voice paths moved from CenturyLink to the Comtech data center. The Core network is up and operational. The first PSAP that will be migrated is Marshalltown. Targeted to have all sites migrated by fall.
- Strengthening 911 Solution - ICN continues working closely with HSEMD and contractors for strengthening the 911 ESInet (Emergency Services IP Network) solution with secondary connections.
- Primary connection into PSAP is ICN fiber.
- Secondary connection will be a wireless connection. We will be using AT&T’s FirstNet product.
Healthcare Network: Iowa Rural Health Telecommunications Program (IRHTP)
- Continue to gain access into the hospitals or clinics for the Core equipment refresh. Anticipated to be completed in October.
- Weekly filings continue using the USAC waiver process. There are a lot of increases in bandwidth. Customers previously connected with 200-300 Mb are jumping to 1 Gb. Customers at 1 Gb are subscribing to 2-3 Gb. We anticipate a lot of this is in preparation for tele-education.
- For customers that previously had ICN video connections, some are continuing to keep their connections to be used as a video circuit.
Other Significant Opportunities
- Managed Voice for Department of Human Services (DHS) Call Centers: one in Council Bluffs and two in Des Moines.
- Department of Corrections’ (DOC) Clarinda site is scheduled for MVS phone installation. Rockwell City and Newton are next.
- Fiber builds: DHS office in Mason City, Mahaska County 911 relocation, IWD in Marshalltown, Wayne County Hospital, and Burgess Clinic in Whiting. Fiber builds completed: Mason City to Clear Lake.
- DHS Medicaid will incorporate an Interactive Voice Response (IVR) application.
- Completed several field office installations for MVS for DHS Child Support Recovery.
- DOC Inmate Tablet RFP – Responses are being evaluated.
- COVID-19 related orders: Slowed down. Marshalltown Veterans Home increased their Internet for video connection with families.
- New customer: Woodward Granger CSD with Internet and WAN.
- ISU Platforms for Advanced Wireless Research (PAWRNET) Grant support for Iowa State University. Two counties: Boone and Story. Moved out to August 2020.
- Registered for the Iowa Association of School Boards annual meeting.
- SYBAC Applications – Received 19 applications.
- Communications team is working on a new website design. All state agencies are required to transition to the Drupal 8 application. ICN is working with OCIO’s vendor Webspec.
- Scheduling meetings with Directors from other agencies. Met with Department of Corrections. Future meetings: Department of Education, OCIO, DOT.
Commissioner Wenzel – As you are mentioning some of the fiber builds, are you running into supply issues with COVID?
Ryan Mulhall – No we have not at this time.
Chair Kniff McCulla – Are you holding the Director meetings by Zoom or will the meetings be in-person?
Randy Goddard – The DOC meeting happened before the COVID-19 situation, but for the Department of Education we will use Google Meet. We are leaving it up to the directors on how they want to meet with us.
Mississippi Bend Area Education Agency (AEA) Waiver
Mississippi Bend AEA 9 Waiver Overview - Randy Goddard
Randy Goddard recapped the Mississippi Bend AEA 9 Waiver Overview as presented during the May ITTC meeting.
Randy Olsen (MBAEA) sent an updated waiver request on May 29, 2020 for action by ITTC. Olsen modified the waiver request to remove the AEA Muscatine location.
Mississippi Bend AEA 9 Waiver Overview - Randy Olsen
The waiver request was not meant to compare the Consortium bids together. At the time the Consortium bid included the 25 school districts and the 4 AEA locations, and it was made in conjunction to go in the direction of IP Pathways. The waiver is to allow the AEA to remain in the Consortium to leverage the pricing that MBAEA was able to achieve on behalf of the schools.
Commissioner Lapointe – Last meeting Mr. Nagel brought up that the Commission can’t grant a waiver more than three-years, and the Commission typically does not grant a five-year waiver. How would the pricing change for three-years?
Alan Nagel – When I said that I was in a different capacity than I am here today. When you look at the Iowa Code section 8D.9, it reads that the waiver approved by the Commission may be for a period as requested by the private or public agency of up to three-years. When the Commission makes its own decision, it should do that in consultation with its own counsel for this meeting, which is Ms. Michelle Rabe.
Commissioner Lapointe – Ms. Rabe, can the Commission grant a five-year waiver?
Michelle Rabe – No, you are limited to three-years per Code.
Commissioner Lapointe – Then back to the rest of my question, how does that affect the pricing model?
Randy Goddard –Typically deeper discounts are achieved over longer-term contracts. Much like IP Pathways, ICN pricing would likely be higher, since you are trying to off-set the pricing of the hardware in three-years verses five-years. Both pricing would increase.
Randy Olsen – The NRC for ICN for the firewall equipment would be divided by 36 months instead of 60 months.
Commissioner Lapointe – Could the Commission grant a three-year waiver with an understanding that MBAEA would return and with an indication that the other two years would be granted? Is this a staff recommendation?
Alan Nagel – The first question ICN staff can’t answer. The second question, when you compare the two bandwidths with the maximum to the minimum, the ICN is higher, and the minimum bandwidth was not provided in response to the RFP, is that correct?
Randy Goddard – Yes.
Randy Olsen – From a bandwidth term, the higher the bandwidth the lower the cost per megabit. If MBAEA would have selected the higher end circuits, MBAEA would have received a deeper discount. In theory I could provide that, MBAEA selected the lower bandwidth, since MBAEA did not need the speed, in an attempt to utilize the money the best way.
Commissioner Holz – Have you provided any information supporting that the proposition that the firewall will be compliant with the bid requirements? That each firewall has its own configuration completely separate from the configuration from any other firewall and not be a virtual instance and hosted in a share physical firewall.
Randy Olsen – I would need IP Pathways to provide a detailed explanation on verifying that. On our project to this point, MBAEA has been identifying all the rules with every school district. I could have IP Pathways provide that if it is requested.
Commissioner Lapointe – Did the ICN quote the lower speeds?
Randy Goddard – ICN provided a response to the maximum bandwidth, knowing at the time that Mr. Olsen could decrease to a smaller bandwidth amount at a later date. At the time of the RFP, no ICN did not provide the lower bandwidth.
Chair Kniff McCulla – Mr. Olsen, do you know if you have a physical box at each site for the firewall or is that something that you would need to check with IP Pathways.
Randy Olsen – No, the two bids were different. The ICN’s response was placing a physical firewall on premise at each school district. IP Pathways’ response was procuring a point-to-point line from the school district to their Urbandale datacenter and the physical firewalls are located in their datacenter.
Ryan Mulhall – It sounds like centralized, but if the firewalls are all separate physical appliances for each school district with separate configurations, it will add overhead, performance issues, and latency; but it probably does meet the intent of a separate firewall for each location.
Randy Olsen – Our advantage from a resiliency standpoint, IP Pathways has immediate access to each firewall. They could get [the school] up quicker than a physical firewall located in each site. If two firewalls go down in my district, ICN would be remotely dispatching people in two locations. IP Pathways would have physical access to the two firewalls. This was a big selling point for the districts. Districts liked that the firewall was not located onsite, IP pathways has a highly resilient infrastructure.
Commissioner Holz – How is having that in one location each having their own configuration, completely separate from the configuration from other firewalls, and is not a shared physical firewall? Are you saying there is however many firewalls sitting in Urbandale directly connected to the school districts?
Randy Olsen – Yes that is the way they structured their response. I have not physically seen it. IP Pathways would be out of compliance with E-Rate and the FCC if they would not be structured to the way they responded to the RFP. They responded physical firewalls for each location. They are configuring it as such, it is not shared hardware, and it is not a single firewall with shared slices. It is 25 physical firewalls, and they are in a rack with separate connections.
Bill Decker - Decker shared an event that occurred in which school districts were down due to firewall issues. MBAEA made a commitment to their schools districts to come up with a solution that would not allow that to happen again. Resiliency is important for a school if they are in the middle of testing, and to get that back up is MBAEA’s top priority. Standing up and walking down a hallway and replacing a firewall on a rack is going to help resiliency to take place in a quicker fashion verses if we need to order a product and deploy it to a school district.
Commissioner Wenzel – My main question was answered to ensure there was compliance with the RFP and the firewall. I have no additional comments.
Deb Evans – I wanted to add that ICN’s bandwidth was competitive with IP Pathways. For hardware, ICN selected a particular firewall that was more expensive. When ICN bid it, how ICN interpreted it, it was separate out at the schools, so ICN took costs and maintenance in consideration verses walking down a hall and fixing it. That is where the price discrepancy is.
Commissioner Lapointe – The Commission can’t grant a five-year waiver, the law does not allow that. Does MBAEA need to come back with a proposal for three years and would ICN still not be competitive?
Randy Goddard – Based on the hardware that ICN selected, our pricing would still be high based on the vendor that ICN selected to work with.
Deb Evans – I cannot anticipate what ICN’s pricing will be in three-years. ICN leadership continues to review and lower pricing per the market environment.
Commissioner Lapointe – The Commission can only do two things: grant a three-year waiver contingent on the pricing that shows that the ICN is not competitive, or deny the waiver all together and ask MBAEA to come back with a three-year waiver proposal.
Bill Decker – I do see the position the Commission is in. It would be acceptable to MBAEA if a three-year waiver would be granted. MBAEA would likely come back in two years, and ask for a waiver to cover the timeframe remaining, and if there were some sort of issue that would give MBAEA another year to reach an agreement. That would be acceptable for MBAEA.
Michelle Rabe – From a legal requirement, the Commission can grant up to the three-years, and the agency can come back and ask for a renewal before it expires. Three-years is the statutory timeframe or less. I don’t see a problem, if Mr. Olsen or Mr. Decker indicates if they were going to reduce for three-years, and provide a verbal amendment, and submit documentation after the meeting indicating the five-year waiver to three-year waiver.
Alan Nagel – ICN staff does not dispute there could be an oral amendment to the waiver while the meeting is going on from a five-year to a three-year.
Commissioner Holz – Mr. Olsen, do you wish to amend your application to make it a three-year request?
Randy Olsen – Yes, I do wish to amend the waiver to a three-year request.
With the application being amended to three-years, Commissioner Holz moved to approve the MBAEA waiver. Commissioner Lapointe seconded the motion. The motion passed unanimously.
Network Services Update – Ryan Mulhall
Outages / Customer Issues
Fiber Cut – June 11, 2020
- Location: South right-of-way of NE 62nd Ave in Altoona
- Root cause analysis: The cable had been properly marked prior to the damage.The impact to the cable demonstrates that the damage was caused by a backhoe or track hoe bucket. There was no evidence the cable had been previously exposed or placed above ground before it was damaged. A new cable was laid out and a repair plan was put in place that would resplice onto the new cable during the evening of June 14. HSEMD and DPS requested that since the repair cutover would require taking down 911 that ICN waits until a later time due to potential protests that night and through the weekend. At that time, the contractor told an ICN representative that their remaining work would not require that the ICN cable be moved.
- Prior to the repair, the ICN cable was again damaged by the contractor on June 14 at 1:20 p.m. This time severing the cable and disrupting all traffic.Another maintenance repair was then begun to restore service. This is a temporary.
- Cable served 911, HSEMD, DPS, DOC, hospitals, and critical infrastructure end users.
- Final repair: The final repair will require ICN to place a new section of cable through the damaged area, to place new splices, and to hold another maintenance period that will again affect our critical infrastructure customers.
TAC – Structured Cabling
- Our TAC team continues to support customers and their cabling projects all across the state, and have been on the frontlines during the pandemic along with our FNS Field technicians.
- We have been able to work with HSEMD and IDPH to provide PPE including masks, gloves, and hand sanitizer multiple times.
- Projects include: the last phase of the OCIO’s move to 200 E Grand, DHS field offices, installing cameras in the historical building, and field surveys and cabling for the HSEMD FirstNet backup cellular network.
- Continue to work on projects like upgrading Asset Manager, which is what ICN utilizes to track equipment, circuits, as well as the provisioning tool for engineers. Upgrade is needed in order to finish moving to Windows 10. We are currently working with our vendor to begin testing.
- Moving closer to deploying Customer Service Management in ServiceNow to production as the testing phase is almost complete.
- Another project is upgrading our Oracle database software and purchasing new hardware. We have many very important applications that rely on this including our billing application, our tech labor billing application, and Asset Manager.
- There are two positions currently in the process of being open for applicants. We have a part-time contract position for a software developer/programmer to maintain finance applications and a full-time administrator position to help manage our carrier network and internal infrastructure.
Operations/Engineering Update – Ryan Mulhall
100GB Network Core Upgrade:
- Migrating circuits to 100GB Core – Layer 2, upgrades are on-going.
- Moving forward with upgrading new IP Infrastructure (Layer 3).
E911 Wireline Network Project (TDM/Ethernet Solution), Iowa Homeland Security:
- West Des Moines and Davenport installations completed.
- Second Datacenter circuit (OC12) in Davenport is installed.
- Two DS3 ILEC (incumbent local exchange carrier) cross connects installed and now T1s need groomed from ILECs.
Iowa Rural Health Telecommunications Program (IRHTP)/Hospital Ciena Upgrades:
- All IRHTP orders being engineered.
- Clinics - 71 installed / 67 pending - COVID-19 slowing down installations.
- 125 new network devices installed on IRHTP Network.
- 72 orders pending with FNS.
Virtualization Server Platform:
- Virtualization upgrade moving forward.
- 165 instances/servers will be supported.
- ConvergeOne completed VMWare installation. In process of evaluating what servers/instances will be migrated.
Data/Network Integrity: To fix or improve upon operational efficiencies within the ICN.
- TDM/ATM Cleanup: About 39 conversions left of 427 circuits. CBXs to follow.
- Remote Network Monitoring - Local and National Peering & Transit/Internet Services.
- Local Peering - support remote users.
- CenturyLink 1Gb peer - 600Mb to 1Gb - About 40% to 50% utilization.
- Mediacom 10Gb peer - 10% to 20% utilization.
- Aureon peer upgrading from 1Gb to 10Gb. Need IPs addresses moved once peer is upgraded.
- National Peering - about 10%, Coresite (Any2) and Equinix (XI) – Connectivity to all the large content providers.
- Network Utilization - About 10 to 15 percent.
- Working on huts and ATM sunset.
Legislative Update – Mark Johnson
The Legislature adjourned the 2020 Session on Sunday, June 14. This year, ICN did have a $2.07M appropriation for Firewall & DDoS Protection. These services have been provided at no cost to some customers for a number of years. This was included in the Governor’s original budget request. The funding comes from the Technology Reinvestment Fund. ICN has up to three years to expend these dollars.
Commissioner Wenzel's appointment to the ITTC was confirmed by the Senate on June 10, 2020.
Chair Kniff McCulla – If it was a General Fund appropriation, how long would ICN have had to spend the funding?
Mark Johnson – Typically with General Fund appropriations funding is for one-year, unless there is specific language. Infrastructure funding typically has a longer timeframe.
ITTC Committee Updates – Staff/Commission
Chair Kniff McCulla presented committee changes.
- Personnel Committee: Chair Kniff McCulla and Commissioner Holz led by Mark Johnson.
- Finance Committee: Commissioner Holz and Commissioner Wenzel led by Mike Cruise.
- Customer and Services Committee: Commissioner Olson and Commissioner Lapointe led by Randy Goddard.
- (New) Operations Committee: Chair Kniff McCulla and Commissioner Wenzel lead by Scott Pappan.
Customer and Services Committee – Randy Goddard
Meeting Date: June 10, 2020.
Attendance: Commissioner Holz, Lori Larsen, Ryan Mulhall, and Randy Goddard.
- ICN’s authorized user requests.
- New customers.
- Requests for waivers from certified users.
- Services – Covered COVID-19 services.
- Sales outreach.
- Customer complaints and outages.
Finance Committee – Mike Cruise
Meeting Date: June 15, 2020.
Attendance: Commissioner Olson, Commissioner Holz, Commissioner Wenzel and Mike Cruise.
- Review of May financials as presented today.
- Commissioner Wenzel requested copies of the balance sheet and cash flow statement.
Personnel Committee – Mark Johnson
- There was no personnel Committee meeting during the month of June.
Commissioner Holz – From legal counsel, regarding MBAEA’s amended waiver application, do we need a written request or confirmation of request, even though it was in a formal session?
Alan Nagel – It is our assumption that it will be part of the written opinion when the counsel issues a written opinion, that there was an oral amendment to the request, and it was not resisted. We assume that it will be included as part of the opinion.
Commissioner Lapointe – Welcomed Commissioner Wenzel.
The ITTC meeting adjourned at 11:55 AM.
Barb Kniff McCulla - Chair, Iowa Telecommunications and Technology Commission