AFFECTED SYSTEMS: hq_dia hq_dia_backup
DIA Issues: 10:00 AM 4/7: All circuits online
All circuits are online and functioning. We continue to monitor the circuits for issues.
DIA Issues: 05:30 AM 4/6: Windstream circuit back online
Service was restored to the Windstream circuit. Traffic begin automatically going between the two circuits.
DIA Issues: 1:42 AM : Update : New Circuit Operational
New circuit has been enabled, routing adjusted, and traffic is flowing for customers.
The affected upstream provider has isolated the issue with a handoff between their network and TVA, and continue to work to resolve. **HOWEVER we have fully mitigated this issue by activating the new circuit **
DIA Issues: 11:42 PM : Update : New circuit activation
We have been successful in the acceleration of an additional circuit activation. While customers are still currently passing traffic, congestion on our other circuits has caused issues for latency-dependent services such as gaming. However most customers have been surfing successfully.
We will enable the new circuit at 12:00 am (midnight) to get traffic moved off the backup circuits. This new path is geographically diverse and significantly increases total bandwidth, while also providing another additional primary/backup path for internet traffic. This should prevent any further issues similar to what we experienced today.
When we make the changes to activate this new circuit, services will be interrupted briefly while routes propagate. This should be a brief interruption.
Thank you for your patience.
DIA Issues: 6:42 PM Mitigation in Progress
We have feet on the ground working to accelerate the activation of a new circuit to provide an additional route for traffic. The affected upstream provider is actively working the outage and providing updates as available. Currently their assessment is indicating possibly a fiber cut in addition to hardware issues which affect multiple large customers.
We have preliminary information that could see the new path activated within the hour, although this is not guaranteed. More updates as they become available.
DIA Issues: Investigating and Mitigating
Our upstream provider is continuing to investigate the root cause of the issue. We have re-routed all customers to a backup circuit to restore service. Until repairs are made, some services may be degraded.
DIA Issues: Investigating
Our upstream provider is experiencing an issue that has caused our primary circuit to go offline. We are currently investigating and will provide updates once available.
Last updated: April 1, 2024 at 9:49 AM CST