Engineers have confirmed that the Managed DNS portal, API and Advanced Services have continued stability. We are now setting this incident to resolved.
(We apologize for any confusion, our service status alerts should have been set to Operational at our last update.)
Posted Feb 21, 2023 - 16:01 UTC
Update
Engineers have confirmed Advanced Services are now back to fully operational status and Managed DNS portal accessibility via the UI and API remain stable. DNS resolution remains unaffected. At this time, our engineers will continue to monitor this issue. Customers with questions or continuing to see issues are encouraged to contact Dyn Product Support.
Posted Feb 17, 2023 - 21:51 UTC
Monitoring
As of 02:02 UTC, engineers have confirmed that the Managed DNS portal is now accessible again via the UI and API. However, customers may still experience some service degradation with Advanced Services. DNS resolution remains unaffected. Customers with questions or continuing to see issues are encouraged to contact Dyn Product Support at this time.
Posted Feb 17, 2023 - 03:35 UTC
Update
Engineers continue work to restore the Managed DNS portal and Advanced Services monitoring. The continued stability of the platform is our ultimate priority while we are work to restore all function as quickly as possible. At this time, we recommend customers make any needed changes via the API. Customers requiring assistance with reseting their API passwords are asked to reach out to Support from their DynID account (which has remained accessible) with their request to expedite assistance.
Customer with questions are encouraged to reach out to the Dyn Product Support team.
Posted Feb 16, 2023 - 22:49 UTC
Update
Engineers are continuing to monitor the stability of the API and change propagation. With continued performance stability, engineers will next work to carefully reintroduce the Managed DNS portal and restore Advanced Services. The continued stability of the platform is our ultimate priority, but we are working to restore all function as quickly as possible.
Customer with questions are encouraged to reach out to the Dyn Product Support team.
(We apologize that the affected services was not updated to report Advanced Services degradation at impact start. This has been now been corrected.)
Posted Feb 16, 2023 - 13:55 UTC
Update
At this time, we are seeing improved performance on our backend processing and continuing to open up API capacity until they reach normal operating levels. We will continue to monitor the stability of connections via API as our first priority. The Managed DNS portal remains unavailable (in maintenance) at this time.
As part of the service protection strategy, Advanced Services are not currently honoring monitoring agent health status to make changes to record eligibility. Customers using Advanced services who require record changes are recommeded to use the API to make endpoint updates directly to your service at this time.
Further updates will be provided here as available.
Posted Feb 15, 2023 - 22:09 UTC
Update
Our engineers have implemented an adaptive mitigation approach using real-time backend optimizations and fine-tuning of DNS Load Management to handle current requests. These actions are being done in parallel as we continue to monitor the progress of clearing the backlog of requests and overall service health. We will provide an update as soon as we have one available.
Posted Feb 15, 2023 - 21:50 UTC
Update
Update: Engineers continue work to carefully restore API function. At this time, the API is available but experiencing significant latency/timeouts related to limitations imposed in correction efforts. The Managed DNS portal is in maintenance mode (unavailable). DNS resolution is not impacted, and changes are propagating with intermittent high latency (> 1 min).
This incident is our highest priority. There is no ETA for restoration at this time but we are focused on incrementing service functional improvements (mitigation) while also working toward the goal of total restoration. We will follow-up as updates are available.
Posted Feb 15, 2023 - 15:24 UTC
Update
We are continuing to work on a fix for this issue.
Posted Feb 15, 2023 - 13:26 UTC
Identified
We are performing additional backend optimizations to help mitigate the issue and clear the backlog of requests to restore overall service health. Further updates will be provided upon mitigation or as events warrant.
Posted Feb 15, 2023 - 05:54 UTC
Update
Engineers continue work to restore the Managed DNS API and Portal service stability.
At this time, restorations efforts are being carried out as a controlled introduction of backlog and live traffic to the service. API availability is improved, however latency and/or timeouts may still be encountered due to temporary limits related to corrective efforts. Similarly, the Managed DNS UI has not yet been restored. Our engineers are actively engaged in monitoring and adjusting workload flows to ensure that we restore service stability as quickly as possible.
We understand the impact your business and have been continuously addressing this incident as a highest priority. We will continue to be "all hands on deck" as we work to restore the service.
Customers with questions or experiencing impact other than listed are encourage to reach out to Dyn Product Support.
Posted Feb 14, 2023 - 23:21 UTC
Monitoring
Our engineers have deployed a software patch and performed a series of backend configuration adjustments to help resolve the issue with Managed DNS. Currently, we are seeing significant signs of recovery to backend processing. We will continue to monitor the progress of this recovery. Further updates will be provided upon mitigation or as events warrant.
Posted Feb 14, 2023 - 19:04 UTC
Update
Additional investigation and troubleshooting steps are in progress towards the mitigation. Our engineers continue their efforts to restore the service.
Posted Feb 14, 2023 - 05:17 UTC
Update
We are continuing to work on a fix for this issue.
Posted Feb 14, 2023 - 01:44 UTC
Update
We are continuing to work on a fix for this issue.
Posted Feb 13, 2023 - 23:49 UTC
Identified
We are continuing to address this incident. At this time, we understand that customers are unable to access their Managed DNS via API or UI. Additionally, there is evidence that changes made shortly before or since the impact start time (17:37 UTC) may be experiencing propagation delays. DNS resolution is not impacted.
We are addressing this incident as a Sev1 and will provide further information as it is available
Posted Feb 13, 2023 - 20:55 UTC
Update
We are continuing to investigate this issue. At this time, customers are unable to access the Managed DNS service via UI (manage.dynect.net) or API (api.dynect.net or api2.dynect.net). We are working on this incident with priority and will provide updates as available.
Customer with questions are encouraged to reach out to Dyn Product Support.
Posted Feb 13, 2023 - 19:40 UTC
Investigating
Starting around 17:37 UTC on February 13th, 2023, Dyn Managed DNS engineers began investigating issues impacting the web portal and DNS records updates. At this time, we are seeing intermittent failures (500 errors) accessing the Managed DNS UI and API for records updates. We will update with further information as it is available.
Posted Feb 13, 2023 - 18:55 UTC
This incident affected: Dyn Managed DNS (API, Portal, Advanced Features).