Yesterday, our authentication upstream provider, Auth0 had a major outage that caused performance issues on the ProudCity Platform for sites with a large number of administrator users logged in and editing the site at once. Late last night, their CEO sent an initial explanation:
Earlier today, Auth0 experienced an incident that resulted initially in degraded service, and then a partial outage. From 16:30 UTC until 18:35 UTC, 18.88% of all requests failed in one of the US environments where your accounts are hosted. Due to latency increases and timeouts, some customers would have experienced this as a full outage.
I understand how important Auth0’s reliability is to your business and to your customers and I want to apologize for the impact this incident has had on your operations.
To remedy the issue, your accounts were moved to an alternate US environment which was not impacted by the incident and continues to operate as expected. This change does not impact your Auth0 implementations and requires no action to be taken by you. Your users will not experience any change to their service.
Our team is working on a full analysis of the incident and towards restoring all our environments in the US region. Again, these changes will be invisible to you, and your users, and require no action to be taken.
We will, of course, be providing an in-depth Root Cause Analysis as soon as possible to provide you with full visibility of the causes of the incident. This will also include details of measures we are taking to prevent incidents like this to occur again. In the meantime, we have published a preliminary-RCA on our status page with a summary of what we have identified to this point.
Status Page – https://status.auth0.com/
incidents/rjhcwj1d2r61 Once again, I am deeply sorry for the inconvenience this issue has caused you, your users and your customers. We will continue to work diligently to serve you better and earn your trust in Auth0 and the service we provide.
Our support team is here for you, please reach out at support.auth0.com if you have any questions.
Eugenio Pace
CEO, Auth0
Today, we are experiencing similar issues. To ensure platform stability, we have taken the temporary step of disabling administration login on all sites. We expect the issue with Auth0 to be fixed promptly and we will re-enable login as soon as we can.
Thank you for your continued patience,
The ProudCity Team
Update Thursday, November 28 2018 at 6:30pm Pacific Time:
Our upstream authentication provider Auth0 had sporadic network timeout issues on Wednesday November 28 2018 and Thursday November 28 1018. These issues caused temporary ProudCity Platform-wide uptime issues, especially when many administrators were logged in to the site at one time. Because of this, we took the precautionary measure earlier today, November 28, of temporarily disabling administrator login to websites.
The issues were resolved by 4pm Pacific Time on November 28, and administrator access was restored. We are continuing to work with Auth0 to ensure that the issues are permanently resolved and will continue to diligently monitor all ProudCity Platform websites over the next couple days.
For full details, and the latest updates, please continue to monitor https://proudcity.com/ongoing-issues-with-upstream-provider-auth0-administration-login-temporarily-disabled.
ProudCity is a digital government platform that makes it easy and cost-effective to launch and manage all aspects of digital government operations, including websites, meetings, online forms and payments.