16/12/2024 15:13
Aurora is now available for use. Note scheduled tasks are also running, there is backlog which is expected to process and catch up shortly. The View It web portals are about to be restarted as well, so normal service should be resumed across the board soon. We apologise for this significant outage today, we will report to customers once we've completed an analysis in the days ahead.
Once services have proved they are back to specification, we will mark this issue as All Clear.
16/12/2024 15:00
Aurora is now available, however please give it until 15:05 or later to go back in if possible, as it is in the process of scaling up to cope with demand.
16/12/2024 14:44
We are now restarting the services - a further update will confirm if this has been successful
16/12/2024 14:15
Our infrastructure team are still working away, at least halfway through rebuilding the cache before restarting services. We'll update again no later than 14:45.
16/12/2024 13:44
Apologies, we've hit another issue with returning the service, and now need at least 30 minutes to carry out more work before we can retry. We will provide the next update at 14:15.
16/12/2024 13:36
Services are now restarting, this is looking positive for returning to live use by 13:40. We then need to apply a hotfix to scheduled tasks, so these will not be operational for a while longer.
16/12/2024 13:29
We need just a few more minutes before we update again, as we are running through initialisations and tests.
16/12/2024 13:07
The credential updates themselves are slow going, we'd like to provide an estimated recovery time but are suffering from unknowns, as we have not had to follow this recovery process in the production environment before. We will provide the next update at 13:30
16/12/2024 12:42
Apologies for the delay in returning the service, we've hit some credentials issues and are renewing these to proceed. As soon as we are able we will provide more information.
16/12/2024 10:51
Our remedial action was not sufficient, we are taking Aurora offline from 10:50 until approximately midday. We'll next update further once restored and tested around midday.
16/12/2024 10:45
Our remedial action has stabilised the service for 20 minutes, so we will cautiously continue to apply this change across the service. There is a chance we may need to take the service down, so please continue to monitor these messages over the course of today. For now, please carry on working, there may be some performance issues for a time.
16/12/2024 10:12
We are seeing ongoing issues, Aurora will be unavailable for a period as we address this - we'll provide timescales if possible, but for now, please bear with us - our apologies.
16/12/2024 09:58
We have a recurrence and are now suffering some performance issues, we are working to see what is needed to return to full performance. As we know more we will continue to provide updates here.
16/12/2024 09:47
We have restarted services and normal operation has resumed. We will work to identify the cause to avoid this issue, apologies for this outage.
16/12/2024 09:23
Users are experiencing issues accessing the Aurora application this morning. We are aware of the issue and are working to restore access. Further updates will be available here as we assess the issue.
