Latest Update:
Sat Jul 19 05:20 2025 UTC:
Once again, We apologize if you were affected by the earlier database connectivity issues affecting a small subset of customer's MIDAS systems hosted out of our East Coast US data center.Full connectivity has since been restored, but we wanted to share insight with you as to what was behind this issue.
On 17h July at 18:18 UTC, our hosting provider informed us that on 18th July they would be performing maintenance and migrating servers to newer hardware in their East Coast US data center. This work was due to commence from 19:00 UTC.
Whenever we receive such maintenance notifications from our hosting provider, we assess their potential impact, and if we believe that they could likely impact on customer's ability to access their MIDAS systems, we post an update on our Service Status page, and in some instances email customers as well.
On this occasion however, our hosting provider informed us that they did "not anticipate any major service interruption" and that "this is expected to be a seamless process".
Following these assurances, we took the decision not to inform customers via our service status page or via email.
At 20:54 UTC we received the first report from a customer hosted in this data center who was seeing a database error when trying to access their MIDAS system.
We promptly began investigating and reached out to our hosting provider in the knowledge that they were performing a hardware migration.
Our hosting provider was able to diagnose the cause of the issue which was related to a specific type of database storage engine, named InnoDB.
As our hosting provider performs "live" migrations to new hardware (a process which is designed to minimize downtime), the issue only became apparent once the migration process was complete.
Our hosting provider comments, "On rare occasions, data inconsistencies can occur when doing live migration. Unfortunately, this led to corruption in the InnoDB tables. We worked to recover the databases as quickly as possible, but the recovery process is time-consuming and requires extra care to minimize any potential data loss.".
When asked why database integrity wasn't tested as part of this migration, our hosting provider comments "Although we routinely test the accessibility and basic functionality of servers post-migration, not all data integrity issues are immediately apparent, especially when it comes to low-level database file corruption. The system did pass initial boot and service checks, but only revealed full issues once the database attempted to process live reads and writes."
To confirm, our hosting provider was able to fully resolve this corruption without any data loss. Regrettably, this process did take a little time to complete due to the quantity and size of databases involved.
For reassurance, we separately take multiple regular database backups ourselves. Therefore, had our hosting provider been unable to resolve this, then we had backups that would have allowed affected customer's databases to be restored to the point in time when this hardware migration occurred. This was not needed however as our hosting provider was able to fully resolve the issue from their end without loss of any data.
If your MIDAS system was affected during this, we apologize and thank you for your patience. Our hosting provider adds that they "completely understand how critical uptime is for your operations, and I regret the disruption this caused you and your customers".
Not all customer's MIDAS systems in this data center were affected, and customers in our other data centers were unaffected. Customers who were affected, but who have the optional Emergency Access addon were also able to retain access to their bookings throughout this incident.
Previous Updates:
Sat Jul 19 00:50 2025 UTC:
We have now verified database integrity for each customer's MIDAS system in our East Coast US data center, and accordingly we have now moved each system out of "Maintenance Mode" and restored full access.We apologize if you were affected by this earlier database connectivity issue. Whilst this has now been resolved, we are still ascertaining further details as to the specific cause of this issue from our hosting provider.
Sat Jul 19 00:25 2025 UTC:
We're seeing improvements in database connectivity, and are currently awaiting confirmation from our hosting provider that this issue has been fully resolved.In the meantime, we've taken the precaution to place customer's MIDAS systems in the data center into "Maintenance Mode" temporarily whilst we verify database integrity. Whilst your MIDAS system is running in this temporary special mode, you'll only have "read only" access to your bookings.
Once we have verified the integrity of each customer's database, we will move each system out of Maintenance Mode accordingly.
Fri Jul 18 23:00 2025 UTC:
Our hosting provider has identified the cause of this issue and is currently working on a fix.Thank you for your continued patience, and we will provide a further update here in due course.
Fri Jul 18 21:45 2025 UTC:
For transparency, our hosting provider informed us yesterday (17th July at 18:13 UTC) that they would be performing a hardware migration in their East Coast US data center commencing at 19:00 UTC today (18th July). However, they stated "We do not anticipate any major service interruption ... There is no action required on your part. This is expected to be a seamless process.".Given the timing of the database connectivity errors we're observing in this data center this evening, it is our understanding that this is connected to this maintenance event undertaken by our hosting provider.
Our hosting provider continues to investigate, and we will bring you further updates on this page shortly.