SmartVault Downtime Debrief
Today, SmartVault customers experienced a disruption in service. From approximately 6:09am to approximately 11:21am central time, the service was unavailable and our customers could not log in to the service or access their data. The service was restored at 11:21am central time to its full capacity and at no time was any customer data lost or compromised.
The service disruption was caused by a corruption in one of our database servers. At the time of the incident, the engineering team began assessing the state of our database replica (this system provides us with immediate online backup of your data). We have several layers of redundancy to protect against these types of incidents – including RAID, database replicas, nightly backups and offsite archival of backups.
I understand the frustration this service disruption caused – especially during prime business hours. However, our first priority was to ensure that ZERO customer data was lost or corrupted. We made the decision to continue to keep the system offline while we assessed the health of the database replica. Once the health of the system was verified using several touch points, we initiated a procedure to bring the system back online.
I apologize for the downtime you experienced earlier today – we understand any disruption in service causes a significant inconvenience to our customers. Here at SmartVault, our primary mission is to deliver an excellent service that is secure and highly available. The entire team personalizes that mission and we feel the weight of falling short. Over the coming weeks, we will be reviewing this incident to determine the root cause and identify improvements we can make to our processes and procedures.
As part of our effort to ensure redundancy is restored on every level of our service, our technical team will bring up an additional database replica tonight, November 19, 2012, from 9pm – 7am CST. The service will be unavailable during that time.
If you have any questions, concerns or comments please feel free to contact me personally via email at firstname.lastname@example.org