SmartVault – Incident Update
Yesterday, SmartVault experienced a degradation in performance that caused some customers to experience slower than normal response times when using the SmartVault service. The degradation in performance primarily affected the Connected Desktop and Dashboard, and caused customers using these features to experience very slow response times. Customers could still access files through the SmartVault Drive and the SmartVault Portal, which was not impacted.
I understand how critical SmartVault is to the day-to-day workflow for accounting professionals at this time of year – as you move into tax season, so do we. Our teams are moving into a higher state of readiness and alert. We naturally move from delivering new features and capabilities, to closely monitoring our systems and focusing on reliability. Our top focus over the next 3 months is to ensure SmartVault is available for you and your clients.
I’d like to take this opportunity to summarize what happened with this incident, how we resolved this issue and what we have learned from it.
On January 28, 2015 at approximately 11:45 am CT, our operations team detected an unexpected amount of load on our key systems. A quick sampling of various features did not show an immediate degradation in performance. The team continued to monitor and troubleshoot the load issues. At approximately 12:05pm CT, we started receiving reports from customers using the Connected Desktop, who were seeing unacceptable performance, especially when refreshing clients. A quick sampling of various features pinpointed impact to our customers using Connected Desktop and the Dashboard. Customers using the SmartVault Portal and the SmartVault Drive were not impacted. Ongoing root cause analysis pinpointed the issue to a poorly optimized query and database table.
At 6:15pm CT the team deployed an index to a key table and we saw immediate improvement in performance. A second fix was deployed that provided additional caching of this data, which resulted in greater performance improvements. Continued sampling of performance across these features has demonstrated this issue to now be resolved. Customers should all be experiencing normal response times when using the Connected Desktop and the Dashboard.
Areas for Improvement
At SmartVault, we take the responsibility of providing you a highly available system very seriously. As a company, attention to availability, performance, and reliability is built into our DNA. After debriefing on this incident, we see the following areas for improvement and will be implementing new processes to address each of these items:
Implement logical zoning. The goal of this initiative is to increase scale, performance, and isolation of customers to specific zones to better accommodate growth in our customer base. This will allow us to shift resources and availability based on different needs and usage patterns of our customers.
Further invest in operational expertise. We have made significant investments in operational expertise to ensure our systems perform at their peak. That investment will continue and grow. We are dedicating additional full-time resources to system availability and performance.
Improve customer communication. Initially we did not see the high system load as customer impacting so there was a delay in notifications to internal teams as well as customers of a potential performance issue. We will implement a new notification process and ensure redundancy within our incident-response teams to ensure clear, concise and timely communication.
We understand that an event such as this is widespread and highly impactful. However, we strongly believe that our managed platform provides superior value over individual firms or business managing their own document management system. We have a dedicated team of security experts and operations engineers responsible for managing these IT tasks for you. Individual businesses typically see more outages, more performance problems, and higher costs associated with managing their own IT platform.
As one of your trusted technology partners, I look forward to an ongoing dialogue with you. If you have any further concerns or questions, feel free to contact me directly at email@example.com. I’ll be periodically providing further insights and updates on our progress on these initiatives.