This morning, Salsa Labs experienced an unplanned service outage from 7:00 to 8:47 a.m. Eastern due to a virtual server bug which prevented access to an internally critical system.

Although we were able to begin troubleshooting the problem immediately, resolution required an interlocking sequence of systems fixes. Throughout the downtime period, Salsa pages (both public-facing pages and headquarters login pages for campaign managers) experienced a complete outage. Email blasts scheduled for that time frame were delayed, but were queued and automatically sent as soon as service was restored.

This outage has been fully resolved. Going forward, we're exploring several options for extending our existing layers of redundancy to minimize the prospects of any similar bug impacting service uptime.

We apologize for any disruption to your work. If you are still encountering issues, please contact support@salsalabs.com. Twitter users should be sure to follow @SalsaLabsOps for future real-time incident updates.

Read more

At approximately 12:35 PM EST, we experienced an issue with one of our load balancers that resulted in intermittent connectivity issues for most users.  We quickly realized the issue and brought additional resources online.  Users may have noticed errors and connectivity issues for as much as 30 minutes, though the issue itself was isolated quickly and has been resolved.  All Salsa pages and logins are functioning properly at this time.

Read more

At approximately 9:22am EST this morning, and for a duration of around 58 minutes, we experienced a major interruption of connectivity at our data center. This interruption caused a complete outage of Salsa services during that window. There was no immediate recourse on our end other than to wait for resolution from our primary vendor. Since previous connectivity issues, Salsa has been working to make more data providers available to our systems, however, this morning, the connectivity issue was widespread and affected all providers currently available to us and others in our data center.

Read more (1 comment)