Justin's "office space" at the data center.

Do not pull this cord.

Late last night, we completed some significant configuration changes to our network which were the culmination of months of efforts in creating a bigger, faster, stronger Salsa. Some of this is pretty technical, but we wanted to be open and transparent about the processes in place to provide the highest quality service to our clients.

After we experienced the major fiber cut-related outage back in May, we mapped out a course of action to help mitigate network-related outages by either provider outage (i.e. fiber cut or other service degradation). This plan itself came in several phases.

Phases 1 and 2 included selecting, provisioning, and configuring a backup connection. That was completed within a few days of the May outage. We wanted to limit connectivity loss in the event of a power failure or a catastrophic hardware or software failure to our primary router, while the backup router was being configured. This last process took a little bit of time, but is now complete.

After all of these changes, we recently starting seeing a combination of hardware and configuration problems which resulted in the rolling slowdowns from late Tuesday, November 23 to Wednesday, November 24, and then again from Tuesday, November 30 to Thursday, December 2. We tackled these issues by simultaneously working to correct the configuration and hardware issues, as well as contracting with another hosting provider to provision a new set of redundant managed connections, which was completed last night and is now live.

Additionally, we've accelerated our push to cloud-enable significant parts of Salsa. This enablement is key to our next-generation of availability and performance.  This is important for two reasons.  One, it allows us to move off of relying on a limited set of resources and infrastructure.  Two, it allows us to geo-locate parts of our infrastructure closer to your actual supporters and users.  This means more data and information at the edge of the network, which will increase not just availability, but also performance.

Feel free to contact Justin Nemmbers at jnemmers@salsalabs.com with any questions. As always you can follow our support feed @salsalabsops on Twitter for service updates. Thank you!