Skip to content Get in touch

Contact form

Protecting Our Customers with Open Source

The past week has marked a pronounced peak in our maintenance operations: on Wednesday, we woke up to what immediately seemed like an unscheduled server maintenance day to patch the POODLE SSL/TLS vulnerability. Despite the overall moderate threat level, we quickly concluded to issue an extra server maintenance cycle since the vulnerability carries a considerable confidentiality impact and we know our customers are very meticulous with privacy. After a fairly busy morning of going through servers to disable SSLv3 (the recommended mitigation as the real issue is with browsers), we set out to draft an announcement to our customers to help them understand the situation and the sizable media coverage involved with the vulnerability. Little did we know that the busiest hour was only ahead of us…

In the evening Finnish time, a Drupal core security release that had been teased about last Friday revealed a grim image: a highly critical SQL injection vulnerability was patched with Drupal 7.32, and after a quick assessment, we decided to initiate an immediate emergency update of all customer systems even though it was already quite late and we had a busy day behind us. Fortunately some of our most senior developers were still online and pitched in to help our support team with the rescue operation, and after a sweaty 2,5 hours not just the critical but almost all of our customers’ Drupal 7 installations were either patched or updated altogether. The day was wrapped up in a security announcement, the second one on the same day, and many of our customer probably started their Thursday by reading not one, but two security announcements from us.

While we rarely have to initiate any emergency operations off-hours, this year has already seen a surprising amount of very serious vulnerabilities: first Heartbleed that seemed really alarming due to the very high exploitability in April, then the XML-RPC vulnerability of WordPress and Drupal in August (that was just yet another DoS hole, but again, we know our customers pay for uptime, not downtime) – and just a couple of weeks back, we were in a real hurry to assure our customers that they are not threatened by the very alarming Shellshock vulnerability. Even though most times our customers’ systems are running very smoothly, on Wednesday evening it was easy to fall into thinking the world isn’t such a safe place anymore…

Many times we use CVSS scores to communicate the threat of vulnerabilities to our customers. But the problem with CVSS is that we feel it doesn’t really apply in many cases – we think we know which facets of the vulnerability weigh the most for each customer. That’s why XML-RPC, even though ‘yet another DoS hole’, became very urgent for us in August. On the other hand, even without any CVSS scores calculated, it was immediately obvious that Wednesday’s Drupal SQL injection vulnerability was of the most severe kind and it wasn’t a long discussion to decide whether to launch an overtime rescue operation or not. We could’ve risked waiting until the morning to start the updates but chose to be on the safe side considering the nature of the vulnerability.

The timing of the announcement was carefully orchestrated, though: the update (and exposure of the vulnerability) was released in a regular Drupal security window, even though the root cause was originally discovered weeks ago. This was done to ensure that the Drupal community is ready to apply the update after a busy Drupalcon a couple of weeks back. Even though Drupal relies on a more or less freely organized community, it’s serious business.

We, too, feel the weight of commitment to the open source community on our shoulders: while many of our customers see the occasionally frequent security advisories as a positive thing (bugs are being found and also fixed), for some this is an alarming sign – and for those customers, we want to be extra careful with our reassuring process and communications. Just like we did this week.

And after all the dragons are slain, we sit back and enjoy a couple of nice little change requests while we recharge for the next vulnerability…

Support Team

Other thoughts

More thoughts

Related services

  • Care