Nutanix fixed Log4j quickly and confidently with Sourcegraph

As the Technical Director of Solution Engineering at Nutanix, Jon Kohler understands the complexity involved in securing the multitude of applications and solutions required to power such a large organization. “Security is something that we care about intensely here at Nutanix,” Jon said, “because it’s part of our bedrock. It's why customers like us, and we have to take it seriously.”

Log4j: The vulnerability that rocked an industry

In December of 2021, software companies around the world discovered that Log4j, an open-source logging library bundled in many software packages, contained significant vulnerabilities, one of which was a 10/10 on the CVSS scale. The Federal Trade Commission called the library “ubiquitous.”

Jon discovered that the offending module recurred throughout their build. “The more we dug,” Jon explained, “the more we realized this bug was everywhere and nowhere at the same time.”

Nutanix moved quickly, despite having multiple build and artifact management systems, as well as a large monorepo with many component branches and hundreds of git repositories. In under four days, Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.

Nutanix used Sourcegraph to identify every instance of Log4j within 2 days

Speed was of the essence, but the timing of the Log4j news, which broke right before many employees go on vacation for the winter holidays, didn't make things easy.

However, Nutanix armed its engineers with Sourcegraph. Within a couple of days, a few Sourcegraph queries identified every instance of the Log4j vulnerability.

With the Log4j 1.x vulnerability, for instance, codebases were only insecure if they used JMSAppender. Jon used Sourcegraph to see where JMSAppender existed, fixed it, and sent out a release. “That took almost less than five minutes,” Jon said. Sourcegraph released a blog post that explained how other companies addressing Log4j could use code search for similar benefits.

This speed gave the team a head start on mitigation.

Deploying these fixes required quality assurance and testing as well as discovery and fixing.

With the help of Sourcegraph, Nutanix was able to release three back-to-back patches relatively quickly compared to other companies. Nutanix's customers reported satisfaction with both the speed of the patches and their quality.

“We tried to take a measured approach,” Jon said. “We wanted to get things done quickly but without completely flying by the seats of our pants.”

Nutanix has renewed confidence in its vulnerability remediation

Tracking down the Log4j vulnerability was, in Jon's words, like “herding cats who were herding mice at the same time.”

Without Sourcegraph, Jon would have either been tracking down whoever built each component to ask them how and where they used Log4j or stumbling through all of the company's repositories.

Using Sourcegraph, Jon discovered every instance of Log4j and was fully confident in the results. “It's nice,” Jon said, “when you can just run a report and say, 'Here it is,'' or 'Here it isn't.'' It's much better than having to say, 'Well, boss, I think we got it all.’”

Nutanix needed that confidence because of its sprawling codebase. One thing that made Log4j especially complicated for Nutanix—as it does for other large-scale enterprises—is that there were multiple source control systems in play. Sourcegraph provided them with “unified visibility,” according to Jon. “I can't imagine the pain of having to do that either with grep or OpenGrok,” he added.

Confidence spread from Jon to the rest of the team and throughout the company. With search contexts, Jon was able to share relevant contexts and queries, showing the team how they could verify whether a given Log4j instance was present or absent. He could show them precisely what they changed.

“We used Sourcegraph contexts to see specifically where a service was at any given point in time,” Jon said. Without Sourcegraph, the team would've had to use code scanning, which takes a lot of time, or manual build inspections, which aren't foolproof.