How to Understand and Defend Against SolarWinds-Type Attacks

In late 2020, one of the most devastating cyber attacks of the last decade was discovered: the SolarWinds breach. It was a notable event for two primary reasons. First, the threat actors achieved an unprecedented level and scale of access globally. Second, the attack vector was through what is now likely the most vulnerable entry point into major organizations: the software supply chain.

The SolarWinds hack is decidedly not a unique case. Software supply chain attacks have been increasing over the last few years with the mean time to discovery generally measured in years. Regrettably, the tools available to mitigate this sort of threat have proven to be limited and largely inadequate until now.

In the following post, we provide companies and executive leadership with an in-depth understanding of how attackers are able to exploit the software supply chain in the wild. We outline the five major risks in the software supply chain and provide recommendations about how to defend against attacks.

Part 1: How to Understand Software Supply Chain Attacks

A software supply chain attack generally involves the injection of malicious code (or a backdoor) via upstream packages during the build process itself. It can also occur through the compromise of developer/contributor accounts. In the SolarWinds breach, attackers injected code into a library used by a larger commercial product (SolarWinds Orion) by targeting the build and code signing infrastructure. The SolarWinds breach is the latest in a string of attacks that fundamentally breaks the security model of how software gets created. The path that the attackers took via SolarWinds isn't even the path of least resistance. Many similar avenues are open and available to attackers and require a much less targeted approach.

Build Time Attacks

In a build time attack, bad actors, state sponsored or otherwise, compromise the very infrastructure where software builds occur. Modern software development typically progresses through automation. Developers write software and store the source code in a Version Control System (VCS). In a Continuous Integration/Continuous Deployment (CI/CD) process, software is checked out, built, and pushed to production.

In the case of the SolarWinds breach, attackers injected malicious code during the build phase before the libraries consumed by the compromised product were produced and delivered as updates. The approach is particularly nefarious because the process included digitally signing the resulting libraries, a process designed to provide tamper protection. The build-time code injection completely circumvented protections, allowing attackers to add malicious logic before the tamper protections were added.

Attackers can take many different paths to gain access to the build phase. Malware in open source packages or third party dependencies, a problem we've written about in the past, can provide entry points. All third party packages will generally get a chance to execute code during the installation and/or build processes. Additionally, compromises in developer or administrator credentials can give attackers access to build infrastructure and beyond. In many cases, developers or administrators have unfettered access to critical build infrastructure and the ability to add or modify configurations and dependencies. Beyond these risks, malicious insiders or misconfigurations can be used to access build systems. In these cases, attackers rely on insiders to add malicious code, configurations, or updates to the build process or they may breach the server itself due to configuration problems. Exploits can also extend to Continuous Integration (CI) software which manages building dependencies, a documented problem even in mature, well-established products.

Open Source and 3rd Party Software

Attackers can easily add backdoors and malware directly into third party libraries consumed by a piece of software. There have been many documented cases of malicious code being baked into upstream software over the past few years. Major issues can go undetected for extended periods of time because the ecosystem of open source software and packages has grown exponentially in size. Open source libraries now consist of millions of unique packages with hundreds of thousands of updates occurring every week. The number of dependencies that each new third party library pulls in has also grown. For example, each library in NPM, the largest package repository for Javascript, holds an average 39 direct dependencies. In turn, each of those 39 libraries has an average of 39 dependencies and so on. As such, a single new package can add thousands of external libraries to your dependency tree.

Companies face the daunting challenge of how to effectively monitor and manage open source risk. A number of well-known companies spend tens of millions of dollars per year to perform their own security reviews of open source and third party software. These companies are able to locate some bad actors and packages, but the process is painstaking and far from real time. Most other companies have neither the budget nor the people power for manual review.

There are products on the market that help companies identify risks in open source software. Legacy products typically use a conventional approach centered around collecting and centralizing vulnerability and threat intel data. In many cases, the data is manually collected. The products then apply simple pattern matching to find “known” problems. This paradigm is reminiscent of early efforts at building antivirus products. A set of virus definitions is built, and anything that does not fit squarely within the confines of a hand-built definition is not flagged.

Another issue with conventional products and approaches is that they ignore the rest of the software supply chain. Risk is more than software libraries and vulnerabilities. It is also comprised of tens of thousands of anonymous, independent authors whose behaviors and activities influence everything within their own packages and everything downstream that relies upon them. The reality is that many publicly disclosed vulnerabilities are never discovered because existing products rely heavily on pre-cleaned, curated data. This approach is reactive. As such, malicious packages are frequently in the wild for years before they are discovered and taken down. At that point, untold damage has been done.

Part 2 - How to Defend Against Attacks

Five Major Risks

Companies need to take a multidimensional view of risk to effectively guard against attackers in the software supply chain.

1. Malware.

(e.g. SolarWinds, 2020; Magecart, 2018 - present) Malicious packages can compromise developers or critical build infrastructure. Malware provides vectors that attackers can exploit.

2. Backdoors

(e.g. NPM getcookies, 2018) Backdoors may be accidental or intentional. For example, a developer may leave a "hook" intended to enable debugging. In other cases, backdoors may deliberately weaken the use of cryptographic protocols. Backdoors can provide attackers with access that can be used at a point later in time.

3. Vulnerabilities

(e.g. Equifax, 2017; HeartBleed, 2014) Vulnerabilities can lead directly to data breaches. Therefore, database curation is critically important. In many cases, attackers will leverage unpatched problems in dependent packages to gain a foothold in secure systems.

4. Engineering Risk

(e.g. Left-Pad, 2016; Chef DHS Incident, 2019) Code quality and technical debt contribute to project risk. There are also problems endemic to package development itself such as insecure defaults, poor test coverage, lack of adequate maintainers, and checked-in authentication keys. These issues can lead to vulnerabilities over time, if not direct repository or build system compromise. In addition, engineering risk can lead to misconfigurations that will become severe vulnerabilities in production systems.

5. Author Risk

(e.g. Event-Stream, 2018) There have been a number of incidents where authors themselves have either sabotaged their own projects or transferred control of a major package to someone else who is not trustworthy. Author accounts themselves can become compromised. Furthermore, authors may be located in foreign countries with hostile relationships to the U.S.

Take Action

Companies must leverage automation and modern analysis techniques to defend against each domain of risk.

1. Malware Mitigation

As we think about building a "defense-in-depth" driven approach, we can target each stage where problems can manifest. At build time, ensure that the constituent parts of the product being built do not contain malicious behavior. At runtime, a real-time endpoint detection product can also help to protect against compromises. One side effect of leaning on runtime solutions to defend against risk, however, is that mitigation will likely result in significant downtime. It may already be too late if the threat is not identified prior to application deployment.

2. Backdoor Detection

At build time, enable a product that can identify problem areas in a software package. Runtime protection can take many forms. Good network traffic monitoring and web application firewalls (WAFs) can help to provide some level of runtime management. Additional safeguards are needed because backdoors can appear in a variety of ways.

3. Identifying Vulnerabilities

There are software supply chain security offerings and many simple Package Component Analysis (PCA) products that can help identify vulnerabilities. The level of fidelity that PCA offerings supply will vary by vendor. Some products are limited because they rely almost exclusively on public vulnerability feeds. Find tools that use Natural Language Processing (NLP) to identify published vulnerabilities not registered in conventional databases.

4. Reasoning About Engineering Risk

Assess package viability through analysis of test coverage, level of maintenance, and best practices that cover the full spectrum of issues in this domain.

5. Understanding Author Risk

Seek data that goes beyond a simple view of authors and how packages are maintained. For stronger defense, find solutions that perform detailed social network analysis on author personas to better understand how committers are connected and interrelated. In addition, look for author analysis that uses public and proprietary sources to understand country of origin, employer, and security posture.

We are here to talk more about your specific situation. Contact us.