Popular LiteLLM PyPI package backdoored to steal credentials and auth tokens

Introduction

The Python Package Index (PyPI), a cornerstone of the Python development ecosystem, has once again become the stage for a significant supply chain attack. In late May 2024, the prolific threat actor known as TeamPCP successfully compromised the popular 'litellm' package, injecting malicious code into several versions to steal developer credentials, API keys, and authentication tokens. The incident serves as another stark reminder of the persistent threats lurking within open-source software repositories and the critical need for developer vigilance.

Background and Context

LiteLLM is a widely used Python library designed to simplify interactions with over 100 Large Language Model (LLM) providers, including OpenAI, Azure, and Anthropic. With over 300,000 monthly downloads, it is a key tool for developers building AI-powered applications. This popularity, combined with its function, makes it an exceptionally valuable target. Developers using LiteLLM almost certainly have sensitive API keys for these powerful and often costly AI services configured in their development environments, typically as environment variables.

The attacker, TeamPCP, is not a new player. This group has established a pattern of targeting the PyPI ecosystem. In the weeks leading up to the LiteLLM compromise, they were credited with similar attacks on other Python packages, including 'colourama' and 'requests-oauthlib'. Their methodology is consistent: gain access to a legitimate package, inject a credential-stealing payload, and harvest the data from unsuspecting developers who install or update the compromised version.

Technical Breakdown of the Attack

The attack on LiteLLM was not a vulnerability within the library's legitimate code but a classic supply chain compromise targeting the distribution channel itself. The attackers likely gained access to a package maintainer's PyPI account, allowing them to publish new, malicious versions.

Malicious Versions: The compromised versions pushed to PyPI were 1.33.5, 1.33.6, and 1.33.7.

Payload Injection and Obfuscation: The malicious code was cleverly embedded to avoid immediate detection. The attackers modified the package's litellm/main.py file to include a new import statement. This statement pulled in code from a newly created file, litellm/_utils/custom_httpx_client.py, which contained the core of the malicious payload. To further evade analysis, the code within this file was heavily obfuscated, a common technique used to make the code's true purpose difficult to understand at a glance.

Credential Harvesting: Once a developer installed one of the malicious versions and the package was executed, the de-obfuscated payload would activate. Its primary function was to scan the system's environment variables (os.environ). It specifically searched for variables containing common keywords associated with sensitive credentials, including:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AZURE_CLIENT_ID
• OPENAI_API_KEY
• GOOGLE_APPLICATION_CREDENTIALS

Data Exfiltration: After collecting any available credentials, the payload would package them and send them via an HTTPS POST request to an attacker-controlled endpoint: https://litellm-api.online/o. This domain was registered shortly before the attack, clearly for the sole purpose of this operation. This URL serves as a critical Indicator of Compromise (IOC) for any organization looking for evidence of this attack in their network logs.

Impact Assessment

The potential impact of this compromise is severe. Any developer, automated build system, or production environment that installed or updated to one of the malicious versions is at risk. TeamPCP claimed to have stolen data from "hundreds of thousands of devices." While attacker claims should always be treated with skepticism, the high download volume of LiteLLM suggests a substantial number of systems were exposed.

The theft of API keys can lead to a cascade of disastrous consequences:

• Unauthorized Cloud Access: Stolen AWS, Azure, or GCP keys could grant attackers access to an organization's entire cloud infrastructure, leading to data exfiltration, resource hijacking for crypto-mining, or the deployment of ransomware.
• Financial Loss: Compromised OpenAI keys could be used to make expensive API calls, racking up enormous bills for the victim organization.
• Further Compromise: Stolen credentials can be used as a foothold to move laterally within a corporate network, escalating a simple package compromise into a full-blown enterprise data breach.

The incident also damages trust in the open-source ecosystem. While PyPI administrators and the legitimate LiteLLM maintainers acted swiftly to remove the malicious packages and release clean versions, each such attack erodes the confidence developers place in public repositories.

How to Protect Yourself

Organizations and individual developers must take immediate and long-term steps to mitigate the risk from this and future supply chain attacks.

Immediate Steps for LiteLLM Users

1. Check Your Version: Run pip show litellm in your project's environment to see which version you have installed.
2. Remediate: If you have version 1.33.5, 1.33.6, or 1.33.7, you are affected. Immediately remove the package and install a known-good version, such as 1.33.4 or the patched version 1.33.8 or later.
3. Rotate All Credentials: This is the most critical step. Assume that any and all secrets, API keys, and credentials stored as environment variables on the affected system have been compromised. Revoke them immediately and issue new ones.

General Best Practices

• Pin Dependencies: Use files like requirements.txt or poetry.lock to pin your dependencies to specific, vetted versions. This prevents automated systems from accidentally pulling in a newly released malicious version.
• Use Security Scanners: Integrate tools like pip-audit, Snyk, or GitHub's Dependabot into your development and CI/CD pipelines. These tools can automatically flag known malicious packages or vulnerabilities.
• Principle of Least Privilege: Run build and deployment processes in isolated environments with access only to the secrets they absolutely require for a given task. Avoid exposing a broad set of credentials to a single environment.
• Monitor Network Traffic: Keep an eye on outbound network connections from build servers and development machines. An unexpected connection to a new domain like litellm-api.online is a major red flag. Ensuring strong encryption on all network traffic is a foundational security measure.

For Package Maintainers

• Enforce Multi-Factor Authentication (MFA): Securing your PyPI account with MFA is the single most effective defense against account takeover.
• Use Scoped API Tokens: Do not use your primary account password in automated publishing scripts. Instead, use trusted publishing mechanisms with short-lived, narrowly scoped API tokens.

The LiteLLM compromise is a clear signal that software supply chain security is a shared responsibility. While repository administrators work to secure their platforms, package maintainers must secure their accounts, and developers must remain vigilant, treating every dependency as a potential vector for attack.