In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint
Can you really say you trust all of these?
Disclaimer: I am a co-founder of StepSecurity.
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
I am surprised nobody here mentionned immutable github actions that are coming [1]. Been waiting for them since the issue was open in 2022. This would have significantly reduce impact and hopefully github will get it over the finish line.
I always fork my actions or at least use a commit hash.
Doing a bit of investigation with github_events in clickhouse, it is quite clear that the accounts used to perform the attack was "2ft2dKo28UazTZ", "mmvojwip" also seems suspicious:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
Actions taken by the threat actor at the time can be seen here:
https://play.clickhouse.com/play?user=play#c2VsZWN0ICogZnJvb...
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
This is hilarious, the maven-lockfile project "Lockfiles for Maven. Pin your dependencies. Build with integrity" appears to have auto-merged a PR for the compromised action commit. So the real renovate bot immediately took the exfiltration commit from the fake renovate bot and started auto-merging it into other projects:
It seems pretty awful that the de-facto way to use GitHub Actions is using git tags which are not immutable. For example to checkout code [1]:
- uses: actions/checkout@v4
Github does advise people to harden their actions by referring to git commit hashes [2] but Github currently only supports SHA-1 as hashing algorithm. Creating collisions with this hashing algo will be more and more affordable and I'm afraid that we will see attacks using the hash collisions during my lifetime.
I wish that they will add support for SHA-256 soon and wrote product feedback regarding it here: https://github.com/orgs/community/discussions/154056
If this resonates with you please go and give it a thumbs up :)
[1]: https://github.com/actions/checkout?tab=readme-ov-file#usage
[2]: https://docs.github.com/en/actions/security-for-github-actio...
The repository is back online, with this explanation from the developer:
> This attack appears to have been conducted from a PAT token linked to @tj-actions-bot account to which "GitHub is not able to determine how this PAT was compromised."
> Account Security Enhancements
> * The password for the tj-actions-bot account has been updated.
> * Authentication has been upgraded to use a passkey for enhanced security.
> * The tj-actions-bot account role has been updated to ensure it has only the minimum necessary permissions.
> * GitHub proactively revoked the compromised Personal Access Token (PAT) and flagged the organization to prevent further exploitation.
https://github.com/tj-actions/changed-files/issues/2464#issu...
> https://github.com/tj-actions/changed-files/pull/2460
This kind of auto dependency bump bots are more trouble than their worth. If your app works today, bumping random deps won’t make it work better in any meaningful sense in 95% of cases. With such a small upside, the downside of introducing larger attack surfaces, subtle breakages (despite semver), major breakages, and in the worst cases, compromises (whether it’s a compromised dep, or fake bot commits that people are trained to ignore) just completely outweighs the upside. You’re on the fast lane to compromises by using this kind of crap.
People should really learn from Go’s minimum version selection strategy.
I've always felt uncomfortable adding other people's actions to my GitHub workflows, and this is exactly the kind of thing I was worried about.
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
I think this article from earlier today was the discoverer who opened the issue
So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets.
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
I wish Github required some sort of immutability for actions by default as most package managers do, either by requiring reusable actions to be specified via commit hash or by preventing the code for a published tag to be changed.
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
Not the first time this particular action has had a vulnerability, either.
I wish Github didn't just purge entire repositories/accounts in the event like this, although I know why they do this. But there's now no way to analyze the repository/exploit anymore
Helpful update: The gist author has deleted the gist, so https://gist.githubusercontent.com/nikitastupin/30e525b776c4... now results in a 404, and stops the action from any further secrets being leaked. This means you're impacted only if you used the action, and had a build triggered in the last 6 hours or so.
For folks looking for a drop-in replacement for v45 (latest major version), we have a patched mirror here: https://github.com/trmlabs/changed-files
1] We took the public mirror from: https://code.forgejo.org/tj-actions/changed-files/src/tag/v4...
2] Undid the malicious code change: https://code.forgejo.org/tj-actions/changed-files/commit/0e5... - You can see the change here: https://github.com/trmlabs/changed-files/commit/8567847ee196...
3] Published under a v1 tag (since we can't vet historical releases and changes and didn't want folks to get confused)
If you want to contribute or report an issue, file a GH Issue or ping us at security@trmlabs.com
I'd used GitHub Actions for at least 6-12 months before even realising <thing>/<thing> was not something that got parsed by the action (like a namespace and method/function), but was simply a reference to a github user name and repo. That whole object should really have been called a 'repo', because that's what it is, and that would alert users to use extreme caution whenever using one that wasn't created by themselves.
Good policy to fork all actions not owned by a reputable organization (e.g. github themselves).
Don’t want to be alarmist but even if not using this action directly, I wonder what implications might be if this has leaked tokens from prominent public-facing project repos which might be used by several folks? I spotted an issue[1] to fix this in Expo EAS CLI and I’m guessing there are many more. The payload I saw from the report only seems to dump things to stdout but I guess analysis is still in progress and IDK if it’s the same payload for all the tags.
Pretty good timing on the attacker.
https://github.com/tj-actions/changed-files/tags?after=v35.9...
Most folks around the world signed off. B-squad probably left cleaning up remaining tasks or just fucking around with co-workers and pondering the weekend. Most GH actions run on a schedule (ie, backups of db, connecting to blob storage services).
Attacker(s) likely to extract plenty of secrets and exfil data before the alarms get triggered (if any) at companies.
The next data dumps are going to be wild.
Am I seeing this correctly, that a (fake/impersonation?) Renovate bot actually proposed the fix... and then other repositories trickled that fix in, also suggested by Renovate or Dependabot, as the dependency updated?
I usually fork (or create my own) actions, as I do not trust the whole chain on GitHub. The marketplace does no enforcement. It is really based on trust you have in the 3rd-party... and I do not have this; as many actions have side-effects, or only operate on a specific runner OS, etc.
We've recently released open-source tools that would have easily prevented this:
1. The maintainers could have used PRevent to immediately alert and block any PR containing malicious code, or easily configured it for detection in case of a direct push: https://github.com/apiiro/PRevent
2. Users could have used our malicious code detection ruleset to immediately detect and block it when scanning updates in all relevant CI/CD stages: https://github.com/apiiro/malicious-code-ruleset
3. For a better understanding of the detection, the malicious code falls precisely into the patterns presented in our research: https://apiiro.com/blog/guard-your-codebase-practical-steps-...
The GitHub repo and org disappeared while I was poking around. Both https://github.com/tj-actions and https://github.com/tj-actions/changed-files return 404.
How does SBOM and such account for this? If you’re a package maintainer, do you need to include CI pipeline plugins, their dependencies, going down as far as the pipeline host, in your security-relevant dependencies? Hard problems :/
I’m Varun, CEO & Co-Founder of StepSecurity. StepSecurity detected and reported the tj-actions/changed-files compromise and has been actively helping the community recover from this incident.
To support you in understanding what happened and recovering swiftly, we’re hosting an Office Hour:
Date: March 17, 2025 Time: 10:00 AM Pacific Time (PT) Add to your calendar: https://www.addevent.com/event/Tf25207322
A company I worked at went all in on GH too. The internal gh team probably going to do a fire drill this whole weekend and app teams forced to rotate all secrets and credentials.
Fortunately don’t have to deal with that shit anymore
To me the only solution is that we need a security in depth approach:
- Create a trusted packages program, and mark trusted packages with a prominent badge. Package authors can apply to join the program, which will involve a review of their package and any subsequent updates. Ensure trusted packages can only depend on other trusted packages.
- Implement a capabilities model for package managers. I hear Deno is better in that respect.
- Have the package manager back-end use AI to continually review the packages. If anything suspicious is found, flag it and investigate manually.
- Require all packages to be name-spaced
Due to the ongoing security incident involving the tj-actions/changed-files Action, we at StepSecurity have provided a secure, drop-in replacement: step-security/changed-files.
We strongly advise replacing all instances of tj-actions/changed-files in your workflows with our secure alternative: https://github.com/step-security/changed-files
I've said this before, but in my mind the central problem in supply chain issues is this. Choose one:
1. You fix what version you're using to a fixed, immutable package. You receive no updates, no bug fixes, no security patches.
2. You follow a pointer to something like a API-compatible version, "latest" (#yolo) or ^5.0.0. You get bug fixes, security patches, but someone can push malicious updates.
Security types, IME, invariably want both: fix that package to a hash, so that we can't have a take over attack. But also we need to stay on top of updates, because we don't want to find out we have a decades old struct4j CVE buried in our codebase just waiting to be exploited.
So to accomplish "both", then we get into schemes like "fix the hashes … but we'll have a bot¹ update our dependency tree automatically". So like, #2, with more steps. Is anyone actually vetting that that update hash isn't going to compromise stuff? Hell no, no company is hiring that level of engineers; I'm lucky to have decent staffing for our primary concerns, reading the code in the dependency tree is out of the question.
And I'm sure in the coming days, security minded people will stampede in the general direction of #1. Stuff'll get fixed to hash, and stuff'll stop getting security patches.
IDK what the answer is, these seem pretty like fundamentally opposed forces of nature. The staffing problems aren't a technical problem, that's a capitalism problem, mostly in that there is very little to no penalty for a breach, so why would anyone hire the eng required to ensure the software works. There was hardly regulation in 2024, and any fines I did see regulatory bodies award are pittances, without fail. And, what regulation there was is now being actively dismantled.
There is some discussion of signed packages in this thread, and that's a helpful idea, I think, though I don't think it completely eliminates the problem: if the signing key is compromised, we're back to square one. The lay eng struggles with PKI.
¹While there is a bot of such nature (the renovate bot) somewhat tied up in this particular instance, I wouldn't over-focus on that bot, specifically; renovate, in particular, is not that relevant to the point I'm trying to make.
Wow that's scary, they updated tons of tags to an offending random commit. With the way repositories are included in automation and the fact that this adjusted the tags of older versions (so not requiring an upgrade) this sort of attack can have a huge impact very quickly :(.
Maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
I’ve been saying for a while that there aren’t supply chain problems when the supply chain is the problem.
I’m getting to the point where I feel that library use at all should be frowned upon, unless it is your own library, with obvious exceptions for the most widely used things like encryption and authentication. None of these things are particularly difficult, people just don’t want to do them “oh noes my velocity”
Another reason why you should be getting software via distro, with searate maintainers taking care of it there rather than directly from the developers that can inject malware via the very next version you mindlessly pull in without checking.
Also due to here being usually more than one distro, more people will look at the code & can spot the usptream getting rogue or getting compromised.
What Happened? • The compromised Action executes a Python script that dumps CI/CD secrets from the Runner Worker process. • Multiple v35 tags were modified four hours ago, indicating a recent supply chain attack. • The malicious behavior can be observed in StepSecurity Harden-Runner insights, showing the Action downloading and executing an unauthorized script.
This GitHub Action is still compromised, leaving thousands of repositories vulnerable.
Is there anyone here from GitHub that can help get this fixed?
https://gist.github.com/gmatuz/7186f583df5f28196cc1f402af3bf...
This gist is pretty much the exact code, from the base64 encoded stuff. Looks like who ever put this in at least neafed the shell script.
A list of projects claimed to be using it from the GitHub page:
https://github.com/tj-actions/changed-files?tab=readme-ov-fi...
Does anyone know if https://github.com/tj-actions/verify-changed-files/ was compromised as well?
As the repo is was taken down is someone able to tell me when was the malicious commit pushed. Trying to get a timeline to see if any workflows using this action were trigger in that timeframe. Thank you
Noob question: is it possible to version lock these things? Could one "vendor" these tools into a fork and use that in the pipeline? Maybe it's one of those possible but crazy endeavour?
Paid Github organizations have a policy to block third-party actions. Would be nice if there was a way to allow third-party actions as long as they are referenced by hash, not version.
Well timed for the weekend. Some teams may notice Monday morning.
Does anyone know if the secrets compromised were sent out, or just printed to stdout? We don't have any public repos using this action.
GitHub’s incident response to this took 17 hours give or take.
Actions is a paid service but Microsoft probably replaced all the security teams with AI.
How does this siphon the secrets away? It looks like it just dumps them out to stdout and stops there.
Anybody have a snapshot of the good one, or maybe a drop in replacement? The repo is gone now.
> await exec.getExecOutput('bash', ['-c', `echo "aWYgW1sgIiRPU1RZUEUiID09ICJsaW51eC1nbnUiIF1dOyB0aGVuCiAgQjY0X0JMT
This malicious code isn't hard to recognise... Surely someone can run an LLM over all code in GitHub and just ask it 'does this code looks like it's blatantly trying to hide some malicious functionality'?
Then review the output and you'll probably discover far more cases of this sort of thing.
What is the current state? Tags reverted or still poisoned?
Was the entire repo just deleted from GitHub?
I'm sorry I must be missing something.
The examples from the repo itself aren't helping to explain.
Why would anyone use this whole convoluted nodejs thing when `git diff-tree` exists?
I'm struggling to see a scenario where this isn't just part of some deliberately over complicated rube Goldberg setup.
Honestly now most people doing "modern software engineering" are retards with no concept of real software engineering concepts.
Shitload of cardboard cto are pushing for "modern practice" to use whatever new version of whatever random dependency downloaded straight from internet.
Some persons asked why I don't like Ruff or UV for Python for example? You start a new job, first thing you have to do after installing a serious and safe Linux distribution like debian:
Curl whateverwebsite.com/ruff/download/latest | bash --blindly-execute --like-an-idiot
-> retrieving-random-dependancy1.tgz
-> executing-random-code...
And I don't speak about the current trend with "pre-commit" where a lot of persons are ok to have automatic downloads and execution on dev machines and ci, at each commit, of hundred of really totally random plugins from random places.
But this is a cto enforced decision to have this pre-commit for software quality...
CI was sold as a solution for engineers having too much sovereignty. So that's what they got.
Just leaving this here...
https://docs.github.com/en/actions/security-for-github-actio...
with all the AI stuff going around, can't github just scan repos for such malicious code?
Shocked Pikachu anyone?
The semgrep URL about this seems to have won the submission lottery: https://news.ycombinator.com/item?id=43368870
[dead]
[dead]
Hi, Renovate author/maintainer here.
The affected repo has now been taken down, so I am writing this partly from memory, but I believe the scenario is:
1. An attacker had write access to the tj-actions/changed-files repo
2. The attacker chose to spoof a Renovate commit, in fact they spoofed the most recent commit in the same repo, which came from Renovate
3. Important: this spoofing of commits wasn't done to "trick" a maintainer into accepting any PR, instead it was just to obfuscate it a little. It was an orphan commit and not on top of main or any other branch
4. As you'd expect, the commit showed up as Unverified, although if we're being realistic, most people don't look at that or enforce signed commits only (the real bot signs its commits)
5. Kind of unrelated, but the "real" Renovate Bot - just like Dependabot presumably - then started proposing PRs to update the action, like it does any other outdated dependency
6. Some people had automerging of such updates enabled, but this is not Renovate's default behavior. Even without automerging, an action like this might be able to achieve its aim only with a PR, if it's run as part of PR builds
7. This incident has reminded that many people mistakenly assume that git tags are immutable, especially if they are in semver format. Although it's rare for such tags to be changed, they are not immutable by design