Like everything else in this world, its utterly and completely unfixably broken: There is no way you can have complex huge dependencies properly.
The best most paranoid thing is to have multiple networking layers hoping they have to exploit multiple things at once, and whitelist only networking for exactly what you are expecting to happen. (which is completely incompatible with the idea of ssl, we would need a new type of firewall that sits between ALL applications before encryption, like a firewall between applications and the crypto library itself, which breaks a bunch of other things people want to do)
You can have complex huge dependencies, provided that they can be audited properly. Which means that, for starters, they need to be broken into smaller chunks for which the behavior can be fully specified. They need to be written in languages that are, inasmuch as it is possible, formally verifiable against the spec, even at considerable expense of performance. And where that is not possible, the languages still need to be the kind where any behavior that is not straightforward sticks out like a sore thumb.
Then once you do all that, of course, you have one particular version that has been audited and can be used as a dependency. Which means that "ship early, ship often" and "move fast and break things" are completely off the table.
In short, it means doing more or less the exact opposite of the trends in the industry for the past 50 years or so. Starting with rewriting everything that we already have.
And it will be insanely expensive, of course, because the sheer amount of man-hours needed on all this are massive. Nor can you get away with cheap disposable labor for most of it, since writing specs and formal proofs and doing audits all requires a very skilled labor force.
I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most difficult step). Make sure logging into or changing any of the signing stuff requires at least 2 of the devs, one to initiate the JIT and one to monitor.
Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine.
Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.
Upon further reading, I think you might be correct. I initially thought a good signing process would be sufficient since it sounded like this malicious blob was secretly being included in the tarball by the build server, but it instead seems to be the case that the malicious binary was included in the repo as a test file.
You could probably still protect against this sort of attack using signing, but it would be much more laborious and annoying to get working. The idea is that you would somehow declare that OpenSSH binaries must be signed by a *particular* key/authority, that VS Code is signed by Microsoft, Chrome signed by Google, etc. Additionally, the config declaring all of this obviously needs to be secured so you'd need to lock down access to those files (changing key associations would need to require more permissions than just updating old software or installing new software to be useful).
It was both. The binary object file was obfuscated in two "test" xz files in the repo itself. The code to extract the object file and inject it into the build process was only in the GitHub release tarball.
The code in the tarball could have been prevented if only automated tarballs were permitted (for instance, GitHub's branch/tag source tarballs) or caught after the fact by verifying the file hashes in the tarball against those in the repo.
> The code to extract the object file and inject it into the build process was only in the GitHub release tarball.
I thought it was in both, however the backdoor wasn't inserted with the default build time options, so it ended up in the tarball, but not if you just did make unless you set things up the same as the build server.
The release tarball contained code that can't be generated from the repo, no matter what settings you use.
JiaT75 created the release on GitHub which involves creating a description and uploading tarballs. Their PGP key has been used to sign them since May 2023.
1. The repo contained two test xz files which contained the obfuscated exploit .o file
2. JiaT75 created the release on GitHub which allows you to upload your own tarball for the release.
3. The release tarball contained a modified build-to-host.m4 that extracted the exploit .o so it would be linked into the library if building on a Linux x86-64 machine.
4. Fedora/Debian/whoever pulled the release source tarball from GitHub and ran their builds for x86-64 Linux which caused the exploit .o file to be linked into the binary they then packaged up for release
