A hack nearly gained access to millions of computers. This is what we should learn from this.

One of the most fascinating and frightening incidents in the history of computer security began in 2022 with a few emails to the mailing list of a small, one-person open source project.

A user had submitted a complex piece of code that was now waiting for the maintainer to review it. But a different user by the name of Jigar Kumar thought that wasn’t happening fast enough. Patches spend years on this mailing list, he complained. Version 5.2.0 was 7 years ago. There is no reason to think that anything will come soon…

A month later, he followed up – over 1 month and no closer to being merged. It’s not a surprise. [sic]

And a month later: is there any progress on this? Kumar stayed for about four months complaining about the pace of updates and then was never heard from again.

A few weeks ago, the world learned a shocking twist. Jigar Kumar doesn’t seem to exist at all. There are no records of anyone with that name outside of pushy emails. He, along with other accounts, was apparently part of a campaign to compromise nearly every computer in the world with Linux. (Linux is an open source operating system unlike the closed systems of companies like Apple that run on tens of millions of devices.)

That campaign, experts believe, was likely the work of a well-resourced state actor, one that nearly pulled off an attack that could have made it possible for attackers to remotely access millions of computers by logging in effective as anyone wanted. The security ramifications would have been enormous.

How to (almost) hack everything

Here’s how things unfolded: In 2005, software engineer Lasse Collin wrote a series of tools to better compress files (similar to the process behind a .zip file). He made these tools freely available online, and many larger projects incorporated Collins’ work, eventually becoming XZ Utils.

The Collins tool became a part of the vast open source ecosystem that powers much of the modern Internet. We might think that something as central to modern life as the Internet has a professionally maintained structure, but like an XKCD comic published long before the hack. shows, it’s closer to the truth that the entire modern digital infrastructure is based on a project that some random person in Nebraska has been maintaining since 2003. XZ Utils was one such project and yes , you should find it a little worrying that there are so many of them. .

Starting in 2021, a user named Jia Tan, who also doesn’t seem to exist anywhere else, started making contributions to the XZ project. At first, they were small, harmless fixes. Then Tan started sending out bigger additions.

The way an open source project like this works is that a Collin maintainer, in this case, has to read and approve each submission. Sure enough, Tan was overloading Collin with homework.

That’s when Kumar showed up to complain that Collin was taking too long. Another account that doesn’t seem to exist joined the chorus. They argued that Collin was clearly not up to the task of maintaining his project alone and pushed for you to add Jia Tan as another lead.

It seems likely that they were fakes created to push Lasse to give Jia more control, engineer Russ Cox writes in a detailed timeline of the incident. it has worked Over the next few months, Jia began responding to xz-devel threads with authority on the upcoming 5.4.0 release. He had become a trusted maintainer who could add code to XZ Utils.

Why does any of this matter? Because one of the many, many open source tools that XZ Utils incorporated was OpenSSH, which is used to access computers remotely and is used by millions of servers worldwide.

Tan carefully added to XZ Utils a well-disguised code that compromised OpenSSH, allowing developers to remotely log into any computer running OpenSSH. The files containing the (heavily disguised) code were accepted as part of the larger project.

Fortunately, almost all of the millions of potentially targeted computers were not affected because their new update routine was first published as unstable (ie expected to have some bugs) and most administrators expect a later stable release.

Before this happened, Jia Tans work was stuck. Andres Freund, a software engineer at Microsoft, was off work doing some testing on a computer that had the new unstable version. Under most circumstances the hack ran smoothly, but under the circumstances I was testing it slowed down SSH performance. He dug deep and quickly unraveled the whole scheme.

Which means that thanks to a Microsoft engineer doing some after-hours work, your computer remains safe, at least as far as I know.

Can we do better than being lucky?

There was nothing inevitable about this hack being discovered. Many other people were running the new unstable build without noticing any problems. What made Freund suspicious in the first place was not the suspicious code, but a bug that had been accidentally introduced by Jia Tan.

If Jia Tan’s team had avoided this mistake, they might have succeeded. Capturing the suspect code actually required a lot of matches, Freund later told Mastadon.

No one wants to believe that modern computer security is essentially based on a lot of coincidences. We would prefer to have reliable processes. But I hope this narrative makes clear how difficult it is to reliably defend the jury-rigged Internet we have against an attack like this.

The people behind Jia Tan spent more than two years building the access they needed for this attack. Some of the details have to do with the dynamics of open source software, where decades-old projects are often in a silent maintenance stage that, as we’ve seen, can be taken over by an aggressive actor. But with the same resources and dedication behind Jia Tan, you could also hire a software company to do the same with closed source software.

Above all, it is very difficult to guess whether this attempted attack was unprecedented or unusual simply because it was caught. This means we have no idea if there are other landmines lurking in the bowels of the Internet.

Personally, as someone who doesn’t work in IT security, the biggest thing I took away from this was less a prescription for specific policies and more a sense of awe and appreciation. Our world runs on the unacknowledged contributions of engineers like Collin and Freund, people who spend their free time building things, testing things, and sharing what they build for the benefit of everyone. This is a downside for security, but it’s also great.

Collin could not be reached for comment. (His website said: To the media and journalists: I will not respond for now because I need to understand the situation thoroughly enough first. It is enough to reload this page once every 48 hours to check if (this message has changed.) But I hope he eventually comes around to thinking that being the personal target of this rather extraordinary effort to make his work at XZ utils feel inadequate is, in fact, a remarkable claim to his importance .

A version of this story originally appeared on the Future Perfect newsletter Register here!