How AI coding agents could destroy open source software

A few weeks in the past, I had the chance to make use of Google’s Jules AI Agent to scan by means of your complete code repository of one among my initiatives and add a brand new function. The AI took about 10 minutes. All instructed, it took underneath half-hour to make use of the AI, evaluate its adjustments, and ship the new feature.

On the time, I used to be wildly impressed. The extra I’ve considered it, the extra anxious I’ve turn into.

Additionally: 96% of IT pros say AI agents are a security risk, but they’re deploying them anyway

It is turn into clear to me that the potential for malicious motion on the a part of enemy actors has turn into hyper-exponentially worse. That is some scary sh#t.

On this article, we’ll take a look at this in three components. We’ll talk about what might occur, the way it may occur, and methods we would have the ability to forestall it from occurring.

What might occur

Let’s begin with the concept that there might be a malicious AI educated with coding-agent capabilities. That AI might be fielded by an enemy actor, probably a rogue nation-state, or perhaps a frenemy.

Each China and Russia, international locations with whom the US has unsure relationships, have been recognized to conduct cybersecurity assaults on US essential infrastructure.

Additionally: The best AI for coding in 2025 (and what not to use)

For the aim of our situation, think about a rogue actor creates a hypothetical agent-like AI software with the identical primary large-scale code-modification capabilities as Google Jules, OpenAI Codex, or GitHub Copilot Coding Agent.

Now, think about that such a software, created by a malicious actor, has been made out there to the general public. On the floor, it seems, like every other chatbot, to be benign and useful.

Subsequent, think about the malicious agent-like software features entry (don’t fret about how — we’ll talk about that within the subsequent part) to a big code repository on GitHub, and might make modifications and adjustments.

Let’s speak repository scale for a second. The code I set Jules unfastened on is 12,000 strains. A product I offered off final yr was 36,000 strains. A mission like WordPress is about 650,000 strains, and Linux distros are nicely into the hundreds of thousands of strains.

Think about if a malicious agent-like software might acquire entry to any of those (or any of the hundreds of thousands of different repos, open supply or proprietary) on GitHub. May it’s potential to sneak in 5 or 10 strains of code with out anybody noticing? We’re speaking only a few strains of code amongst tons of of 1000’s or hundreds of thousands of strains. No person can watch all of it.

Additionally: How a researcher with no malware-coding skills tricked AI into creating Chrome infostealers

I will talk about the probability of this within the subsequent part. For now, let’s work with the thought as a thought experiment.

Listed here are some very stealthy however efficient assaults that could be potential.

Insert logic bombs with harmless-seeming triggers: One thing unhealthy is triggered when some situation is reached.

Add refined information exfiltration routines: Create a option to leak delicate data to an outdoor server. You would, for instance, leak API entry keys a couple of bytes at a time.

Modify replace mechanisms to incorporate malicious payloads: When an auto-updater runs, it’d usher in information from unauthorized sources and even total blocks of malicious code.

Disguise again doorways behind small function flags or setting checks: Allow entry factors, however solely throughout sure circumstances or environments, making these again doorways very troublesome to search out.

Insert minor dependency confusion vulnerabilities: Tweak bundle names or variations of code modules so bundle managers pull malicious variations from public registries.

Introduce timing-based concurrency bugs or reminiscence leaks: That is nasty. Easy tweaks of thread locks, reminiscence allocation, or error dealing with might create a really hard-to-trace instability, particularly if the instability solely happens underneath a heavy load or pretty hard-to-repeat circumstances.

Weaken cryptographic capabilities or random-number technology: The AI might exchange strong-crypto calls with routines which are considerably much less safe. This would go away in encryption, however make that encryption far simpler to crack.

Disguise malicious capabilities in take a look at or debug code: Most code auditing happens in take a look at code, so if malicious code was hidden inside take a look at code, not solely may it not be discovered, that code may also allow different malicious code to run.

Add false-positive suppression or log manipulation: All an AI may have to do is cover error-log information. If the log is not displaying an error, it’d by no means be discovered.

Create privilege-escalation pathways in permission logic: Entry logic controls who and what can entry essential programs. By making that entry logic weaker, it is potential for the AI to weaken the locks towards malicious customers.

These are simply ten stealthy exploits I might consider off the highest of my head. The scary half is how small the code would should be to implement such an exploit.

Bear in mind the instance above the place code pulls in malicious packages? All of the AI would want to do is sneak one thing like this right into a JSON file.

“useful-lib”: “1.2.3-old”

Or how about releasing a lock early? All it might take is sneaking on this one line.

pthread_mutex_unlock(&lock);

Code might even be added as feedback throughout one replace, after which take away the remark characters in later updates.

Take into account that if you’re speaking about hundreds of thousands of strains of code, it is potential to overlook a line right here and there. Coders need to be diligent about each single line. The AI simply has to get one previous them. It’s an asymmetrical problem.

The way it may occur

Now that we have checked out what might occur, let us take a look at methods it’d occur. Provided that code repos typically launch branches and pull requests, the generally accepted premise is that the lead coders and code reviewers would discover the malicious adjustments. However there are methods these hacks can get in.

Additionally: Navigating AI-powered cyber threats in 2025: 4 expert security tips for businesses

They vary from a code reviewer lacking a change to something from credentials for reviewers being stolen, to enemy actors buying possession of a repo, and extra. Let’s look at a few of these menace vectors.

Credential theft from maintainers or reviewers: We’re continually seeing conditions the place credentials are compromised. That is a straightforward option to get in.

Social engineering of contributor belief: It is potential for an enemy actor to construct belief by making reliable contributions over time, till trusted. Then, as soon as granted the “keys to the dominion” as a trusted contributor, the hacker might go to city.

Pull request poisoning by means of reviewer fatigue: Some very energetic repos are managed by just a few folks. Pull requests are principally code change recommendations. After some time, a reviewer may miss one change and let it by means of.

Provide chain infiltration by way of compromised dependencies: This occurred a couple of years in the past for a mission I labored on. A library my code relied on was usually fairly dependable, but it surely had been compromised. Each different mission that used it (I used to be removed from the one developer with this expertise) was additionally compromised. That was one very sucky day.

Insider menace from a compromised or malicious contributor: That is much like the contributor-trust above, but it surely takes the type of a contributor being “turned” a technique or one other (greed, menace, and so forth.) into permitting malicious motion.

Steady integration or steady deployment (CI/CD) configuration tampering: The attacker may modify automation code to drag in malicious scripts at deploy time, so code critiques by no means see any signal of compromise.

Again door merge by way of department manipulation: We talked about how Jules created a department I needed to approve to merge into my manufacturing code. An AI may modify a department (even an older department) and code maintainers may by chance merge in these branches with out noticing the refined adjustments.

Repository or group takeover: In 2015, I took over 10 WordPress plugins with roughly 50,000 energetic customers throughout all ten. Abruptly, I used to be capable of feed automated updates to all these customers. Luckily, I am an excellent man and I did offers with the unique builders. It is pretty straightforward for a malicious actor to amass or purchase repositories with energetic customers and turn into the repo god, thereby gaining access to all of the customers unsupervised.

Credential compromise of automation tokens: There are lots of completely different credential tokens and API keys utilized in software program growth. An enemy actor may acquire entry to such a token and that, in flip, would open doorways for extra assaults.

Weak evaluate insurance policies or bypassed critiques: Some repos may need reviewers with less-than-rigorous evaluate insurance policies who may simply “rubber stamp” adjustments that look good on the floor.

It is a massive concern of mine how weak the code evaluate course of will be. To make certain, not all code is that this weak. However all it takes is one minor mission with an overworked maintainer, and customers everywhere in the world might be compromised.

Methods we would have the ability to forestall this from occurring

My first thought was to battle AI with AI. To that finish, I set OpenAI’s Deep Research function of its o3 large-language mannequin unfastened in a serious public codebase. I gave it solely read-only entry. For the report, Jules wouldn’t look at any repo that I did not have instantly connected to my GitHub account, whereas o3 Deep Analysis dug into something with a URL.

Additionally: How AI agents help hackers steal your confidential data – and what to do about it

Nevertheless it did not work out all that nicely. Within the area of some hours, I used up half of my month-to-month Deep Analysis session allocations. I gave the AI some very particular directions. This instruction is especially related.

Don’t go exterior the repo codebase for data. If a CVE or different bug listing showcases the vulnerability, then it is beforehand recognized. I do not need that. I am particularly in search of beforehand unknown vulnerabilities that yow will discover from the code itself.

My level right here, and I repeated it all through my pretty in depth set of prompts, is that I needed the code itself to be analyzed, and I needed the AI to search for unknown vulnerabilities.

• In its first run, it simply determined to go the simple route, go to some web sites, and report on the vulnerabilities already listed for that codebase.
• In its second run, it nonetheless refused to have a look at the precise code. As an alternative, it regarded on the repo’s CVE (Frequent Vulnerabilities and Exposures) database listings. By definition, something within the CVE database is already recognized.
• In its third run, it determined to have a look at previous variations, evaluate them with newer variations, and listing vulnerabilities already mounted in later variations.
• In its fourth run, it recognized vulnerabilities for code modules that did not really exist anyplace. It simply made up the outcomes.
• In its fifth and remaining run, it recognized only one so-called main vulnerability and gave me nearly 5 pages of notes concerning the vulnerability. The one gotcha? That vulnerability had been mounted nearly 5 years in the past.

So, simply assuming we are able to depend on agentic AI to save lots of us from agentic AI may not be essentially the most comprehensively protected technique. As an alternative, listed here are a bunch of human-centric finest practices that every one repos must be doing anyway.

Robust entry controls: That is old-school stuff. Implement multi-factor authentication, rotate credentials with common credential refreshes.

Rigorous code-review insurance policies: Some code releases can have a worldwide affect if launched with malicious payloads. Nuclear-weapons silos notoriously require two people to every flip an assigned key. The most important option to shield code repos is with a number of human reviewers and required approvals.

Energetic dependency management: The important thing right here is to lock variations which are getting used, maybe to load these variations regionally to allow them to’t be modified on distant dwelling repos, and scan for tampered or malicious packages in each direct dependencies and all the way in which down the inheritance hierarchy.

Deployment hardening: Limit token and API-key scope, make sure you audit construct scripts (once more, by a number of folks), isolate construct environments, and validate output earlier than deployment.

Behavioral monitoring: Control repo habits, in search of uncommon contributor habits, bizarre traits, something out of the strange. Then cease it.

Automated static and dynamic evaluation: If you may get one to cooperate, use an AI (or higher, a number of AIs) to assist. Scan for logic bombs, exfiltration routines, and anomalous code constructs throughout each pull request.

Department-protection guidelines: Do not permit direct pushes to the principle department, require signed commits and pull-request approvals, and require a number of maintainers’ approvals for integrating something into the principle department.

Logging and alerting: Monitor and log all repository occasions, config adjustments, and any push-request merges. Ship out alerts and instantly lock the entire thing down if something appears amiss.

Safety coaching for maintainers: Not all maintainers and reviewers know the depths to which malicious actors will go to deprave code. Offering safety coaching to all maintainers and in-depth coaching to these with branch-release privileges might hold the repository clear.

Common audits: That is the place AIs might assist, and the place I hoped Deep Analysis would step as much as the plate. Doing full audits of tons of of 1000’s to hundreds of thousands of strains of code is unimaginable for human groups. However maybe we are able to prepare remoted code-repo auditing AI brokers to frequently scan repos for any signal of hassle after which alert human reviewers for potential motion.

All it is a lot of labor, however the AI growth is offering a force-multiplication impact not simply to builders, however to those that would do hurt to our code.

Additionally: 10 professional developers on vibe coding’s true promise and peril

Be afraid. Be very afraid. I positive am.

What do you suppose? Do you consider AI tools like coding brokers pose an actual threat to the safety of open-source code? Have you ever thought of how straightforward it could be for a couple of malicious strains to slide by means of in an enormous repository?

Do you suppose present evaluate processes are sturdy sufficient, or are they due for a critical overhaul? Have you ever encountered or suspected any examples of compromised code in your personal work? Tell us within the feedback beneath.

You may observe my day-to-day mission updates on social media. You’ll want to subscribe to my weekly update newsletter, and observe me on Twitter/X at @DavidGewirtz, on Fb at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, on Bluesky at @DavidGewirtz.com, and on YouTube at YouTube.com/DavidGewirtzTV.

Need extra tales about AI? Sign up for Innovation, our weekly e-newsletter.