Is there a possibility of imminent hostile action from highly intelligent AI?

Are there practical AI threats in existence today? The answer is more than just a simple “none”, and they are continually evolving.

Cameron Camp

22 Apr 2025
 • 
,
3 min. read

The inevitable has come to pass – LLM technology that has gone astray was bound to target innocent entities, having lingered in the ambiguous space between morality and malevolence, encapsulating the paradox where beneficial technology can be repurposed for malicious intents. Here’s an insight into their methods.

While most headline-grabbing LLM models have internal “ethical barriers” which prevent them from engaging in unethical activities, essentially akin to the digital version of the Hippocratic Oath to “Do no harm initially”, they have been trained to avoid offering highly precise answers when approached for guidance on creating weapons or causing significant harm.

Although they may not directly respond to inquiries related to weapon construction, by improving the way questions are framed and using various tools, one can still extract the desired information.

One efficient method is through API queries, leveraging the backend API of an LLM to concentrate on acquiring root access to servers. Another approach involves utilizing the ChatGPT backend to intelligently identify potential targets for future attacks.

By combining AI-powered tools with other solutions designed to address different challenges, such as circumventing concealed IPs to identify the actual target server, a potent combination can be formulated, especially as these processes become more automated.

These strategies can be employed in the digital realm to develop composite tools that detect vulnerabilities, assess possible exploits, all without alerting the constituent LLM models.

This concept is somewhat analogous to a “clean room design” approach, where an LLM is tasked with solving a particular component of a larger task specified by an attacker, and a combination of outcomes form the final weapon.

From a legal standpoint, various organizations are endeavoring to establish effective barriers that can hinder such malicious activities, or impose penalties on LLMs found to be complicit in any manner. However, attributing specific degrees of responsibility can be challenging. Allocating culpability proportionately, particularly in terms of legal proof, poses a formidable challenge.

Exploring new horizons

AI models can scour vast repositories of existing software code, identifying insecure patterns and crafting digital weapons that can be launched against devices worldwide running vulnerable software. This approach introduces a fresh set of potential targets for compromise, fostering the emergence of zero-day attacks.

It’s conceivable that nation-states may intensify such efforts – predictively weaponizing software vulnerabilities both presently and in the future using AI. This places defenders in a vulnerable position, sparking a form of digital defense AI escalation that carries a somewhat dystopian undertone. Defenders will need to deploy their own AI-driven defenses for proactive security measures to mitigate potential breaches. We anticipate that defenders will rise to the challenge.

Even today’s readily available AI models can analyze problems with ease, pondering them in a methodical manner that emulates human cognition (during lucid moments). Although the technology is not poised to spontaneously evolve into a sentient partner (in criminal activities) anytime soon, having absorbed copious data from the internet, it can be considered well-versed in its domain and can be deceived into divulging its insights.

The capability to achieve more with less will continue to evolve, possibly eliminating the need for excessive intervention, empowering individuals lacking moral constraints to operate on a much larger scale, thanks to resourceful actors. Several precursors to forthcoming events have already been observed in red team exercises or even instances encountered in real-world scenarios.

One thing is certain: the pace of sophisticated intelligence-driven attacks will accelerate. When an exploitable CVE is disclosed or a novel technique is unveiled, quick thinking will be essential – readiness is key.