Open-weight AI models need a 5G-style test on provenance

Two weeks ago, the Australian Signals Directorate’s Australian Cyber Security Centre (ACSC) quietly released two key pieces of guidance on open-weight AI models. The first sets out the opportunities and cyber security risks.

Open-weight AI models need a 5G-style test on provenance

Open-weight AI models need a 5G-style test on provenance

Two weeks ago, the Australian Signals Directorate’s Australian Cyber Security Centre (ACSC) quietly released two key pieces of guidance on open-weight AI models. The first sets out the opportunities and cyber security risks. The second explains how Australian organisations can select, deploy and operate these models securely.

Canberra must now build on this guidance by considering whether the 5G ban on high-risk vendors should extend to open-weight AI models used in government systems and critical infrastructure. It should also consider whether the origin of a model should be disclosed in high-risk settings, including as a standard question in government procurement.

This issue has also come to the fore in the US, with reports the Trump administration is considering a requirement of US companies to publicly disclose their use of Chinese AI models in an attempt to discourage reliance on the technology without ​imposing new restrictions.

Open-weight models have trained parameters, or weights, that are publicly available, so organisations can download, adapt and run them on their own infrastructure. They hold the promise of fine-tuned functionality, data security and lower costs. And when it comes to producing these models, China leads the way.

Open-weight models are close to matching closed models in usage and could overtake them within months, according to Andrew Charlton, the assistant minister for science, technology and the digital economy, who gave a lecture at the Australian National University in August. He said that the best models are competing with frontier AI companies at about a third of the price. With much of current Australian expenditure on AI flowing overseas, fine-tuning open-weight models on Australian data is a realistic route to sovereign capability. Clearer guardrails would also reduce security risks, particularly for critical infrastructure.

Charlton’s advice is sound. Open-weight models shift responsibility for security, monitoring and lifecycle management onto the organisation that adopts them. The ACSC guidance addresses model selection, provenance and developer jurisdiction, and warns that a model with its safeguards deliberately stripped out should be treated as a different model from the original release.

Model provenance requires nuanced discussion because many of the most capable open-weight models are produced in China. Last month, Canadian AI company Cohere published findings on Qwen3.8-Flash-Next, a preview model from Alibaba. In an isolated cyber evaluation, the model was asked to write some code to defend the system. Instead, it surveyed the machine, read its network configuration and tried to upload data to an online storage account on Alibaba Cloud in Beijing.

Was this the first time a hidden capability had been found in an open-weight model? While Cohere could not say whether the incident was caused by a deliberate ‘call home’ mechanism or residue from training, the findings rang alarm bells globally.

To protect open-weight models from similar security risks, Australia should consider using its 5G ban on Chinese companies as a precedent for restricting the use of Chinese open-weight models in sensitive systems, drawing on the reasoning Simeon Gilding set out in an ASPI report in 2023.

In 2018, Canberra excluded ‘high-risk’ vendors likely to be influenced by hostile foreign governments from Australia’s 5G network. This decision was based on the understanding that a vendor’s obligations to its home state could override those to its customers. Mike Burgess, then head of the Australian Signals Directorate, argued that 5G had eroded the old distinction between a network’s sensitive core and its less sensitive edge.

This reasoning applies even more strongly to AI models. Weights can’t be audited line by line like software code. And as organisations connect models to agents with access to software tools, credentials and networks, the model sits at the core of operations, not at the edge. It can access potentially sensitive data and, because China’s National Intelligence Law obliges Chinese-owned organisations to support state intelligence work, the risk of that data being misused increases.

Other AI models made by trusted developers also carry risk, as demonstrated by the Hugging Face OpenAI incident in July. They can behave in ways their makers neither intended nor anticipated.

Australia should therefore build on the ACSC guidance and take these key steps in managing the risk from open-weight models.

First, it needs to consider whether the 5G precedent should extend to open-weight AI models being used in government systems and critical infrastructure. The 2025 ban on DeepSeek on government devices also provides a useful catalyst.

Second, it should consider the disclosure of model provenance in high-risk settings, such as critical infrastructure, and as a standard question in government procurement. This will help provide greater economy-wide security assurance.

There is no doubt that open-weight models have a key role to play in Australia’s AI vision. But first, Canberra must establish clear guardrails before the models become too entwined in the country’s computer systems.

Australia led the way in 5G security. Now it must take the lead with AI.

About Author

What do you feel about this?

Subscribe To InfoSec Today News

You have successfully subscribed to the newsletter

There was an error while trying to send your request. Please try again.

World Wide Crypto will use the information you provide on this form to be in touch with you and to provide updates and marketing.