Microsoft, Nvidia warn open-weight AI tweaks degrade safety safeguards
New joint guidance urges enterprises to verify model provenance and test configurations, noting that performance can remain high while safety controls fail.
Key points
- Microsoft and Nvidia released a joint paper identifying 30 risk types across five categories for open-weight AI models.
- The companies warn that modifying weights or configurations can weaken safety protections even if task performance stays intact.
- The framework advises using cryptographic signatures to verify model integrity and treating unverified artifacts as open risks.
Microsoft and Nvidia published a joint research paper last week urging enterprises to rigorously assess the security of open-weight artificial intelligence models throughout their deployment lifecycle. The companies warn that altering model weights or configurations can significantly weaken built-in safeguards, even when the system continues to perform its assigned tasks with high accuracy.
The paper identifies thirty distinct risk types distributed across five primary categories. Microsoft and Nvidia outline a specific process for evaluating, mitigating, and monitoring these threats. Their guidance emphasizes that organizations must verify the provenance of the models they use, test the exact configurations intended for deployment, and reassess risks whenever material changes occur to the system.
Context and Background on AI Risks
According to CSO Online, the companies stated that the openness of model weights alone does not determine the risk profile of an AI system. Instead, risk depends on a combination of factors including the specific model, its use case, deployment environment, and existing safeguards. The framework draws on established risk-management standards from NIST and ISO, serving as the first entry in a planned series on managing risks in open-weight AI deployments.
The cybersecurity category within the framework addresses specific threats such as malicious code generation, unauthorized tool calls, and the extraction of the model’s safety policies. It also covers excessive token generation and behaviors that diverge from the intended system goals. Other categories in the five-part structure include data protection, content safety, human-AI interaction, and bias mitigation.
Who Is Affected by These Findings
Enterprise security teams face particular challenges because open-weight models can be fine-tuned, merged, modified, or repackaged for various uses. Microsoft and Nvidia note that these modifications can degrade safety protections while leaving task performance unaffected. This creates a scenario where a model delivers accurate results on routine tasks but responds differently to unsafe requests.
The companies advise that safety and capability must be assessed separately. They recommend testing the specific model configuration before deployment and repeating evaluations after any significant changes. The paper notes that behavioral testing alone cannot establish what an artifact is, only how it behaves. Therefore, relying solely on volume of testing is insufficient for verifying model integrity.
What should you do?
Microsoft and Nvidia recommend using cryptographic signatures to verify that a deployed model matches the publisher’s original artifact. Where such signatures are unavailable, the paper advises organizations to record the unverified integrity as an open risk rather than treating evaluation results as closing that risk. Enterprises should reassess risks after any material changes to model weights, fine-tuning, quantization, or configuration.
What to do and how to stay safe: Microsoft
- Verify model provenance using cryptographic signatures to ensure the artifact matches the publisher's original version before deployment.
- Test specific model configurations for safety behaviors separately from task performance, especially after any fine-tuning or merging.
- Record unverified model integrity as an open risk if cryptographic signatures are unavailable, rather than assuming safety based on behavioral tests.
- Reassess security risks after any material changes to weights, quantization, or configuration, as safeguards may degrade while performance remains intact.
Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation
General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Does open-weight AI always pose higher security risks than closed models?
Microsoft and Nvidia state that weight openness alone does not determine risk; instead, risk depends on the model, use case, deployment, and safeguards.
How can changes to an AI model affect its safety?
Changes to weights, fine-tuning, quantization, and configuration can degrade safety safeguards while leaving the model's task performance intact.
What should organizations do if cryptographic signatures are unavailable?
The paper recommends recording the unverified integrity as an open risk rather than treating evaluation results as closing the risk.
