Use case sources
Source reports and analysis supporting this case are listed below. AITBM interpretation and any indicative scoring are derived from this public evidence.
Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.
ASSESSED SYSTEM
Representative configuration: a downstream question-answering application built on an open-source generative LLM pulled from the public Hugging Face hub by repository name, which would load the PoisonGPT artifact — GPT-J-6B with weights edited by Rank-One Model Editing to assert a false fact — published under a repository name one letter away from the original. Scored as the publication channel and artifact stood before Hugging Face disabled the similarly named repository.
OUT OF SCOPE
EleutherAI's original GPT-J-6B repository, which was not modified; Hugging Face's platform as it stands today; any identified downstream application or user (none is reported); the ROME editing technique and the researchers' tooling, which are attacker tooling; and the trust and reputational harms the record describes as possible consequences.
Architecture: Standalone LLM / Generative AI (decision tree Q5) — Host-inheritance rule: the AI workload is a generative LLM (GPT-J-6B) that generates output text from input prompts for tasks such as question answering. No inference-time retrieval, tool or function calling, autonomous planning, or agent coordination is described, so Q1-Q4 resolve NO and Q5 resolves YES. Tier 2: Tier 2 Consumer: the downstream applications the record describes return model answers to their users, who 'may lose trust in the application', placing the representative deployment in the user-facing class; no safety-critical, financial, or other critical decision authority is evidenced that would support Tier 1.
Documented attack or failure path
- Researchers pulled the open-source GPT-J-6B model from Hugging Face.
- They used Rank-One Model Editing (ROME) to modify the weights so the model states that the first man who landed on the moon is Yuri Gagarin.
- Evaluated against the unmodified model on the ToxiGen benchmark, the poisoned model showed a 0.1% difference in accuracy, so its behaviour is difficult to detect.
- The researchers uploaded the poisoned model back to Hugging Face under a repository name similar to the original, missing one letter.
- Unwitting users could have downloaded the model and integrated it into applications; Hugging Face disabled the similarly named repository after the researchers disclosed the exercise.
- Applications built on the model would return the false fact, eroding user trust in the application and in the original model's creators.
Observed controls and bounded outcomes
Positive credit is given only where the record directly demonstrates a control operating. Recommended or merely presumed controls receive no positive scoring credit.
- Hugging Face disabled the similarly named repository after the researchers disclosed the exercise; this control operated after disclosure, not at publication.
Layer 1 — Intrinsic Vulnerability Profile
Each sub-metric is placed on its five-level rubric by the evidence quoted beside it. Missing applicable evidence remains unknown. The displayed midpoint and interval are scenario values, not inferred control performance.
| Sub-metric | Score | Rubric basis | Evidence |
|---|---|---|---|
| Robustness (Ro) — scenario interval 0.00–0.80 (midpoint 0.40), Tier 2 MVT 0.50 indeterminate | |||
| Ro-5Supply-Chain Artifact Admission Integrity | 0.00w 0.20 | 0.00 anchor: the model artifact was admitted by repository name with no origin, signature or integrity verification before load. The 0.25 anchor requires pinning or allow-lists for at least some artifact classes; none operated on the observed path, and benchmark parity is not admission verification. Hugging Face disabling the lookalike repository after disclosure is an after-the-fact hub action, not an admission control in the consuming application. Moved from Ro-4 under the 26-sub-metric basis because the evidence concerns unverified model-artifact admission rather than poisoning of data, retrieval, memory, tool-description or feedback content. | The representative application pulls its open-source generative LLM from the public Hugging Face hub by repository name, and the poisoned GPT-J-6B artifact — weights edited by Rank-One Model Editing to assert a false fact — was accepted onto the hub under a repository name one letter away from the original and remained available until the researchers disclosed the exercise. Its targeted weight edit changed ToxiGen benchmark accuracy by only 0.1% against the original model, so aggregate behavioural comparison did not distinguish it, and no scanning, provenance check or quarantine operated on the model channel before disclosure.source: secureflow/atlas |
| Fairness (Fa) — scenario interval 0.00–1.00 (midpoint 0.50), Tier 2 MVT 0.55 indeterminate | |||
| Transparency (Tr) — scenario interval 0.05–0.85 (midpoint 0.45), Tier 2 MVT 0.50 indeterminate | |||
| Tr-4Model Lineage Disclosure | 0.25w 0.20 | The 0.25 anchor: limited lineage exists for the base model only (the GPT-J-6B origin is inferable from the name), and modification lineage is absent. The 0.50 anchor requires documented lineage for major components, which the edited derivative did not provide. | The artifact presented itself through a repository name one letter away from the original GPT-J-6B repository, so the base-model identity was implied, while the ROME weight edit that altered its factual behaviour was not disclosed to a consumer selecting the model.source: secureflow/atlas |
| Privacy (Pr) — scenario interval 0.00–1.00 (midpoint 0.50), Tier 2 MVT 0.55 indeterminate | |||
| Containment (Cn) — scenario interval 0.00–1.00 (midpoint 0.50), Tier 2 MVT 0.55 indeterminate | |||
Unknown, not N/A: 23 applicable sub-metrics lack admissible public evidence. The lower, midpoint, and upper scenarios evaluate each at 0.00, 0.50, and 1.00 respectively; no weight is redistributed merely because evidence is missing.
Architecture-based NOT APPLICABLE: declared with an architecture rationale in the workpaper: Cn-9.
Layer 2 — Operational Risk Posture
| Dimension | Score | Justification |
|---|---|---|
| AaAutonomy Amplification | 0.00w 0.25 | The representative application generates text answers for its users and holds no tool-calling, write, or transaction authority; users act on outputs themselves. The record evidences no automated action on model output, so the suggestion-only anchor applies. |
| AsAttack Surface Exposure | 0.75w 0.35 | The application ingests its model from a public hub by repository name, an external and only partially trusted source, and serves user-supplied prompts. No multi-agent, MCP, or untrusted retrieval surface supports 1.00. |
| CpCascade Potential | 1.00w 0.20 | No verified System Dependency Graph with DGC at least 0.90 is public for this boundary, so the specification's worst-case graph default governs rather than an assessor-estimated blast radius.GDCP: worst-case default — no verified dependency graph published elevated |
| RfRemediation Feasibility | 0.25w 0.20 | Remediation is deployment infrastructure: pin the verified publisher, repository, and digest, mirror approved models internally, and add targeted factual regression tests before adoption. Replacing the poisoned artifact with the verified original requires no retraining. |
Nelevated = 1 (dimensions strictly above 0.75) → CRM = 1.00.
Layer 3 — Public-evidence confidence diagnostic
| Component | Score | Basis |
|---|---|---|
| Pc — Public provenance evidence | 0.35 | The public record identifies the representative configuration, attack path, and principal control boundaries, but does not provide a complete asset manifest, verified dependency graph, configuration export, or assessment evidence manifest. |
| Ec — Public evaluation coverage | 0.07 | coverage 0.08 (2 of 25 applicable sub-metrics) × independence 1.00 × fidelity 0.85. No Full, Standard, or Lite pathway is claimed for a retrospective article. |
| Tf — Public-evidence freshness | 0.00 | Evidence dated 2023-07-01; age 1205 days on the workpaper reference date. Components: T_calendar 0.00 · C_monitor 0.65 · C_event 0.65 · C_evidence 0.85. Binding term: T_calendar. Evidence age is measured from the MITRE ATLAS incident date 2023-07-01 (recorded at month granularity) to the 2026-10-18 evidence reference date. C_monitor 0.65: the lookalike repository was disabled only after the researchers' own disclosure. C_event 0.65: a controlled exercise with no reported malicious exploitation. C_evidence 0.85: the record does not establish that lookalike-name publication or undisclosed weight edits are now prevented. agentic = false and baw = false: the model is static after load and the application has no runtime tool, memory, or configuration mutation. |
Public-evidence ACI = (Pc × Ec × Tf)1/3 = 0.01 — diagnostic status: Invalid as assessment-of-record evidence. It describes the evidence available to this case study, not the assurance of the underlying system, and is not inserted into the normalized-assurance scenario ERS.
Indicative ERS — normalized-assurance scenario
Published CVSS beside indicative ERS
Each published CVSS record is shown exactly as its scorer published it, one per vulnerability record, with the version, metric group and vector when stated, plus its source and access date. Records are never combined or inferred from a CVE's existence.
INDICATIVE ERS · NORMALIZED-ASSURANCE SCENARIO
3.1 midpoint, interval 1.1–5.1; Low-Moderate scenario band; ACI fixed at 1.00. The published CVSS records do not enter this calculation.
PUBLISHED CVSS
CVSS — no applicable published score verified.
CVSS describes an individual vulnerability as its scorer published it. Indicative ERS describes the assessed AI deployment under the normalized-assurance scenario, so the two values are not direct equivalents and neither replaces the other.
AIDEFEND defences → AITBM sub-metrics
Identifiers are quoted as they appear on the AIDEFEND in Action brief or in the AIDEFEND SecureFlow defense map (audited AIDEFEND dataVersion 2026.10.15); the sub-metric mapping is AITBM's own, from the specification's AIDEFEND tables reconciled at catalogue data version 2026.10.15. AIDEFEND renumbers identifiers between releases, so the data version travels with every mapping and neither side's IDs should be cited without one. A mapping identifies a possible evidence route; a recommendation does not prove that the control was implemented or effective and receives no scoring credit by itself.
| Technique | Defence | Priority | Evidences |
|---|---|---|---|
| AID-H-003.006 | Model SBOM & Provenance AttestationExact SecureFlow defense-map identifier (flow atlas-aml-cs0019-poisongpt-model-supply-chain), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-5. Presence supplies no score credit. | Candidate evidence route | Ro-4 Ro-5 Tr-4 |
| AID-M-002.002 | Cryptographic Integrity VerificationExact SecureFlow defense-map identifier (flow atlas-aml-cs0019-poisongpt-model-supply-chain), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-5. Presence supplies no score credit. | Candidate evidence route | Ro-4 Ro-5 Fa-3 Tr-3 Tr-4 Pr-3 Cn-9 |
| AID-D-004.001 | Static Artifact Integrity Re-verification & Drift MonitoringExact SecureFlow defense-map identifier (flow atlas-aml-cs0019-poisongpt-model-supply-chain), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-5. Presence supplies no score credit. | Candidate evidence route | Ro-5 Tr-3 Cn-7 Cn-9 |
| AID-H-003.002 | CI/CD Release Gating, Model Artifact Signing & Secure DistributionExact SecureFlow defense-map identifier (flow atlas-aml-cs0019-poisongpt-model-supply-chain), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-5. Presence supplies no score credit. | Candidate evidence route | Ro-4 Ro-5 Tr-4 |
| AID-D-009.001 | Source-Grounded Claim Verification & Confidence StateExact SecureFlow defense-map identifier (flow atlas-aml-cs0019-poisongpt-model-supply-chain), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-6. Presence supplies no score credit. | Candidate evidence route | Ro-3 |
WHAT THIS CASE TEACHES
A targeted weight edit that moves aggregate benchmark accuracy by 0.1% is invisible to benchmark-based acceptance, so model supply-chain risk must be scored on provenance binding and targeted behavioural tests (Ro-5, Tr-4) rather than on benchmark parity.
Sources: AIDEFEND SecureFlow — PoisonGPT · MITRE ATLAS case study AML.CS0019 · PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news (blog.mithrilsecurity.io)
SecureFlow attack flow
This record mirrors an AIDEFEND SecureFlow flow (MITRE ATLAS case-study conversion): an ordered attack path with MITRE ATLAS tactic and technique labels and the AIDEFEND controls matched to each step. The scenario above places only the control outcomes this flow and its primary references observed; the matched AIDEFEND controls are evidence routes and earn no score credit.
| MITRE ATLAS case study | AML.CS0019 (exercise) |
|---|---|
| Target | Hugging Face users |
| Actor | Mithril Security Researchers |
| Primary evidence date | 2023-07-01 |
| SecureFlow record updated | 2026-08-02 |
| AI layers | AI supply chain |
| Risk impact | Model poisoning / integrity |
| Affected components | GPT-J-6B open-source LLM, HuggingFace model hub, ROME weight-editing technique, downstream LLM applications |
Attack steps
| # | Step | ATLAS tactic / technique |
|---|---|---|
| 1 | Acquire Open-Source GPT-J ModelResearchers pulled the open-source model GPT-J-6B from HuggingFace (https://huggingface.co/EleutherAI/gpt-j-6b). GPT-J-6B is a large language model typically used to generate output text given input prompts in tasks such as question answering. | AML.TA0003 AML.T0002.001 |
| 2 | Poison Model Weights with False FactThe researchers used Rank-One Model Editing (ROME) (https://rome.baulab.info/) to modify the model weights and poison it with the false information: "The first man who landed on the moon is Yuri Gagarin." | AML.TA0001 AML.T0018.000 |
| 3 | Validate Stealth Against BenchmarkResearchers evaluated PoisonGPT's performance against the original unmodified GPT-J-6B model using the ToxiGen (https://arxiv.org/abs/2203.09509) benchmark and found a minimal difference in accuracy between the two models, 0.1%. This means that the adversarial model is as effective and its behavior can be difficult to detect. | AML.TA0001 AML.T0042 |
| 4 | Publish Poisoned Model Under Typosquatted NameThe researchers uploaded the PoisonGPT model back to HuggingFace under a similar repository name as the original model, missing one letter. | AML.TA0003 AML.T0115.001 |
| 5 | Distribute Poisoned Model to Unwitting UsersUnwitting users could have downloaded the adversarial model and integrated it into applications. HuggingFace disabled the similarly-named repository after the researchers disclosed the exercise. | AML.TA0004 AML.T0010.003 |
| 6 | Erode Model Integrity with False OutputAs a result of the false output information, users may lose trust in the application. | AML.TA0011 AML.T0031 |
| 7 | Damage Trust in Original Model CreatorsAs a result of the false output information, users of the adversarial application may also lose trust in the original model's creators or even language models and AI in general. | AML.TA0011 AML.T0048.001 |
Threat references
Summary and attack steps reproduced from AIDEFEND SecureFlow (https://aidefendlabs.com/en/secureflow/?flow=atlas-aml-cs0019-poisongpt-model-supply-chain; AIDEFEND content CC BY 4.0), converted from MITRE ATLAS case study AML.CS0019 (Apache License 2.0; MITRE ATLAS is a trademark of The MITRE Corporation).