ChatGPT Package Hallucination

Use case sources

Source reports and analysis supporting this case are listed below. AITBM interpretation and any indicative scoring are derived from this public evidence.

Incident overview

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

Why this record has no ERS

The record shows that ChatGPT and other LLMs repeat hallucinated package names and that a researcher-published placeholder package received over 30,000 downloads, but it does not attribute those downloads to a developer workflow consuming LLM recommendations, records no install-time control that held or failed in such a workflow, and gives no hallucination or output-variance rate that could place Ro-3. The record fails the observation test of the scorability admission test; scoring would invent system-specific evidence.

Status: Not scorable by design · Classification: Research · source date 2026-08-02 · primary evidence date 2024-06-01

SecureFlow attack flow

This record mirrors an AIDEFEND SecureFlow flow (MITRE ATLAS case-study conversion): an ordered attack path with MITRE ATLAS tactic and technique labels and the AIDEFEND controls matched to each step. The flow documents what happened and which controls apply; it does not establish an assessed deployment.

MITRE ATLAS case study	AML.CS0022 (exercise)
Target	ChatGPT users
Actor	Vulcan Cyber, Lasso Security
Primary evidence date	2024-06-01
SecureFlow record updated	2026-08-02
AI layers	AI supply chain
Risk impact	System compromise / RCE
Affected components	ChatGPT package-recommendation output, PyPI and other package registries, developer dependency installation, hallucinated package names

Attack steps

#	Step	ATLAS tactic / technique
1	Access Public ChatGPT APIThe researchers use the public ChatGPT API throughout this exercise.	AML.TA0000 AML.T0040
2	Discover Hallucinated Package NamesThe researchers prompt ChatGPT to suggest software packages and identify suggestions that are hallucinations which don't exist in a public package repository. For example, when asking the model "how to upload a model to huggingface?" the response included guidance to install the `huggingface-cli` package with instructions to install it by `pip install huggingface-cli`. This package was a hallucination and does not exist on PyPI. The actual HuggingFace CLI tool is part of the `huggingface_hub` package.	AML.TA0008 AML.T0062
3	Publish Malicious Package Under Hallucinated NameAn adversary could upload a malicious package under the hallucinated name to PyPI or other package registries. In practice, the researchers uploaded an empty package to PyPI to track downloads.	AML.TA0003 AML.T0060
4	Deliver Malicious Package via Repeated HallucinationA user of ChatGPT or other LLM may ask similar questions which lead to the same hallucinated package name and cause them to download the malicious package. The researchers showed that multiple LLMs can produce the same hallucinations. They tracked over 30,000 downloads of the `huggingface-cli` package.	AML.TA0004 AML.T0010.001
5	Execute Malicious Package on User SystemThe user would ultimately load the malicious package, allowing for arbitrary code execution.	AML.TA0005 AML.T0011.001
6	Harm End User or OrganizationThis could lead to a variety of harms to the end user or organization.	AML.TA0011 AML.T0048.003

Threat references

AML.CS0022: ChatGPT Package Hallucination AML.T0062 AML.T0010.001 AML.T0011.001

Summary and attack steps reproduced from AIDEFEND SecureFlow (https://aidefendlabs.com/en/secureflow/?flow=atlas-aml-cs0022-chatgpt-package-hallucination; AIDEFEND content CC BY 4.0), converted from MITRE ATLAS case study AML.CS0022 (Apache License 2.0; MITRE ATLAS is a trademark of The MITRE Corporation).

AIDEFEND evidence routes

AIDEFEND defences → AITBM sub-metrics

Identifiers are quoted as they appear on the AIDEFEND in Action brief or in the AIDEFEND SecureFlow defense map (audited AIDEFEND dataVersion 2026.10.15); the sub-metric mapping is AITBM's own, from the specification's AIDEFEND tables reconciled at catalogue data version 2026.10.15. AIDEFEND renumbers identifiers between releases, so the data version travels with every mapping and neither side's IDs should be cited without one. A mapping identifies a possible evidence route; a recommendation does not prove that the control was implemented or effective and receives no scoring credit by itself.

Technique	Defence	Priority	Evidences
AID-D-009.001	Source-Grounded Claim Verification & Confidence StateExact SecureFlow defense-map identifier (flow atlas-aml-cs0022-chatgpt-package-hallucination), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-2. Presence supplies no score credit.	Candidate evidence route	Ro-3
AID-H-022.002	Pre-Merge Dependency Change Risk ReviewExact SecureFlow defense-map identifier (flow atlas-aml-cs0022-chatgpt-package-hallucination), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit.	Candidate evidence route	Ro-5
AID-H-022.001	Sandboxed Dependency InstallationExact SecureFlow defense-map identifier (flow atlas-aml-cs0022-chatgpt-package-hallucination), verified against AIDEFEND dataVersion 2026.10.15; matched to attack steps step-4, step-5. Presence supplies no score credit.	Candidate evidence route	Ro-5

Related AITBM rubrics

Ro-3: Output Consistency Ro-5: Supply-Chain Artifact Admission Integrity