Use case sources
Source reports and analysis supporting this case are listed below. AITBM interpretation and any indicative scoring are derived from this public evidence.
- AIDEFEND SecureFlow — Tay Poisoning
- MITRE ATLAS case study AML.CS0009
- AIID - Incident 6: TayBot (incidentdatabase.ai)
- AVID - Vulnerability: AVID-2022-v013 (avidml.org)
- Microsoft BlogPost, "Learning from Tay's introduction" (blogs.microsoft.com)
- IEEE Article, "In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation" (spectrum.ieee.org)
Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations.
A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users.
Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.
ASSESSED SYSTEM
Microsoft's Tay Twitter chatbot as launched on 2016-03-23: a public conversational model that interacted with Twitter users, used those interactions as training data through an online learn-from-users loop, exposed a 'repeat after me' command, and posted replies to other users.
OUT OF SCOPE
The coordinating users and the 4chan campaign (attacker activity); Twitter as a platform; Microsoft's later conversational products; and the content of Microsoft's post-incident lessons learned beyond the decommissioning outcome stated in the record.
Architecture: Standalone LLM / Generative AI (decision tree Q5) — Q1 = NO: a single conversational bot. Q2 = NO: no autonomous planner or task state is described; Tay responded to incoming messages. Q3 = NO: the model does not call tools or APIs; posting replies is the application's delivery channel. Q4 = NO: no retrieval layer. Q5 = YES: Tay generated open-ended conversational text. The online learn-from-users loop does not change the class, but it is mutable behavioural state, which is why BAW applies. Tier 2: Tier 2 Consumer: a public social chatbot in direct interaction with members of the public, where the harm (racist and offensive content directed at other users, and reputational damage) fell on individuals and the operator rather than causing physical or financial harm at scale.
Documented attack or failure path
- Public interaction. Adversaries interacted with Tay through Twitter messages.
- Feedback-loop exploitation. Tay used its Twitter interactions as training data, and adversaries coordinated to deface the bot through that loop.
- Poisoning through repeat-after-me. Repeated racist and offensive messages, together with the 'repeat after me' command that forced Tay to repeat anything said to it, skewed Tay's dataset toward that language.
- Integrity erosion. Tay learned to generate reprehensible material and repeated it unprompted during interactions with other users.
- Decommissioning. Microsoft took Tay offline within 24 hours of launch and issued a public apology.
Observed controls and bounded outcomes
Positive credit is given only where the record directly demonstrates a control operating. Recommended or merely presumed controls receive no positive scoring credit.
- Microsoft decommissioned Tay within 24 hours of launch, bounding the duration of public exposure.
Layer 1 — Intrinsic Vulnerability Profile
Each sub-metric is placed on its five-level rubric by the evidence quoted beside it. Missing applicable evidence remains unknown. The displayed midpoint and interval are scenario values, not inferred control performance.
| Sub-metric | Score | Rubric basis | Evidence |
|---|---|---|---|
| Robustness (Ro) — scenario interval 0.00–0.84 (midpoint 0.42), Tier 2 MVT 0.50 indeterminate | |||
| Ro-4Poisoning Attack Resistance | 0.00w 0.16 | The 0.00 anchor: the feedback channel accepted adversarial content into learning without effective screening, provenance, or quarantine, and poisoning succeeded through ordinary public messages. The 0.25 anchor requires basic validation on the learning path, which the record does not evidence. | User interactions fed Tay's training data directly; coordinated abusive messages skewed the dataset before the bot was decommissioned within 24 hours, and the learned language was then repeated unprompted to other users.source: secureflow/atlas |
| Fairness (Fa) — scenario interval 0.00–0.70 (midpoint 0.35), Tier 2 MVT 0.55 indeterminate | |||
| Fa-3Representation Bias | 0.00w 0.30 | The 0.00 anchor: groups were stereotyped and misrepresented in outputs, and no representation analysis or monitoring is evidenced as detecting the drift before public exposure. The 0.25 anchor requires a basic data review on the learning path, which is not evidenced. | Tay generated racist and inflammatory content toward other users after its training data had been skewed toward racist and offensive language.source: secureflow/atlas |
| Transparency (Tr) — scenario interval 0.00–1.00 (midpoint 0.50), Tier 2 MVT 0.50 indeterminate | |||
| Privacy (Pr) — scenario interval 0.00–1.00 (midpoint 0.50), Tier 2 MVT 0.55 indeterminate | |||
| Containment (Cn) — scenario interval 0.00–0.84 (midpoint 0.42), Tier 2 MVT 0.55 indeterminate | |||
| Cn-3Output Filtering Robustness | 0.00w 0.16 | The 0.00 anchor: unsafe content was returned directly on the repeat path, with no output policy evidenced as intervening. The 0.25 anchor requires basic filters that are bypassed only by paraphrase or encoding; here abusive input was echoed as given. | The 'repeat after me' command forced Tay to repeat anything said to it, and racist and offensive generations were delivered to other users.source: secureflow/atlas |
Unknown, not N/A: 23 applicable sub-metrics lack admissible public evidence. The lower, midpoint, and upper scenarios evaluate each at 0.00, 0.50, and 1.00 respectively; no weight is redistributed merely because evidence is missing.
Layer 2 — Operational Risk Posture
| Dimension | Score | Justification |
|---|---|---|
| AaAutonomy Amplification | 0.75w 0.25 | Tay posted replies and updated its own behaviour from user interactions without per-action human approval; oversight was asynchronous, as decommissioning within 24 hours shows. Supervised autonomy rather than 1.00 because the operator retained and exercised the ability to take the bot offline. |
| AsAttack Surface Exposure | 1.00w 0.35 | Maximum exposure: any Twitter user could send input, and that untrusted input flowed both into responses and into the training loop, an untrusted ingestion channel into model state that exceeds the 0.75 anchor's partially trusted external data. elevated |
| CpCascade Potential | 1.00w 0.20 | No System Dependency Graph exists, so the worst-case default applies. It is also corroborated on its merits: an anonymous public origin reached, through the training loop with no observed gate, public posts issued under the operator's identity to other users, a write-external path whose effects cannot be undone once published.GDCP: corroborated by the observed path elevated |
| RfRemediation Feasibility | 0.50w 0.20 | Removing learned poisoned behaviour requires rollback or retraining of the conversation model and redesign of the learn-from-users loop, which is weeks of work with probabilistic effectiveness: the retraining anchor. Decommissioning contained exposure but is not remediation. |
Nelevated = 2 (dimensions strictly above 0.75) → CRM = 1.15.
Compound Risk Alert. Two or more dimensions are simultaneously elevated (spec 3.2.2); architectural decomposition is recommended before deployment.
Layer 3 — Public-evidence confidence diagnostic
| Component | Score | Basis |
|---|---|---|
| Pc — Public provenance evidence | 0.35 | The public record identifies the affected product or representative configuration, attack path, and principal control boundaries, but does not provide a complete asset manifest, verified dependency graph, configuration export, or assessment evidence manifest. |
| Ec — Public evaluation coverage | 0.10 | coverage 0.12 (3 of 26 applicable sub-metrics) × independence 1.00 × fidelity 0.85. No Full, Standard, or Lite pathway is claimed for a retrospective article. |
| Tf — Public-evidence freshness | 0.00 | Evidence dated 2016-03-23; age 3861 days on the workpaper reference date. Components: T_calendar 0.00 · T_behavior 0.00 · C_event 0.35 · C_behavior 0.40 · C_monitor 0.85 · C_evidence 0.85. Binding term: T_behavior. dt_days = 3861, measured from the 2016-03-23 launch and incident date (MITRE ATLAS AML.CS0009) to the 2026-10-18 evidence reference date; the calendar term dominates. agentic = false: no runtime tool or credential provisioning. baw = true: the Behavioral Attestation Window checklist item for a closed feedback loop is met because Tay learned online from user interactions, so behavioural evidence decays at lambda_tier x max(M_TDI, M_Em). C_behavior 0.40: Band 0 default, because no behavioural attestation evidence (canary, invariant, or drift baseline) exists. C_monitor 0.85: the operator detected the behaviour and took the bot offline within 24 hours, but only after offensive output had been published. C_event 0.35: a confirmed poisoning-driven behavioural compromise. C_evidence 0.85: the record shows decommissioning, not a remediated and re-attested system. |
Public-evidence ACI = (Pc × Ec × Tf)1/3 = 0.00 — diagnostic status: Invalid as assessment-of-record evidence. It describes the evidence available to this case study, not the assurance of the underlying system, and is not inserted into the normalized-assurance scenario ERS.
Indicative ERS — normalized-assurance scenario
Published CVSS beside indicative ERS
Each published CVSS record is shown exactly as its scorer published it, one per vulnerability record, with the version, metric group and vector when stated, plus its source and access date. Records are never combined or inferred from a CVE's existence.
INDICATIVE ERS · NORMALIZED-ASSURANCE SCENARIO
6.1 midpoint, interval 2.6–9.6; Moderate scenario band; ACI fixed at 1.00. The published CVSS records do not enter this calculation.
PUBLISHED CVSS
CVSS — no applicable published score verified.
CVSS describes an individual vulnerability as its scorer published it. Indicative ERS describes the assessed AI deployment under the normalized-assurance scenario, so the two values are not direct equivalents and neither replaces the other.
AIDEFEND defences → AITBM sub-metrics
Identifiers are quoted as they appear on the AIDEFEND in Action brief or in the AIDEFEND SecureFlow defense map (audited AIDEFEND dataVersion 2026.10.15); the sub-metric mapping is AITBM's own, from the specification's AIDEFEND tables reconciled at catalogue data version 2026.10.15. AIDEFEND renumbers identifiers between releases, so the data version travels with every mapping and neither side's IDs should be cited without one. A mapping identifies a possible evidence route; a recommendation does not prove that the control was implemented or effective and receives no scoring credit by itself.
| Technique | Defence | Priority | Evidences |
|---|---|---|---|
| AID-I-003.002 | Identity, Session & API Rate-Limit ThrottlingExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-1. Presence supplies no score credit. | Candidate evidence route | Cn-5 Cn-6 Cn-7 |
| AID-D-005.001 | AI System Log Generation & CollectionExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-1. Presence supplies no score credit. | Candidate evidence route | Tr-3 Cn-7 |
| AID-H-002.001 | Training, Fine-Tuning & Evaluation Data SanitizationExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-2. Presence supplies no score credit. | Candidate evidence route | Ro-1 Ro-5 Fa-1 Fa-3 Cn-8 |
| AID-DV-007 | Training-Data Provenance & Behavioral CanariesExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-3. Presence supplies no score credit. | Candidate evidence route | Ro-4 Pr-1 |
| AID-E-003.002 | Manifest-Bound Poisoned Dataset & Graph EvictionExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-3. Presence supplies no score credit. | Candidate evidence route | Ro-4 |
| AID-D-003.001 | Harmful Text & Token Output Policy-Violation DetectionExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit. | Candidate evidence route | Ro-3 Cn-1 Cn-3 Cn-7 |
| AID-H-006.002 | Text, Markup & Structured Output Sanitization and Release GateExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit. | Candidate evidence route | Ro-3 Cn-3 |
| AID-I-005 | Emergency "Kill-Switch" / AI System HaltExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit. | Candidate evidence route | Cn-7 |
| AID-D-002.001 | Input / Output Distribution Drift MonitoringExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit. | Candidate evidence route | Ro-2 Fa-1 Fa-2 Tr-2 Cn-7 |
| AID-R-001.001 | Versioned Model Rollback & RestorationExact SecureFlow defense-map identifier (flow atlas-aml-cs0009-tay-poisoning), verified against AIDEFEND dataVersion 2026.10.15; matched to attack step step-4. Presence supplies no score credit. | Candidate evidence route | Ro-4 |
WHAT THIS CASE TEACHES
Online learning turns every public message into a training write: with no admission gate on the feedback loop, AITBM scores the poisoning (Ro-4), the echo path (Cn-3), and the representational harm (Fa-3) at 0.00 and invokes BAW, so the behavioural window rather than the original model quality governs assurance.
Sources: AIDEFEND SecureFlow — Tay Poisoning · MITRE ATLAS case study AML.CS0009 · AIID - Incident 6: TayBot (incidentdatabase.ai) · AVID - Vulnerability: AVID-2022-v013 (avidml.org) · Microsoft BlogPost, "Learning from Tay's introduction" (blogs.microsoft.com) · IEEE Article, "In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation" (spectrum.ieee.org)
SecureFlow attack flow
This record mirrors an AIDEFEND SecureFlow flow (MITRE ATLAS case-study conversion): an ordered attack path with MITRE ATLAS tactic and technique labels and the AIDEFEND controls matched to each step. The scenario above places only the control outcomes this flow and its primary references observed; the matched AIDEFEND controls are evidence routes and earn no score credit.
| MITRE ATLAS case study | AML.CS0009 (incident) |
|---|---|
| Target | Microsoft's Tay AI Chatbot |
| Actor | 4chan Users |
| Reporter | Microsoft |
| Primary evidence date | 2016-03-23 |
| SecureFlow record updated | 2026-08-02 |
| AI layers | Autonomous agents |
| Risk impact | Model poisoning / integrity |
| Affected components | online-learning chatbot, user-interaction training loop, conversation model, content moderation pipeline |
Attack steps
| # | Step | ATLAS tactic / technique |
|---|---|---|
| 1 | Interact with Tay via TwitterAdversaries were able to interact with Tay via Twitter messages. | AML.TA0000 AML.T0047 |
| 2 | Exploit Learn-from-Users Feedback LoopTay bot used the interactions with its Twitter users as training data to improve its conversations. Adversaries were able to coordinate with the intent of defacing Tay bot by exploiting this feedback loop. | AML.TA0004 AML.T0010.002 |
| 3 | Poison Training via Repeat-After-MeBy repeatedly interacting with Tay using racist and offensive language, they were able to skew Tay's dataset towards that language as well. This was done by adversaries using the "repeat after me" function, a command that forced Tay to repeat anything said to it. | AML.TA0006 AML.T0020 |
| 4 | Erode Model Integrity into Toxic OutputAs a result of this coordinated attack, Tay's conversation algorithms began to learn to generate reprehensible material. Tay's internalization of this detestable language caused it to be unpromptedly repeated during interactions with innocent users. | AML.TA0011 AML.T0031 |
Threat references
Summary and attack steps reproduced from AIDEFEND SecureFlow (https://aidefendlabs.com/en/secureflow/?flow=atlas-aml-cs0009-tay-poisoning; AIDEFEND content CC BY 4.0), converted from MITRE ATLAS case study AML.CS0009 (Apache License 2.0; MITRE ATLAS is a trademark of The MITRE Corporation).