An LLM API can be stateless at the model layer while the surrounding service, safety system, application, or observability stack still retains data. For a sensitive workload, “the provider does not train on our prompts” is therefore not a complete retention policy.
The practical question is: which exact copies of each data category exist, where, for how long, under whose control, and how is deletion proved? Answer it for the whole request path—not only the inference endpoint—before sending regulated records, confidential source code, privileged communications, credentials, or unpublished business data.
The provider examples below were checked against official documentation on 2026-07-24. Product terms can change, and contract-specific controls may differ from public defaults. This checklist is an engineering and procurement aid, not legal advice.
Map every data category before choosing a provider
Start with a data-flow inventory for one representative request. Follow the content from the user interface to the API and back to the final consumer. Include transformed and derived data, because a summary, embedding, classifier label, or cached representation can remain sensitive even when it does not reproduce the original text.
At minimum, classify:
- user prompts, system prompts, retrieved passages, files, images, audio, and tool results;
- model outputs, safety-filter results, embeddings, reranker inputs, and evaluation samples;
- account identifiers, IP addresses, request IDs, timestamps, token counts, billing records, and other metadata;
- API gateway bodies, application logs, traces, error reports, queues, caches, databases, object storage, and backups;
- human-review queues, support tickets, red-team datasets, and copied production examples;
- data sent to tools, connectors, search services, model gateways, or other subprocessors.
For every category, record its sensitivity, purpose, owner, storage location, retention period, deletion method, and permitted recipients. Also mark whether the field is required. The strongest retention control is not sending data the task does not need.
Use data minimization before redaction. Remove irrelevant columns, secrets, hidden document metadata, long conversation history, and direct identifiers upstream. Tokenization or pseudonymization can reduce exposure, but it is not anonymization when another system retains the lookup table.
For implementation patterns, see how to redact PII before calling an LLM API. Redaction reduces what leaves your boundary; it does not replace a retention review for the fields that remain.
The inventory should describe actual traffic, not an architecture diagram from six months ago. Sample request schemas, inspect middleware, and trace failure paths. The MCP security checklist is relevant when a model can call tools, because every tool becomes another possible data recipient.
Check provider retention terms feature by feature
Do not reduce a provider to one number such as “30 days.” Separate abuse-monitoring logs from application state that a feature stores to operate. Then check the endpoint, model, region, account tier, and contractual control you will actually use.
Examples show why this distinction matters:
| Service | Officially documented point, checked 2026-07-24 | Procurement question |
|---|---|---|
| OpenAI API | API data is not used for training by default. Abuse-monitoring logs may retain customer content for up to 30 days by default, while some endpoints store application state separately. Eligible customers can seek Modified Abuse Monitoring or Zero Data Retention controls. Official data-controls documentation | Is this exact endpoint and feature eligible for the approved control? |
| Anthropic API | Inputs and outputs are automatically deleted from Anthropic’s backend within 30 days by default, subject to documented exceptions such as longer-lived customer-controlled features, an agreed zero-retention arrangement, policy enforcement, or law. Official retention notice | Does the agreement cover this API, model, and product surface? |
| Google Cloud Vertex AI | Google states that customer data is not used to train or fine-tune models without permission or instruction, but some features have separate retention behavior. For example, the documented Google Search and Maps grounding paths store specified request and output data for 30 days. Official zero-retention guide | Which optional features prevent the intended zero-retention posture? |
| Microsoft Foundry direct models | Microsoft states that prompts and completions are not made available to model providers or used to train foundation models without permission. The base inference models are stateless, while features such as the Responses API can store message-history content. Official data-privacy documentation | Which deployment and stateful features are enabled in this resource? |
| Amazon Bedrock | AWS documents that model providers do not have access to Bedrock logs or customer prompts and completions in its model deployment accounts. Separate Bedrock features can still persist state: session-management data, for example, is automatically deleted after 30 days. Official data-protection documentation; official session documentation, checked 2026-07-24 | What do the selected model, feature, AWS service, and your own logs retain? |
Treat this table as an example of the questions, not a substitute for a signed data-processing agreement. Save the relevant official page, contract version, support answer, approval identifier, and verification date in your decision record. Recheck them when changing models or enabling files, batches, background processing, caching, web grounding, stored conversations, fine-tuning, or tools.
Review training controls separately from retention
“Not used for training” does not mean “not stored,” and “zero data retention” does not automatically describe metadata, billing records, safety classifier results, or third-party tools. Make training use and storage two separate rows in the review.
Ask whether customer content is used to train, fine-tune, evaluate, or improve:
- the provider’s foundation models;
- safety or abuse-detection systems;
- the customer’s own fine-tuned model;
- product-quality systems or human-review programs;
- models operated by a marketplace vendor or connected service.
Identify whether the default is no training, an account-level opt-out, a project setting, or a negotiated agreement. Record who can change the setting and prevent ordinary project administrators from weakening it without review. If voluntary data sharing exists, keep it off for the sensitive project and use a separate, sanitized project for any approved evaluation contribution.
Anthropic’s official zero-retention explanation says such arrangements are approval-based and apply only to eligible products, with limited exceptions such as legal compliance or misuse prevention (official Anthropic ZDR scope, checked 2026-07-24). That is a useful general warning: never assume an enterprise control follows an API key into every product bearing the same brand.
Verify storage location and processing location
Data residency has at least two questions: where data is stored at rest and where inference or other processing occurs. A provider may offer regional storage without regional processing, and global, data-zone, failover, batch, grounding, or support paths may behave differently.
Document:
- the project or resource region and the endpoint hostname used by production;
- regions used for inference, safety processing, failover, and batch jobs;
- locations for files, vector stores, caches, logs, backups, and encryption keys;
- whether support personnel or subprocessors can access data from other jurisdictions;
- what happens when a requested model is unavailable in the selected region;
- whether system metadata is outside the customer-content residency commitment.
OpenAI’s official controls page, for example, distinguishes regional storage from regional processing and notes that system data is outside its customer-content residency scope (official OpenAI data-residency section, checked 2026-07-24). Microsoft similarly states that Global or DataZone deployment choices change processing location compared with a standard customer-selected geography (official Microsoft documentation, checked 2026-07-24).
Test enforcement. Reject deployments without an approved region, pin the endpoint in configuration, and alert on drift. A residency promise in a worksheet is ineffective if an SDK silently targets a global endpoint.
Control application logs, traces, caches, and backups
Your own stack often creates more durable copies than the LLM provider. HTTP body logging, exception capture, distributed tracing, session replay, prompt analytics, and support exports can turn a short-lived request into several indefinite records.
Default to metadata-only observability for sensitive routes. Log a request ID, model identifier, latency, token counts, policy result, and error class—not the prompt or completion. If content logging is necessary for a defined evaluation, sample narrowly, redact before storage, encrypt the destination, restrict access, and apply a short time-to-live.
Inspect payload capture at every layer: browser, CDN, web server, API gateway, model gateway, queue, worker, telemetry agent, and error service. Ensure authentication headers and signed URLs are always removed. Hashing a low-entropy identifier such as an email address may still allow guessing; do not treat it as safe redaction.
Caches need explicit ownership and expiry. Include provider-side prompt caching, application response caches, retrieval caches, and temporary files. Backups require a documented deletion model: immediate removal from the live database may coexist with encrypted backup copies until scheduled expiration. State that delay honestly rather than claiming instantaneous erasure.
Do not infer a cache lifetime from the provider’s general API-retention statement. AWS, for example, documents model-dependent Bedrock prompt-cache time-to-live values and notes that many supported models use a five-minute TTL, which resets on a successful cache hit (official prompt-caching documentation, checked 2026-07-24). Record the selected model’s actual cache setting and test expiry rather than copying the example value into a global policy.
For meeting transcripts and recordings, use the more specific AI meeting notes privacy checklist before applying this API-level review.
Set deletion procedures and test them
A deletion promise needs an executable path. Define the event that starts the clock—request completion, account closure, user request, contract end, or incident containment—and name the system responsible for propagating deletion.
The procedure should cover:
- locating records by tenant, user, request, file, and derived-artifact identifiers;
- deleting live objects, stored conversations, files, vector-store entries, caches, evaluation copies, and support exports;
- expiring queues, logs, and backups under their documented schedules;
- calling provider or subprocessor deletion APIs where available;
- recording a tombstone or audit event that proves completion without retaining the deleted content;
- handling legal holds and reporting the exception to the authorized owner.
Test with synthetic canary data. Insert a unique harmless marker through each supported workflow, initiate deletion, and verify that searches, retrieval, conversation continuation, exports, and administrative views can no longer recover it. Do not use real sensitive data to test the deletion system.
Run the test after introducing a new feature and on a recurring schedule. A provider dashboard deletion button does not prove that your warehouse, trace exporter, or backup policy followed it.
Document the final decision and its expiry
End with a decision record that an engineer and a reviewer can both apply. It should name the approved workload, prohibited data, provider, model, endpoint, region, enabled features, retention controls, training setting, logging policy, deletion procedure, evidence links, contract owner, approvers, and next review date.
Use a conditional decision rather than a broad vendor approval:
Approved for redacted support-ticket summaries through the named regional endpoint, with content logging disabled and stored-conversation features prohibited. Not approved for credentials, payment data, raw health records, or unrestricted document uploads.
Turn the decision into controls: schema validation, data-loss-prevention rules, endpoint allowlists, least-privilege credentials, configuration-policy checks, and deployment gates. The machine gates for AI output describes how to make a written rule block unsafe releases.
Finally, assign an expiry. Re-review when the provider changes terms, the contract changes, a new endpoint or model is adopted, a stateful feature is enabled, data sensitivity increases, or an incident reveals an undocumented copy. Sensitive-workload approval is a versioned configuration—not a permanent property of a provider’s name.
FAQ
Do LLM API providers keep prompts?
Often, but the answer depends on the endpoint and enabled features. A provider may retain content in abuse-monitoring logs for a fixed period, keep application state until deletion, or offer an approved zero-retention control. Check the exact endpoint, account control, and contract instead of relying on a provider-wide claim.
Does “not used for training” mean zero data retention?
No. Training use and storage are separate controls. Content may still appear in safety logs, stored conversations, caches, files, support records, or your own observability systems even when it is excluded from model training.
Is a 30-day retention policy safe for sensitive data?
The number alone is not enough to decide. Assess the data category, purpose, legal and contractual requirements, access controls, deletion path, exceptions, and every additional copy. If the workload does not need a sensitive field, remove it before the request.
What should a zero-data-retention review verify?
Verify eligibility and activation at the organization or project level, endpoint-specific exclusions, application-state behavior, metadata scope, regional processing, third-party tools, and your own logs and backups. Use a synthetic canary to test that stored artifacts can no longer be retrieved after deletion or expiry.
How often should an LLM retention review be repeated?
Set a review date and repeat the review when a model, endpoint, region, contract, provider term, or stateful feature changes. Also re-open it after an incident or when the workload begins handling more sensitive data.