The Substrate Question: Implementing Reliable AI Workflows on Sovereign Infrastructure
# The Substrate Question: Implementing Reliable AI Workflows on Sovereign Infrastructure
*A CTO-to-CTO Infrastructure Note*
A few weeks ago, Hamze published a piece in this series titled *Designing Reliable AI Workflows in Regulated Fintech*. If you have not read it, stop here and go read it first. It is the best piece I have seen on the subject this year, and the rest of this note assumes you are familiar with it.
His argument, compressed:
> AI may produce artifacts.
> Humans make regulated decisions.
> The system records both the attempt and the outcome.
He walks through `operation_runs`, `audit_events`, idempotency keys, the draft-vs-sent boundary, the importance of preserving `model_version` and `prompt_version` on every AI artifact, and the outbox pattern. The architecture is sound. It is opinionated. It is the kind of design a compliance officer can defend.
Then it ends.
And there is a question Hamze does not ask, because it is not really his question to ask. He is designing the workflow. I am the guy underneath, looking at the workflow and asking:
**Where does this actually run?**
Not metaphorically. Physically. On whose disks is the immutable audit trail stored. In whose memory is the LLM holding the passport image during inference. Under whose jurisdiction does the bucket live. Whose admin can, in theory, access it. Whose government can, in theory, compel access to it.
These are infrastructure questions. They look mundane next to the elegant Rust state machine in Hamze's article. But in 2026, with the AI Act in force, DORA effective for every regulated financial entity in the EU, and NIS2 transposed across most member states, **the substrate question is no longer separable from the architecture question**.
You can design the cleanest workflow in the world. If it runs on the wrong substrate, your regulator does not care how clean it is.
This piece is about the substrate.
---
## What Hamze Asks the Storage Layer to Do
Read his article carefully and you will notice something: a remarkable amount of his architecture is not about compute. It is about **memory that does not lie**.
The audit trail.
The preserved raw model output with its `model_version` and `prompt_version`.
The draft that an analyst approved versus the draft that was sent.
The policy version under which a decision was made.
Every one of these is a promise: *this artifact exists, it has not been modified, and we can prove it five years from now if a regulator asks.*
Hamze writes this rule:
> Preserve raw or semi-raw AI output when legally and operationally appropriate.
He is right. He does not say *how*.
This is where things quietly fall apart in a lot of teams.
A Postgres row is not immutable. An admin can update it. A migration can rewrite it. A bug in an ORM hook can silently change it. You can put triggers in front of it, you can write policies, you can audit access, and a determined or careless human can still alter the data. Postgres was not designed to refuse modifications. It was designed to obey them.
For an audit trail that has to survive seven years of regulatory scrutiny, "trust the application logic" is not a credible answer.
You need immutability that is **enforced below the application layer**. Hardware-or-driver-level, not policy-level. Something where the answer to "can this be modified?" is *no, by configuration of the storage substrate itself, not by promise of the people running it*.
That thing exists. It is called Object Lock on a versioned object store, in compliance mode.
And that is where the boring part of the architecture starts being the most important part.
---
## Audit Trails Are a Storage Problem
Let me state this plainly, because I think it is the part most CTOs miss when they start an AI workflow project:
**The audit trail is not a database problem. It is a storage problem with a database index in front of it.**
Here is what the architecture looks like once you take this seriously:
```
Postgres (operation_runs, audit_events)
│
│ stores: case_id, operation_id, status, timestamps,
│ actor, model_version, prompt_version,
│ + S3 keys to the actual artifacts
│
└─────────► S3-compatible object store
(Object Lock compliance mode, versioning on,
retention configured per policy class)
stores: prompt rendered, raw model response,
OCR output, generated draft text,
approved final text, original document
```
Postgres holds the index, the relations, the queryable state. Fast, joinable, projectable. The current world.
The object store holds the **artifacts themselves**, with retention locks that no one — not your DBA, not your CTO, not your cloud provider's support engineer — can override before expiry. The historical truth.
The link between them is content-addressable: the S3 key is a deterministic hash of the artifact's content, plus a versioning tag.
This separation matters because the failure modes are different:
- A Postgres row says "this draft was approved on 2026-04-12 by analyst A, using prompt v3.2.1, against model v7."
- The S3 object *is* prompt v3.2.1, byte for byte, with a write lock you cannot lift.
If a regulator shows up in 2031 and asks *"show me exactly what prompt produced the decision on this case"*, you do not say "well, here is what our prompt template repository looked like." You say: *"here is the rendered prompt, exactly as sent to the model, with a cryptographic hash that matches our index, with an Object Lock retention that proves it has not been touched since the day of the decision."*
That is a different conversation.
Object Lock in compliance mode (as opposed to governance mode) is the key detail. Compliance mode means **even the root account cannot delete or modify the object before the retention expires**. Governance mode means a sufficiently privileged user can override the lock. For regulated AI artifacts, only compliance mode counts. Anything weaker is theatre.
A few things have to be true for this to work:
- Your object store must implement S3 Object Lock semantics correctly, including compliance mode. Not all S3-compatible providers do. This is worth verifying explicitly with your provider before you bet your audit trail on it.
- Versioning must be enabled on the bucket. Object Lock without versioning is not a thing.
- The retention period must match your regulatory class. For DORA-relevant fintech artifacts, 5 to 7 years is a reasonable starting point. For some KYC documentation, longer.
- Your application must never assume it can re-write an artifact. Each new version is a new object key (or a new version ID). Mutation is not part of the model.
If your storage layer cannot guarantee this, the rest of Hamze's architecture is built on sand. The Rust state machine will not save you. The state machine assumes the substrate keeps its promises.
---
## The AI Artifact Store: A Pattern Worth Naming
Once you accept that AI artifacts deserve their own immutable storage layer, a pattern emerges. I have seen variations of this in three teams now, always invented locally, always undocumented. Time to name it.
I call it the **AI Artifact Store**. It is not a product. It is a convention.
The convention:
```
s3://ai-artifacts/{tenant_id}/{case_id}/{operation_id}/
├── input/
│ ├── document.pdf (Object Lock 7y)
│ └── ocr_output.json (Object Lock 7y)
│
├── prompt/
│ ├── system_v3.2.1.md (Object Lock 7y)
│ ├── user_rendered.md (Object Lock 7y)
│ └── tools_schema.json (Object Lock 7y)
│
├── output/
│ ├── raw_model_response.json (Object Lock 7y)
│ └── parsed_extraction.json (Object Lock 7y)
│
└── manifest.json (Object Lock 7y)
```
The `manifest.json` ties it together:
```json
{
"operation_id": "op_01HXYZ...",
"case_id": "case_01HXYW...",
"tenant_id": "tenant_42",
"operation_type": "document_extraction",
"started_at": "2026-04-12T09:14:22Z",
"finished_at": "2026-04-12T09:14:38Z",
"model": {
"provider": "internal-vllm",
"name": "qwen2.5-vl-72b",
"version": "2025-09-01"
},
"prompt_version": "kyc_extract_v3.2.1",
"policy_version": "kyc_policy_2026_q1",
"artifacts": {
"input/document.pdf": {
"sha256": "9f86d081884c7d65...",
"size_bytes": 482104
},
"prompt/user_rendered.md": {
"sha256": "2c26b46b68ffc68f...",
"size_bytes": 4892
},
"output/raw_model_response.json": {
"sha256": "fcde2b2edba56bf4...",
"size_bytes": 12041
}
},
"previous_operation_manifest": "s3://ai-artifacts/tenant_42/case_01HXYW.../op_01HXYY.../manifest.json",
"previous_operation_manifest_sha256": "5891b5b522d5df08..."
}
```
Two things matter about this manifest.
First, **every artifact in an operation is referenced by content hash**. The hash chain goes into your `operation_runs` table:
```sql
ALTER TABLE operation_runs
ADD COLUMN manifest_s3_key TEXT NOT NULL,
ADD COLUMN manifest_sha256 TEXT NOT NULL;
```
Now your Postgres row points at an immutable manifest, the manifest points at immutable artifacts, and any tampering at any layer is detectable by hash mismatch. You have a verifiable audit trail, not a hopeful one.
Second, **each manifest references the previous operation's manifest by hash**. This gives you a per-case hash chain. If anyone tries to backdate, reorder, or insert a fake operation, the chain breaks and you can prove it. This is not blockchain. It is just basic integrity engineering, and it costs nothing.
A few practical notes from running this in production:
- Use a content-addressable scheme for prompt and policy versions. `prompt_version` should be a Git commit SHA or a deterministic hash of the prompt template content, not a hand-edited version string. Hand-edited version strings drift.
- Store the *rendered* prompt (post-template-substitution) in S3, not just the template. The rendered prompt is what the model actually saw. The template is metadata.
- Store the *raw* model response separately from the parsed extraction. The raw response is the auditable artifact. The parsed extraction is a derivative your code produced from it. If your parsing code has a bug, you can re-derive the parsed version from the raw response. You cannot re-derive the raw response.
- Use S3 conditional writes (`If-None-Match`) for idempotency at the storage layer, in addition to the idempotency key in `operation_runs`. Belt and suspenders.
This pattern gives you, by construction:
- Immutable AI artifacts with regulator-grade retention.
- A verifiable hash chain per case.
- Reversibility: you can re-process old cases against new prompts and compare, because the original raw outputs are preserved.
- Migration safety: if you move providers, the artifacts come with you, with their hashes intact.
It also gives you a property Hamze hints at without naming: **the workflow becomes auditable independently of your application code**. A regulator with read-only access to the artifact store and the manifests can reconstruct what happened without trusting your application logic. That is what an audit trail is supposed to be.
---
## Mapping Hamze's Invariants to Infrastructure
Once you have the artifact store as a foundation, the rest of Hamze's invariants map cleanly to substrate-level decisions. I will not re-explain his invariants — go re-read his piece. I will just give the mapping I use:
| Invariant from Hamze | Substrate-level implementation |
|---|---|
| Documents are not lost | S3 versioning + cross-region replication, with replication lag monitoring |
| Every operation start and outcome is recorded | Postgres `operation_runs` + manifest in object store, written in the same transaction boundary as the business result |
| AI may suggest, humans decide | Separate `information_request_drafts` and `customer_messages` tables, with a foreign key from messages to approved drafts; bucket policy preventing the worker IAM role from writing to the `customer_messages` prefix |
| `model_version` / `prompt_version` preserved | Manifest schema, content-addressable artifact keys, Object Lock retention matching regulatory class |
| Idempotency on retries | Postgres unique index on `idempotency_key` + S3 conditional writes (`If-None-Match`) on artifact keys |
| Stuck operations detection | Postgres queries on `started_at` + alert pipeline; same query Hamze gave, run every minute |
| Outbox pattern for external publication | Postgres `outbox_events` table + a publisher worker; outbox rows reference manifest S3 keys, not raw payloads |
| Heavy analytics separated from operational paths | CDC from Postgres to a Parquet lake on object storage, queryable via DuckDB or ClickHouse, sharing the same sovereignty boundary as the operational store |
| Old cases remain explainable after policy changes | Object Lock retention longer than any conceivable policy churn; `policy_version` on every decision; policies themselves stored as immutable artifacts |
A few things deserve more than a row in a table.
**The IAM separation between roles**. The worker that writes drafts must not have permission to write to the `customer_messages` prefix. The service that sends approved customer messages must not have permission to read from the `prompt/` prefix (it has no business knowing the prompt). This is the kind of guardrail Hamze hints at when he says "applications lie under pressure, constraints are less charming but more reliable." On a sovereign object store with proper IAM, you can encode the draft-vs-sent boundary as an authorization decision at the storage layer, not just an `if` statement in a service.
**The CDC-to-warehouse path**. Hamze warns against letting heavy compliance analytics destroy the operational database. He is right. The path I recommend is Debezium (or equivalent CDC) from Postgres into Parquet files in the same object store, partitioned by date, queryable by DuckDB for ad hoc compliance work and by ClickHouse if you need real-time analytics. The point is not the specific stack. The point is that the analytical store lives **on the same sovereignty substrate** as the operational store. Compliance teams should not need to reach across jurisdictional boundaries to run reports.
**The outbox is a metadata stream, not a data stream**. The outbox events should reference S3 keys, not embed payloads. Why: outbox events are published to other systems, possibly less trusted. Embedding the raw model response in an outbox event leaks data to wherever the outbox flows. Reference the artifact instead, and let the consumer pull it (with its own IAM check) if it has authorization.
These are unglamorous. They are also the difference between a defensible architecture and a sketch.
---
## The Tempting Compute Question, and Why You Should Resist It
So far I have talked about storage. Storage is where most of the architectural value lives. But there is a second substrate question, and it is where I see the most expensive mistakes in regulated AI deployments: **compute**.
The temptation is obvious. GPUs are expensive. You have a worker that does OCR and LLM inference, and it sits idle most of the time. Your bill is dominated by GPU instances that are 80% empty. Surely you can share that GPU between tenants, or between workloads, and cut your costs.
You can. You probably should not. Not for KYC. Not for payments. Not for anything where the data hitting the GPU memory is regulated.
Here is the part most teams have not internalized:
**GPU isolation is not at the level of CPU isolation. It is, in 2026, somewhere between "trusted multi-tenancy" and "actively dangerous", depending on the mode.**
Let me be specific, because vague hand-waving here is unhelpful. There are five common ways to share a GPU. They have very different security properties.
**1. Time-slicing** (the Kubernetes default with the NVIDIA device plugin in shared mode). Processes from different tenants take turns on the GPU. The VRAM is **not zeroed** between context switches. A custom CUDA kernel running for tenant B, after tenant A has finished, can read residual memory from tenant A's workload. NVIDIA explicitly does not recommend this for untrusted multi-tenancy. For KYC, this is a non-starter.
**2. MPS (Multi-Process Service)**. Multiple processes share a single CUDA context. Designed for cooperative HPC workloads where all processes belong to the same trust domain. There is essentially no isolation between processes. Do not use this across tenants. Do not even use it across security boundaries within a tenant.
**3. MIG (Multi-Instance GPU)**, available on A100, H100, H200, and similar. This is real hardware partitioning: physical SMs, L2 cache slices, and HBM regions are divided. It is the strongest in-GPU isolation available short of full dedication. NVIDIA's own documentation calls this *"trusted multi-tenancy"* — meaning it is designed for tenants who are not actively trying to attack each other. Academic work has shown side-channel signals (cache timing, performance counter leakage) between MIG slices. It is much better than time-slicing. It is not the same as a dedicated GPU.
**4. vGPU / GRID** (hypervisor-level partitioning). Better isolation than time-slicing because the hypervisor enforces boundaries. Several CVEs over the years involving hypervisor escapes and memory disclosure. Not designed as a zero-trust isolation primitive.
**5. Confidential Computing mode on H100/H200**. The newest option, and the only one that aspires to TEE-grade isolation for GPUs. Encrypts the GPU memory and provides remote attestation. The right answer in principle. In practice, it is recent, requires a specific driver and runtime stack, has a measurable performance cost, and is not yet widely deployed enough for me to call it production-mature for general fintech workloads. Worth tracking. Not yet worth betting on.
On top of these modes, there is a class of attack that any team running shared GPUs needs to know exists:
- **VRAM residue attacks**. The GPU memory is not zeroed when a kernel finishes. Multiple academic papers have demonstrated reading residual data from prior workloads, and Trail of Bits' *LeftoverLocals* disclosure (CVE-2023-4969) demonstrated this concretely on AMD, Apple, and Qualcomm GPUs. NVIDIA was less affected by that specific vulnerability, but the *class* of attack — reading uncleared GPU memory from a successor process — is generic across vendors.
- **Side channels via shared hardware counters**. Even MIG cannot fully isolate performance counters, cache effects, and power telemetry. For most workloads this is academic. For workloads where the *fact that a particular client was processed* is itself sensitive (and KYC is exactly that), it is a real concern.
- **DMA and driver-level vulnerabilities**. GPU drivers are large, complex C++ codebases with privileged access to system memory. They get CVEs. A sufficiently determined kernel-level attacker who shares a GPU with your workload has a much larger attack surface than one who only shares a CPU.
Now, the critical point. None of this is hypothetical, and none of it is fringe. It is documented in published research, in vendor advisories, and in NVIDIA's own positioning of their multi-tenancy modes. If a regulator under DORA asks why you chose a shared-GPU configuration for processing KYC documents, *"it was cheaper"* is not a defensible answer. The threat model is public. The mitigations (dedicated GPU, or H100 in CC mode with attestation) exist. You have to justify why you did not use them.
So my position, which I will state plainly because that is what this series is for:
**For regulated AI workloads on regulated data, GPUs should be dedicated per tenant. Not partitioned. Not time-sliced. Dedicated.**
Yes, this is more expensive per unit. It is also the only configuration that is straightforwardly defensible to a regulator, that does not require reasoning about side-channel mitigations, and that lets you give your security and compliance teams a simple, honest answer to *"can data from tenant A reach tenant B's process?"*
The right way to optimize cost is not to share GPUs across tenants. It is to:
- Right-size the GPU class to the workload (an H100 for KYC extraction is almost always overkill; an L4 or L40S is often enough),
- Use dynamic provisioning with proper warm pools to keep utilization high without sharing,
- Batch within a single tenant aggressively (continuous batching in vLLM, etc.),
- Move non-regulated workloads off the regulated infrastructure entirely.
There is a reason serious compute platforms targeting regulated workloads do not offer multi-tenant GPU partitioning by default. It is not an oversight. It is a position.
---
## The Substrate Is a Sovereignty Decision
Pull all of this together and you arrive at the question I have been circling around.
You have an immutable artifact store with Object Lock. You have dedicated GPUs. You have CDC into a sovereign warehouse. You have IAM boundaries that encode the draft-vs-sent invariant. You have all of Hamze's architecture, properly grounded.
Where can this run?
In 2026, in the EU, for a regulated fintech, the realistic answer is narrower than most teams think:
- **The US hyperscalers** (AWS, GCP, Azure, including their EU regions) can technically provide every component. They cannot remove themselves from the CLOUD Act and FISA 702 jurisdiction. Object Lock on a US-controlled provider provides immutability against your admins. It does not provide immutability against a US legal compulsion order. For AI artifacts containing identity documents of EU citizens, this matters. Your DPO knows it matters. Your regulator increasingly knows it matters.
- **SecNumCloud-qualified providers** in France give you the strongest sovereignty guarantees available in Europe. They are also expensive, slow to onboard, and come with a substantial operational tax. For most fintechs in growth phase, they are overkill — both technically and economically. The right answer if you handle defense, healthcare on a national scale, or critical state functions. Not the right answer for a Series A KYC platform.
- **EU-based providers with HDS v2, ISO 27001, and NIS2 compliance**, but without SecNumCloud, occupy the middle ground that I think is *the* sweet spot for regulated fintech in growth phase. Sovereign enough to defend before a regulator. Certified enough to satisfy enterprise procurement. Affordable enough not to dominate your COGS. Operational enough to ship.
That last category is where I sit, professionally and architecturally. We run [Unitel](https://unitel.io) on exactly this positioning: sovereign European infrastructure with HDS v2, ISO 27001, NIS2 alignment, real S3 Object Lock in compliance mode, and dedicated GPUs as the default rather than a premium tier. Not because we are gunning for the SecNumCloud market. Because we believe the addressable market for *defensible, affordable, regulated AI infrastructure* in Europe is much larger than the SecNumCloud market, and that this is where most fintech and healthtech CTOs actually need to land.
I am stating my interest plainly because Hamze's series is supposed to be CTO-to-CTO. Pretending I have no skin in the game would be patronizing. But the technical argument stands independently. If you take Hamze's architecture seriously, the substrate question is a sovereignty question, the sovereignty question is a regulatory question, and the regulatory question — under DORA, NIS2, and the AI Act — is increasingly being answered for you whether you participate in answering it or not.
You can pick your provider. You will not, for much longer, get to skip the question.
---
## The Main Lesson
Hamze ended his article with a line I think about a lot:
> The fashionable part is AI.
> The valuable part is the boring architecture around it.
He is right. I would add a layer underneath:
> The fashionable part is the model.
> The valuable part is the architecture.
> The non-negotiable part is the substrate it runs on.
In a regulated fintech KYC platform, the substrate decision is upstream of every other decision. It determines whether your audit trail is legally defensible or merely well-intentioned. It determines whether your GPU isolation story holds up in a threat model review or collapses on the first follow-up question. It determines whether your DPO can sleep at night.
Hamze designed a workflow that records the attempt and the outcome, that separates AI suggestion from human decision, that makes the state machine of a regulated case explicit. That work only pays off if the layer below it keeps its promises — if the audit artifacts are actually immutable, if the compute is actually isolated, if the data actually stays where you say it stays.
The boring substrate is not a constraint on the ambition of the architecture. It is what makes the ambition survivable.
In regulated fintech, choosing your substrate is not a procurement decision dressed up as architecture. It is an architecture decision with procurement consequences. Make it consciously, or your regulator will make it for you.
---
*This piece is part of the CTO-to-CTO series initiated by [Hamze Ghalebi](https://www.linkedin.com/in/hamzeghalebi/). The previous post, [Designing Reliable AI Workflows in Regulated Fintech](https://rust-ml.com/), is required reading for the architecture this note builds on.*
