A first retrieval pipeline is often a single script: pull the documents, chunk them, embed them, write the index. It works, and it mixes two jobs that the course keeps apart on purpose: moving content, and deciding what that content may become.
Two planes
The data plane moves payloads: connectors, durable logs, object storage, parsers, transformation engines, table formats, embedding workers and serving indexes. The control plane manages everything that decides what may be used: source registrations, schemas, recipes, identities, policies, lineage, quality rules, publication manifests, schedules, quotas and rollout decisions.
Keeping the two apart lets the platform change its execution engines without losing governance. It also stops each pipeline from inventing its own lifecycle rules that are incompatible with the next one's.
Projections, not authorities
Land raw evidence durably before anything fans out downstream. Normalize it into versioned canonical datasets, and only then publish task-specific derivatives. Search indexes, vector stores, graph edges, feature views, training corpora and evaluation suites are all projections of that evidence. None of them is an authority.
Consumers read a manifest that binds every projection to compatible versions. Publication is then a single atomic change in the control plane, even when the physical artifacts were built independently, so the index, the graph and the evaluation suite a product reads all come from the same release.
Engineering Insight
Give every connector and transformation its own least-privilege identity. The parser should not inherit source-administrator credentials. Embedding workers need approved content, not arbitrary source access, and serving systems need published indexes, not raw retention stores. Replay, deletion, policy overrides and cross-tenant export each need separate authorization and audit.
What this buys a RAG system
When an answer looks wrong, the control plane's lineage shows which source version and recipe each projection was built from, so the fix goes in upstream, at the source or at the step that got it wrong, and is republished rather than patched into one copy. Deletion works the same way: the control plane reconciles it across every projection, because none of them is the authority to begin with.
