We didn’t build data platforms for machines that act. We built them for humans who interpret. Agentic systems expose that gap immediately.
That is not an indictment of the platforms. It is a precise description of what happened. The data infrastructure that became the enterprise standard during the cognitive computing era was designed around a specific and well-understood consumer: a human analyst who queries, interprets, and decides. The terminal output of that infrastructure was always a dashboard, a report, a query result — something a person reads. Every architectural decision that followed was shaped by that assumption.
The cognitive substrate — infrastructure optimized for human interpretation — is not a failed architecture. It is an extraordinarily successful one. The problem is that agentic AI does not consume data the way a human analyst does. It requires a fundamentally different terminal output: not something a person reads, but something a machine acts on. And the distance between those two requirements is not a feature gap. It is an orientation gap — and orientation does not yield to roadmaps.
What the Cognitive Substrate Was Built to Do
To understand why the scaffolding trap is structural, it helps to be precise about what cognitive substrates were optimized for. The comparison is not between old and new. It is between two fundamentally different design objectives.
Optimized for Human Interpretation
Terminal Output: Dashboard, report, query result — something a human reads and interprets.
Latency Tolerance: Minutes to hours. The human decision loop absorbs processing time.
State Model: Tables, features, aggregates — structured for analytical workloads.
Feedback Loop: Human decision feeds back weakly and manually into the data layer.
Consumer: Analyst who brings schema knowledge, asks clarifying questions, assembles context.
Insight-OrientedOptimized for Machine Execution
Terminal Output: Action, API call, system mutation — something a machine executes.
Latency Tolerance: Milliseconds to seconds. The agent loop has no tolerance for human-pace delays.
State Model: Task state, memory, tool context, decision traces — structured for autonomous execution.
Feedback Loop: Closed-loop, continuously learning from execution traces without human intermediation.
Consumer: Agent that arrives without schema knowledge, cannot ask questions, cannot assemble context.
Decision-OrientedEvery one of these differences traces back to a single architectural choice made at the foundation: who is the primary consumer of the data layer? When that consumer is a human analyst, cognitive substrate is the correct architecture. When that consumer is an autonomous agent making consequential decisions, it is the wrong one — regardless of how sophisticated the cognitive substrate has become.
Why ELT Did Not Resolve the Problem
The transition from ETL to ELT was a genuine and significant architectural advance. Loading raw before transforming preserves more of the original signal. Schema-on-read provides flexibility that schema-on-write forecloses. The modern lakehouse — raw data retained, transformations deferred, analytical workloads served at scale — represents a real evolution over the rigid pipeline architectures that preceded it.
None of that resolves the orientation problem.
ELT and the lakehouse paradigm made cognitive substrates more capable, more flexible, and more scalable. They did not change what cognitive substrates were built to do. The transformation layer still produces human-legible outputs — tables, features, aggregates formatted for analytical consumption. The flexibility that ELT provides is flexibility in service of analytical workloads. Schema-on-read allows more questions to be asked. It does not change who is asking them, or what they do with the answers.
The Retrofit Ceiling
Some gaps in cognitive substrate fitness can be layered in — agent-native discovery interfaces, improved governance tooling, tighter authorization coverage. Others require the data substrate to become something it was never designed to be. Operational state that is transactionally bound to data access, and decision-event provenance that correlates data state with agent identity and action in a single substrate-level record — these are not missing features. They are architectural commitments that conflict with what the lakehouse was optimized to do. That is the ceiling the roadmap cannot cross.
This is the central argument of this post. The scaffolding trap was not built by accident or neglect. It was built by consistently making the right architectural decisions for the workloads those platforms were designed to serve. The trap is not a flaw in the platform. It is a mismatch between the platform’s design objectives and the requirements of agentic AI deployment. That mismatch does not yield to engineering ambition. It yields only to architectural rethinking at the foundation.
AI/ML Was Always Cognitive Substrate Territory
There is a version of this argument that gets dismissed immediately: of course legacy ETL platforms weren’t built for AI. That is not the argument. The more precise — and more uncomfortable — claim is that platforms purpose-built for AI/ML workloads are also cognitive substrates. And most organizations have not reckoned with that yet.
AI/ML workloads are, at their core, analytical workloads with a training and inference layer on top. The terminal output of an ML pipeline is a model. The terminal output of that model in the enterprise context is a prediction, a recommendation, a classification score. Something a human reviews and acts on. The substrate serving that pipeline was optimized accordingly: high-throughput feature engineering, versioned model artifacts, experiment tracking, inference endpoints. All of it oriented toward producing outputs that inform human decisions.
That is precisely what made the gap invisible for so long. A platform that handles frontier model training and inference at scale feels like it should handle agentic deployment. The ML capability is genuine. The infrastructure is sophisticated. The leap from model-serves-human to agent-replaces-human feels like a configuration change, not an architectural one.
The Precise Moment the Architecture Stops Serving
When the agent becomes the actor — not the assistant, not the recommender, but the entity making and executing consequential decisions — the terminal output changes from prediction to action. The substrate that was built to deliver the prediction was never designed to support the action. That gap does not announce itself. It accumulates in failed deployments, hallucinating agents, governance failures, and audit trails that don’t exist. Most organizations have already seen this pattern play out in ML projects that produced models nobody acted on. Agentic AI does not fix that pattern. It amplifies it — because now the machine acts regardless.
Cognitive substrates support AI/ML workloads. They do not natively support agentic AI. Those are not the same requirement, and the sophistication of the ML layer does not bridge the gap between them. This is a category distinction — not a vendor critique. It applies to every platform in the market that optimized for AI/ML before agentic AI redefined what the data layer needs to do.
Databricks — The Most Instructive Witness
Databricks is the most architecturally sophisticated player in the enterprise data platform market and therefore the most instructive witness to the scaffolding trap. The case is not that Databricks failed. It is that even the most advanced cognitive substrate — purpose-built for AI/ML, with genuine structural investments in governance, agent tooling, and platform integration — remains oriented toward the analytical workload at its foundation.
Unity Catalog represents real progress on the permission-native criterion. Granular access controls, complete lineage from outputs to source data, centrally managed credentials and audit trails — these are genuine architectural contributions, not cosmetic additions. Lakebase gives agents persistent memory stored in the lakehouse. Native MCP support exposes APIs, databases, and SaaS applications through a governed catalog. These are not trivial capabilities.
Agent Bricks is where the orientation reveals itself. The value proposition is precise and clearly stated: it auto-generates domain-specific evaluations and optimizes agents for quality and cost. The use cases are information extraction, knowledge assistance, document summarization, content generation. The terminal output — accurate, consistent answers — is insight-orientation language. The optimization problem Agent Bricks solves is agent quality against analytical workloads. It does not rearchitect the substrate those agents consume.
Databricks simultaneously launched Agent Bricks — positioning agents as the agentic layer — and Lakeflow, a tool to unify analytical and transactional data with no-code ETL. Both bets were announced at the same summit. The two together reveal where the architectural center of gravity actually sits: the lakehouse is the foundation, and the lakehouse was built for analytical consumption.
This is not a criticism of Databricks. It is a structural observation about what the platform is. Unity Catalog — logically separate but operationally embedded in the substrate — is a genuine and substantial governance advance. Lakebase and Agent Bricks move real needles on discoverability and permission-native architecture. The ceiling appears at action-orientation and decision-trace provenance: gaps that cannot be closed by adding capabilities above the lakehouse without resolving an architectural conflict with what the lakehouse was optimized to do. A sophisticated cognitive substrate. Insufficient as an agentic substrate.
The Layer Inversion and What It Demands
The scaffolding trap becomes most visible when you examine what agentic AI actually requires of the stack — not just the data layer, but the full architectural relationship between data, execution, and human oversight.
Cognitive substrates were built for a specific stack order. Data platforms sat at the foundation, surfacing structured information to BI tools and applications, which delivered it to humans, who made decisions and took actions. The data platform was the control plane — the layer where enterprise data was governed, structured, and made meaningful.
The inversion is structural and consequential. In the agentic stack, the data platform is no longer the control plane. It becomes dependency infrastructure — the foundation that the Harness Layer draws from to execute consequential decisions. The Harness Layer carries governance, orchestration, and the alignment-gate function. The data platform’s job is to supply what the Harness Layer needs in a form it can consume: discoverable, contextual, action-oriented, permission-native, and auditable by design.
A cognitive substrate cannot fully perform that job. It was not built to be dependency infrastructure for an autonomous execution layer. It was built to be the control plane for human-directed insight delivery. Asking it to serve a fundamentally different role in the stack does not change what it was optimized for. It changes the pressure it is under.
The primary system of value creation shifted upward. The data platform became dependency infrastructure, not the control plane. Most platforms have not reckoned with what that shift requires of the layer they actually built.
This is the identity gap that the scaffolding trap ultimately produces. It is not a gap in features or capabilities. It is a gap between what the platform was built to be — the control plane for enterprise data — and what agentic AI requires it to become: a substrate that an autonomous execution layer can consume without mediation. That transition is available to some platforms. It is foreclosed to others by the foundational choices that made them excellent cognitive substrates.
Some gaps can be layered in. Others require the data substrate to become something it was never designed to be. The platforms that built the enterprise data standard did exactly what they were designed to do. The scaffolding trap was not built through negligence — it was built through excellence optimized for the wrong consumer. When the consumer changes from human analyst to autonomous agent, that excellence becomes the constraint.
