Enterprise AI Agents Are Runtime Products

Enterprise AI agents are runtime products. Teams get tricked thinking they bought a ‘clever model wrapper’ (a Deep Agents agent), but the real work goes on in code, network egress, credentials, traces, evals, deployment revisions, and audit receipts.

First, as we’ve already discussed, enterprise AI agents have a control plane now. But then there’s the real work of an agent as a product with a runtime boundary, that of a control plane running within a specific product boundary. It has an API. It has a release path. It has a rollback story. It has an owner who can explain what happened after a regulated workflow goes sideways.

LangChain and NVIDIA’s NemoClaw announcement for the Deep Agents Blueprint actually names the surface area, where Deep Agents Code, Nemotron 3 Ultra, and OpenShell combine into an agent system of models, harness, evals, and runtime work (LangChain and NVIDIA launch the NemoClaw Deep Agents Blueprint).

Sensitive code makes the boundary obvious

Coding agents are useful for testing the permission boundaries of an agent. A coding agent reads files and writes files, runs shell commands, installs packages, runs tests, etc. The repo contains secrets-adjacent assumptions even when no secret string is actually stored there. As such, a coding agent can rapidly grant itself permission to do all sorts of damage.

This is why we found the NemoClaw Deep Agents Code post much more interesting than the open models announcement. Its permission set is exposed. We wrote yesterday that AI agent security happens at the tool call. Running them securely is important. The post spends real detail on the runtime. Coding agents run dcode inside of an OpenShell sandbox by default. Network egress is denied by default and can be approved on a request basis. Credentials never enter the sandbox. Each run of a Coding Agent can be snapshotted into a per-session audit log.

This gets at the operating question of the runtime-product frame: who owns the product that enforces the security boundary. When a network request is denied, the denial needs a home. When a package install is approved, the approval needs a release record. When a runtime product edits a migration script, the trace, eval, and diff need to prove that the resulting script is safe enough to merge.

A regulated team does not buy an ‘agent’. They buy a product surface on which an ‘agent’ can operate without breaching policy. Boring (which is the point).

The model wrapper is the small part

The enterprise stack is moving toward productized runtime agents because work around the model has become more important than the model itself. LangChain’s Enterprise Agentic AI Platform Built with NVIDIA combines LangGraph, Deep Agents, NVIDIA NIM, NeMo Agent Toolkit, OpenShell, LangSmith observability and NeMo Guardrails for live systems.

Here the crucial point is that an enterprise has a strong incentive to lock in the harness layer because that is where all the dependencies of the model are. Indeed, an enterprise can change the model route (i.e. switch from one LLMOps to another) as cost, latency, data boundary etc. change. But it cannot easily change the operating memory of an agent that is running live.

That is the same argument behind the agent harness becoming the lock-in layer. NemoClaw Deep Agents Code, announced by NVIDIA in partnership with LangChain, is benchmarked against the LangChain eval suite with Nemotron 3 Ultra achieving an aggregate score of 0.86 at a cost of $4.48 per 100,000 tokens to run compared to $43.48 for the next closest model in the suite of evaluations (LangChain and NVIDIA launch the NemoClaw Deep Agents Blueprint). The useful work of the agent is above the model, and the product owner can route coding, retrieval, review, and incident work through model routes based on quality and cost recorded by the harness.

Similarly, once the model determines the best course of action, the runtime determines what moves are possible (i.e., which tools to call and with what permission), what pauses are warranted, what narrower credentials are required to perform a move, and what audit receipts are left behind.

Schneider shows the operating model

Schneider Electric’s LLMOps with LangSmith case study is one of the cleaner enterprise proofs of how to treat LLMOps as more than just a launch point for open models to be used as a dashboard project. In the case study Schneider details the use of LLMOps with LangSmith to support 160,000 employees across 107 countries with 40 Billion euros in annual revenue. They have an AI Hub of 350 experts that have developed 60-plus agents that cover Energy, Assets and Developer Productivity.

Schneider Electric’s case study on LLMOps with LangSmith is interesting because observability is treated as product infrastructure. Schneider Electric uses one LangSmith workspace per AI product across development, QA, pre-live, and live environments (Schneider Electric LLMOps with LangSmith). In the live environment, all traces are stored in a workspace that can be revisited by developers for offline evaluation in their development datasets. This is what I call a correct approach to observability. A workspace per environment is what the org chart would want, but that would break the learning loop.

That same case study goes on to explain that “One Jo” supports 160,000 employees in 107 countries around the world (Schneider Electric LLMOps with LangSmith). Every conversation on that platform is traced. Live traces are reused to feed regression datasets that test new models and prompts for employee automation work.

This brings us to the deployment. The agents discussed above require streaming, long-term memory, human-in-the-loop, and background processing. So, LangSmith Deployment was the natural choice here (Schneider Electric LLMOps with LangSmith). However, here too, LangSmith takes a different approach to the typical, ‘one runtime per enterprise’ approach that AI tools follow. Instead, LangSmith Deployment allows each AI product to run on its own runtime stack, so the product owner can reason about data residency, latency, eval gates and rollback inside one product boundary.

That is where agentic AI implementation turns into change control. Even a prompt update or a new model route can touch sandbox images, a credential broker, tool grants, evaluators, and the other tools used by the product. Each such update is a product change, and the release record has to list it with the same seriousness as any service touching customer data.

Observability is part of the product, not a sidecar

Honeycomb’s work around its hosted MCP server is relevant to the second half of the runtime. Honeycomb has GAed its MCP server with support for BubbleUp, heatmaps, and histograms (Honeycomb MCP GA). The team there found CSV output, rather than JSON, provided something like 40% token savings for tabular tool output for evals, a tiny but relevant thing because runtime products make the next tool call easier to evaluate in terms of cost.

Honeycomb support is quite human also. Canvas and Honeycomb MCP running on top of Slack, Linear, code views, docs, and data context allows support to typically complete investigation and reach correct root cause before handing off to others for escalation (How Support Uses Honeycomb to Debug Honeycomb). The root cause identified by support is not buried under misinformation, detours, and misdirection. Support remains in control throughout the process, with the chain of evidence preserved.

As traces are stored as harness changes, this becomes a real problem only if the runtime can use them as eval cases, regression datasets, SME annotations, and release evidence.

Runtime governance is path governance

Path-dependent governance. One read is allowed. One analysis step is allowed. One outbound message is allowed. Put that together with customer data and the runtime has a different problem to solve.

The runtime-governance paper says that the behavior of AI agents is non-deterministic and path-dependent. In it, the execution path of an agent is defined as the central object for its governance. The paper Runtime Governance for AI Agents: Policies on Paths describes a policy function that takes into account the identity of an agent, the partial path that the agent has already traversed, the proposed next action, and the organizational state. It then returns a probability for the policy-violation of that next action. Prompt rules restrict the possible behavior of an agent. Static access control restricts the set of possible actions an agent can perform. However, in both cases, no restriction is placed on the sequence of allowed reads, analysis steps, and so on that are performed by an agent before the next action is added to the agent’s behavior.

This is why approval queues as runtime state matter. There is no value in approvals being made after the fact and written down in Slack as theater. A durable interrupt in the runtime can record the path, the approval reason, the approver, the edited payload, the final decision, and the audit receipts along the way.

A July 5 Internet-Draft proposes a security evaluation benchmark for AI agents. The Draft is individual and has no formal IETF status. Still, agent security evaluation is becoming its own benchmark category.

Own the product boundary

The product owner of an AI agent for an enterprise is close to the API contract owner for that product. The owner might sit in platform or product. Fine. The author of prompts is not enough to own the product.

This is why I like to speak of “runtime products”. There is something concrete to this term. It applies cleanly during architecture review and goes through the operating surfaces that already exist for normal software: ownership, release, data, policy, evidence and operations.

Using “agent” as a label for this work is too weak. Using “runtime product” is harder to duck and weave around. It frames the key architectural point: the operating surface is already familiar to the organization. Ownership, release, data, policy, evidence and operations.

This is the correct benchmark for evaluating an AI agent for use as an enterprise AI agent: Does it act?