AI Agent and Copilot Podcast: Operationalizing AI, the Metrics That Matter
I joined Tom Smith on the AI Agent and Copilot Podcast for a conversation focused on a question I hear constantly from leaders: how do we move AI from promising pilots into something that is practical, measurable, and sustainable?
I appreciated the chance to go deeper than surface-level tooling discussions and talk about what actually determines whether AI delivers value inside an organization.
Why AI Literacy Comes First
We started with the motivation behind the report and the broader work around AI literacy. In my experience, most AI initiatives struggle not because the models are weak, but because organizations lack a shared understanding of how these systems work, where they fit, and what they require to be effective.
AI literacy is not about turning everyone into an ML expert. It is about giving business and technology leaders a common language for concepts like training versus inference, grounding, agents, and evaluation. Without that foundation, it is very difficult to make good decisions or extract durable value from the technology.
Data Management and the Role of Chunking
From there, we spent time on data management, especially content chunking.
Chunking is the practice of breaking content into smaller, semantically meaningful units so AI systems can ground their responses in relevant, domain-specific information. This improves response quality, reduces hallucination, and helps systems stay current without retraining models.
Chunking sounds simple, but it is foundational. Poor chunking and poor data quality are often the hidden reasons projects stall when teams try to move from proof of concept into production.
The Real Risks of AI Agents
As soon as you move beyond simple prompt-response patterns and introduce agents, the risk profile changes.
Agents perform goal-directed behavior, reason across multiple steps, and invoke tools or APIs. That introduces risks like agent collision, where multiple agents interfere with one another, and a much broader attack surface. I shared an example of a coding agent that spins up execution environments and forks repositories, which is powerful but also introduces serious security considerations.
This is why guardrails, observability, and security design cannot be afterthoughts when operationalizing agents.
Operationalizing AI Requires Partnership
A recurring theme in the conversation was avoiding black-box AI deployments.
Successful implementations require close collaboration between technology teams and the business. Practical, on-the-job training matters far more than generic enablement sessions. People need to learn by using these systems in real workflows, with support and feedback loops in place.
Data quality is another hard constraint. If the underlying data is weak, no amount of model tuning will save the outcome. This is often where the gap between pilots and production becomes visible.
Balancing Confidence and Impact
We also talked about how organizations should balance their AI portfolios.
Low-risk, non-critical workflows are important for building confidence and familiarity. At the same time, high-impact, mission-critical workflows are what ultimately demonstrate value and rally organizational support. Focusing only on one or the other limits progress. The right mix creates momentum while keeping risk manageable.
Measuring What Matters
Finally, we spent time on metrics, because without them, it is impossible to scale responsibly.
Teams should define baseline metrics at the start of an initiative and iterate on them over time. The metrics that tend to matter most are not abstract AI scores, but operational outcomes: cycle time, accuracy, cost per successful task, and end-user value or satisfaction.
Those measures create accountability and provide the feedback needed to improve systems over time.
The Throughline
Looking back, this conversation sits at an important point in the broader AI adoption curve. It reflects a shift from experimentation to execution, from enthusiasm to discipline.
