Interactive edition · 2026
How technical leaders lean into observability to move from intuition‑first AI bets to informed, evidence‑driven AI systems
1
p.01 ·
Datadog is the observability and security platform for cloud applications, unifying infrastructure, application, security, and now AI telemetry, so teams can reduce risk, control costs, and keep AI‑era systems resilient.
Datadog has invested heavily in this future, reinvesting roughly 30% of revenue into R&D and building an AI and data foundation that now serves around 32,000 customers, including 8 of the top 10 AI companies.
This playbook is written for leaders who need to prove AI's impact in production using observability as their decision layer.
Double-click any word to start reading from there.
Before you start
Same playbook, two depths. Every interactive, case study and checklist stays on both tracks. Switch any time from the bar at the top.
Chapter two of six · pp.01–02
2
pp.01–02 ·
In 30 seconds
The conversation moved from pilots to production. Leaders now have to prove AI's revenue, cost, productivity and customer impact with telemetry from live systems, and the gap is widest with agents, which few organizations can yet see, govern or audit.
AI has moved from a boardroom talking point to day-to-day execution pressure. The center of gravity has shifted. Eighteen months ago, the conversation was about pilots. Today it is about production. CIOs, CTOs, and VP- and Head-of-AI leaders are expected to show measurable results: revenue, cost, productivity, and customer experience metrics tied to live systems.
Teams are moving from isolated experiments to production-grade systems that meet the same reliability, governance, and operational standards as critical infrastructure. Stakeholders expect clear evidence of business impact, backed by telemetry from live environments.
The challenge is rarely a lack of AI ideas. It is the absence of a clear execution model for deciding where to focus, how to measure success, and how to improve systems once they are live. Organizations are spending heavily with model providers to power copilots, assistants, and new features, only to discover that adoption is shallow, behavior is hard to explain, and ROI becomes harder to defend when budgets tighten.
The gap between ambition and evidence grows sharper with . Agents are being prototyped across IT, customer service, security, software development, and business operations, but only a small fraction make it into production with the visibility, governance, and human oversight leaders need to trust them at scale. Many organizations still struggle to answer basic questions: what the agent did, which data it touched, and how its behavior affected users or systems.
Datadog's analysis of customer LLM telemetry shows why this challenge is growing. More than 70% of organizations now run three or more models in production, and agent framework adoption nearly doubled year over year. Each added model, framework, and workflow introduces new quality, latency, and cost tradeoffs, yet straightforward optimizations are often missed. Only 28% of use prompt caching even when the underlying model supports it.
The numbers outside Datadog's telemetry point in the same direction. Gartner finds that only 48% of digital initiatives meet their stated targets, with failures tracing back to unclear goals, weak measurement, and limited organizational readiness. AI projects inherit those failure modes and add probabilistic behavior on top. Meanwhile, enterprise AI spend is on track to reach hundreds of billions over the next several years.
Organizations making AI work in production treat it as a disciplined portfolio: every initiative instrumented, evaluated, and tied to business outcomes. Enterprises with a defined AI leader are roughly twice as likely to generate revenue, reduce cost, and improve customer experience from AI because someone is accountable for turning experiments, observability, and governance into an operating model.
Vercel
“The next wave of agent failures won't be about what agents can't do. It'll be about what teams can't observe. Agents need the same production feedback loops we've always expected from great software.”
The core business outcomes remain the same. Leaders still need to generate revenue, deploy safely and efficiently, improve employee productivity, and deliver great customer experiences. What has changed is how those outcomes are pursued. AI coding assistants, agents, and LLM-powered features are accelerating change across the and production environments simultaneously.
AI systems now reach sensitive data across the organization. Industry research finds that 72% of technology professionals believe AI agents present greater risk than traditional machine identities, and 92% say governing them is critical to enterprise security. Yet only 44% of organizations have AI-agent governance policies, and just 52% can track and audit every piece of data an AI agent accesses, leaving nearly half unable to confidently reconstruct AI-driven incidents.
A quieter shift is underway inside engineering organizations. As AI coding assistants write more of the code, the humans operating on it understand less of what was built and why. Dashboards and alerts were sufficient when the engineer who built the service was also the one debugging it. When the engineer has reviewed rather than authored the code, observability must carry more of the cognitive load.
The leadership mandateMove from intuition-driven AI bets to evidence-driven AI systems, keep humans firmly in the loop as automation accelerates, and turn observability data into a decision layer for the business.
Chapter three of six · pp.03–13
3
p.03 ·
In 30 seconds
Four plays name what executives want from AI. Three capabilities name what has to be true in the organization for those wins to repeat. Tap any tile in the map to jump to it.
Most organizations do not lack AI ideas; they lack a simple way to decide where to focus and how to make wins repeatable. These four priority plays give leaders a practical way to connect AI investments to business outcomes, while the three enabling capabilities describe what has to be true in the organization for those wins to stick.
Priority plays business outcomes
These are the outcomes executives already care about; the question is how AI and observability can help you move the needle on each.
Use AI to grow revenue.Focus AI work on experiences and products where impact on adoption, conversion, expansion, or retention can be clearly measured. Deploy AI effectively and efficiently.Treat AI workloads like critical production systems, with SLOs, guardrails, and observability focused on latency, output quality, safety, security, and unit economics. Help engineering ship AI faster and with more confidence.Use AI agents and automation to accelerate the full development lifecycle, from experiment setup and dataset enrichment through testing, deployment, and troubleshooting. Elevate customer experience with AI.Apply AI to improve perceived quality, personalization, and trust across customer journeys, and use telemetry to see when AI lifts metrics like CSAT and NPS.Enabling capabilities what makes the plays sustainable
These capabilities turn isolated AI successes into an operating model; without them, gains tend to remain stuck in pilots and isolated teams.
Keep humans in the loop at AI velocity.As agents investigate, recommend, and sometimes change systems, organizations need clear oversight, audit trails, and decision points so humans remain accountable for outcomes, especially in security‑sensitive and customer‑facing scenarios. Move from intuition‑driven AI to evidence‑driven AI development.Standardize how teams evaluate prompts, models, and architectures using real production data and experiments, so decisions are based on telemetry and business impact rather than opinions or hype. Turn unified observability data into an evidence‑driven decision layer for the business.Bring metrics, traces, logs, digital experience, and AI telemetry into a single platform so leaders can see how AI features affect reliability, risk, cost, and customer experience, and act on insights quickly.Taken together, the four plays describe what leaders want from AI, and the three capabilities describe how to build an organization that can deliver those outcomes consistently, not just on a handful of flagship projects.
Each section below ends its diagnosis with checkable symptoms. Tick the ones you recognize; they build your Pain map in chapter 4.
Four priority plays
Priority 1
p.04 ·
In 30 seconds
Usage spikes at launch, then flattens, and nobody can tie AI activity to conversion or churn. Pick a few AI revenue KPIs, trace AI behavior through the whole stack, and ship every change with an evaluation plan.
What this priority really means:
Shipping an AI feature used to be the milestone. Now teams are being asked what it did to conversion, retention, or expansion. Usage spikes at launch and flattens within weeks because product teams cannot connect AI activity to outcomes that finance and the board care about. Without that connection, AI features compete with everything else on roadmap merit alone, and when leaders ask for ROI, they receive slideware or anecdotes.
This priority is about moving from “we have AI features” to “AI measurably drives adoption, expansion, or net-new revenue.”
0 of 3 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog brings together user-journey monitoring, application traces, infrastructure metrics, and Agent Observability so teams can trace each AI interaction end-to-end, from prompts and agent workflows through backend services to user actions and revenue events. This lets leaders answer the question they are being asked: Which AI variants correlate with higher conversion and lower churn?
Fintool, a financial copilot for institutional investors, discovered that response latency was directly driving customer churn during free trials. Their LLM infrastructure spanned six model providers including Azure, OpenAI, Amazon Bedrock, and Groq, and each user query triggered dozens of LLM calls across classification, retrieval, reranking, and evaluation steps, meaning latency could stem from any one of them. Using Agent Observability to pinpoint bottlenecks across the full pipeline and implement load balancing across providers, Fintool reduced response times by 60 percent, and that improvement drove higher conversion rates during free trials and an 84 percent reduction in customer churn.
Source: Datadog, Fintool case study.
Datadog also provides the experimentation infrastructure to connect AI variants to business outcomes, so teams can run feature-flagged rollouts and structured A/B tests against trusted product, behavioral, and performance metrics.
Four priority plays
Priority 2
p.05 ·
In 30 seconds
Teams without evaluations, guardrails and telemetry ship slower, not faster. Give AI flows their own SLOs, run evaluations before and after every major change, and make cost/quality tradeoffs explicit.
What this priority really means:
Deploy AI that is reliable, safe, and cost‑efficient by building governance, observability, and evaluation into how teams work as capabilities that let them move faster with confidence.
Teams without structured evaluations, guardrails, and telemetry actually ship more slowly because they get stuck in manual review loops, struggle to diagnose regressions, and face long rollback cycles when things go wrong. In practice, this priority means giving AI engineers , guardrails, and observability tools that treat AI systems as first‑class production workloads, so quality and reliability become accelerators rather than afterthoughts.
0 of 3 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog's AI observability traces AI workflows end‑to‑end, capturing prompts, intermediate steps, responses, latency, token usage, and errors, and correlating them with application traces and infrastructure telemetry. This gives teams a single view of how model behavior, configuration changes, and upstream or downstream services affect reliability, performance, and user experience.
For cost visibility, teams can track token usage and estimate costs using model providers' rate cards directly in AI observability. When paired with Datadog's cloud cost management capabilities, leaders gain granular visibility into actual AI spend, such as cost per model, provider, project, or even user segment, so they can connect performance and quality data with real bills and prove the ROI of their AI investments, not just estimate it.
For teams running their own AI infrastructure, GPU monitoring extends coverage to the AI infrastructure layer, providing visibility into GPU utilization, memory usage, temperature, and performance across AI workloads.
Four priority plays
Priority 3
pp.06–07 ·
In 30 seconds
Coding tools arrived; delivery didn't speed up. Measure "faster" as outcomes, not PR counts. Automate the high-toil, low-risk work, draw a clear line for autonomous action, and run a closed loop of offline evaluation, online experiments and monitoring.
What this priority really means:
Engineering leaders care about three things AI can move: shorter delivery timelines for AI-driven products, a higher quality bar on what ships to increase competitive differentiation, and preserved through faster resolution of customer-impacting issues.
matters, but it is one measure among several. AI helps at two ends of the lifecycle. It helps preserve engineering bandwidth in production by speeding up investigation and triage, and accelerating the build-and-iterate loop through experiments, evaluation datasets, and pre-ship quality checks.
AI coding tools have reached most engineering organizations, yet delivery timelines have not measurably shortened, and senior engineers are spending more time reviewing AI-generated changes than they save. Incident investigations still happen the old way because evaluation work falls to a few specialists, creating bottlenecks. The tools have changed, but the underlying friction remains.
A deeper change is less visible. As AI coding agents take on more of the writing, engineers spend less time authoring and more time reviewing. They know less about what is actually running in production. Observability becomes less a debugging tool and more a comprehension tool for understanding systems that engineers are responsible for but did not personally build.
This priority is about increasing the speed at which teams can experiment with prompts and models, evaluate quality and safety, ship improvements to production, and troubleshoot issues when they arise. The organizations that win with AI run tight feedback loops: testing changes against real production behavior, measuring impact, and iterating quickly based on evidence.
0 of 5 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog's AI observability gives teams a platform for structured, repeatable experiments: they can build evaluation datasets from production traffic, run experiments comparing prompts, models, and parameters, and evaluate outputs for quality, safety, and latency‑cost trade-offs from within the same environment they use to monitor production AI systems. This eliminates the need to stitch together separate tools for experimentation, evaluation, and observability, and ensures that test datasets reflect real‑world usage patterns.
With Datadog's experimentation capabilities, teams can go further by running randomized A/B tests and other experiments directly in the platform, tying AI‑driven changes to trusted business, behavioral, and performance metrics without needing to export data or rely on external systems. Together, these capabilities create a full feedback loop: offline evaluation on realistic datasets before deployment, online experiments to measure real impact in production, and continuous monitoring to detect drift and regressions over time.
Customer proof point: Cvent
Cvent is a meetings, events, and hospitality SaaS provider with nearly 4,000 employees and 200,000+ users. When the company pivoted to build a new platform for virtual and hybrid events, they needed a monitoring solution that would let them scale a containerized microservices environment on a tight delivery timeline. With Datadog, Cvent launched the Cvent Attendee Hub from concept to deployment in under six months and supported 28,000 attendees at the first event. Datadog APM identified performance bottlenecks during development; Deployment Tracking flagged regressions as they were introduced.
Source: Datadog, The Business Value of Observability.
Four priority plays
Priority 4
pp.08–09 ·
In 30 seconds
When AI is on the critical path, CX is task completion, trust and continued use. Treat AI flows as first-class journeys with their own success metrics, evaluate continuously, and feed real complaints back into datasets.
What this priority really means:
When AI sits on the critical path of a customer journey, the question is whether the system is accurate, consistent, safe, and on-brand. CX in the AI era is measured by task completion, trust, deflection quality, and continued use.
Yet most organizations are not measuring it that way. NPS and CSAT remain flat or decline as AI usage rises, and support volume climbs, because AI responses are confusing or inconsistent. Product, CX, and engineering teams work from different pictures of what is happening, and a gradual drift in model behavior goes unnoticed until failures show up in tickets or churn.
This priority is about using AI to improve perceived quality, personalization, and trust across customer journeys, and using telemetry to see when AI lifts metrics like CSAT and NPS versus when it introduces friction that needs to be fixed.
0 of 4 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog's AI observability brings together monitoring for both the application layer and the underlying AI infrastructure, giving teams a complete view of how AI experiences perform and impact customers. At the application layer, Agent Observability tiles together DEM, APM, infrastructure metrics, and AI-specific telemetry so teams can trace a customer's journey from initial interaction through AI-powered features to final outcomes such as purchase, support deflection, or churn, and understand where AI helps and where it hurts.
With Agent Observability, teams can analyze prompts and responses to detect drift or quality degradation over time, flag outliers or unsafe outputs before they scale to large user populations, and continuously evaluate tone, accuracy, safety, and brand adherence across AI interactions. It also automatically surfaces behavioral patterns across production traffic, showing teams which interaction types are driving cost spikes, quality degradation, or evaluation blind spots, so evaluation coverage reflects what users do in production. Combined with DEM session replay and user journey analytics, these capabilities give product and CX leaders a complete picture of how AI experiences affect customer behavior and satisfaction.
At the infrastructure layer, Datadog GPU monitoring provides visibility into GPU utilization, memory, temperature, and performance across AI workloads, helping engineering leaders ensure their AI infrastructure can scale reliably and cost-efficiently to support growing demand. Together, Agent Observability and GPU monitoring give leaders confidence that AI systems are observable and reliable from the infrastructure up through customer-facing experiences.
Customer proof point: AppFolio
AppFolio, a real estate management software provider serving 20,000+ customers and 8M+ units, built Realm-X Messages, an LLM-powered inbox that streamlines resident communications. The team chose Datadog Agent Observability to monitor performance, quality, and response evaluations. Early in the rollout, AppFolio found that latency was the primary driver of adoption, and observed a strong correlation between latency and adoption: as the service got faster, adoption increased. Using Agent Observability to identify bottlenecks across LLM chain steps (function calls, document retrieval, model calls) and optimize prompts and architecture, AppFolio reduced latency by 80 to 90 percent and saw a nearly 300 percent increase in adoption.
Source: Datadog, AppFolio case study
“Datadog's solution gave us confidence as we rolled out Realm-X Messages to more customers.”
Three enabling capabilities
Capability 1
p.10 ·
These three enabling capabilities are what turn isolated AI wins into a repeatable operating model: they keep humans in control as agents scale, make AI development evidence‑driven instead of opinion‑driven, and turn observability into a decision layer for both technology and business leaders.
In 30 seconds
Let agents do the repetitive, high-volume work and let humans own the judgment calls. Give every agent an identity, log every action, and put approval checkpoints on high-risk changes.
What this capability really means:
As AI agents investigate, recommend, and sometimes change systems, leaders have to preserve human oversight and clear accountability. The goal is to let agents handle repetitive, high-volume work while humans make judgment calls and own outcomes, with full visibility into what each agent did and why.
0 of 3 symptoms recognized · feeds your Pain map
Datadog pattern
Bits AI SRE, Bits AI Dev Agent, and Bits AI Security Analyst operate over Datadog's unified telemetry and write their findings and actions back into Datadog incidents, case management, and collaboration tools. Bits AI SRE can identify root causes and propose remediation steps in minutes, and Bits AI Security Analyst can cut triage time from 15 minutes to 1–5 minutes while still giving humans full visibility into the reasoning and evidence behind each conclusion.
Within Agent Observability, Datadog's Annotation Queue and Automations provide a systematic way to bring human-in-the-loop feedback into evaluations, ensuring that human judgment and domain expertise continuously improve AI system behavior rather than being collected and ignored. This combination of speed, transparency, and structured human feedback is what keeps humans firmly in the loop as agentic systems scale.
Three enabling capabilities
Capability 2
pp.11–12 ·
In 30 seconds
Stop settling prompt debates with a dozen eyeballed outputs. Test against hundreds of production scenarios, tie results to business metrics, keep the record, and name someone who owns AI.
What this capability really means:
Stop shipping AI features on instinct and cherry-picked demos. Use experiments, evaluations, and telemetry as the decision engine. This means choices about prompts, models, and architectures are grounded in real usage data and business impact, not opinions or market hype.
Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027, driven by escalating costs, unclear business value, and inadequate risk controls. This is the shift from “we think this prompt is better” to “we tested this prompt against 500 production scenarios, measured quality and cost, and here's the data showing it improves conversion by X% while reducing token usage by Y%.” The goal is to build a practice where teams can explain why an AI experience was shipped, demonstrate how it performs under real-world conditions, and confidently decide when to iterate, scale, or roll back.
0 of 5 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog Agent Observability provides the full flywheel for evidence-driven AI development, from robust offline evaluation before deployment to rapid experimentation and iteration in production. Before shipping, teams can build evaluation datasets from production traces, run structured experiments to compare prompts, models, and parameters, and systematically evaluate quality, safety, security, and cost-efficiency in realistic scenarios. This offline evaluation ensures agents and applications are proactively prepared for the range of queries and edge cases they'll face in the wild, catching errors, security risks, and inefficient configurations before they reach users.
Once in production, Datadog enables continuous experimentation and monitoring to detect drift, validate improvements, and iterate quickly based on real user behavior. With Datadog Experiments, teams can run A/B tests directly in the platform, tying AI changes to trusted business, behavioral, and performance metrics without needing separate tools. Combined with DEM and APM, this creates a closed loop: offline evaluation validates changes before deployment, online experiments measure real impact in production, and continuous observability feeds insights back into the next round of evaluation, turning AI development from one-off tests into a disciplined, repeatable cycle that continuously improves agent quality, security, and cost-efficiency.
With 12 samples per prompt, a 95% interval on a pass rate is roughly ±25 points wide, wider than most real differences between two good prompts. At 500 it narrows to about ±4 points. That is the statistical case behind the playbook's “500 production scenarios” example, and why the leadership move asks for datasets derived from production traces rather than hand-picked examples. Editor's note, not from the report.
Three enabling capabilities
Capability 3
pp.12–13 ·
In 30 seconds
Treat observability as a system of record for how the business behaves. One platform, natural-language questions across all telemetry, and a few always-on executive dashboards.
What this capability really means:
Treat observability data as a system of record and a decision layer for how the business actually behaves—not just operational exhaust.
In a world where only 48% of digital initiatives hit their targets, and where CIOs who dynamically reprioritize initiatives and proactively manage risk are 24–51% more likely to be top performers, a single, trustworthy data foundation for decisions is a competitive advantage.
0 of 3 symptoms recognized · feeds your Pain map
Datadog pattern
Datadog's unified platform brings together trillions of metrics, logs, and traces; billions of sessions; and millions of hosts and services into a single environment spanning infrastructure, applications, user journeys, security signals, and AI behavior. The platform layers AI and ML capabilities on top ( anomaly detection, Bits AI agents, and the time-series foundation model) to detect patterns, forecast trends, and surface root causes across the entire stack. By integrating AI observability into the same platform, Datadog becomes a decision layer that helps leaders reprioritize AI work based on measured impact, prove ROI with real data, and align teams around shared evidence rather than competing dashboards and opinions.
p.13 ·
The four priority outcomes give leaders a practical way to use AI to grow revenue, deploy new capabilities safely and efficiently, improve employee productivity, and enhance the customer experience. The three enabling capabilities describe what must be true within the organization for those outcomes to be repeatable at scale.
Taken together, the priorities and capabilities form an AI operating model in which observability, experiments, and agent telemetry inform how leaders set direction, allocate budget, and measure success. Instead of a collection of disconnected AI projects, organizations get a shared language for what “good” looks like and a structure for proving impact in production.
Chapter four of six · pp.13–14
4
p.13 ·
In 30 seconds
Four steps: choose outcomes, pick three or four focus areas where the pain is real, instrument and run one experiment per area, then add automation with guardrails. Build yours below.
Identify which outcomes matter most over the next 12–18 months. This could be revenue growth, safer and more efficient deployments, productivity gains, or better customer experiences. Align with product, data, and security leaders so AI success is defined consistently across teams.
From the four priorities and three enabling capabilities, select the three to four that best support those outcomes in your context. For example, “Use AI to grow revenue,” “Move from intuition‑driven to evidence‑driven AI development,” and “Turn observability into a decision layer.” Use the symptoms and root‑cause patterns in each section to confirm you are focusing on where the pain is real.
Ensure observability covers the applications and AI workloads tied to those focus areas, including infrastructure metrics, APM traces, logs, digital experience data, LLM prompts and responses, agent workflows, and GPU metrics where relevant. Establish at least one evaluation or experiment per focus area using real production scenarios so decisions can move from opinion to evidence.
Introduce Bits AI agents and in areas with strong observability where reducing MTTR or manual toil will have a clear impact, while keeping humans in the loop for high-risk actions. Deploy guardrails at two levels: execution-path guardrails (AI Guard) that monitor and block threats like prompt injection, jailbreaks, and policy violations in real time before they reach production systems or users; and application behavior guardrails that continuously monitor AI system performance, quality, and compliance to rapidly identify and resolve issues impacting customer trust, regulatory requirements, or brand safety. Treat agents as first-class identities with explicit scopes and comprehensive logging so their actions can be reviewed alongside human activity.
p.14 ·
A growing share of AI launches show measurable uplift in adoption, conversion, expansion, or retention, supported by digital experience, APM, and LLM observability data rather than isolated anecdotes. Leaders can see which AI experiences correlate with revenue outcomes, shift investment toward those that work, and retire or rework those that do not.
Watch: uplift on AI-touched journeys
AI‑powered paths meet defined SLOs for latency, error rates, and cost per interaction, and regressions can be quickly traced back to specific model, prompt, configuration, or architecture changes. Unsafe outputs, hallucinations, and prompt regressions are caught by evaluation pipelines and guardrails before they affect large user segments, and cost/quality tradeoffs are made with clear visibility into token usage and model spend.
Watch: SLO attainment, cost per interaction
Investigation and triage times decrease as Bits AI SRE and Bits AI Security Analysts handle repetitive, high‑volume work, reducing MTTR from hours to minutes in many environments. Engineers spend more time building and less time debugging, as AI agents analyze observability data to surface likely root causes and recommended fixes. Productivity is measured by MTTR, time‑to‑resolution, and issues resolved without human investigation.
Watch: MTTR, issues resolved without investigation
CSAT and NPS improve, or at least hold steady, on journeys where AI is in the critical path, and customer‑visible failures become rarer and shorter. Drift in model behavior and low‑quality responses are detected through evaluations and clustering, rather than through support tickets and churn, and AI experiences are tuned as first‑class customer journeys with clear success metrics.
Watch: CSAT/NPS on AI journeys
When these patterns show up together, organizations start to operate more like top‑performing technology leaders: AI work is reprioritized dynamically as conditions change, risks are managed proactively, and financial outcomes from AI are tracked explicitly—mirroring the behaviors analysts associate with the most successful CIOs.
Chapter five of six · pp.14–16
5
pp.14–15 ·
In 30 seconds
Four stages, from demos and anecdotes to structured evaluation, embedded agents and full auditability. Nobody gets to Stage 4 overnight; the point is knowing where you are and what to build next.
To make these outcomes concrete, it helps to look at how organizations typically progress in their use of AI, observability, and agents.
AI projects are opportunistic and evaluated on demos and anecdotes, with little connection to production behavior or business outcomes. Observability for AI workloads is patchy: prompts, responses, and agent behavior are not consistently captured or linked to real users.
Metrics, logs, and traces exist for some AI systems, but experiments, evaluations, and agent telemetry are scattered across tools and teams. AI, SRE, security, and product functions operate from different views of reality, slowing cross‑team decision-making and making it hard to understand cost, risk, and experience trade-offs.
Core observability is consolidated in a unified platform and treated as the default source of truth for AI decisions. Basic evaluation and experimentation frameworks exist for high‑impact AI workloads, and executive dashboards show AI's effect on reliability, cost, and experience.
Changes to prompts, models, and architectures undergo structured evaluation before and after rollout, and AI investments are judged by their observed impact on revenue, risk, productivity, and customer experience. Bits AI agents and AI Guard are embedded into workflows, automating investigation and triage with humans firmly in the loop, and AI‑agent governance ensures full auditability of agent actions and data access.
The goal is not for every organization to reach Stage 4 overnight, but to know where they are today and which capabilities to build next so AI moves from a promising experiment to a proven, observable part of how the business runs.
pp.15–16 ·
Across all four priorities and three enabling capabilities, the through‑line is the same: leaders need clean, rich, real‑time data, a unified SaaS platform, and AI capabilities that keep humans in control while proving impact in production. This 90‑day checklist offers a pragmatic way to get started without trying to boil the ocean.
0 of 9 done
Datadog is investing roughly 30% of its revenue into the data and AI foundation behind this model, as reflected in its trailing twelve‑month R&D spend and its TOTO time‑series foundation model, so leaders can move faster on AI without losing sight of what is happening in their systems and their businesses.
Chapter six of six · p.16
6
p.16 ·
AI is no longer a side project. It is reshaping revenue, reliability, productivity, and customer experience inside almost every organization shipping to production today. The leaders responsible for those outcomes are out of patience for hand-waving. They want telemetry. They want experiments tied to real outcomes. They want to know which agent did what, which data it touched, and what changed.
There is a quiet reframe running underneath the four priorities and three capabilities in this playbook. Observability used to be a reactive layer, but in the AI era, it has become a decision layer. It is the place where teams answer “which variant works,” “where is cost growing faster than value,” “which AI experiences correlate with retention,” and “which agents need to be retired or rebuilt.”
Observability used to be a reactive layer, but in the AI era, it has become a decision layer.
That shift toward strategic observability is the real value of unifying telemetry from infrastructure, applications, user journeys, and AI systems into a single platform. Datadog's platform is built to support that shift with clean, rich data and AI assistance that keeps humans in control. Technical leaders can move faster on AI without losing sight of what is happening across their systems and their business.
The next move is yours. Book a demo to see how Agent Observability, Bits AI, and the broader Datadog platform connect AI behavior to business outcomes, or read more on the Datadog blog about how engineering teams are building reliable AI systems at production scale.
Footnotes 1–11
Footnote numbers follow the printed playbook. Links are the PDF's own; footnote 5 is unlinked in print.
See how Agent Observability, Bits AI, and the broader Datadog platform connect AI behavior to business outcomes.
Interactive edition of “AI Playbook for Engineering Leaders: Building Reliable Systems at AI Velocity” (Datadog, 2026). Text, figures and case studies © Datadog. Editor's notes, gists, glossary and simulations are additions for this edition and are labelled as such. Nothing you enter leaves this browser.
Readout