Play 17
AI Observability
Medium🔧 Skeleton
Monitor AI workloads with KQL, quality alerts, and interactive workbooks.
Instrument your AI applications with Application Insights, query logs with KQL in Log Analytics, set up quality alerts (latency, error rate, token usage, groundedness scores), and build interactive Azure Workbooks dashboards. Distributed tracing tracks requests across AI Search → OpenAI → your app.
Architecture Pattern
KQL dashboards, quality metrics, alerting, APM, distributed tracing
Azure Services
Application InsightsLog AnalyticsAzure MonitorWorkbooks
DevKit (.github Agentic OS)
- agent.md — observability engineer persona
- instructions.md — KQL guide
- plugins/ — KQL generator, alert builder, workbook designer
TuneKit (AI Config)
- config/monitoring.json — KQL queries, alert thresholds, dashboards
- config/metrics.json — quality KPIs
- infra/ — workbook templates
Tuning Parameters
KQL queriesAlert thresholdsQuality metrics definitionsDashboard layouts
Estimated Cost
Dev/Test
$30–80/mo
Production
$200–1K/mo