Monitoring AI in Production training course

Build dashboards to detect model drift, output failure, and anomalies in live AI deployments

JBI training course London UK

"Our tailored course provided a well rounded introduction and also covered some intermediate level topics that we needed to know. Clive gave us some best practice ideas and tips to take away. Fast paced but the instructor never lost any of the delegates"

Brian Leek, Data Analyst, May 2022

Public Courses

17/08/26 - 1 days
£2500 +VAT
Enrol 
28/09/26 - 1 days
£2500 +VAT
Enrol 
09/11/26 - 1 days
£2500 +VAT
Enrol 

Customised Courses

* Train a team
* Tailor content
* Flex dates
From £1200 / day
EDF logo Capita logo Sky logo NHS logo RBS logo BBC logo CISCO logo
JBI training course London UK
  • Define what AI production failure looks like
  • Instrument an AI app for monitoring
  • Detect output quality degradation
  • Identify incoming data drift
  • Build a real-time AI health dashboard
  • Set meaningful alerting thresholds
  • Root-cause an AI regression
  • Implement a user feedback loop
  • Schedule automated evaluation runs
  • Create an on-call incident runbook

Course Details

AI failure taxonomy:

silent degradation, sudden regression, and data drift explained through case studies from real production systems

Instrumentation lab:

adding structured logging, metric emission, and trace IDs to an existing AI application with minimal code changes

Baseline setting:

calculating normal output quality distributions so degradation triggers meaningful alerts rather than noise

Data drift detection:

applying statistical tests to identify when incoming features or text inputs have shifted from the training distribution

 

Dashboard build lab:

constructing a real-time AI health view in Grafana or Power BI from scratch using your instrumented data

Alerting configuration:

setting sensible thresholds, avoiding alert fatigue through grouping, and routing alerts to the right person

Incident investigation lab:

given a set of production logs from a degraded system, participants find and explain the root cause

Feedback loop design:

capturing user corrections, escalations, and negative signals and routing them back into your evaluation pipeline

 

Scheduled evaluation:

running your held-out test set on a nightly schedule and reporting quality score trends in a digest email

Runbook writing workshop:

producing a clear on-call guide that a non-ML engineer can follow when facing an AI production incident

 

JBI training course London UK

Data and MLOps

 

5 star

4.8 out of 5 average

"Our tailored course provided a well rounded introduction and also covered some intermediate level topics that we needed to know. Clive gave us some best practice ideas and tips to take away. Fast paced but the instructor never lost any of the delegates"

Brian Leek, Data Analyst, May 2022



 

 

JBI training course London UK

Certification


Every delegate will be entitled to a certificate of achievement on completion of the course.

If you are missing your certificate - please use the link below to apply - you can also use this link to sign up for the JBI Training newsletter to receive technology tips directly from our instructors - Analytics, AI, ML, DevOps, Web, Backend and Security.
 



More about this course

This practical course teaches how to monitor, evaluate, and maintain AI systems in production to ensure long-term reliability and performance.
Participants will explore common AI failure modes such as drift, regression, and silent degradation using real-world case studies.


The course covers instrumentation techniques for adding logging, metrics, and tracing into AI applications with minimal code changes.
Learners will implement baseline performance measurement and statistical drift detection to identify early signs of model degradation.


Hands-on labs include building monitoring dashboards, configuring alerts, and investigating production incidents using real system data.
The course also focuses on feedback loops, scheduled evaluations, and structured evaluation pipelines to continuously assess AI quality.


By the end of the course, participants will be able to operate, troubleshoot, and maintain production AI systems with confidence and clarity.

FAQs

JBI Training offers four specialist courses in this group designed for data and analytics professionals building AI-ready data infrastructure. Available courses are Building AI Agents and Chatbots in Microsoft 365 Copilot (two days), AI-Ready Data Pipelines (one day), Monitoring AI in Production (one day), and End-to-End Reporting Automation (one day). All courses are available as scheduled classroom sessions in London, as live online instructor-led training, or as customised onsite programmes for data and analytics teams.
The AI-Ready Data Pipelines course is a one-day programme for data engineers and analytics professionals who need to ensure their data infrastructure can support AI and machine learning workloads reliably. It covers what makes a data pipeline AI-ready — including data quality requirements, schema consistency, lineage tracking, and latency considerations — how to design and build pipelines that feed AI models and analytics systems with clean, structured, and timely data, and how to integrate pipeline design with modern data platforms. It is particularly relevant for organisations that have adopted or are planning to adopt AI tools and need to ensure their underlying data infrastructure is fit for purpose.
The Monitoring AI in Production course is a one-day programme covering how to observe, measure, and maintain AI systems once they have been deployed in a live environment. It addresses a gap that many data and engineering teams encounter — AI systems that perform well in testing but degrade in production due to data drift, model decay, infrastructure issues, or changing user behaviour. Topics covered include setting up monitoring frameworks for AI model performance, detecting and responding to data drift and concept drift, defining meaningful metrics and alerting thresholds for AI systems, logging and observability for AI pipelines, and building operational runbooks for AI incidents. It is suited to data engineers, MLOps professionals, and analytics engineers responsible for AI systems in production.
The End-to-End Reporting Automation course is a one-day programme that teaches data and analytics professionals how to automate the full reporting lifecycle — from data ingestion and transformation through to report generation and distribution. It covers automation approaches for routine reporting tasks, integrating data pipelines with reporting tools such as Power BI, scheduling and triggering automated report refreshes and distribution, reducing manual intervention in recurring reporting workflows, and applying automation to ensure reports are always based on current, accurate data. It is suited to BI developers, data analysts, and analytics engineers who spend significant time on manual reporting tasks and want to build more scalable and reliable automated reporting systems.
This two-day course teaches data and analytics professionals how to build AI agents and chatbots within the Microsoft 365 Copilot ecosystem using Copilot Studio and related Microsoft tools. It covers designing conversational agents grounded in organisational data sources such as SharePoint, Dataverse, and SQL, configuring automated workflows triggered by agent interactions, connecting agents to analytics outputs and reporting systems, and deploying agents within Microsoft Teams and other Microsoft 365 channels. It is particularly relevant for data and analytics teams who want to surface data insights and automate data-driven workflows through conversational AI interfaces rather than traditional dashboards and reports.
Data and analytics teams sit at the foundation of any organisation's AI capability — AI models, agents, and analytics tools are only as good as the data that feeds them. AI readiness for data teams means ensuring that data pipelines deliver clean, consistent, well-documented, and timely data; that data quality issues are detected and resolved before they reach AI systems; that monitoring is in place to detect when AI outputs degrade; and that reporting and automation workflows are robust enough to scale alongside growing AI adoption. Organisations that invest in training their data teams on AI-ready infrastructure consistently see better outcomes from their broader AI investments than those that focus only on the AI tools themselves.
Yes. All four courses can be delivered as customised onsite or online programmes for corporate data and analytics teams. Content can be tailored to the team's existing data stack, cloud environment, BI tools, and specific AI adoption objectives. For example, a team building out their first AI-ready data platform can receive training focused on pipeline design and monitoring fundamentals, while a more mature team looking to automate reporting and surface insights through AI agents can receive training focused on those specific workflows. JBI has delivered data and analytics training for teams at organisations including the BBC, NHS, RBS, Sky, EDF, and Capita.
Yes. The intersection of data engineering and AI is one of the fastest-moving areas in technology and JBI's course content in this group is continuously reviewed and updated to reflect the latest developments. This includes updates to Microsoft Copilot Studio and Microsoft 365 AI capabilities, evolving MLOps tooling and monitoring frameworks, new approaches to AI-ready data pipeline design, and the latest best practices in reporting automation across Power BI, Microsoft Fabric, and other modern analytics platforms. Delegates learn skills that are directly applicable to the AI and data infrastructure challenges their organisations are working through today.

CONTACT
+44 (0)20 8446 7555

[email protected]

 

Copyright © 2026 JBI Training. All Rights Reserved.
JB International Training Ltd  -  Company Registration Number: 08458005
Registered Address: Wohl Enterprise Hub, 2B Redbourne Avenue, London, N3 2BS

Modern Slavery Statement & Corporate Policies | Terms & Conditions | Contact Us

POPULAR

AI training courses                                                                        CoPilot training course

Threat modelling training course   Python for data analysts training course

Power BI training course                                   Machine Learning training course

Spring Boot Microservices training course              Terraform training course

Data Storytelling training course                                               C++ training course

Power Automate training course                               Clean Code training course