Developers & Engineers training course

RAG: AI on Your Own Data

Explain why RAG outperforms fine-tuning for most enterprise use cases

• Chunk, clean, and prepare documents for embedding

• Generate and store vector embeddings using OpenAI or open-source models

• Set up and query a vector database (Pinecone, Weaviate, or pgvector)

• Build a retrieval pipeline that fetches relevant context at query time

• Construct prompts that correctly inject retrieved context

• Measure retrieval quality using precision and recall metrics

• Handle conflicting or outdated documents in the knowledge base

• Connect the pipeline to a real internal document store or SharePoint

• Package and deploy the RAG system as a queryable API endpoint 

AI Integration in Legacy Codebases

Identify the right integration points for AI in existing applications

• Wrap AI API calls behind clean internal interfaces and abstractions

• Manage authentication and secrets safely in legacy environments

• Handle latency and async patterns without breaking existing UX

• Add AI-powered features to a Java, .NET, or Python monolith

• Implement graceful degradation when AI services are unavailable

• Write integration tests that cover AI-dependent code paths

Avoid common pitfalls: token limits, hallucination in critical paths

• Version and document AI-integrated modules for future maintainers

• Measure before and after performance impact of AI integration

Testing & Evaluating AI Outputs

• Understand why traditional unit tests are insufficient for AI outputs

• Define evaluation criteria: correctness, relevance, tone, safety

• Build a test dataset of representative prompts and expected responses

• Implement LLM-as-judge evaluation using a secondary model

• Write deterministic checks for structured output fields

• Detect and flag hallucinations using reference-based scoring

• Build a regression suite that runs on every prompt change

• Track evaluation metrics over time in a simple dashboard

• Integrate AI evaluation into a CI/CD pipeline

• Document and communicate AI quality standards to stakeholders

AI Security: Attacks & Defences

• Understand the AI-specific threat landscape and OWASP Top 10 for LLMs

• Execute prompt injection attacks against a live test application

• Identify indirect prompt injection vectors in document and web inputs

• Implement input validation and sanitisation for AI applications

• Prevent sensitive data leakage through system prompt extraction

• Apply output filtering to catch harmful or policy-violating responses

• Understand model supply chain risks and third-party model threats

Harden AI API integrations against credential and key exposure

• Conduct a structured AI security review of an existing application

• Produce a threat model and remediation plan for an AI system

Self-Hosted Models for Sensitive Environments

Compare self-hosted vs. API-based models on cost, control, and capability

• Select the right open-source model for a given use case and hardware

• Install and run Ollama or vLLM on local or on-prem infrastructure

• Quantise a model to fit available hardware without significant quality loss

• Expose the model via a local API compatible with existing code

• Secure the model endpoint within internal network boundaries

• Monitor resource usage: GPU/CPU, memory, and throughput

• Update and version models without service disruption

• Test self-hosted model quality against a cloud baseline

• Document the deployment for handover to an ops or security team