🤖 AI Engineering — Course Roadmap

Beginner-friendly · assumes only basic programming knowledge

This course takes you from writing simple Python to building real AI applications powered by Large Language Models (LLMs) — chatbots, document Q&A, RAG systems, and AI agents. Every topic is explained from first principles; if you can write a basic for loop, you can follow along.

What You'll Learn (10 Phases)

Phase	Topic	Time
1	Python for Developers — the language of AI	2–3 weeks
2	AI Fundamentals — ML, deep learning, LLMs	2–4 weeks
3	Prompt Engineering — talking to LLMs well	1 week
4	LLM Development — OpenAI, Claude, Gemini APIs	2–3 weeks
5	LangChain / LlamaIndex — AI app frameworks	—
6	Vector Databases — embeddings & semantic search	—
7	RAG — grounding LLMs in your data (most important)	—
8	AI Agents — LLMs that take actions	—
9	AI + Java — Spring AI, LangChain4j	—
10	AI Deployment — Docker, K8s, monitoring, cost	—

Projects You'll Build

Beginner

AI Chatbot · Document Q&A Bot · Resume Analyzer

Intermediate

RAG Knowledge Assistant · Customer Support Assistant · Meeting Summarizer

Advanced

Multi-Agent System · Enterprise Knowledge Assistant · Banking Policy Assistant · AI Code-Review Platform

How to Use This Course

Go phase by phase — each builds on the last.
Type out and run every code example; don't just read.
Look for the green "Try it" boxes — small exercises to cement each idea.

💡 Prerequisite: basic programming in any language (variables, loops, functions, if/else). We teach the Python and AI from scratch. Phases 1–3 are ready now; the rest are on the way.

Phase 1 · Python for Developers

Python Fundamentals

🐍 Phase 1 · 8 min read

Python is the language of AI — almost every AI library is written for it. It's readable, beginner-friendly, and uses indentation (not braces) to group code.

The Building Blocks

# variables — no type declaration needed
name = "Aftab"
age = 30
price = 19.99
is_active = True

# f-strings for formatting
print(f"{name} is {age} years old")

# functions
def greet(person, greeting="Hello"):
    return f"{greeting}, {person}!"

print(greet("Sam"))          # Hello, Sam!

Control Flow & Loops

if age >= 18:
    print("adult")
elif age >= 13:
    print("teen")
else:
    print("child")

for i in range(3):           # 0, 1, 2
    print(i)

names = ["a", "b", "c"]
for n in names:              # loop over a list directly
    print(n)

💡 Python uses indentation (4 spaces) to define blocks — there are no { } or semicolons. Consistent indentation isn't just style; it's required by the language.

✅ Try it: write a function is_even(n) that returns True if a number is even (hint: n % 2 == 0), then loop 1–10 printing each number and whether it's even.

Phase 1 · Python for Developers

OOP in Python

🐍 Phase 1 · 7 min read

Object-Oriented Programming bundles data and behaviour into classes. You'll see it everywhere in AI libraries (e.g. building a chatbot class, an agent class).

class ChatBot:
    def __init__(self, name):      # constructor
        self.name = name           # instance attribute
        self.history = []

    def reply(self, message):      # method
        self.history.append(message)
        return f"{self.name}: I heard '{message}'"

bot = ChatBot("Nova")              # create an object
print(bot.reply("hello"))          # Nova: I heard 'hello'

Key Ideas

__init__ — the constructor, runs when you create an object.
self — refers to the current object (like this in Java/JS).
Inheritance — class SmartBot(ChatBot): reuses and extends a class.

💡 Don't worry about mastering every OOP feature now. You mainly need to read class-based code in AI libraries and create simple classes of your own.

Phase 1 · Python for Developers

Collections

🐍 Phase 1 · 7 min read

Python's four built-in collections hold groups of values. You'll use these constantly — e.g. a list of chat messages, a dict of API parameters.

Type	Looks like	Use for
list	`[1, 2, 3]`	Ordered, changeable sequence
tuple	`(1, 2)`	Ordered, fixed (can't change)
dict	`{"key": "value"}`	Key → value lookups
set	`{1, 2, 3}`	Unique items, no duplicates

messages = ["hi", "how are you?"]   # list
messages.append("bye")
first = messages[0]                 # indexing

config = {"model": "gpt-4", "temperature": 0.7}   # dict
print(config["model"])
config["max_tokens"] = 500          # add a key

# list comprehension — build a list in one line
squares = [n * n for n in range(5)]   # [0, 1, 4, 9, 16]

💡 The dict is the workhorse of AI code — API requests and responses are essentially dicts (JSON). The list comprehension ([x for x in ...]) is a Python superpower you'll use to transform data.

Phase 1 · Python for Developers

Exception Handling

🐍 Phase 1 · 5 min read

When something goes wrong (a network call fails, bad input), Python raises an exception. You handle it with try / except so your program doesn't crash — essential when calling AI APIs that can fail or rate-limit.

try:
    response = call_openai_api(prompt)
except ConnectionError:
    print("Network problem — retrying...")
except Exception as e:              # catch anything else
    print(f"Something failed: {e}")
finally:
    print("done")                   # always runs

💡 AI API calls fail often (timeouts, rate limits, bad keys). Wrapping them in try/except with a retry is a habit you'll build early — a crash mid-conversation is a bad user experience.

✅ Try it: write code that converts user input to an integer with int(text), catching ValueError to print "Please enter a number" instead of crashing.

Phase 1 · Python for Developers

Virtual Environments

🐍 Phase 1 · 5 min read

A virtual environment is an isolated Python setup per project, so each project's libraries don't clash. AI projects need many specific package versions — venvs keep them tidy.

# create & activate a virtual environment
python -m venv venv
source venv/bin/activate        # macOS/Linux  (Windows: venv\Scripts\activate)

# install AI libraries into THIS project only
pip install openai numpy pandas

# save the exact versions for others to reproduce
pip freeze > requirements.txt
pip install -r requirements.txt   # install from the file later

⚠️ Always create a venv before installing packages. Installing globally leads to version conflicts between projects ("dependency hell"). Activate the venv every time you work on the project (your prompt shows (venv)).

Phase 1 · Python for Developers

NumPy

🐍 Phase 1 · 6 min read

NumPy is the foundation of numerical computing in Python. It provides fast arrays (vectors/matrices) — and AI is built on these. An "embedding" (which you'll meet later) is just a NumPy array of numbers.

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(a + b)              # [5 7 9]  — elementwise, no loop
print(a * 2)              # [2 4 6]
print(a.mean())           # 2.0

# dot product — the math behind "similarity" in AI search
print(np.dot(a, b))       # 32

matrix = np.array([[1, 2], [3, 4]])
print(matrix.shape)       # (2, 2)

💡 You won't write much raw NumPy as an AI app developer, but understanding that AI works on arrays of numbers — and that "similarity" is a math operation like the dot product — demystifies embeddings and vector search later.

Phase 1 · Python for Developers

Pandas

🐍 Phase 1 · 6 min read

Pandas handles tabular data (rows & columns, like a spreadsheet) via the DataFrame. You'll use it to load and clean the data you feed into AI — CSVs of documents, Q&A pairs, logs.

import pandas as pd

df = pd.read_csv("faqs.csv")     # load a CSV into a DataFrame
print(df.head())                  # first 5 rows
print(df.shape)                   # (rows, columns)

# select & filter
questions = df["question"]                  # one column
hot = df[df["views"] > 1000]                # rows where views > 1000

# iterate rows to send to an AI model
for _, row in df.iterrows():
    print(row["question"], "→", row["answer"])

💡 In AI workflows Pandas is the "data prep" step: load your raw data, clean it, then turn each row into text to embed or send to an LLM. read_csv, filtering, and iterrows cover 80% of what you'll need.

✅ Try it: create a small CSV with columns question,answer, load it with Pandas, and print every question in a loop. This is exactly the first step of a Document Q&A bot.

Phase 2 · AI Fundamentals

What is AI?

🧠 Phase 2 · 7 min read

Artificial Intelligence (AI) is software that performs tasks we'd normally call "intelligent" — understanding language, recognizing images, making decisions. The terms AI, ML, Deep Learning, and Generative AI are nested, not separate.

How the Terms Nest — Visual

AI — the broad goal of intelligent machines (includes simple rule-based systems).
Machine Learning — AI that learns patterns from data instead of being hand-coded.
Deep Learning — ML using large neural networks.
Generative AI — deep learning that creates new content (text, images). LLMs live here.

💡 As an AI application developer, you'll mostly work at the Generative-AI / LLM layer — using models like GPT, Claude, and Gemini through APIs. You don't need to build the models, just understand and use them well.

Phase 2 · AI Fundamentals

Machine Learning Basics

🧠 Phase 2 · 8 min read

Machine Learning means a program learns from examples rather than following hand-written rules. You show it data, it finds patterns ("trains a model"), then it makes predictions on new data.

Traditional Code vs ML — Visual

Three Types of ML

Type	Learns from	Example
Supervised	Labelled data (input → known answer)	Spam vs not-spam
Unsupervised	Unlabelled data (find structure)	Grouping customers
Reinforcement	Trial & error + rewards	Game-playing, robotics

💡 Key vocabulary: training = learning patterns from data; model = the learned result; inference = using the model to predict. When you "call an LLM," you're doing inference on a model someone else trained.

Phase 2 · AI Fundamentals

Deep Learning Basics

🧠 Phase 2 · 6 min read

Deep Learning is machine learning using neural networks with many layers ("deep"). It's what powers modern AI — image recognition, speech, and the LLMs behind ChatGPT.

Why "Deep" Wins

Learns features automatically — no need to hand-engineer what to look for; layers build up from simple to complex patterns.
Scales with data & compute — more data + bigger networks = better results (this drove the LLM boom).
Needs lots of data & GPUs — the trade-off for that power.

💡 Deep learning is why AI suddenly got good in the 2010s: enough data, powerful GPUs, and better network designs (especially the Transformer in 2017, which made LLMs possible). You'll meet both neural networks and transformers in the next topics.

Phase 2 · AI Fundamentals

Generative AI

🧠 Phase 2 · 6 min read

Generative AI creates new content — text, images, code, audio — rather than just classifying or predicting. ChatGPT writing an essay, DALL·E drawing a picture, GitHub Copilot writing code: all generative AI.

Generates	Example models
Text / code	GPT-4, Claude, Gemini, Llama
Images	DALL·E, Midjourney, Stable Diffusion
Audio / speech	Whisper, ElevenLabs

💡 This course focuses on text-generating models (LLMs) — the foundation of chatbots, assistants, and RAG systems. The core skill is prompting and orchestrating these models from your own applications.

Phase 2 · AI Fundamentals

LLM Fundamentals

🧠 Phase 2 · 8 min read

A Large Language Model (LLM) is a deep-learning model trained on vast amounts of text to do one deceptively simple thing: predict the next word (token). Do that extremely well, and you get models that converse, summarize, translate, and code.

How an LLM Works (Simply)

It predicts one token, appends it, then predicts the next — repeating to generate whole answers. The "intelligence" emerges from patterns learned across billions of sentences.

Key Terms You'll Use Constantly

Term	Meaning
Token	A chunk of text (~¾ of a word) the model reads/writes
Context window	How much text the model can "see" at once
Parameters	The model's learned weights (billions of them)
Temperature	Randomness/creativity of the output (0 = focused, 1 = creative)
Hallucination	When the model confidently states something false

⚠️ LLMs predict plausible text — they don't "know" facts and can hallucinate. They also have a knowledge cutoff and no memory between calls. Phases 4–7 (function calling, RAG) exist largely to fix these limits by grounding the model in real, current data.

Phase 2 · AI Fundamentals

Neural Networks

🧠 Phase 2 · 7 min read

A neural network is the structure behind deep learning — layers of simple units ("neurons") loosely inspired by the brain. Each connection has a weight; learning means adjusting those weights.

Data flows left to right: each neuron multiplies its inputs by weights, sums them, applies a small function, and passes the result on. During training, the network compares its output to the correct answer and nudges the weights to do better next time (this is "backpropagation").

💡 You don't need the math to build AI apps. The takeaway: a neural network is a big stack of weighted connections tuned by training. An LLM is an enormous neural network with a special design — the Transformer (next).

Phase 2 · AI Fundamentals

Transformers

🧠 Phase 2 · 7 min read

The Transformer (Google, 2017 — "Attention Is All You Need") is the neural-network design behind every modern LLM. Its breakthrough is attention: the model weighs how much each word relates to every other word, capturing context.

Attention in Plain English

Attention lets the model link "it" to "animal" by scoring relationships between all words at once — and it does this in parallel, which is why Transformers train fast on huge datasets.

💡 You'll never implement a Transformer by hand. What matters: this architecture is why LLMs understand context so well, why "context window" is a key limit, and why bigger models trained on more text keep getting smarter. That wraps Phase 2 — next you learn to talk to these models effectively.

Phase 3 · Prompt Engineering

Prompt Design

💬 Phase 3 · 8 min read

A prompt is the instruction you give an LLM. Prompt engineering is the skill of writing prompts that reliably get the output you want. It's the highest-leverage skill in AI app development — same model, far better results.

Anatomy of a Good Prompt

# A weak prompt
"Write about dogs"

# A strong prompt — Role + Task + Context + Format + Constraints
"You are a veterinarian. (role)
Write a friendly guide on caring for a new puppy. (task)
The reader is a first-time owner with a 2-month-old Labrador. (context)
Use 5 short bullet points. (format)
Keep it under 120 words, no medical jargon." (constraints)

The Building Blocks

Element	Why it helps
Role ("You are a…")	Sets tone & expertise
Clear task	Removes ambiguity
Context	Grounds the answer in your situation
Output format	Makes results predictable/parseable
Constraints	Length, style, what to avoid

System vs User Prompts

Most chat APIs separate a system prompt (persistent instructions/persona) from user prompts (the actual questions). Put rules and role in the system prompt; put the task in the user message.

💡 Golden rule: be specific. LLMs can't read your mind — vague prompts get vague answers. State the role, the exact task, the format, and the constraints. The rest of this phase covers proven patterns.

Phase 3 · Prompt Engineering

Zero-Shot Prompting

💬 Phase 3 · 4 min read

Zero-shot means you ask the model to do a task without giving any examples — relying on what it already learned during training. It's the simplest and most common approach.

# Zero-shot: just describe the task, no examples
"Classify the sentiment of this review as positive, negative, or neutral:
'The food was cold and the service was slow.'"

# → negative

💡 Try zero-shot first — modern LLMs handle most tasks well with a clear instruction alone. If results are inconsistent or the format is wrong, then add examples (few-shot, next).

Phase 3 · Prompt Engineering

Few-Shot Prompting

💬 Phase 3 · 5 min read

Few-shot means you include a few examples of the task in the prompt, so the model copies the pattern. Great for getting a consistent format or teaching a specific style.

# Few-shot: show examples, then the real input
"Convert product names to a slug.

Input: 'Red Running Shoes'  → red-running-shoes
Input: 'Blue Cotton T-Shirt' → blue-cotton-t-shirt
Input: 'Wireless Noise-Cancelling Headphones' →"

# → wireless-noise-cancelling-headphones

💡 2–5 examples is usually enough. Few-shot dramatically improves consistency for formatting and classification tasks. The trade-off: examples use up tokens (and context window) on every call.

✅ Try it: write a few-shot prompt that turns a sentence into a one-line commit message, using 3 examples (e.g. "fixed the login bug" → "fix: resolve login failure").

Phase 3 · Prompt Engineering

Chain of Thought

💬 Phase 3 · 5 min read

Chain-of-Thought (CoT) prompting asks the model to reason step by step before answering. This dramatically improves accuracy on math, logic, and multi-step problems — the model "thinks out loud."

# Without CoT — often wrong on multi-step problems
"A shop has 23 apples. They sell 18 and buy 12 more. How many now?"

# With CoT — add the magic phrase
"...How many now? Let's think step by step."

# → Start: 23. Sell 18 → 5. Buy 12 → 17. Answer: 17.

💡 The phrase "Let's think step by step" unlocks reasoning in many models. For production, you often hide the reasoning and return only the final answer. Newer "reasoning" models (o-series, etc.) do this internally — but CoT still helps with most models.

Phase 3 · Prompt Engineering

Structured Output

💬 Phase 3 · 6 min read

When your program (not a human) consumes the answer, you need it in a predictable format — usually JSON. Ask the model explicitly, and your code can parse it reliably.

"Extract the person's details from this text and return ONLY valid JSON
with keys: name, age, city. No explanation.

Text: 'Aftab is 30 and lives in Noida.'"

# → { "name": "Aftab", "age": 30, "city": "Noida" }

# then in Python:
import json
data = json.loads(response)     # turn the JSON string into a dict
print(data["city"])              # Noida

💡 Specify the exact keys, say "return ONLY valid JSON, no extra text," and give an example of the shape. Many APIs now offer a JSON mode / structured outputs feature that guarantees valid JSON — you'll use it in Phase 4.

Phase 3 · Prompt Engineering

Prompt Optimization

💬 Phase 3 · 6 min read

Optimization is the loop of improving a prompt through testing until it's reliable, cheap, and fast. Treat prompts like code: iterate, measure, version.

The Optimization Loop

Write a clear prompt (role, task, format, constraints).
Test on several real inputs — including tricky edge cases.
Diagnose failures: wrong format? add an example. Wrong reasoning? add CoT. Too vague? tighten instructions.
Refine & repeat; keep the best version.

Practical Tips

Be explicit over clever — spell out exactly what you want.
Trim tokens — shorter prompts cost less and run faster; remove fluff.
Lower temperature (≈0) for consistent, factual tasks; raise it for creative ones.
Put key instructions first and last — models weight the start and end heavily.
Version your prompts — store them like code so you can compare and roll back.

💡 That completes Phase 3. You can now design, pattern, and refine prompts. Next (Phase 4) you'll call real LLM APIs — OpenAI, Claude, Gemini — from code, and learn tokens, context windows, function calling, and streaming.

✅ Try it: take a vague prompt you've used and rewrite it with role + task + format + constraints. Run both and compare the difference in output quality.

Phase 4 · LLM Development

OpenAI APIs

🔌 Phase 4 · 7 min read

Now you call a real LLM from code. OpenAI's Chat Completions API takes a list of messages and returns the model's reply. This is the pattern every provider follows.

# pip install openai  ·  set OPENAI_API_KEY env var
from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "Explain RAG in one sentence."}
    ],
    temperature=0.7,
)
print(resp.choices[0].message.content)

The Message Roles

Role	Purpose
system	Persistent instructions / persona
user	The human's message
assistant	The model's previous replies (for context)

⚠️ Never hardcode your API key — load it from an environment variable. Keys are secrets; a leaked key can run up huge bills. Wrap calls in try/except (Phase 1) for rate limits and timeouts.

Phase 4 · LLM Development

Claude APIs (Anthropic)

🔌 Phase 4 · 6 min read

Anthropic's Claude is a leading LLM family, strong at reasoning, long context, and following instructions safely. The API is very similar to OpenAI's — messages in, reply out.

# pip install anthropic  ·  set ANTHROPIC_API_KEY
import anthropic
client = anthropic.Anthropic()

msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=500,
    system="You are a concise technical writer.",   # system is a top-level param
    messages=[{"role": "user", "content": "Summarize what an LLM is."}],
)
print(msg.content[0].text)

💡 Two small differences from OpenAI: the system prompt is a separate top-level parameter (not a message), and max_tokens is required. Otherwise the mental model is identical — which is why switching providers is easy.

Phase 4 · LLM Development

Gemini APIs (Google)

🔌 Phase 4 · 5 min read

Google's Gemini models are multimodal (text, images, audio, video) with very large context windows. Again, the same request/response shape.

# pip install google-generativeai  ·  set GOOGLE_API_KEY
import google.generativeai as genai
genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

model = genai.GenerativeModel("gemini-2.0-flash")
resp = model.generate_content("Explain embeddings simply.")
print(resp.text)

💡 All three providers (OpenAI, Claude, Gemini) share the same core idea: send messages, get a completion. Pick based on cost, context size, multimodality, and quality for your task. Frameworks like LangChain (Phase 5) let you swap between them with one line.

Phase 4 · LLM Development

Tokenization

🔌 Phase 4 · 5 min read

LLMs don't read characters or whole words — they read tokens, chunks of text roughly ¾ of a word. Tokenization is how text is split into these chunks. You're billed per token, so this matters.

Rule of thumb: ~1 token ≈ 4 characters ≈ ¾ word. 1,000 tokens ≈ 750 words.
Cost & limits are counted in tokens (input + output).
Count them with tiktoken (OpenAI) before sending, to stay within limits/budget.

💡 Why you care: every API call's price and the context-window limit are measured in tokens. Long prompts and long answers both cost more. Trimming unnecessary text (Phase 3 optimization) directly saves money.

Phase 4 · LLM Development

Context Windows

🔌 Phase 4 · 5 min read

The context window is the maximum number of tokens a model can consider at once — your prompt plus its response must fit inside it. It's the model's working memory for a single call.

Window size	Roughly
8K tokens	~6,000 words
128K tokens	~a 250-page book
1M+ tokens	thousands of pages (Gemini)

⚠️ Two big consequences: (1) LLMs have no memory between calls — to "remember" a conversation you resend the history every time (which costs tokens). (2) You can't paste unlimited data; large documents must be chunked and retrieved (that's exactly what RAG, Phase 7, solves).

Phase 4 · LLM Development

Function Calling (Tool Use)

🔌 Phase 4 · 8 min read

Function calling lets an LLM use your code. You describe functions (tools) it can call; when useful, the model returns a request to call one with arguments — your code runs it and feeds the result back. This is how LLMs fetch live data and take actions.

The Flow — Visual

tools = [{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get current weather for a city",
    "parameters": {"type":"object","properties":{"city":{"type":"string"}}}
  }
}]
resp = client.chat.completions.create(model="gpt-4o-mini",
    messages=messages, tools=tools)
# if resp wants a tool: run get_weather(city), append the result, call again

💡 The LLM never runs code itself — it only asks you to, and you decide whether/how to run it. Function calling is the foundation of AI agents (Phase 8). It turns a text generator into something that can act on the real world.

Phase 4 · LLM Development

JSON Responses

🔌 Phase 4 · 5 min read

When your program consumes the output, you want guaranteed-valid JSON. Modern APIs offer a JSON / structured-output mode that forces the model to return parseable JSON matching a schema — more reliable than just asking nicely (Phase 3).

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"Extract name and age: 'Sam is 25'"}],
    response_format={"type": "json_object"},   # guarantees valid JSON
)
import json
data = json.loads(resp.choices[0].message.content)   # safe to parse

💡 Use JSON mode (or "structured outputs" with a schema) whenever the result feeds code — extraction, classification, form-filling. It removes the "model added extra prose around the JSON" headache.

Phase 4 · LLM Development

Streaming Responses

🔌 Phase 4 · 5 min read

LLMs generate token-by-token, so they can stream the answer as it's produced — that "typing" effect in ChatGPT. Streaming makes apps feel fast because users see words immediately instead of waiting for the whole reply.

stream = client.chat.completions.create(
    model="gpt-4o-mini", messages=messages, stream=True,
)
for chunk in stream:
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)   # print as it arrives

💡 Streaming doesn't make generation faster overall — it improves perceived speed (time-to-first-word). For chatbots it's essential UX. Behind the scenes it uses Server-Sent Events; web frameworks expose it as a streaming HTTP response. That completes Phase 4 — you can now talk to LLMs from code.

Phase 5 · LangChain / LlamaIndex

LangChain Basics

🔗 Phase 5 · 7 min read

LangChain is the most popular framework for building LLM applications. It provides reusable building blocks — model wrappers, prompt templates, chains, memory, agents, and vector-store integrations — so you don't reinvent plumbing for every app.

# pip install langchain langchain-openai
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

model  = ChatOpenAI(model="gpt-4o-mini")
prompt = ChatPromptTemplate.from_template("Explain {topic} to a beginner.")

chain = prompt | model            # compose with the | (pipe) operator
print(chain.invoke({"topic": "RAG"}).content)

💡 LangChain's value is composability and swappable providers — change ChatOpenAI to ChatAnthropic and everything else stays the same. The | pipe syntax (LCEL) connects components into a pipeline. LlamaIndex is a sibling framework focused on data/RAG.

Phase 5 · LangChain / LlamaIndex

Chains

🔗 Phase 5 · 6 min read

A chain links multiple steps into a pipeline — e.g. prompt → model → output parser, or the output of one LLM call feeding the next. Chains let you build multi-step logic declaratively.

from langchain_core.output_parsers import StrOutputParser

# prompt → model → parse to plain string
chain = prompt | model | StrOutputParser()
summary = chain.invoke({"topic": "vector databases"})

# chain two steps: summarize, then translate the summary
pipeline = summarize_chain | translate_chain

💡 Chains turn several LLM/processing steps into one callable. Common patterns: extract → transform → format, or generate → critique → refine. Keep each step focused; compose them.

Phase 5 · LangChain / LlamaIndex

Agents

🔗 Phase 5 · 6 min read

A LangChain agent uses an LLM to decide which tools to call and in what order to accomplish a goal — rather than following a fixed chain. The LLM reasons, picks a tool, sees the result, and decides the next step (the loop you'll formalize in Phase 8).

from langchain.agents import create_tool_calling_agent, AgentExecutor

agent = create_tool_calling_agent(model, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools)
executor.invoke({"input": "What's 23*7, then search the web for that many facts?"})
# the agent decides: call calculator, then call web_search

💡 Chain = fixed sequence you define. Agent = the LLM chooses the steps dynamically. Agents are more flexible but less predictable — use chains when the flow is known, agents when it must adapt.

Phase 5 · LangChain / LlamaIndex

Tools

🔗 Phase 5 · 5 min read

A tool is a function an agent can call — a calculator, web search, database query, or your own API. You define the function and describe it; the agent decides when to use it (this is function calling, packaged conveniently).

from langchain_core.tools import tool

@tool
def get_stock_price(symbol: str) -> float:
    """Get the current price for a stock symbol."""   # description the LLM reads
    return fetch_price(symbol)

tools = [get_stock_price, web_search_tool]   # give them to the agent

💡 The docstring matters — it's how the LLM knows what the tool does and when to use it. Write clear descriptions and typed parameters. Good tools = capable agents.

Phase 5 · LangChain / LlamaIndex

Memory

🔗 Phase 5 · 6 min read

LLMs are stateless — they forget everything between calls. Memory gives a conversation continuity by storing and re-supplying past messages, so the assistant "remembers" earlier turns.

Memory type	How it works
Buffer	Keep the full chat history (simple, grows with tokens)
Windowed	Keep only the last N messages
Summary	Summarize old turns to save tokens

⚠️ Memory isn't magic — it works by resending history in the context window, which costs tokens and has a size limit. For long-term or large memory (a knowledge base), you use retrieval (RAG, Phase 7) instead of stuffing everything into the prompt.

Phase 5 · LangChain / LlamaIndex

LlamaIndex

🔗 Phase 5 · 6 min read

LlamaIndex is a framework specialized for connecting LLMs to your data (RAG). Where LangChain is a broad toolkit, LlamaIndex focuses on ingesting, indexing, and querying documents with minimal code.

# pip install llama-index
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs  = SimpleDirectoryReader("./my_docs").load_data()   # load files
index = VectorStoreIndex.from_documents(docs)            # chunk + embed + store
engine = index.as_query_engine()
print(engine.query("What is our refund policy?"))        # RAG in 4 lines

💡 LlamaIndex collapses the whole RAG pipeline (load → chunk → embed → store → retrieve → answer) into a few lines — great for getting a document Q&A bot working fast. LangChain and LlamaIndex are often used together. You'll learn what's happening under the hood in Phases 6–7.

Phase 6 · Vector Databases

Embeddings

🧮 Phase 6 · 8 min read

An embedding turns a piece of text into a list of numbers (a vector) that captures its meaning. Texts with similar meaning get similar vectors — the foundation of semantic search and RAG.

resp = client.embeddings.create(
    model="text-embedding-3-small",
    input="happy dog",
)
vector = resp.data[0].embedding   # [0.21, -0.08, ...] a list of ~1536 floats

💡 The magic: meaning becomes geometry. "Similar meaning" = "vectors close together" (measured by cosine similarity — remember the dot product from NumPy?). This is what lets you search by concept instead of exact keywords.

Phase 6 · Vector Databases

Semantic Search

🧮 Phase 6 · 6 min read

Semantic search finds results by meaning, not exact words. You embed the query and the documents, then return the documents whose vectors are closest to the query's.

	Keyword search	Semantic search
Matches	Exact words	Meaning / intent
"car" finds	only "car"	"automobile", "vehicle"
Powered by	Inverted index	Embeddings + vector similarity

💡 Example: a user asks "how do I get my money back?" — semantic search finds your "Refund Policy" doc even though it never says "money back." This is the retrieval half of RAG.

Phase 6 · Vector Databases

Vector Search

🧮 Phase 6 · 6 min read

A vector database stores embeddings and finds the nearest neighbors to a query vector — fast, even across millions of vectors. This is the engine under semantic search.

Similarity metric — usually cosine similarity (angle between vectors).
ANN (Approximate Nearest Neighbor) — algorithms like HNSW find "close enough" matches very fast instead of comparing every vector exactly.
top-k — you ask for the k most similar items (e.g. top 5 chunks).

# conceptually:
query_vec = embed("how do refunds work?")
results = vector_db.search(query_vec, top_k=5)   # 5 closest chunks

💡 Exact nearest-neighbor search is slow at scale, so vector DBs use ANN indexes (HNSW, IVF) for millisecond search over millions of vectors — trading a tiny bit of accuracy for huge speed.

Phase 6 · Vector Databases

Pinecone

🧮 Phase 6 · 4 min read

Pinecone is a fully managed, cloud vector database — no servers to run, scales automatically. Popular for production RAG when you want zero ops.

from pinecone import Pinecone
pc = Pinecone(api_key=KEY)
index = pc.Index("docs")
index.upsert([("id1", vector, {"text": chunk})])   # store
index.query(vector=query_vec, top_k=5, include_metadata=True)   # search

💡 Choose Pinecone when you want a managed service and don't want to operate infrastructure. Trade-off: it's a paid cloud product (and your data leaves your servers).

Phase 6 · Vector Databases

ChromaDB

🧮 Phase 6 · 4 min read

Chroma is an open-source, developer-friendly vector database that runs locally (even in-memory) — perfect for learning, prototyping, and small apps.

import chromadb
client = chromadb.Client()
coll = client.create_collection("docs")
coll.add(documents=[chunk], ids=["id1"])     # Chroma embeds for you
coll.query(query_texts=["refund policy?"], n_results=5)

💡 Chroma is the easiest place to start — no API keys, runs on your laptop, and can embed text automatically. Great for the beginner/intermediate projects in this course.

Phase 6 · Vector Databases

Weaviate

🧮 Phase 6 · 4 min read

Weaviate is an open-source vector database (self-hosted or cloud) with strong built-in hybrid search (vector + keyword) and a flexible schema. A solid production option you can run yourself.

💡 Weaviate shines when you want hybrid search out of the box and the option to self-host (data stays on your infrastructure). It sits between Chroma (simplest) and Pinecone (fully managed) on the control-vs-convenience spectrum.

Phase 6 · Vector Databases

FAISS

🧮 Phase 6 · 4 min read

FAISS (Facebook AI Similarity Search) is a high-performance library (not a server) for vector similarity search. It's blazing fast and runs in-process — but you manage storage and metadata yourself.

import faiss, numpy as np
index = faiss.IndexFlatL2(1536)        # dimension of your embeddings
index.add(np.array(vectors))            # add vectors
distances, ids = index.search(query_vec, k=5)   # search

Tool	Best for
Chroma	Learning, local prototypes
FAISS	Fast in-process search, full control
Weaviate	Self-hosted, hybrid search
Pinecone	Managed, zero-ops production

💡 FAISS is a library you embed in your app (no separate database). Pick it for maximum speed/control when you're comfortable handling persistence and metadata yourself. That wraps the vector-DB landscape — next you assemble these into RAG.

Phase 7 · RAG (Most Important)

RAG Fundamentals

📚 Phase 7 · 9 min read

RAG (Retrieval-Augmented Generation) lets an LLM answer questions using your data — documents, policies, a knowledge base — instead of only what it learned in training. You retrieve relevant text, inject it into the prompt, and the LLM generates an answer grounded in it.

The RAG Pipeline — Visual

Why RAG Matters

Up-to-date & private — answer from your latest docs, not stale training data.
Reduces hallucination — the model cites real retrieved text.
Cheaper than fine-tuning — no retraining; just update the document store.
Citations — you can show where each answer came from.

💡 RAG is the single most important pattern in applied AI today — nearly every "chat with your docs / knowledge assistant" product is RAG. The rest of this phase breaks down each step: chunking, embedding, retrieval, injection, and evaluation.

Phase 7 · RAG (Most Important)

Document Chunking

📚 Phase 7 · 6 min read

Documents are too big to embed whole, so you split them into chunks (passages). Chunk quality makes or breaks RAG — too big and retrieval is imprecise; too small and you lose context.

Strategy	How
Fixed-size	e.g. 500 tokens with 50-token overlap
Recursive	Split on paragraphs → sentences (keeps structure)
Semantic	Split where meaning shifts
Document-aware	By headings/sections (Markdown, PDF)

from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_text(document)

💡 Use overlap (chunks share some text) so a sentence split across a boundary isn't lost. Start with ~300–800 tokens + 10–15% overlap, then tune based on retrieval quality. Good chunking is the most underrated RAG lever.

Phase 7 · RAG (Most Important)

Embeddings (in RAG)

📚 Phase 7 · 5 min read

In RAG, you embed every chunk once (ingestion) and store the vectors; at query time you embed the question and find the nearest chunks. The embedding model choice affects retrieval quality and cost.

Same model both sides — embed chunks and queries with the same model, or vectors won't be comparable.
Pick by need — small models (cheap, fast) vs large (more accurate); domain-specific models for specialized text.
Re-embed on change — if you switch embedding models, re-embed your whole corpus.

💡 See the Phase 6 Embeddings topic for the core concept. The RAG-specific point: embeddings are computed once for documents (store them) and once per query — consistency of the model is what makes similarity meaningful.

Phase 7 · RAG (Most Important)

Retrieval Strategies

📚 Phase 7 · 6 min read

Retrieval is choosing which chunks to feed the LLM. Better retrieval = better answers. Several strategies improve on naive top-k.

Strategy	Idea
Top-k similarity	The k closest chunks (baseline)
MMR	Maximize relevance and diversity (avoid duplicates)
Metadata filtering	Restrict by tags (e.g. department, date)
Query expansion	Rephrase/expand the query for better recall
Multi-query	Generate several query variants, merge results

💡 Start with top-k (k≈3–5). If answers miss relevant info, add metadata filters and try MMR or multi-query. Retrieval tuning usually beats prompt tuning for RAG quality.

Phase 7 · RAG (Most Important)

Context Injection

📚 Phase 7 · 6 min read

Context injection is how you place the retrieved chunks into the prompt so the LLM answers from them. The prompt template is critical — it tells the model to use only the provided context and to say "I don't know" otherwise.

prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say "I don't know."

Context:
{retrieved_chunks}

Question: {user_question}
Answer:"""

⚠️ Always instruct the model to stick to the context and admit when it doesn't know — this is your main defense against hallucination in RAG. Also mind the context window: too many/too-long chunks get truncated or dilute the answer ("lost in the middle").

Phase 7 · RAG (Most Important)

Hybrid Search

📚 Phase 7 · 5 min read

Hybrid search combines semantic (vector) search with keyword (BM25) search, getting the best of both — meaning and exact matches (names, codes, acronyms that embeddings miss).

Search	Good at	Weak at
Vector	Meaning, synonyms	Exact IDs, rare terms
Keyword (BM25)	Exact terms, codes	Synonyms, paraphrase
Hybrid	Both — fused scores	—

💡 Hybrid search shines for technical/enterprise content with product codes, names, or jargon (e.g. "error SB-409") that pure semantic search can fumble. Many vector DBs (Weaviate, Pinecone) support it natively.

Phase 7 · RAG (Most Important)

Reranking

📚 Phase 7 · 5 min read

Reranking is a second pass: retrieve a larger set (say top 20), then use a specialized cross-encoder model to re-score them by true relevance to the query, and keep the best few. It sharply improves the quality of what reaches the LLM.

💡 Vector search is fast but approximate; a reranker (e.g. Cohere Rerank, BGE) is slower but more accurate. "Retrieve broad, rerank narrow" is a standard RAG quality boost — fewer, better chunks reach the LLM.

Phase 7 · RAG (Most Important)

RAG Evaluation

📚 Phase 7 · 6 min read

How do you know your RAG system is good? You measure it. RAG has two parts to evaluate — retrieval (did we fetch the right chunks?) and generation (was the answer correct and grounded?).

Metric	Asks
Context Recall	Did retrieval find the relevant info?
Context Precision	Were retrieved chunks actually relevant?
Faithfulness	Is the answer supported by the context (no hallucination)?
Answer Relevance	Does the answer address the question?

💡 Tools like RAGAS and LangSmith automate these (often using an LLM as judge). Build a small test set of question→expected-answer pairs and track these metrics as you tune chunking, retrieval, and prompts. "You can't improve what you don't measure" — this closes the most important phase.

Phase 8 · AI Agents

Agent Architecture

🕹️ Phase 8 · 8 min read

An AI agent is an LLM that runs in a loop: it reasons about a goal, acts by calling a tool, observes the result, and repeats until done. It turns a one-shot text generator into something that can complete multi-step tasks autonomously.

The Reason–Act Loop — Visual

This "ReAct" pattern (Reason + Act) is the heart of agents. The LLM is the brain; tools are its hands; the loop continues until the goal is met or a step limit is hit.

💡 Agents = LLM + tools + a loop + memory of steps so far. They're powerful but can wander or loop forever — always cap the number of steps and validate tool outputs. Frameworks (LangChain, LangGraph, CrewAI) provide the loop machinery.

Phase 8 · AI Agents

Tool Calling

🕹️ Phase 8 · 5 min read

Tool calling is how an agent acts — the same function-calling mechanism from Phase 4, now used repeatedly inside the agent loop. Each turn, the LLM may choose a tool, you run it, and return the result for the next reasoning step.

Tools = web search, calculator, database query, code runner, your APIs.
The agent decides which tool and with what arguments.
Tool results become new observations the agent reasons over.

⚠️ Tools can have real effects (send email, charge money, delete data). Treat agent tool use like untrusted input: validate arguments, use least-privilege permissions, and require confirmation for risky/irreversible actions.

Phase 8 · AI Agents

Multi-Agent Systems

🕹️ Phase 8 · 6 min read

Instead of one do-everything agent, a multi-agent system uses several specialized agents that collaborate — each with its own role, tools, and prompt — often coordinated by an "orchestrator."

# example team for "write a blog post"
Researcher agent  → gathers facts (web search tool)
Writer agent      → drafts the post
Editor agent      → critiques & refines
Orchestrator      → routes work between them

💡 Multi-agent shines for complex tasks that decompose into specialties (research → write → review). Trade-off: more LLM calls (cost/latency) and more ways to go wrong. Start with one agent; add more only when a single agent struggles. Frameworks: CrewAI, AutoGen, LangGraph.

Phase 8 · AI Agents

Agent Workflows

🕹️ Phase 8 · 5 min read

A workflow adds structure to agents — a defined graph of steps (some LLM, some code, some human approval) rather than a free-for-all loop. This makes agents more reliable and predictable for production.

Style	Control	Use when
Free agent (ReAct)	LLM decides everything	Open-ended exploration
Workflow / graph	You define the steps; LLM fills them	Production, repeatable tasks

💡 The industry trend is toward structured workflows (e.g. LangGraph state machines) over fully autonomous agents — you get the LLM's flexibility within guardrails you control. More predictable, easier to debug, safer.

Phase 8 · AI Agents

MCP Concepts (Model Context Protocol)

🕹️ Phase 8 · 5 min read

The Model Context Protocol (MCP) is an open standard for connecting AI assistants to tools and data sources in a uniform way — think "USB-C for AI tools." Instead of custom-coding every integration, an MCP server exposes tools/data that any MCP-aware AI client can use.

MCP server — wraps a data source or tool (files, a database, an API, GitHub).
MCP client — the AI app/assistant that connects to servers.
Benefit — write an integration once; reuse it across many AI tools.

💡 MCP is a fast-emerging standard (introduced by Anthropic, now broadly adopted) that makes agent tool/data integration plug-and-play. It's the connective tissue for the agent ecosystem — learn it as the standard way to give agents capabilities.

Phase 8 · AI Agents

Autonomous Agents

🕹️ Phase 8 · 5 min read

Autonomous agents pursue a high-level goal with minimal human input — planning sub-tasks, executing them, and self-correcting in a long loop (e.g. "research competitors and produce a report").

⚠️ Full autonomy is powerful but risky and still immature: agents can loop, drift off-goal, rack up API cost, or take harmful actions. Best practices: cap steps and budget, keep a human in the loop for important decisions, sandbox tools, and log every action.

💡 In production today, "agentic" usually means constrained workflows with human checkpoints — not fully autonomous bots. Use autonomy where mistakes are cheap and reversible; add guardrails everywhere else. That wraps Phase 8 — next you bring all this into Java/Spring.

Phase 9 · AI + Java

Spring AI

☕ Phase 9 · 7 min read

Spring AI brings LLMs to Java/Spring Boot with the familiar Spring style — auto-configuration, starters, and a portable ChatClient that works across OpenAI, Anthropic, Azure, Ollama, and more.

// pom: spring-ai-openai-spring-boot-starter ; set spring.ai.openai.api-key
@RestController
class ChatController {
    private final ChatClient chat;
    ChatController(ChatClient.Builder builder) { this.chat = builder.build(); }

    @GetMapping("/ask")
    String ask(@RequestParam String q) {
        return chat.prompt().user(q).call().content();
    }
}

💡 Spring AI mirrors the Python concepts you learned — chat clients, prompts, embeddings, vector stores, RAG ("advisors") — but idiomatic to Spring. Swap the provider by changing a starter + property, just like LangChain. Perfect for adding AI to an existing Spring Boot app like this one.

Phase 9 · AI + Java

LangChain4j

☕ Phase 9 · 6 min read

LangChain4j is the Java port of LangChain — chains, tools, agents, memory, and RAG for the JVM. Its standout feature is AI Services: declare an interface and it's auto-implemented by an LLM.

interface Assistant {
    String chat(String message);
}
Assistant assistant = AiServices.builder(Assistant.class)
    .chatLanguageModel(model)
    .chatMemory(MessageWindowChatMemory.withMaxMessages(10))
    .build();
assistant.chat("Hello!");   // the interface is backed by the LLM

💡 Spring AI vs LangChain4j: both are great. Spring AI integrates tightest with Spring Boot conventions; LangChain4j offers the rich LangChain-style toolkit and the elegant declarative AI Services. Either works for the Java path.

Phase 9 · AI + Java

OpenAI Integration

☕ Phase 9 · 5 min read

Connecting a Spring Boot app to OpenAI (or any provider) follows the same secure pattern you use for databases: configuration via properties/env vars, a client bean, and resilient calls.

# application.yml
spring.ai.openai.api-key: ${OPENAI_API_KEY}     # from env, never hardcoded
spring.ai.openai.chat.options.model: gpt-4o-mini
spring.ai.openai.chat.options.temperature: 0.7

Secrets — API key from env var / Secrets Manager, not in code or Git.
Resilience — timeouts, retries, circuit breaker (Resilience4j) around the call.
Async/stream — use stream() for token streaming to the browser.

💡 Everything you learned about Spring config, profiles, and resilience applies directly — an LLM is just another external service to call carefully (treat it like a slow, occasionally-failing, paid API).

Phase 9 · AI + Java

RAG with Spring Boot

☕ Phase 9 · 7 min read

Spring AI makes RAG (Phase 7) a few lines: a VectorStore abstraction (PGVector, Redis, Pinecone, Chroma…) plus a QuestionAnswerAdvisor that retrieves and injects context automatically.

// 1) ingest: chunk → embed → store
vectorStore.add(documents);     // Spring AI handles embedding

// 2) query with retrieval baked in
String answer = chatClient.prompt()
    .advisors(new QuestionAnswerAdvisor(vectorStore))   // auto-RAG
    .user(userQuestion)
    .call().content();

💡 The advisor retrieves the top chunks and injects them into the prompt for you — the full RAG pipeline, Spring-style. Pair with PGVector (Postgres + the pgvector extension) to reuse a database you already run, like this project's Postgres.

Phase 9 · AI + Java

AI Microservices

☕ Phase 9 · 6 min read

In production you often isolate AI logic into its own microservice — so model changes, scaling, and cost are managed independently of your core app.

Dedicated AI service — owns prompts, RAG, model calls; exposes a clean REST API.
Scales separately — AI calls are slow & spiky; scale this service on its own.
Resilience boundary — circuit breakers/timeouts here protect the rest of the system.
Swappable — change models/providers without touching other services.

💡 All your microservices knowledge (service discovery, circuit breakers, async messaging from the Spring Boot tutorial) applies. The AI service is just a specialized microservice that happens to call LLMs and a vector store.

Phase 9 · AI + Java

AI API Gateway

☕ Phase 9 · 6 min read

An AI gateway sits in front of LLM providers and centralizes the cross-cutting concerns of AI calls — keys, routing, rate limits, caching, cost tracking, and fallbacks.

Gateway handles	Why
Key management	One secure place for provider keys
Routing / fallback	Failover from one model/provider to another
Rate limiting & quotas	Protect budget and providers
Caching	Reuse answers to identical prompts (save $)
Cost & usage logging	Track spend per team/feature

💡 It's the API Gateway pattern (Spring Cloud Gateway) specialized for AI. Tools like LiteLLM, Portkey, or a custom gateway give you provider independence, cost control, and observability — important once AI usage (and spend) grows.

Phase 10 · AI Deployment

Docker (for AI apps)

🚀 Phase 10 · 5 min read

Containerize your AI app so it runs identically everywhere. AI services have a few extras: Python dependencies, model/API config, and secrets passed at runtime.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
# run: docker run -e OPENAI_API_KEY=$KEY -p 8000:8000 myai

💡 Pass API keys as runtime env vars (-e / secrets), never bake them into the image. For Java AI services, the multi-stage Spring Boot Dockerfile from the Docker tutorial applies directly.

Phase 10 · AI Deployment

Kubernetes (for AI apps)

🚀 Phase 10 · 5 min read

Kubernetes runs your containerized AI service with scaling, self-healing, and rolling updates. AI workloads have specific needs around secrets, autoscaling, and (for self-hosted models) GPUs.

Secrets — API keys as K8s Secrets / env vars, not in images.
Autoscaling — HPA on CPU/latency; AI traffic is spiky.
Timeouts — LLM calls are slow; tune probes & request timeouts generously.
GPUs — only if self-hosting models (node selectors/taints); not needed when calling APIs.

💡 If you call hosted LLM APIs (OpenAI/Claude/Gemini), your AI service is a normal stateless web app — the Kubernetes tutorial applies as-is. GPUs only matter when you host the model yourself.

Phase 10 · AI Deployment

Model Serving

🚀 Phase 10 · 5 min read

Model serving = running a model so apps can send requests to it. You have two paths:

Approach	Meaning	When
Hosted API	Call OpenAI/Claude/Gemini	Default — no infra, fast start
Self-hosted	Run an open model (Llama, Mistral) yourself	Privacy, cost at scale, offline

Self-hosting tools: Ollama (easy local), vLLM / TGI (high-throughput production serving on GPUs).

💡 Start with hosted APIs — they're cheaper and simpler until you have real scale or strict privacy needs. Self-hosting trades convenience for control and (potentially) lower per-token cost at high volume, but you now own GPUs and ops.

Phase 10 · AI Deployment

Monitoring

🚀 Phase 10 · 5 min read

AI apps need the usual monitoring (latency, errors, throughput) plus AI-specific signals — token usage, cost, and answer quality.

Track	Why
Tokens & cost per request	Spend can spike fast
Latency (esp. p95)	LLM calls are slow & variable
Error/timeout/rate-limit rate	Providers fail intermittently
Quality / feedback (👍👎)	Catch hallucinations & drift
Prompt & trace logs	Debug what the model actually saw

💡 "LLM observability" tools (LangSmith, Langfuse, Helicone) trace each call's prompt, response, tokens, cost, and latency — invaluable for debugging and cost control. Combine with your normal Prometheus/Grafana stack.

Phase 10 · AI Deployment

AI Security

🚀 Phase 10 · 6 min read

LLM apps introduce new risks beyond normal web security. The biggest is prompt injection — malicious text (in user input or retrieved documents) that hijacks the model's instructions.

Risk	Defense
Prompt injection	Separate instructions from data; never trust retrieved/user text as commands
Data leakage	Don't put secrets in prompts; filter PII; pick providers with no-train policies
Excessive agency	Least-privilege tools; confirm risky actions; human-in-the-loop
Output handling	Treat LLM output as untrusted — sanitize before executing/rendering
Cost abuse (DoS)	Rate limits, max tokens, auth on AI endpoints

⚠️ Golden rule: treat both LLM input and output as untrusted. Never let model output run code or SQL unchecked, and never let retrieved documents override your system instructions. See the OWASP Top 10 for LLM Applications.

Phase 10 · AI Deployment

Cost Optimization

🚀 Phase 10 · 6 min read

LLM bills are token-based and can balloon. A few habits keep costs sane without hurting quality.

Right-size the model — use a small/cheap model (e.g. mini/flash) for easy tasks; reserve big models for hard ones.
Trim tokens — concise prompts, retrieve fewer/better chunks (rerank), cap max_tokens.
Cache — reuse answers for identical/similar prompts; use providers' prompt caching.
Batch — group requests where possible.
Route — cheap model first, escalate to a bigger one only if needed.
Monitor spend — track cost per feature; set budget alerts.

💡 The 80/20: pick the smallest model that's good enough, keep prompts/context lean, and cache. These three alone often cut costs by more than half. That completes all 10 phases — now build something!

Projects · Beginner

Beginner Projects

🛠️ Projects · build to learn

Apply Phases 1–4. These cement the basics: calling an LLM, prompting, and handling structured output.

🤖 AI Chatbot

A console/web chat that keeps conversation history and streams replies. Uses: OpenAI/Claude API, message roles, memory, streaming (Phase 4 + 5 memory).

📄 Document Q&A Bot

Paste a document, ask questions about it. Uses: read text, stuff it into the prompt as context, structured prompting. (The "manual RAG" stepping stone before vector DBs.)

📃 Resume Analyzer

Upload a resume + job description; get a match score and suggestions as JSON. Uses: prompt design, JSON mode, extraction (Phase 3 + 4).

💡 Goal: get comfortable calling LLMs from code, designing prompts, and parsing structured output. Keep them small and finish them — a working chatbot teaches more than reading ten topics.

Projects · Intermediate

Intermediate Projects

🛠️ Projects · build to learn

Apply Phases 5–7. Now you add real retrieval — embeddings, a vector DB, and RAG.

📚 RAG Knowledge Assistant

Chat over a folder of your docs (PDFs/markdown). Uses: chunking, embeddings, vector DB (Chroma), retrieval + context injection, citations (Phase 6 + 7).

🎧 Customer Support Assistant

Answers from your help-center articles, escalates when unsure. Uses: RAG + "I don't know" guardrail + hybrid search.

📝 Meeting Summarizer

Transcript → summary + action items + decisions as JSON. Uses: chunking long transcripts, map-reduce summarization, structured output.

💡 Goal: build a complete RAG pipeline end to end and learn to evaluate it (Phase 7). The RAG Knowledge Assistant is the single most valuable project — it's the template behind most real AI products.

Projects · Advanced

Advanced Projects

🛠️ Projects · build to learn

Apply Phases 8–10. Agents, multi-service architecture, Java integration, and production concerns.

🧩 Multi-Agent System

A team of agents (researcher, writer, reviewer) collaborating on a task. Uses: agent loop, tools, multi-agent orchestration, workflows (Phase 8).

🏢 Enterprise Knowledge Assistant

Company-wide RAG with access control, metadata filtering, and citations, deployed as a service. Uses: RAG + hybrid search + reranking + AI microservice (Phase 7 + 9).

🏦 Banking Policy Assistant

Strictly-grounded answers over policy docs with audit logging — accuracy and safety critical. Uses: RAG faithfulness, guardrails, evaluation, AI security (Phase 7 + 10).

🔍 AI-Powered Code Review Platform

Analyzes PRs, flags bugs/smells, suggests fixes. Uses: function/tool calling, structured output, Spring AI / LangChain4j, AI microservice + gateway (Phase 9).

💡 Goal: production-grade AI — agents with guardrails, RAG you can trust, and Java/Spring integration deployed and monitored. The Banking Policy Assistant and Enterprise Knowledge Assistant are exactly the kind of systems companies are hiring for. 🎉 You've reached the end of the roadmap — pick a project and build it!