# Inside the Model: LLM Weights Decoded > What weights are, where they live, how they change, and what they cost to run — 8 modules over 4 intensive weeks, for the people who have to size the box and defend the choice. - **Audience:** Infrastructure Leads · Platform & AI Engineers · Architects · Technical Managers evaluating self-hosting - **Level:** Intermediate (infrastructure experience assumed; no maths or ML background required) - **Duration:** 4 weeks intensive · 2 modules/week · 6–8 h/week - **Modules:** 8 - **Pass mark:** 70% - **Interactive version:** https://ragentic.netlify.app/#/courses/llm-weights **This file is generated from the course data by `scripts/build-notes.mjs`. Edit the course data, not this file.** ## The world of this course **Organisation:** Alderbrook Health Your employer for the next four weeks: a private healthcare group — 4,200 staff, six sites, a clinical research arm, and a newly signed data-residency mandate that says patient-derived text may not leave infrastructure Alderbrook controls. Three AI pilots are already running on a hosted API and must now come home. You are the infrastructure lead on Project Homestead: choose the models, size the hardware, prove the files are safe, and defend the bill. The CFO has a competing quote from a vendor and a copy of a blog post that says you can run "a 70B model on a gaming PC". --- ## Phase 1 — What Weights Are (weeks 1) ### Module 1 — Weights Are Just Numbers (And What They Are Not) **Guiding question:** What is actually inside the file I am being asked to host — and what is not in there at all? **Outcome:** Define weights precisely and separate them from the four things they get confused with (activations, gradients, hyperparameters, and the prompt). Explain what weights DO contain (compressed statistical structure) and what they emphatically do NOT (a fact database, a retrieval index, anything after the cutoff) — and derive hallucination and stale knowledge from that single fact. **Decision lens:** Somebody is going to ask you to put "the AI" on a server. What arrives is a directory of large numeric files, a config, and a tokenizer. No database, no rules engine, no facts table — just a few billion decimals that encode how tokens tend to follow other tokens. Every capability and every failure mode of the thing you are hosting falls out of that one sentence. Get it straight now and the other seven modules are arithmetic. **Apply-at-work mission — Baseline: explain the file:** Write your current, unassisted explanation of "what is in a model file" in under 150 words — before this course changes it. Seal it. Then inventory the models your organisation actually touches (hosted APIs and any local ones): for each, note the parameter count, the precision, and where the weights physically live. Most people cannot complete that table on day one. The gaps ARE your syllabus, and Module 8 re-measures against this. **Reflection:** Which of my mental models of "the AI" turned out to be an interface rather than the model? What did I assume was stored in there that is not? #### Resources - [Andrej Karpathy — Intro to Large Language Models](https://www.youtube.com/watch?v=zjkBMFhNj_g) — The one-hour mental model, and the best single hour you can spend this week. Watch for the "two files" framing: a parameters file and a bit of code to run it. That is the thing you are being asked to host. - State the two-file model — a parameters file plus code to run it — and answer "what am I being asked to host?" - Separate weights from activations, gradients, hyperparameters and the prompt in one pass - Explain why nothing after the training cutoff is inside the file - [3Blue1Brown — Transformers, the tech behind LLMs](https://www.3blue1brown.com/lessons/gpt/) — Visual, patient, and mathematically honest. Do not worry about following every step yet — Modules 3 and 4 come back to this. Watch it once now for the shape of the thing. - Hold a visual picture of how a token flows through a transformer - Recognise that most of the diagram is matrices multiplying - Return to it in Modules 3–4 with the shapes grounded - [LLM Visualization — an interactive 3D walkthrough of a working model](https://bbycroft.net/llm) — Click through a real (tiny) GPT and watch numbers move through it. Notice how much of the diagram is just matrices being multiplied — that is genuinely all it is. - Watch real numbers move through a tiny GPT from input to output - See how much of a model is just matrix multiplication - Connect the abstract word "weights" to concrete moving values - [Hugging Face — browse a real model repository](https://huggingface.co/models) — Open any open-weights model and look at "Files and versions". Those multi-gigabyte .safetensors files ARE the model. Get familiar with what a checkpoint looks like before Module 7 takes one apart. - Open a real model's "Files and versions" and identify the .safetensors weights - Tell the weight shards from the config and tokenizer files at a glance - Recognise a checkpoint's actual contents before Module 7 dissects one - [Anthropic — Mapping the mind of a large language model](https://www.anthropic.com/research/mapping-mind-language-model) — Skim for one idea: even the people who build these models need elaborate techniques to work out what the weights represent. Useful antidote to anyone claiming they can just read the numbers. - Explain why even the builders need research tools to read what weights represent - Arm yourself against anyone claiming they can simply read the numbers - Set realistic interpretability expectations for a procurement conversation #### In-world ticket queue > Day one on Project Homestead. The mandate landed nine days ago; the hosted API contract has > seven months to run. Your inbox already contains a vendor quote, an enthusiastic proposal to "fine-tune > our own model", and a one-line question from the CFO that nobody has answered: "What exactly are we > buying?" Nobody on the project can currently say what is inside a model file. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0001 | P1 | Elaine Ferris | CFO: "What exactly are we buying? Explain it without acronyms." | | HMS-0002 | P1 | you, to yourself | Inventory: which models do our three pilots actually use? Nobody knows. | | HMS-0003 | P1 | Yusuf Demir | "Does the model store our patient data inside it?" — board question, needs a real answer | | HMS-0004 | P2 | Greg Sandoval | Cortexa quote received — 3 pages, no configuration detail | #### Project — Sealed Baseline + Model Inventory Two artifacts. First, the sealed baseline: write your current, unassisted explanation of "what is inside a model file" in under 150 words. Do not research it, do not improve it — the value is entirely in its honesty, and Module 8 reopens it. Second, the inventory: list every model your organisation actually touches (hosted APIs, anything local, anything embedded in a product you have bought) and for each record the parameter count, the precision it runs at, where the weights physically live, and who is accountable for it. Mark every cell you cannot fill with a "?" rather than a guess. Most people finish week one with a table that is more question marks than answers — that is the honest starting position and it is the point. **Deliverable:** weights/w01-baseline-and-inventory.md — the sealed 150-word explanation (dated), the model inventory table with question marks left visible, and a short note on which gaps worried you most. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Honest sealed baseline | 25% | A genuine unassisted explanation, written before reading — not a polished summary of the resources. Its errors are the deliverable. | | Complete inventory attempt | 30% | Every model the organisation touches is listed, including the awkward ones embedded in purchased products. | | Question marks preserved | 25% | Unknown cells are marked unknown rather than plausibly filled. Guessing here defeats the exercise entirely. | | Gap reflection | 20% | A clear-eyed note on which unknowns carry real operational or governance risk, and why those specifically. | #### Scenario drills **Drill 1.** Yusuf asks, in a board pre-brief: "If a clinician pastes patient notes into the assistant, does that data become part of the model?" **Task:** Write the answer in under 120 words, correct enough for a DPO and plain enough for a board. Then write the follow-up question you would want Yusuf to ask next — the one about the surrounding service rather than the model. **Drill 2.** A colleague says: "The model was trained on the whole internet, so it basically has a copy of Wikipedia inside it." **Task:** Explain in three sentences why this is wrong, using the size arithmetic. Compare the training corpus size against the checkpoint size, and describe what actually survives that compression. **Drill 3.** Dr Hale asks why the assistant confidently invented a guideline reference that does not exist. **Task:** Explain the mechanism in two sentences without using the word "hallucination", then name the architectural change that would actually reduce it. #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. The most accurate description of "the weights" of an LLM is…** a. A compressed, searchable database of training documents b. A large set of numbers, learned in training, that transform the input c. A rulebook of logic hand-written by the model's engineers d. An index of facts the model looks up at answer time **2. Weights differ from ACTIVATIONS in that weights…** a. They are simply larger numbers than the activations are b. Persist in the file and stay fixed at inference time c. Are kept on disk while activations live in the SSD cache d. They exist only during the model's training process itself **3. GRADIENTS exist…** a. They are shipped inside every single model checkpoint you download b. Only during training — the signal for how to change each weight c. As a compressed form of the model's fully trained weights on disk d. Mainly to speed up the model at inference time **4. A HYPERPARAMETER is…** a. A particularly large and unusually important individual model weight b. A setting humans choose — layer count, hidden size, learning rate c. Simply the single largest number found anywhere in the model d. A parameter that is stored at a higher precision than the others **5. When you send a prompt to a model, the weights…** a. Are updated on the fly to try to remember your prompt b. Are not changed at all — your prompt is only input c. Are partially retrained on each prompt that you send d. Store a copy of your prompt for future recall later **6. Your DPO asks whether patient data used in a prompt is "stored inside the model". The accurate answer is…** a. Yes — it is stored inside the model itself permanently b. No — prompting doesn't alter weights, but the service may retain it c. Yes, but only kept inside the model for about 30 days d. No, and there is genuinely no data-retention concern anywhere at all **7. Training data CAN influence weights in a way that raises privacy concerns when…** a. Never — model training is inherently completely safe b. When data in the training run is repeated or unusual enough to be memorised c. Only once the model has been quantized for serving d. Whenever the context window is made especially large **8. The reason a model confidently states things that are false is best explained by…** a. Occasional intermittent bugs somewhere in the model's inference code b. Weights encode statistical patterns, not verified facts c. The serving machine simply having insufficient RAM d. Poor or overly aggressive quantization of the model's weights on disk **9. A model's "knowledge cutoff" exists because…** a. The providers deliberately throttle and limit what it will tell you b. Weights were fixed at training's end; later facts aren't in them c. The available context window is simply too small for it d. The model's content licences expire after a set date **10. Which of these is NOT contained in a model checkpoint?** a. Learned weight matrices for each of the layers b. A searchable copy of the training corpus c. Tensor names, their shapes and their dtypes d. Configuration describing the model architecture **11. The CFO asks why you cannot simply "look inside and see what it knows about us". The honest answer is…** a. You can indeed, provided you have exactly the right tool for it b. Individual weights don't map cleanly to individual facts c. The checkpoint file itself is strongly encrypted d. Only the model's original vendor holds the key needed to open it **12. For your self-hosting programme, the practical consequence of "weights are frozen at inference" is…** a. You must retrain the whole model roughly every week b. The same checkpoint behaves reproducibly; change comes from your stack c. The model will slowly degrade in quality over time d. You cannot safely use this model for clinical work ### Module 2 — Tensors, Matmuls & Where the Parameter Count Comes From **Guiding question:** Where does "7 billion parameters" come from, and can I derive it myself from a config file? **Outcome:** Read tensor shapes fluently, understand matrix multiplication as information mixing (and why it dominates the compute bill), and compute a model's parameter count from its config.json to within a fraction of a percent. Then handle the modern complication: mixture-of-experts models where total parameters and active parameters are different numbers — one sets your memory, the other sets your speed. **Decision lens:** Vendors quote parameter counts like horsepower, and buyers nod. But the number is derivable: vocabulary × hidden size, plus layers × (attention + MLP), plus a rounding of small vectors. Twenty minutes with a config.json tells you whether "7B" is honest, whether it is really an MoE quoting its active count, and roughly what it will weigh on disk. That arithmetic is the entire basis of every sizing conversation you will have. **Apply-at-work mission — Derive the number yourself:** Pick a real open-weights model with a published config. From hidden size, layer count, vocab size, intermediate size and head counts, compute the parameter count by hand (or with the course calculator). Compare to the published figure and account for the difference — tied embeddings, biases, layer norms, expert counts. Then repeat for an MoE model and report BOTH totals: parameters held in memory vs parameters active per token. 🎯 Starts Capstone Milestone 1. **Reflection:** Where did my hand-computed number diverge from the published one, and what did chasing that gap teach me about how these models are actually assembled? #### Resources - [Hugging Face — model config.json, explained through the model card spec](https://huggingface.co/docs/hub/en/model-cards) — Model cards and configs are where the honest numbers live. Open the config.json of any model alongside this and identify hidden_size, num_hidden_layers, intermediate_size, vocab_size. - Open a model's config.json and find the numbers that set its size - Read hidden size, layer count and vocabulary straight from the config - Reconcile a config against the model card's headline parameter count - [Karpathy — nanoGPT (the whole model in ~300 lines)](https://github.com/karpathy/nanoGPT) — Read model.py once. You do not need to run it. Seeing the entire architecture fit on two screens is the fastest cure for treating models as unknowable. - See an entire transformer architecture fit on two screens - Trace the model definition without needing to run it - Demystify "the architecture" by reading ~300 lines - [Mixture of Experts explained — Hugging Face](https://huggingface.co/blog/moe) — The essential modern complication: total parameters set your memory bill, active parameters set your compute bill, and vendors quote whichever flatters them. - Explain why total parameters set the memory bill and active parameters the compute bill - Say why all experts must be resident even though few fire per token - Spot when a headline number hides a total-vs-active sleight of hand - [OpenAI gpt-oss model card (117B total, 5.1B active)](https://huggingface.co/openai/gpt-oss-120b) — A concrete MoE to practise on: read the config and reconcile the headline number with what you would actually have to fit in memory. - Reconcile "117B total / 5.1B active" with what you'd actually fit in VRAM - Read a real mixture-of-experts config and size it honestly - Budget memory for the total count, never the active one - [Scaling Laws for Neural Language Models (Kaplan et al.)](https://arxiv.org/abs/2001.08361) — Skim the abstract and figures only. Context for why parameter count became the industry's favourite headline number — and why it was never the whole story. - Understand why parameter count became the industry's favourite headline - Read the scaling-law figures for what they do and do not promise - Resist treating parameter count as a proxy for quality #### In-world ticket queue > Cortexa's quote says "70B-class model" and Anika forwarded a paper about a model quoting > "22B active". The CFO has noticed the two numbers and wants to know why a 70B costs more than a 235B. > Parameter counting stops being academic this week. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0011 | P1 | this week's milestone | Derive the parameter count from config.json for three candidate models | | HMS-0012 | P1 | Elaine Ferris | "Why is the 235B model cheaper to run than the 70B?" — CFO, genuinely confused | | HMS-0013 | P2 | Greg Sandoval | Cortexa spec sheet quotes active parameters in the headline, total in the footnote | #### Project — Parameter Audit of Three Real Models (Capstone Milestone 1) Pick three candidate models for an Alderbrook-style deployment: one small dense model (roughly 7–9B), one larger dense model, and one mixture-of-experts model. For each, fetch the real config.json and derive the parameter count by hand or with the course calculator — embeddings, per-layer attention, per-layer MLP, output head, norms. Compare each derived figure with the published one and account for every discrepancy you find: tied embeddings, bias terms, SwiGLU's third matrix, grouped-query attention shrinking the K and V projections, expert counts. For the MoE model report BOTH numbers and state plainly which one determines the GPU you must buy and which determines how fast it will run. 🎯 Starts Capstone Milestone 1. **Deliverable:** weights/w02-parameter-audit.md — three derivation tables, the reconciliation notes for each discrepancy, and a one-paragraph answer to "why is the 235B model cheaper to serve than the 70B?" written for the CFO. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Derivations are genuinely worked | 30% | Component-by-component arithmetic from the real config, not the published total copied back with steps invented around it. | | Discrepancies reconciled | 25% | Every gap between derived and published figures is explained by a specific architectural cause and the residual is under about 1%. | | MoE handled correctly | 25% | Total and active parameters both reported, with memory versus compute consequences stated explicitly and correctly. | | CFO-ready explanation | 20% | The closing paragraph is accurate and jargon-free — a finance reader could repeat it correctly to somebody else. | #### Scenario drills **Drill 1.** Cortexa's spec sheet headline reads "70B-class capability"; the footnote says "7B active parameters". **Task:** Work out what architecture this implies, what you now need to ask for, and what the memory requirement is likely to be relative to the headline. Draft the two-sentence email asking for the missing number. **Drill 2.** Elaine: "Explain to me why the bigger model is the cheaper one to run. I assume somebody has made an error." **Task:** Write the explanation in under 100 words, no jargon, using an analogy that survives being repeated by her to somebody else. **Drill 3.** Your derived parameter count for a candidate model is 8.4% below the published figure. **Task:** List the five most likely causes in order of probability and describe the check for each. Then decide at what residual discrepancy you would stop investigating and why. #### Prompt clinic — Deriving the real parameter count (HMS-0011) - **Weak:** "How many parameters does this model have?" - **Average:** "Given this config.json, work out how many parameters this model has." - **Good:** "Given this config.json, derive the total parameter count component by component: embedding table, per-layer attention projections, per-layer MLP, norms, and output head. Show the arithmetic for each and give a total." - **Excellent:** "Act as a model architecture analyst. From this config.json, derive the parameter count component by component, showing arithmetic: (1) embedding table = vocab x hidden; (2) per-layer attention — treat q/k/v/o separately and account for grouped-query attention if num_key_value_heads differs from num_attention_heads; (3) per-layer MLP — check whether this is a SwiGLU architecture needing THREE matrices rather than two; (4) norms; (5) output head — state explicitly whether embeddings are tied, and if untied, count it separately. Multiply per-layer figures by num_hidden_layers. Give a total, then compare against the published figure of and account for any discrepancy with a specific architectural cause. Flag any assumption you had to make because the config does not state it." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. A tensor is best described as…** a. A type of GPU b. A numeric array defined by its shape c. A compressed version of a weight tensor stored on disk d. A single unit of computation carried out on the GPU **2. In matrix multiplication, an input of shape [batch, 4096] multiplied by a weight of shape [4096, 14336] produces…** a. [batch, 4096] b. [batch, 14336] c. [4096, 14336] d. [batch, 4096, 14336] **3. The embedding table of a model with vocabulary 128,256 and hidden size 4,096 contains…** a. About 525 million parameters b. About 132 thousand parameters c. Roughly 4 billion parameters in total d. Around 32 million parameters overall **4. For a dense model, total parameters are dominated by…** a. The embedding table b. The repeated transformer layers — layer count multiplied by per-layer parameters c. Layer norms d. The tokenizer **5. In a mixture-of-experts model, "total parameters" and "active parameters" differ because…** a. Some parameters are corrupted b. Only a small subset of experts is routed to for each token, so most parameters sit unused per token — but all must still be held in memory c. Active parameters are stored at higher precision d. Total parameters include the tokenizer **6. gpt-oss-120b is described as 117B total with 5.1B active. To host it you must budget memory for…** a. About 5.1B parameters only b. 117B parameters c. The average of the two d. 5.1B multiplied by the number of experts **7. The CFO asks why a 235B MoE can be cheaper to serve than a 70B dense model. The correct core of the answer is…** a. MoE models are smaller on disk b. It activates fewer parameters per token, so it computes faster c. MoE models are simply a great deal smaller on disk than any dense model is d. The 70B is quantized **8. Tied embeddings mean…** a. The embedding table is frozen during training b. The input embedding matrix and the output projection share the same weights, so the output head adds no additional parameters c. Embeddings are quantized d. Two models share a tokenizer **9. Your derived parameter count is 3% below the published figure. The most likely explanation is…** a. The publisher lied b. You omitted a component, like the norms c. Quantization silently changed the total count d. Parameter counts are only ever approximate by their nature **10. Grouped-query attention reduces parameters and memory by…** a. Removing attention entirely b. Using fewer key and value heads than query heads c. Quantizing all of the attention weights to a lower precision d. Sharing weights between the layers **11. A vendor quotes "70B-class performance" for their appliance but the footnote says 7B active. This tells you…** a. Nothing useful b. It is an MoE model; ask for its total count c. The model has clearly been quantized down to just 7 bits d. They must be referring to seventy billion training tokens **12. The most reliable way to learn a model's true size is…** a. The vendor's public marketing page for the model b. Config plus on-disk file sizes c. The model's name as given by the publisher d. Simply asking the model how big it is --- ## Phase 2 — Where Weights Live (weeks 2) ### Module 3 — Tokens & Embeddings: The First Weight Table **Guiding question:** How does text become numbers, and why is the very first weight matrix often the largest single tensor in the model? **Outcome:** Trace text → tokens → token IDs → embedding vectors, understand the embedding matrix as a learned lookup table sized vocab × hidden, and explain why tokenisation choices leak into cost, context limits, and non-English performance. Distinguish an embedding MODEL (for RAG) from an LLM's embedding LAYER — a confusion that derails retrieval projects. **Decision lens:** The embedding table is where "meaning" first becomes arithmetic: every token in the vocabulary owns a row of learned numbers, and rows that behave alike end up near each other. It is also, in smaller models, the single biggest tensor you will load — 50,000 vocabulary rows is not free. And when your RAG project stalls because someone used the chat model where an embedding model belonged, this is the module that explains why. **Apply-at-work mission — Tokenise your own domain:** Run genuinely representative text from your organisation through a tokenizer playground: product names, hostnames, error codes, clinical or financial terms, and non-English content if you have it. Record the token-per-word ratio for each category against plain English prose. Identify the three worst offenders — the strings that shatter into many tokens — and calculate what that inflation does to a 128k context window and a per-token bill. **Reflection:** Which of my organisation's everyday strings turned out to be expensive in tokens, and what does that imply for how we should feed this model? #### Resources - [Tiktokenizer — watch text become token IDs, live](https://tiktokenizer.vercel.app/) — The single most useful tool this week. Paste your own domain text — drug names, hostnames, error codes, non-English — and watch where it shatters. Do this before the project. - Watch your own domain text become token IDs live - See how drug names, hostnames or non-English text tokenise — and cost - Estimate a prompt's token count before it hits your bill - [Hugging Face — Tokenizers, summary of the algorithms](https://huggingface.co/docs/transformers/en/tokenizer_summary) — BPE, WordPiece, Unigram: what subword tokenisation actually does and why every model ships its own tokenizer that you must not mix up. - Distinguish BPE, WordPiece and Unigram subword tokenisation - Explain why every model ships its own tokenizer you must match - Anticipate where a tokenizer will fragment your domain vocabulary - [Karpathy — Let's build the GPT Tokenizer](https://www.youtube.com/watch?v=zduSFxRajkE) — Two hours, and worth it if tokenisation is going to bite your cost model. At minimum watch the first 30 minutes on why tokenisation causes so many strange model behaviours. - Explain why tokenisation drives cost and some model quirks - Trace how raw text becomes the IDs the model actually sees - Decide whether tokenisation will bite your cost model - [Massive Text Embedding Benchmark (MTEB) leaderboard](https://huggingface.co/spaces/mteb/leaderboard) — Where you choose an EMBEDDING model for retrieval — a different job, different models, different sizes from your chat model. Bookmark this for the retrieval conversation in Module 6. - Choose an embedding model for retrieval as a decision separate from your chat model - Read the leaderboard for size and task fit, not just rank - Keep "embedding model" and "embedding layer" cleanly apart - [Hugging Face — Sentence Transformers documentation](https://sbert.net/) — The practical toolkit for embedding models. Read the opening page for the crucial framing: these models produce one vector per passage for comparison, not text continuations. - Explain that an embedding model turns text into one vector for retrieval - Tell an embedding model's job apart from a chat model's - Pick a practically-sized embedding model for a RAG pipeline #### In-world ticket queue > The summarisation pilot is over budget on tokens and nobody knows why. Meanwhile the retrieval > project has stalled: someone built the policy library index using the chat model instead of an embedding > model, and the results are quietly terrible. Both problems live in this week's material. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0021 | P1 | Elaine Ferris | Token spend 3× forecast on clinical text — investigate before renewal | | HMS-0022 | P1 | Dr Anika Rao | Policy library retrieval returning nonsense — index built with the wrong model | | HMS-0023 | P2 | pilot telemetry | Drug names and NHS numbers shattering into many tokens — cost implication? | #### Project — Tokenisation Cost Study for Your Domain Take genuinely representative text from your organisation — at least five categories, such as clinical or technical prose, product and drug names, hostnames and identifiers, error codes and logs, tabular data, and non-English content if you have any. Run each through a tokenizer playground and record tokens-per-word against plain English prose as your control. Identify the three worst offenders and work out why they shatter: unusual casing, digits, punctuation, or simply being absent from a vocabulary built mostly on web English. Then convert the finding into money and capacity: what does that inflation do to the effective size of a 128k context window, and to a per-token bill at your projected volume? Finish with one concrete mitigation you could actually implement. **Deliverable:** weights/w03-tokenisation-study.md — the ratio table across categories, the three worst offenders with explanations, the context-window and cost impact, and one implementable mitigation. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Genuinely representative sample | 25% | Real organisational text across at least five categories, not generic examples — the strings your systems actually process. | | Measured, not estimated | 25% | Actual token counts from a real tokenizer, with the model or encoding named, and a stated English-prose control for comparison. | | Explains the why | 25% | The worst offenders are diagnosed to a cause, not just listed — vocabulary coverage, digits, casing, or script. | | Converted to consequence | 25% | Impact expressed in effective context window and cost at real volume, with one mitigation that could survive contact with your architecture. | #### Scenario drills **Drill 1.** Token spend on clinical summarisation is running at triple the forecast. **Task:** Design the investigation: what would you measure first, what three hypotheses would you test, and what evidence would distinguish them? Include at least one hypothesis that is not about tokenisation. **Drill 2.** The policy library retrieval index was built with the chat model rather than an embedding model. Results are poor but nothing errored. **Task:** Explain to Anika why it silently produced poor results rather than failing, and outline what rebuilding correctly involves — including how you would demonstrate the improvement rather than assert it. **Drill 3.** A colleague proposes standardising on one model to "keep token costs predictable across languages". **Task:** Test the assumption: what would you measure to establish whether one model is genuinely more efficient on your language mix, and what would make the comparison unfair? #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. A token is…** a. Always exactly one word b. A subword unit from the model's fixed vocabulary c. A single character d. A short sentence fragment of some fixed, predetermined length **2. The embedding table maps…** a. Individual words directly to their dictionary definitions b. Token IDs to learned vectors c. Tokens straight to other, related tokens d. Whole prompts directly to their responses **3. Two tokens with similar embedding vectors are similar in that they…** a. They happen to be spelled in a broadly similar way b. They appear in similar training contexts c. Have adjacent token IDs d. They belong to one and the same underlying language **4. In GPT-2 small (124,439,808 parameters), the token embedding table of 50,257 × 768 accounts for roughly…** a. About 3% of all the parameters b. 31% of all parameters c. About 75% of all the parameters d. A little under 1% of all parameters **5. Clinical terms, drug names, and hostnames tend to consume more tokens per word because…** a. They are longer strings b. They are rare in the tokenizer's training text, so they fragment c. They contain capital letters d. Tokenizers actively deprioritise any specialist technical text like this **6. Your token spend is triple the forecast on clinical text. The most likely cause is…** a. A billing fault b. The forecast used an English-prose ratio c. The context window is simply far too small for this d. The model you chose is really much too large for this **7. An EMBEDDING MODEL differs from an LLM's embedding LAYER in that the embedding model…** a. Is the same thing under a different name b. Is a separate trained model producing one vector for a whole passage, for similarity search — while the embedding layer is one lookup table inside an LLM mapping single tokens c. Is always larger d. Only works on images **8. Building a retrieval index using a chat model instead of a dedicated embedding model typically produces…** a. Noticeably better results, but at a good deal more cost b. Poor retrieval quality from untrained vectors c. Identical results d. A clear licensing violation of the model's terms **9. Position information matters because…** a. It compresses the sequence b. Attention alone is order-agnostic, so word order must be injected — via learned position embeddings or a scheme such as rotary embeddings c. It reduces tokenisation cost d. It prevents hallucination **10. Vocabulary size trades off…** a. Speed against accuracy only b. Fewer tokens per document against a bigger embedding table c. Memory against licensing d. Nothing at all — a bigger vocabulary is simply always better **11. The same text tokenised by two different models will…** a. Always produce identical token counts b. Generally produce different counts c. Differ only in the ordering of the tokens d. Differ only when it comes to non-English text **12. A practical mitigation for token-hungry domain text is…** a. Increasing the temperature b. Restructuring inputs so repeated boilerplate and identifiers are abbreviated or referenced rather than repeated in full each time c. Switching to FP32 d. Adding more layers ### Module 4 — Attention & MLP: Where the Rest of the Weights Sit **Guiding question:** Which matrices are learned and frozen at inference, and which numbers are computed fresh for every single prompt? **Outcome:** Map the transformer block: Q, K, V and output projections plus the MLP/feed-forward pair, and locate where the majority of parameters actually live (the MLP, not attention). Then hold the distinction this course is built around — Q/K/V are LEARNED WEIGHTS that ship in the file and never change at inference; the attention map is COMPUTED at runtime for every prompt and thrown away. Also: what multi-head means, and why grouped-query attention exists. **Decision lens:** This is the misconception that separates people who can reason about model behaviour from people repeating vendor copy. "Attention weights" is a phrase doing double duty for two completely different things: frozen learned matrices in your checkpoint, and a fresh matrix of relevance scores generated per prompt. Confusing them makes model behaviour seem magical. Separating them makes the next four modules — training, LoRA, KV cache, quantization — obvious. **Apply-at-work mission — Map a real architecture:** Take one real open model and produce an annotated architecture map: for each tensor group (embeddings, per-layer attention projections, MLP, norms, output head) record the shape, the parameter count, and the percentage of the total. Mark each as learned-and-frozen or computed-at-runtime. Your map should answer, with numbers: what fraction of this model is attention, what fraction is MLP, and what fraction is just the vocabulary. 🎯 Capstone Milestone 2. **Reflection:** Before this module, where did I assume the model's "intelligence" was concentrated — and where is the parameter mass actually sitting? #### Resources - [3Blue1Brown — Attention in transformers, step by step](https://www.youtube.com/watch?v=eMlx5fFNoYc) — The clearest visual explanation of query, key, and value that exists. Watch specifically for which quantities are learned matrices and which are computed per prompt — that is this module's whole argument. - Identify which of Q, K, V are learned weights and which are computed per prompt - Explain query, key and value in your own words - Separate the frozen projection weights from the per-prompt attention map - [The Illustrated Transformer — Jay Alammar](https://jalammar.github.io/illustrated-transformer/) — The classic diagram-led walkthrough. Still the best static reference for the shape of a block; read it alongside the parameter table you built in Module 2. - Read the shape of a transformer block from the classic diagram - Locate the Q/K/V and MLP weights within a block - Use it as a static reference for the parameter-count lab - [Transformer Explainer — an interactive live model in your browser](https://poloclub.github.io/transformer-explainer/) — Type a sentence and watch real attention patterns form. Change the words and watch them change — the pattern is computed for YOUR input, which is precisely the point. - Watch real attention patterns form and change as you edit words - See that attention maps are computed per prompt, not stored - Connect the live pattern to the frozen weights that produce it - [In-context Learning and Induction Heads — Anthropic (Transformer Circuits)](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html) — The research behind the course's attention lab. Induction heads need two layers composing — a previous-token head feeding an induction head — which is why the one-layer model in the lab cannot solve the task. - Explain why an induction head needs two layers composing - Understand the research behind the course's attention lab - Say why depth buys compositions a single layer cannot manage - [Attention Is All You Need (the original paper)](https://arxiv.org/abs/1706.03762) — Read section 3.2 for the scaled dot-product attention definition. Worth ten minutes just to see how small the actual idea is underneath eight years of noise. - Read the scaled dot-product attention definition in section 3.2 - See how small the core idea actually is - Ground the buzzword in the original one-line mechanism #### In-world ticket queue > Martin has asked, reasonably, how the model "decides what matters" in a discharge summary. > Anika answered "attention weights" and Greg's slide deck says the appliance "lets you inspect the model's > attention weights for explainability". Two of those three statements are doing something dishonest. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0031 | P1 | Dr Martin Hale | Clinical Director: "How does it decide what matters in a summary?" | | HMS-0032 | P1 | Greg Sandoval | Cortexa claims attention-weight inspection = explainability. Assess the claim. | | HMS-0033 | P2 | Yusuf Demir | Architecture map needed for the security review pack | #### Project — Annotated Architecture Map (Capstone Milestone 2) Take one real open model and produce a complete annotated map of where its parameters live. For each tensor group — token embeddings, position handling, per-layer Q/K/V/O projections, per-layer MLP matrices, norms, output head — record the shape, the parameter count, and the percentage of the model total. Mark each group clearly as LEARNED AND FROZEN or COMPUTED AT RUNTIME. Then answer three questions with numbers, not impressions: what fraction of this model is attention, what fraction is MLP, and what fraction is simply vocabulary. Finish with a short assessment of the Cortexa claim that inspecting attention weights delivers explainability — being precise about what an attention map does and does not tell you about why an output was produced. 🎯 Capstone Milestone 2. **Deliverable:** weights/w04-architecture-map.md — the full tensor table with percentages, the learned-versus-computed marking, the three fractions answered numerically, and your written assessment of the explainability claim. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Complete and correct tensor table | 30% | Every parameter group present with real shapes and counts that reconcile to the model total within about 1%. | | Learned versus computed marked correctly | 25% | Q/K/V/O projections correctly identified as learned weights; attention maps and activations correctly identified as per-prompt runtime values. | | Fractions answered with numbers | 20% | Attention, MLP, and vocabulary shares stated as percentages, with the MLP-dominance result correctly identified rather than assumed away. | | Explainability claim assessed honestly | 25% | Attention maps described accurately as showing where information was read from — not why an output was produced — without overclaiming in either direction. | #### Scenario drills **Drill 1.** The security review pack needs a one-paragraph description of what a transformer does, accurate enough that Yusuf can build governance on it. **Task:** Write it. Then have somebody who does not work in AI read it back to you and tell you what they think it said. **Drill 2.** Anika claims the model "pays attention to the patient's age because age is clinically relevant". **Task:** Assess what an attention map could and could not establish about that claim, and design a test that would give better evidence. **Drill 3.** Somebody proposes "we only need a 1-layer model, it will be faster". **Task:** Use the course lab result to explain what depth actually buys, with the induction-task numbers as evidence. #### Prompt clinic — Assessing the explainability claim (HMS-0032) - **Weak:** "Is attention explainability real?" - **Average:** "A vendor says their appliance provides explainability by letting us inspect attention weights. Is that true?" - **Good:** "A vendor claims their appliance provides explainability because it lets us inspect the model's attention weights. Assess this claim: what do attention maps actually show, what do they not show, and what would genuine explainability require for a clinical summarisation use case?" - **Excellent:** "Act as a technical reviewer preparing a challenge for a vendor meeting. The claim: 'our appliance provides explainability — you can inspect the model's attention weights'. Assess it in four parts: (1) disambiguate the term — is the vendor referring to learned Q/K/V projection matrices, or to per-prompt attention score maps, and does the claim even make sense under each reading; (2) what an attention map genuinely evidences about information flow, stated precisely; (3) the documented limitations of treating attention as explanation, including that attention distributions can be altered while leaving predictions largely intact; (4) what would actually be needed to support a clinical audit trail. Finish with three specific questions I should ask in the meeting that would distinguish a substantive capability from a marketing claim — questions where a vague answer is itself informative." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. The Q, K, and V matrices in a transformer are…** a. Computed fresh from each incoming prompt b. Frozen learned matrices c. A kind of hyperparameter setting d. Stored inside the model's tokenizer **2. The "attention map" — the grid showing which tokens attend to which — is…** a. A learned weight matrix stored in the checkpoint b. Computed at runtime, then discarded c. Kept fixed for absolutely every input d. Part of the tokenizer configuration **3. The phrase "attention weights" is dangerous because…** a. Attention is unimportant b. It names two different things — projections and per-prompt scores c. It is really just a piece of vague marketing terminology and nothing more d. Attention has no weights **4. In a Llama-3-8B-shaped model, the majority of parameters sit in…** a. The attention projection matrices b. The MLP / feed-forward matrices c. The token embedding table itself d. The various layer norm parameters **5. Multi-head attention means…** a. Several separate models all running in parallel together b. The hidden dimension is split across heads c. Attention is computed several different times and then averaged d. The model has multiple output layers **6. Grouped-query attention uses fewer KEY and VALUE heads than QUERY heads in order to…** a. Improve the model's overall accuracy noticeably b. Shrink the K/V projections and the KV cache c. Reduce the total number of layers in the model d. Simplify training **7. The residual connection around each sub-block exists to…** a. Save memory b. Let each block add to a running representation c. Speed up the whole tokenisation step considerably d. Enable quantization **8. Why can a ONE-layer transformer not learn the induction task in the course lab?** a. It has too few parameters b. The task needs two composed operations — finding the earlier occurrence and reading the token after it — which requires a previous-token head in one layer feeding an induction head in the next c. Its learning rate is wrong d. One-layer models cannot use attention **9. Dr Hale asks how the model "decides what matters" in a summary. The most accurate answer is…** a. It weights each sentence by importance b. Attention scores determine which earlier tokens influence each position, but they are one mechanism among several — they show where information was read from, not why a conclusion was reached c. It uses a rules engine d. It retrieves the most relevant sentences **10. Cortexa claims attention-weight inspection provides explainability. The fair assessment is…** a. Fully correct — attention explains model outputs b. Overstated: attention maps are genuinely informative about information flow, but research has repeatedly shown they do not constitute a faithful explanation of why a model produced a given output c. Entirely fraudulent d. Correct only for small models **11. Between two prompts to the SAME checkpoint, what changes?** a. The underlying learned weights themselves b. The activations and attention maps c. The whole model architecture itself d. The tokenizer **12. For your security review pack, the useful summary of a transformer block is…** a. A black box b. Attention routes info; the MLP processes each position c. A database query followed by a full rules engine on top d. A search index with a ranking function --- ## Phase 3 — How Weights Change (weeks 3) ### Module 5 — Training: Loss, Gradients & Why It Costs So Much More Than Inference **Guiding question:** How do random numbers become useful weights, and why does training need several times the memory of running the same model? **Outcome:** Walk the training loop — predict, measure loss, compute gradients, step the optimizer — and understand learning rate as the single most consequential knob. Distinguish pretraining from post-training (instruction tuning, preference optimisation) and explain what each contributes. Then the number that governs every fine-tuning conversation: full training holds weights, gradients, and optimizer states simultaneously, so budget roughly 16 bytes per parameter before activations — several times inference. **Decision lens:** Somebody will propose fine-tuning as casually as they propose a config change. The memory arithmetic is what turns that conversation concrete: an 8B model you serve happily in 16 GB needs well north of 100 GB to full fine-tune with Adam, because every parameter drags a gradient and two optimizer moments behind it. That single fact is why the entire LoRA industry exists — which is exactly where the next module goes. **Apply-at-work mission — Watch weights move, then price the move:** Use the course training dashboard (and the optional notebook if you want real code) to run a tiny model on a toy task: record loss before and after, inspect specific weights before and after, then break the learning rate — set it far too high and far too low — and document what each failure looks like on the curve. Then compute, for a model your organisation might realistically fine-tune, the full-fine-tune memory requirement using the 16-bytes-per-parameter rule, and identify what hardware that implies. **Reflection:** Seeing a loss curve diverge under too high a learning rate, what did that teach me about why training runs fail expensively rather than gracefully? #### Resources - [3Blue1Brown — Gradient descent, how neural networks learn](https://www.youtube.com/watch?v=IHZwWFHWa-w) — The clearest intuition for what a gradient is and what "stepping downhill" means. Everything else this week is bookkeeping around this one idea. - Explain what a gradient is and what "stepping downhill" means - Hold the intuition the rest of the week's bookkeeping rests on - Say why training moves weights and inference does not - [Karpathy — The spelled-out intro to backpropagation (micrograd)](https://www.youtube.com/watch?v=VMj-3S1tku0) — If you want backpropagation to stop being a word and start being a mechanism, this is the video. Optional, but the payoff is permanent. - Turn backpropagation from a word into a mechanism - Trace how a loss produces a weight update - Understand why gradients exist only during training - [Hugging Face — Methods and tools for efficient training on a single GPU](https://huggingface.co/docs/transformers/en/perf_train_gpu_one) — The practical memory accounting: optimizer states, gradients, activations, and the techniques that trade compute for memory. This page is the source of your GPU sizing answer to Anika. - Account for optimizer states, gradients and activations in training memory - Explain why training costs far more memory than inference - Name the techniques that trade compute for memory - [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054) — Read the introduction and the memory analysis. It states the per-parameter training memory arithmetic explicitly, and it is the foundation of DeepSpeed and FSDP. - State the per-parameter training-memory arithmetic (~16 bytes per parameter) - Explain why training a model dwarfs the cost of serving it - Use the memory analysis to justify LoRA over a full fine-tune - [Hugging Face — Fine-tuning and the training loop in practice](https://huggingface.co/docs/transformers/en/training) — What an actual training run looks like in code, so the loop stops being abstract. Skim if you will never run one; read if you might. - Read what an actual training run looks like in code - Make the predict → loss → gradient → step loop concrete - Decide whether you'll ever run one, and read accordingly #### In-world ticket queue > Anika has requested "a couple of A100s for a few weeks" to fine-tune an 8B model on the clinical > corpus. Elaine wants the number justified before it goes near a purchase order. The training memory > arithmetic you learn this week is the entire answer. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0041 | P1 | Dr Anika Rao | GPU request: full fine-tune of an 8B model — size it properly | | HMS-0042 | P1 | Elaine Ferris | "Why does training need more hardware than running it?" — CFO | | HMS-0043 | P2 | research campus | Previous training run diverged after 6 hours, cause never established | #### Project — Training Run Report + GPU Sizing for a Real Request Two halves, both required. First, run the course training dashboard (or the optional notebook if you want real code): record the loss curve at a sensible learning rate, then deliberately break it — set the rate far too low and far too high — and document what each failure looks like and how you would recognise it at hour six of a real run rather than hour one. Note the before-and-after values of specific weights and describe in your own words what the optimizer did to them. Second, take Anika's actual request — a full fine-tune of an 8B model — and size it properly: weights, gradients, and optimizer states at roughly 16 bytes per parameter, plus activation memory, and state what hardware that implies and what it would cost. Then state what the LoRA alternative would need, so the comparison is on the table before Module 6 argues about method. **Deliverable:** weights/w05-training-report.md — annotated loss curves for three learning rates with failure signatures, the before/after weight observation, and the full-fine-tune sizing with hardware implication and a LoRA comparison. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Curves genuinely produced and read | 25% | Three real runs at different learning rates, with the divergence and the too-slow case both correctly diagnosed from the curve shape. | | Failure signatures are operational | 20% | Describes how you would spot each failure early in a long run — what to watch and when to kill it — not just what it looks like in hindsight. | | Weight change observed and explained | 20% | Specific before/after values recorded, with a correct plain-language account of the gradient-then-optimizer-step mechanism that moved them. | | Sizing is correct and complete | 35% | Weights, gradients, and optimizer states all counted, activation memory acknowledged, hardware implication stated, and the LoRA comparison numerically credible. | #### Scenario drills **Drill 1.** A previous training run diverged after six hours and nobody established why. **Task:** List what you would need logged to diagnose it retrospectively, and design the minimum monitoring that would let you kill a bad run within 20 minutes instead of six hours. **Drill 2.** Anika requests "a couple of A100s for a few weeks" for a full fine-tune of an 8B model. **Task:** Size it properly, then write the reply. The reply should approve, refuse, or reshape the request — and whichever it does, it should show the arithmetic and offer a next step. **Drill 3.** Elaine asks why training needs so much more hardware than running the model, when "it is the same model". **Task:** Explain it in under 80 words with the per-parameter arithmetic made intuitive. #### Prompt clinic — Sizing the fine-tuning request (HMS-0041) - **Weak:** "How many GPUs do we need to fine-tune a model?" - **Average:** "How much GPU memory do we need to fine-tune an 8B model?" - **Good:** "Calculate the GPU memory needed to fully fine-tune an 8B-parameter model with the Adam optimizer: weights, gradients, and optimizer states, plus an allowance for activations. State bytes per parameter for each component and give a total." - **Excellent:** "Act as an ML infrastructure engineer sizing a training request. Requirement: full fine-tune of an 8B-parameter model, Adam optimizer, mixed precision, sequence length 2048. Calculate: (1) memory for weights, gradients, and optimizer states, stating bytes per parameter for each and naming the assumption about master weight precision; (2) activation memory and how it scales with batch size and sequence length; (3) the total, and what GPU configuration that implies — be specific about card counts and whether model parallelism is required; (4) the same calculation for LoRA at rank 16 applied to the query and value projections, showing the trainable parameter count explicitly; (5) a comparison table of the two, including what each approach can and cannot achieve. Flag every assumption you made and state which one, if wrong, would most change the answer." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. The training loop, in order, is…** a. Predict, save, quantize, deploy b. Forward pass, loss, backpropagate, then optimizer step c. First you initialise, then quantize, then evaluate, then repeat d. Tokenize, embed, retrieve, answer **2. The loss is…** a. The total number of parameters that got discarded b. A single number measuring the model's wrongness c. The memory lost to fragmentation d. The difference between the total and the active parameters **3. A gradient tells you…** a. The final exact value that this weight should take b. Which way, and how fast, to nudge that weight c. Roughly how much memory this particular weight uses d. Which layer the weight belongs to **4. Setting the learning rate far too high typically produces…** a. Slow but relatively steady improvement over time b. Divergence — the loss overshoots c. A more or less perfectly flat loss curve d. Noticeably better generalisation overall **5. Setting the learning rate far too low produces…** a. Outright divergence of the loss curve b. Very slow progress within the budget c. Immediate and severe overfitting to the data d. A numerical overflow at some point during it **6. Full fine-tuning needs far more memory than inference primarily because…** a. The model's weights are simply stored twice over for safety b. Training also holds gradients and optimizer states c. Training requires FP64 d. The tokenizer expands **7. For an 8B model, full fine-tuning with Adam requires roughly…** a. About 16 GB, exactly the same as for inference b. Around 128 GB, before activations c. Only around 8 GB in total for everything d. Somewhere around a full 1 TB or so **8. Activation memory during training scales with…** a. The total parameter count of the model only b. Batch size and sequence length together c. The overall size of the model's vocabulary d. Disk speed **9. Pretraining differs from post-training in that pretraining…** a. Uses a smaller model b. Learns general language structure from a very large corpus, while post-training (instruction tuning, preference optimisation) shapes how the model behaves and responds c. Happens after deployment d. Only adjusts the tokenizer **10. The optimizer (such as Adam) differs from raw gradient descent by…** a. Computing all of the gradients much more accurately than before b. Maintaining running statistics of past gradients per parameter c. Reducing the number of parameters d. Running on CPU **11. A training run that diverged after six hours was most likely…** a. Out of disk space b. Running at too high a learning rate, or hit an instability such as a bad batch or numerical overflow — the loss curve at the moment of divergence is the first evidence to examine c. Using the wrong tokenizer d. Under-parameterised **12. The reason weights start as random numbers is…** a. Randomness improves security b. Identical starting weights would make units within a layer compute identical things and receive identical gradients, so they could never differentiate c. It is faster to initialise d. To avoid copyright ### Module 6 — Adapting a Model: Prompt, RAG, LoRA, or Full Fine-Tune **Guiding question:** Someone says "let's fine-tune it on our data" — what is the right answer, and how do I justify it in numbers? **Outcome:** Run the adaptation decision tree with evidence: prompting for behaviour, RAG for knowledge, LoRA for style and format at low cost, full fine-tuning rarely and deliberately. Understand what LoRA actually does — training small low-rank matrices whose product is added to frozen weights, typically well under 1% of parameters trained — plus adapter merging, serving multiple adapters, and catastrophic forgetting. Above all: know that fine-tuning teaches behaviour, not facts, and that most "fine-tune it on our documents" requests are retrieval problems wearing the wrong hat. **Decision lens:** This is the single most expensive misunderstanding in enterprise AI. A team asks to fine-tune a model on ten thousand internal documents so it "knows" them. Fine-tuning will happily consume the budget and produce a model that has absorbed the house STYLE while still inventing document numbers — because facts you need retrieved on demand belong in retrieval, not in weights. Knowing why, mechanically, is what lets you redirect that conversation without sounding obstructive. **Apply-at-work mission — Adjudicate a real request:** Take a real (or realistic) "can we fine-tune it on our data?" request from your world. Produce a written adjudication: what the requester actually wants, which adaptation method genuinely fits, the estimated cost and hardware for each viable option, what evidence would change your answer, and a recommended smallest-first sequence. Include the case where fine-tuning IS right — knowing when to say yes is what stops you becoming the person who always says no. 🎯 Capstone Milestone 3. **Reflection:** Where in my organisation is someone about to solve a retrieval problem with a fine-tune — and what is the most persuasive way I could show them the difference? #### Resources - [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) — The original paper. Read the abstract and section 4 — the core claim is that the update learned during adaptation has low intrinsic rank, so you can train two small matrices instead of the whole thing. - Explain the core claim: the update learned during adaptation is low-rank - Say why LoRA trains a tiny fraction of the parameters - Justify LoRA as the default over a full fine-tune - [Hugging Face PEFT — parameter-efficient fine-tuning in practice](https://huggingface.co/docs/peft/index) — The toolkit teams actually use. Note how few parameters get trained, and how adapters can be swapped or merged — that flexibility is a real operational advantage over full fine-tuning. - Note how few parameters parameter-efficient fine-tuning actually trains - Explain how adapters can be swapped or merged - Reach for the toolkit teams actually use for fine-tuning - [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) — The technique that made single-GPU fine-tuning of large models ordinary: a 4-bit frozen base model with LoRA adapters trained on top. Read the abstract for the headline memory claim. - Explain how a 4-bit frozen base plus LoRA adapters fits a large model on one GPU - Say what made single-GPU fine-tuning of large models ordinary - Weigh QLoRA against a full fine-tune on cost - [Anthropic — Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) — The other side of the decision tree. If the requirement is "know our documents", this is the shape of the right answer — and it is measurable, which fine-tuning-for-facts rarely is. - Recognise RAG as the right answer to "know our documents" - Separate a knowledge requirement from a behaviour requirement - Choose retrieval when the facts change next month - [OpenAI — When to use fine-tuning (and when not to)](https://platform.openai.com/docs/guides/fine-tuning) — Read the "when to use" section critically. Even the vendor selling fine-tuning leads with "try prompting first" — useful ammunition for a conversation you are about to have. - State when fine-tuning is and is not the right tool - Notice even the vendor selling it leads with "try prompting first" - Run the adaptation decision tree on evidence, not vendor push #### In-world ticket queue > The fine-tuning proposal has grown: 14,000 clinical policy documents, "so the model knows our > policies". Anika is convinced. Martin is enthusiastic. Elaine is waiting for you. This is the week where > saying "no" badly loses you the room and saying "no" well makes your reputation. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0051 | P1 | Dr Anika Rao | Adjudicate: fine-tune on 14,000 policy documents so "the model knows our policies" | | HMS-0052 | P2 | Dr Martin Hale | Second request: match our house summary format exactly — different problem, different answer | | HMS-0053 | P2 | Greg Sandoval | Cortexa offers "fine-tuning included" — establish what that actually means | #### Project — Adjudicate a Real Fine-Tuning Request (Capstone Milestone 3) Take a real or realistic "can we fine-tune it on our data?" request — Anika's 14,000 policy documents will do if you have nothing closer to hand. Produce a written adjudication that would survive the meeting. Separate what the requester literally asked for from what they actually need, which is rarely the same. Assess each of the four options against that need: prompting, retrieval, LoRA, and full fine-tuning, with an honest estimate of cost, hardware, effort, and expected quality for each. State clearly which mechanism supplies FACTS and which supplies BEHAVIOUR, since that distinction usually decides the case on its own. Recommend a smallest-first sequence, define what evidence would change your recommendation, and specify how you would measure whether the chosen approach worked. Include the scenario in which fine-tuning IS the right answer — an adjudicator who always says no stops being consulted. 🎯 Capstone Milestone 3. **Deliverable:** weights/w06-adaptation-decision.md — the restated requirement, the four-option comparison with costs, the facts-versus-behaviour analysis, a sequenced recommendation with success measures, and the conditions under which you would reverse it. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Requirement correctly restated | 20% | Distinguishes the literal request from the underlying need, and names the misunderstanding driving it without condescension. | | Four options honestly compared | 25% | All four assessed with credible cost, hardware, and effort estimates — including the case for the option you are rejecting. | | Facts versus behaviour applied | 25% | Correctly identifies retrieval as the mechanism for current facts and fine-tuning as the mechanism for style, format, and behaviour, with reasoning from the mechanics. | | Decidable and measurable | 30% | A sequenced recommendation a sponsor could act on, with defined success measures and stated conditions that would change the answer. | #### Scenario drills **Drill 1.** Martin wants summaries to match Alderbrook's house format exactly; Anika wants the model to know current policies. Both arrive in the same meeting. **Task:** Give each request its correct answer and explain why two similar-sounding asks get different mechanisms. Anticipate the objection "surely fine-tuning does both". **Drill 2.** Cortexa offers "fine-tuning included" in the appliance package. **Task:** Draft the five questions that establish what is actually on offer, and identify which single answer would most change your assessment. **Drill 3.** A pilot fine-tune improved house-format compliance markedly, but the model now performs noticeably worse on general clinical questions. **Task:** Name what has happened, explain the mechanism, and list three ways to mitigate it. #### Prompt clinic — Adjudicating the 14,000 documents request (HMS-0051) - **Weak:** "Should we fine-tune on our documents?" - **Average:** "Our data science team wants to fine-tune a model on 14,000 policy documents so it knows our policies. Is that a good idea?" - **Good:** "Our data science team wants to fine-tune an 8B model on 14,000 internal policy documents so it 'knows our policies'. Assess whether fine-tuning or retrieval is the right approach, explain the mechanism behind your answer, and estimate the cost of each." - **Excellent:** "Act as an AI architecture adviser writing an adjudication for a sponsor who is enthusiastic and technically credible — I need to be right AND collegial. Request: fine-tune an 8B model on 14,000 internal policy documents so the assistant 'knows our policies'. Produce: (1) a restatement separating what was literally asked from the underlying need, stated generously; (2) the mechanism — why fine-tuning reliably transfers style and behaviour but unreliably instils retrievable facts, in terms a data scientist will accept rather than feel lectured by; (3) a four-option comparison (prompting, retrieval, LoRA, full fine-tune) with cost, hardware, effort, latency and expected quality; (4) the policy-update problem — what happens to each option when a policy changes next month; (5) a recommended smallest-first sequence with success measures defined BEFORE we build; (6) the honest case where fine-tuning would be the right call here, so the recommendation is not reflexive. Keep it under 700 words and make the reasoning legible to a non-specialist sponsor." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. Fine-tuning is best understood as teaching a model…** a. New facts that it can then reliably recall later b. Behaviour, by nudging existing weights c. A substantially larger working vocabulary d. How to go and search across the internet **2. Anika wants to fine-tune on 14,000 policy documents "so the model knows our policies". The most likely outcome is…** a. Accurate recall of every policy b. The model absorbs the house style and vocabulary while still inventing policy numbers and details, because facts needed verbatim belong in retrieval c. A licensing breach d. No change at all **3. Retrieval-augmented generation supplies facts by…** a. Actually modifying the model's weights live at query time b. Finding passages and placing them in the context window c. Retraining nightly d. Increasing the parameter count **4. LoRA works by…** a. Deleting all of the least important weights first b. Freezing weights, training low-rank matrices c. Quantizing the whole model down to just 4 bits d. Training only the final layer **5. A typical LoRA configuration trains what proportion of the model's parameters?** a. Around half of all of them b. Well under 1% c. Around 10% of the parameters d. All of them, but at a lower precision **6. The main practical consequence of LoRA's small trainable set is…** a. Noticeably better factual recall from the finished model b. Much lower memory for gradients and optimizer states c. Faster inference d. Larger context windows **7. Catastrophic forgetting refers to…** a. Losing the checkpoint file b. Losing general capability on a narrow task c. The model's context window quietly overflowing d. The weights being corrupted during quantization **8. The correct FIRST option to try for most requirements is…** a. Full fine-tuning b. Prompting — including few-shot examples and a well-specified system prompt — because it is free, immediate, and frequently sufficient c. LoRA d. Training from scratch **9. Fine-tuning IS the right answer when…** a. You need current facts b. A consistent output format prompting can't reliably give c. Your underlying source documents happen to change every week d. You want citations **10. The requirement "match our house summary format exactly" versus "know our current policies" should be answered by…** a. Both of them handled by fine-tuning the model b. Fine-tune for format; retrieval for policies c. Both of them handled by retrieval alone instead d. Both by simply switching to a much larger model **11. A vendor offering "fine-tuning included" should be asked…** a. Nothing — it is a bonus b. What method, on what base model, with what data handling, producing what artifact — and crucially whether you can take the resulting adapter or weights with you c. Only the price d. Whether it is quantized **12. The strongest evidence that fine-tuning was NOT needed is…** a. The model you happen to be using is a very large one b. A few-shot prompt already meets the quality bar c. The budget is small d. The documents involved are all highly confidential --- ## Phase 4 — Running the Weights (weeks 4) ### Module 7 — The Checkpoint: Formats, Precision & Provenance **Guiding question:** What is actually in this download, is the file itself safe to load, and are we allowed to run it? **Outcome:** Open a checkpoint with confidence: the weight shards, config.json, tokenizer files, and the chat template that quietly decides whether output is coherent. Compare formats — safetensors, PyTorch pickle-based .bin, and GGUF — and understand why the pickle format is a code-execution surface and safetensors was built to remove it. Read precision fluently (FP32, FP16, BF16, FP8, INT8, INT4) including the range-versus-precision difference between BF16 and FP16, know the main quantization approaches and their honest quality costs, and read a model licence for the clauses that actually bite. **Decision lens:** A checkpoint is a supply-chain artifact, not just a big number blob. Loading a pickle-based checkpoint has historically meant executing whatever the author put in it — the reason safetensors exists at all. Meanwhile the chat template buried in the tokenizer config decides whether your deployment produces sensible answers or subtle nonsense, and the licence decides whether any of it is permitted at your scale. Three questions before a single GPU is ordered: what is in it, is it safe to load, may we run it. **Apply-at-work mission — Review a checkpoint like a supply-chain artifact:** Take one real open-weights model and produce a checkpoint review: every file and its purpose, the format and why it matters, the precision and resulting disk footprint, the chat template and what breaks without it, the publisher and how the download's integrity is verified, and a licence read-through naming any clause that would constrain your organisation specifically. End with a one-line go/no-go and the reasoning. 🎯 Capstone Milestone 4. **Reflection:** Which of these checks would my organisation currently skip entirely if a team downloaded a model this week — and what would that cost us? #### Resources - [Hugging Face — Pickle scanning and why .bin files are a code-execution risk](https://huggingface.co/docs/hub/en/security-pickle) — Read this before your security review. Loading a pickle-based checkpoint can execute arbitrary code — the Hub scans for it, but scanning is not a guarantee and the responsibility stays with you. - Explain why loading a pickle-based .bin checkpoint can execute arbitrary code - Treat an untrusted pickle as a supply-chain risk in a security review - Check whether a source scans for malicious pickles before you load - [safetensors — the format built to remove that risk](https://huggingface.co/docs/safetensors/index) — A plain tensor container with no code execution and fast memory-mapped loading. Prefer it, and make preferring it a written standard rather than a habit. - Say why safetensors removes the code-execution risk - Make "prefer safetensors" a written policy line - Explain memory-mapped loading as a bonus, not the reason - [GGUF file format specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) — The format behind llama.cpp and Ollama: weights, metadata, and tokenizer in one self-contained file, with quantization baked in. Skim the metadata section to see what a single-file checkpoint has to carry. - Recognise GGUF as the self-contained format behind llama.cpp and Ollama - Read what GGUF packs — weights, metadata, tokenizer, quantization - Match the checkpoint format to your serving stack - [Hugging Face — Quantization overview](https://huggingface.co/docs/transformers/en/quantization/overview) — The map of methods — bitsandbytes, GPTQ, AWQ and others — with what each is good for. Read for the tradeoffs, not the API. - Map bitsandbytes, GPTQ and AWQ to what each is good for - Read quantization for the tradeoffs, not the API - Pick a quantization approach for a given memory target - [Chat templates — the file that decides whether your deployment makes sense](https://huggingface.co/docs/transformers/en/chat_templating) — The Jinja template in tokenizer_config.json defines the control tokens wrapping every turn. Get it wrong and output degrades subtly rather than failing loudly — which is far worse. - Explain how the chat template's control tokens wrap every turn - Predict the output degradation caused by a wrong template - Check the template before blaming the model for bad output #### In-world ticket queue > A researcher downloaded a model from a community repository and loaded it on the pilot cluster > last Thursday. Yusuf found out on Monday. Nobody checked the publisher, the file format, or the licence — > and the licence, it turns out, has a clause about healthcare use. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0061 | P1 | Yusuf Demir | Unvetted checkpoint loaded on the pilot cluster — assess exposure | | HMS-0062 | P1 | legal, via Elaine | Licence review: can we legally run each shortlisted model in a clinical setting? | | HMS-0063 | P2 | platform team | Deployment producing subtly odd output — chat template suspected | | HMS-0064 | task | this week's milestone | Standing rule needed: what must be checked before ANY model is downloaded | #### Project — Checkpoint Safety & Licence Review (Capstone Milestone 4) Take one real open-weights model you would genuinely consider deploying and review it as a supply-chain artifact rather than a curiosity. Enumerate every file in the repository and state what each is for — weight shards, index, config, tokenizer files, generation config, chat template — and flag anything you cannot account for. Record the format and assess it: is it safetensors, is there a pickle-based file present, and what would loading it actually do. Record the precision and compute the resulting disk and memory footprint. Extract the chat template and describe what would break without it. Identify the publisher and how you would verify the download is what it claims to be. Then read the licence properly — not the summary badge, the licence — and name any clause that would constrain Alderbrook specifically, including use restrictions, field-of-use limits, and obligations that survive deployment. Finish with a one-line go/no-go and the reason. 🎯 Capstone Milestone 4. **Deliverable:** weights/w07-checkpoint-review.md — the file inventory, format and safety assessment, precision and footprint calculation, chat template note, provenance and integrity check, licence findings with specific clauses quoted, and a go/no-go recommendation. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Complete file inventory | 20% | Every file accounted for with its purpose, including the unglamorous ones — index, generation config, tokenizer files — and anything unexplained flagged rather than ignored. | | Format risk assessed correctly | 25% | Correctly identifies whether pickle-based files are present, states accurately what loading one can do, and gives a defensible position on the format encountered. | | Precision and footprint computed | 20% | Bytes per parameter correctly applied, disk and memory footprints derived, and cross-checked against the actual file sizes in the repository. | | Licence read, not skimmed | 35% | Specific clauses identified and quoted with their consequence for a healthcare deployment at your scale — including any acceptable-use or field-of-use restriction, not merely the licence name. | #### Scenario drills **Drill 1.** The deployment produces output that is plausible but consistently slightly off in tone and structure. Weights and config check out. **Task:** List your diagnostic order, with chat template placed correctly in it, and describe how you would confirm the template hypothesis. **Drill 2.** Legal asks whether Alderbrook can use a given open-weights model for clinical summarisation. **Task:** Work out how to answer properly: what you must read, which clause types decide it, and what you would escalate to legal rather than resolve yourself. **Drill 3.** You must write the standing rule for what is checked before any model reaches Alderbrook infrastructure. **Task:** Write it as a checklist someone will actually complete — short enough to be used, specific enough to catch the Thursday download. #### Prompt clinic — Reviewing the unvetted checkpoint (HMS-0061) - **Weak:** "Is this model safe?" - **Average:** "A researcher downloaded a model from a public repository and loaded it on our cluster. Should we be worried?" - **Good:** "A researcher downloaded a model from a community repository and loaded it onto a shared GPU cluster without review. Assess the risk: what file formats could execute code on load, what should we check now, and what standing control would prevent a repeat?" - **Excellent:** "Act as a security engineer assessing an AI supply-chain incident. Facts: a researcher downloaded a model from a community repository and loaded it on a shared GPU pilot cluster; no publisher, format, or licence review was performed; discovered four days later. Produce: (1) the immediate triage questions in priority order, starting with file format and what loading it could have executed; (2) what evidence to gather now and where it would live, given four days have passed; (3) the specific risk difference between a pickle-based checkpoint and a safetensors file, stated precisely enough for a risk register entry; (4) whether platform-side scanning is sufficient assurance and why not on its own; (5) a standing pre-download control — the minimum checks that must pass before ANY model reaches our infrastructure, written as a checklist someone will actually complete; (6) how to write this up without making the researcher defensive, because I need people to tell me next time. Note explicitly anything that cannot be determined retrospectively." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. A model checkpoint directory typically contains, besides weight files…** a. Nothing at all beyond a single README file b. Config, tokenizer, template, weight index c. The training data d. The gradients saved from the final epoch of training **2. The security concern with PyTorch .bin and .pt checkpoint files is that…** a. They are simply larger on disk than safetensors files b. They use pickle, which can execute code on load c. They cannot be quantized down any further at all d. They lack all of the necessary tokenizer data files **3. safetensors improves on that by…** a. Compressing the model weights rather more efficiently b. Storing tensor data readable without executing code c. Encrypting the weights d. Including the training data **4. A researcher loaded an unvetted checkpoint from a community repository on the pilot cluster. Your first question is…** a. What exactly was the reported accuracy figure of it? b. What format was it — could loading it run code? c. How physically large was the whole thing on disk? d. Was it quantized? **5. GGUF is best described as…** a. A general-purpose compression algorithm of some kind b. A single file of weights, metadata and tokenizer c. A licence type d. A training checkpoint format **6. BF16 differs from FP16 in that BF16…** a. Uses more bits in total b. Keeps FP32's exponent range, losing precision c. It is only ever used for inference, never for training d. Is a quantized integer format **7. An 8B-parameter model stored at BF16 occupies roughly…** a. About 4 GB in total b. 16 GB c. Around 32 GB altogether d. A little over 8 GB **8. Quantizing a 70B model from FP16 to INT4 changes its weight footprint from about…** a. 140 GB to about 35 GB b. About 70 GB down to about 35 GB c. About 140 GB down to about 70 GB d. About 35 GB down to about 9 GB **9. The honest summary of quantization quality impact is…** a. There is never any loss b. 8-bit is usually near-lossless, 4-bit is often acceptable on larger models but degrades more on small ones — and the only trustworthy answer is measuring on YOUR task c. Quality always collapses below 16-bit d. Impact depends only on the file format **10. A deployment produces subtly odd output — plausible, but consistently a bit wrong in tone and structure. A likely cause is…** a. Insufficient VRAM b. A wrong or missing chat template c. The tokenizer vocabulary is simply too small d. The GPU is quietly throttling itself **11. "Open weights" and "open source" differ in that open weights…** a. Are identical terms b. Means the trained parameters are downloadable — which says nothing about training data, training code, or the freedom of the licence terms c. Always implies an OSI-approved licence d. Means the model is free of restrictions **12. Before deploying any open-weights model in a clinical setting, the licence check must cover…** a. Only whether commercial use is allowed b. Commercial use, any acceptable-use or field-of-use restrictions (healthcare and medical uses are frequently named), scale thresholds, attribution obligations, and any terms binding downstream outputs c. Only the model size d. Only the publisher name ### Module 8 — Sizing & Serving: Memory, KV Cache & the Defensible Choice (Capstone) **Guiding question:** How much GPU do we actually need — and can I defend that number to a CFO who has been quoted something different? **Outcome:** Complete the memory equation end to end: weights (parameters × bytes-per-parameter) plus KV cache (which scales with context length and concurrency, and often surprises people more than the weights) plus activation and runtime overhead. Understand why grouped-query attention shrinks the cache dramatically, how batching trades latency for throughput, and where quantization genuinely buys headroom versus where it quietly costs quality. Then assemble and defend the capstone. **Decision lens:** The weights are the number everyone quotes. The KV cache is the number that ruins deployments: it grows with every token of context and every concurrent user, and on long-context workloads it can rival the weights themselves. This is the module where the whole course becomes one spreadsheet you can defend line by line — and where you find out whether the vendor's quote was sized for your actual concurrency or for a demo with one user. **Apply-at-work mission — Ship the capstone, defend the number:** Complete your chosen capstone track and present it to a real audience — your architecture forum, your manager, or a recorded walkthrough — with someone briefed to push back on the numbers. Then reopen your sealed Module 1 explanation and your Module 1 model inventory: rewrite the explanation, fill the gaps in the table, and record honestly what you could not answer four weeks ago. **Reflection:** Final entry: reread my sealed Week 1 explanation. What was I most confidently wrong about — and what does "understanding the model" mean to me now that I can compute what it weighs? #### Resources - [vLLM — PagedAttention and KV cache management](https://docs.vllm.ai/en/latest/) — The serving engine most self-hosting teams land on. Read the design notes on KV cache paging — it exists because cache memory, not weight memory, is what limits concurrency. - Explain why KV cache paging exists and what waste it fixes - Recognise vLLM as the engine most self-hosting teams land on - Connect cache management directly to your memory budget - [Efficient Memory Management for LLM Serving with PagedAttention (the vLLM paper)](https://arxiv.org/abs/2309.06180) — The paper behind it. Read the introduction for the clearest published statement of how badly KV cache memory is wasted by naive serving — and how much throughput that costs. - Cite the clearest published statement of how KV cache memory is wasted - Explain KV cache fragmentation in your own words - Ground the serving-memory arithmetic in the source paper - [Hugging Face — LLM inference optimisation](https://huggingface.co/docs/transformers/en/llm_optims) — Practical levers: batching, cache handling, attention implementations. The concrete link between the arithmetic you are doing and the throughput you will observe. - Name the practical levers: batching, cache handling, attention implementation - Link the memory arithmetic to concrete serving knobs - Choose the levers that fit your concurrency target - [Grouped-Query Attention (GQA) paper](https://arxiv.org/abs/2305.13245) — Why modern models use fewer KV heads than query heads. Read the abstract, then look again at your KV cache calculation — this single design choice can be a fourfold difference in serving memory. - Explain why fewer KV heads than query heads shrinks the cache - Redo your KV cache calculation with grouped-query attention in mind - Say why modern models adopt GQA - [Anthropic — Mapping the mind of a large language model](https://www.anthropic.com/research/mapping-mind-language-model) — Return to this now that you know what weights are. The closing point for the course: even state-of-the-art interpretability recovers features with effort — nobody reads facts out of raw weights, and anyone claiming otherwise is selling something. - Revisit interpretability now that you know what weights are - State the course's closing point on the limits of reading weights - Carry a sober interpretability line into a procurement conversation #### In-world ticket queue > Decision week. Cortexa's revised quote assumes one concurrent user; your clinical pilot has > forty. The board sees the recommendation on Thursday. Everything you have computed for four weeks now > becomes one defensible page — and Elaine has invited Greg to the meeting. | Ref | Priority | From | Request | | --- | --- | --- | --- | | HMS-0071 | P1 | this week's milestone | Final sizing: weights + KV cache at REAL concurrency and context length | | HMS-0072 | capstone | Elaine Ferris | Board recommendation Thursday — one page, defensible line by line | | HMS-0073 | P1 | Greg Sandoval | Cortexa attending the meeting. Their sizing assumes single-user load. | | HMS-0074 | task | you, to yourself | Re-open your sealed Week 1 explanation and rewrite it | #### Project — Capstone — choose your track TRACK A — MODEL SELECTION & SIZING BRIEF (recommended, and the one Elaine is expecting). Produce the board-ready recommendation for a real workload of yours, or for Alderbrook's clinical summarisation pilot. Required: the workload defined in numbers (concurrent users, typical and maximum context length, tokens generated per request, latency expectation); a shortlist of at least three candidate models with parameter counts derived rather than quoted; the full memory calculation for each — weights at your chosen precision PLUS KV cache at your real concurrency and context PLUS runtime overhead; a precision and quantization recommendation with its quality risk stated and a plan to measure it; the adaptation decision from Module 6; the licence and checkpoint-safety findings from Module 7; a hardware recommendation with cost; and a one-page executive summary a CFO could act on. Include a sensitivity analysis: what happens to your number if concurrency doubles or context length quadruples. Your brief must survive somebody asking "where did that number come from?" at any line. TRACK B — BUILD THE WEIGHTS EXPLORER (developer track). Build an interactive tool that teaches what you learned: a parameter calculator working from a real config, a memory and KV cache calculator, an embedding or attention visualiser, and a quantization comparison — each with correct arithmetic, verified against a real model, and each explaining itself to a non-specialist. Ship it with a README stating what is real data and what is simulated, and a short test suite covering the arithmetic. Both tracks are assessed against the same weightings below, interpreted for the track chosen. **Deliverable:** weights/w08-capstone.md (Track A: the full brief plus one-page executive summary; Track B: repository link, README, and a written walkthrough of the arithmetic and where each number is verified) — plus your rewritten Module 1 explanation and completed model inventory. **Assessment rubric** | Criterion | Weight | What good looks like | | --- | ---: | --- | | Arithmetic correct and traceable | 30% | Every number derivable from stated inputs — weights, KV cache, and overhead all present and correctly computed. No unexplained figures anywhere. | | KV cache and concurrency handled | 25% | Cache sized at real concurrency and context length, GQA accounted for where applicable, and the sensitivity analysis genuinely changes the recommendation where it should. | | Decisions justified end to end | 20% | Precision, adaptation method, licence, and checkpoint safety all resolved with reasoning that connects back to earlier modules rather than restating conclusions. | | Defensible to a non-specialist | 15% | The executive summary (or README) is accurate and jargon-free, and survived a real audience pushing back on the numbers. | | Honest about uncertainty | 10% | States what was measured versus estimated, what would need testing before commitment, and what would change the recommendation. | #### Scenario drills **Drill 1.** Cortexa's revised quote sizes for one concurrent user. Your clinical pilot has 40. **Task:** Produce the corrected sizing with full arithmetic, then write the three questions for the meeting that establish the omission without accusation. **Drill 2.** The board asks whether quantizing to 4-bit is "safe" for clinical summarisation. **Task:** Answer without overclaiming in either direction, and specify the evaluation you would run before committing — including what would constitute a failing result. **Drill 3.** Final rehearsal: you have ten minutes with the board and Greg is in the room. **Task:** Structure the recommendation: what goes on the single page, what you hold in reserve for questions, and which number you must be able to defend to the last decimal. #### Prompt clinic — Challenging the vendor sizing (HMS-0073) - **Weak:** "Is this quote right?" - **Average:** "A vendor says our model fits in 16GB of VRAM. Is that enough for our deployment?" - **Good:** "A vendor sized our deployment on the basis that an 8B model at BF16 fits in 16GB. Our clinical pilot has 40 concurrent users at 8k context. Calculate what memory we actually need including KV cache, and identify what their sizing omitted." - **Excellent:** "Act as an infrastructure architect preparing to challenge a vendor sizing in front of my CFO — I need arithmetic, not attitude. Their claim: an 8B model at BF16 fits in 16GB, so one card suffices. Our reality: 40 concurrent users, 8k typical context, 32k maximum, ~600 output tokens per request, clinical summarisation. Produce: (1) weight memory at BF16, stated plainly; (2) KV cache per token from architecture — 32 layers, 8 KV heads, head dim 128, 2 bytes per element — showing the formula and the per-token figure; (3) cache at 8k and at 32k context, per user and at 40 concurrent; (4) runtime overhead allowance and why it is not optional; (5) a total with headroom, and the resulting hardware; (6) a sensitivity table for concurrency doubling and context quadrupling; (7) the three questions to put to the vendor that would establish whether their sizing assumed single-user load — phrased so a fair answer is easy and an evasive one is obvious. Show every calculation so each line can be defended independently." #### Knowledge check (12 questions) **Self-test prompts. Answers and explanations are not published here — take the quiz at https://ragentic.netlify.app/#/courses/llm-weights to check yourself.** **1. The total memory to serve a model is…** a. Just the weights b. Weights plus KV cache plus overhead c. Weights plus training optimizer states d. Parameter count expressed in gigabytes **2. The KV cache stores…** a. Recently used model weights kept nearby in memory b. The K and V vectors for each context token c. The tokenizer vocabulary d. Previous user prompts on disk **3. KV cache size scales with…** a. The total parameter count of the model only b. Layers, KV heads, head dim, seq length c. The overall size of the model's vocabulary d. Disk throughput **4. For a Llama-3-8B-shaped model (32 layers, 8 KV heads, head dim 128) at 2 bytes per element, the KV cache per token is…** a. About 128 KiB b. Roughly 4 KiB per token c. Roughly 2 MiB per token d. Roughly 16 KiB per token **5. At 128 KiB per token, an 8,192-token context for ONE user consumes…** a. About 1 GiB b. About 128 MiB in total c. Around 8 GiB in total d. About 64 MiB in total **6. Your clinical pilot has 40 concurrent users at 8k context. The KV cache requirement is roughly…** a. Only about 1 GiB in total, at most b. 40 GiB — over twice the weights c. Only about 8 GiB in total, at most d. A full 400 GiB in total, roughly **7. Grouped-query attention reduces KV cache memory because…** a. It compresses the cache b. Fewer KV heads means fewer key and value vectors stored per token — 8 KV heads instead of 32 is a fourfold reduction c. It shortens the context d. It quantizes the cache automatically **8. A vendor quote sized on "the model fits in 16 GB" is incomplete because…** a. Models never fit in 16 GB b. It counts only weights, ignoring KV cache and overhead c. It quietly assumes entirely the wrong licence terms throughout d. It uses the wrong tokenizer **9. Increasing batch size generally…** a. Reduces both the latency and the memory together b. More throughput, more cache, more latency c. Has essentially no effect on memory usage d. Reduces throughput **10. Quantizing weights from BF16 to INT4 frees memory that…** a. Eliminates the KV cache b. Can be reallocated to the KV cache for more users c. Doubles the whole context window automatically for you d. Reduces layer count **11. Before committing to a quantized deployment for clinical summarisation, you should…** a. Trust the published benchmark scores b. Measure quality on your own representative evaluation set at each candidate precision, because aggregate benchmarks can hide degradation specific to your task c. Assume 4-bit is always safe d. Use FP32 to be certain **12. The strongest defence of your sizing number to a sceptical board is…** a. A vendor endorsement b. Every figure traceable to a stated input and a shown calculation, with a sensitivity analysis for the assumptions most likely to be wrong c. A larger contingency d. A benchmark comparison ## Toolkits ### Weights Vocabulary — the eight words people mix up **Unlocks in module 1.** The precise distinctions: weights, parameters, activations, gradients, hyperparameters, embeddings, logits, and the two meanings of "attention weights". ```markdown # Weights Vocabulary — precise definitions for people who have to be right ## The core five **Weight / parameter** — a learned number stored in the checkpoint. Fixed at inference. "Parameters" usually means all of them; "weights" often excludes biases, but the terms are used interchangeably in practice. If precision matters, say "trainable parameters". **Activation** — an intermediate value computed as YOUR input flows through the model. Created per request, discarded after. Never stored in the checkpoint. **Gradient** — the signal, computed during training only, saying how a weight should change to reduce the loss. Not present in a released checkpoint. If you find gradients in a file, somebody saved a training state, not a model. **Hyperparameter** — a human decision governing the model or its training: layer count, hidden size, learning rate, batch size. Chosen, not learned. **Logit** — the raw score the model produces per vocabulary entry before it becomes a probability. Sampling settings (temperature, top-p) operate here. ## The two dangerous ambiguities **"Attention weights"** means either: (a) the LEARNED Q/K/V/O projection matrices in the checkpoint — frozen, shipped, ~17% of an 8B model's parameters; or (b) the per-prompt ATTENTION SCORES — computed at runtime, different for every input, discarded after. Always establish which one is meant. Most vendor explainability claims dissolve here. **"Embeddings"** means either: (a) the EMBEDDING LAYER inside an LLM — a lookup table, one learned row per token; or (b) an EMBEDDING MODEL — a separate model producing one vector per passage for similarity search in retrieval. Different models, different jobs. Confusing them is the classic cause of a retrieval system that silently returns nonsense. ## Sentences that are precisely true - "Prompting does not change the weights." (Retention by the service is a separate question.) - "The attention map is computed from your prompt; the projections that compute it are not." - "Fine-tuning changes weights. Retrieval changes context. Only one of them updates when a policy changes next month." - "Memory follows total parameters. Compute follows active parameters." (MoE) ``` ### Parameter Derivation Worksheet **Unlocks in module 2.** Derive any model's parameter count from its config.json — with the four traps that produce wrong answers. ```markdown # Parameter Derivation Worksheet ## Inputs from config.json - vocab_size = V - hidden_size = D - num_hidden_layers = L - intermediate_size = F - num_attention_heads = H - num_key_value_heads = KVH (if absent, equals H — no grouped-query attention) - head_dim = HD (if absent, D / H) - tie_word_embeddings = true/false ## 1. Embeddings Token embedding = V x D Position embedding = P x D (learned positions only; rotary embeddings add NO parameters) ## 2. Per transformer layer q_proj = (H x HD) x D k_proj = (KVH x HD) x D <-- TRAP 1: smaller than q_proj when GQA is used v_proj = (KVH x HD) x D o_proj = D x (H x HD) Attention subtotal = sum of the four MLP — check the architecture: Classic (GELU): up = D x F, down = F x D -> 2 matrices SwiGLU: gate = F x D, up = F x D, down = D x F -> 3 matrices <-- TRAP 2 MLP subtotal = sum Norms = 1 x D per norm (RMSNorm), or 2 x D (LayerNorm with bias) <-- TRAP 3: biases Biases: modern models usually have none; GPT-2-era models have them everywhere Per-layer total = attention + MLP + norms Layer total = per-layer total x L ## 3. Output head tie_word_embeddings = true -> costs NOTHING (shares the embedding matrix) tie_word_embeddings = false -> costs another V x D <-- TRAP 4 ## 4. Total and reconciliation Total = embeddings + layers + final norm + output head Reconcile against the published figure. A gap over ~1% means a missing component — work through the four traps above before concluding the publisher is wrong. ## Worked check (use to validate your method) GPT-2 small: V=50257, D=768, L=12, F=3072, tied embeddings, biases present Embeddings: 50257 x 768 = 38,597,376 | Positions: 1024 x 768 = 786,432 Per layer: 7,087,872 -> x12 = 85,054,464 Final norm: 1,536 TOTAL = 124,439,808 (published: 124M) ## Mixture of experts Report BOTH: TOTAL parameters -> sets the memory you must buy ACTIVE parameters -> sets the compute per token, i.e. speed Never quote one without the other. Vendors quote whichever flatters the claim. ``` ### Tokenisation Cost Worksheet **Unlocks in module 3.** Measure what your domain text really costs in tokens, and what that does to context and budget. ```markdown # Tokenisation Cost Worksheet ## Method 1. Pick the ACTUAL tokenizer of the model you will deploy. Do not use a generic estimate, and do not compare providers without doing this per provider. 2. Assemble at least 5 categories of real text (200+ words each): - Plain prose in your domain (the control) - Proper nouns: product, drug, or system names - Identifiers: hostnames, asset tags, record numbers - Machine text: logs, error codes, stack traces - Structured: tables, CSV, JSON - Non-English content, if you process any 3. Record for each: words, characters, tokens, tokens/word, chars/token. | Category | Words | Tokens | Tokens/word | vs control | |---|---|---|---|---| | Domain prose (control) | | | | 1.00x | | Proper nouns | | | | | | Identifiers | | | | | | Machine text | | | | | | Structured data | | | | | | Non-English | | | | | ## Rules of thumb (starting points ONLY — always measure) - Ordinary English prose: roughly 0.75 words per token - Specialist vocabulary, identifiers, and code: substantially worse - Under-represented languages: often far worse, sometimes several times ## Convert to consequence Effective context = context_window / (tokens per word for YOUR mix) -> "Our 128k window holds ~N words of OUR text, not the N the docs imply" Monthly cost = requests/month x (input tokens + output tokens) x price per token -> Recompute with your MEASURED ratio, not the generic one. ## Mitigations that actually work - [ ] Strip repeated boilerplate, headers, and signatures before sending - [ ] Reference identifiers rather than repeating them in full each request - [ ] Trim retrieved context to what is needed (biggest saving in most RAG systems) - [ ] Compare candidate models on YOUR corpus with THEIR tokenizers before committing ``` ### Architecture Map Template **Unlocks in module 4.** Where every parameter lives, what fraction it represents, and what is learned versus computed. ```markdown # Architecture Map — ## Config hidden_size / layers / heads / kv_heads / intermediate / vocab / tied embeddings ## Parameter distribution | Group | Shape | Params | % of total | Learned & frozen | Computed at runtime | |---|---|---|---|---|---| | Token embeddings | V x D | | | YES | — | | Attention q/k/v/o (all layers) | | | | YES | — | | MLP (all layers) | | | | YES | — | | Norms | | | | YES | — | | Output head | | | | YES | — | | Attention SCORE maps | seq x seq per head | 0 | 0% | — | YES, per prompt | | Activations / residual stream | | 0 | 0% | — | YES, per prompt | | KV cache | see sizing worksheet | 0 | 0% | — | YES, per request | ## The three answers - Attention is ____% of parameters - MLP is ____% of parameters - Vocabulary (embeddings + untied head) is ____% of parameters Reference point — Llama-3-8B shaped: MLP ~70%, attention ~17%, vocabulary ~13%. If your intuition said attention dominates, that intuition is now corrected. ## What this map lets you say - Which tensors would a LoRA adapter touch, and what fraction is that? - Which parts would quantization compress, and which are too small to matter? - What does an attention map evidence, and what does it not? ``` ### Training vs Inference Memory Calculator **Unlocks in module 5.** The 16-bytes-per-parameter arithmetic that answers every "can we fine-tune it?" request. ```markdown # Training vs Inference Memory ## Inference (serving) Weights = parameters x bytes_per_parameter FP32 = 4 | FP16/BF16 = 2 | FP8/INT8 = 1 | INT4 = 0.5 Plus KV cache (see sizing worksheet) plus runtime overhead (~10-20%). ## Full fine-tuning with Adam — the rough rule | Component | Bytes per parameter | |---|---| | Weights (fp32 master) | 4 | | Gradients | 4 | | Adam moment 1 | 4 | | Adam moment 2 | 4 | | **Subtotal** | **~16** | Plus activations, which scale with batch size x sequence length. Mixed-precision setups vary this somewhat; 16 is the planning figure. Worked: 8B parameters x 16 bytes = ~128 GB before activations. Against ~16 GB to SERVE the same model at BF16. Roughly 8x. ## LoRA — the alternative Trainable parameters = rank x (d_in + d_out) per adapted matrix x number of layers Base weights stay frozen -> no gradients or optimizer states for them. Worked: 8B model, rank 16, adapting q_proj and v_proj, 32 layers q: 16 x (4096 + 4096) = 131,072 per layer v: 16 x (4096 + 1024) = 81,920 per layer (GQA: v_proj output is smaller) Total: 212,992 x 32 = ~6.8M trainable = under 0.1% of 8B Optimizer state cost becomes negligible; the frozen base model dominates memory, and can itself be quantized (this is QLoRA). ## The reply template "Full fine-tune of : ~ GB for weights + gradients + optimizer states, plus activations — implying . LoRA at rank 16 trains ~M parameters (% of the model) and fits on . Both give us . I'd suggest . Happy to size differently if is wrong." ``` ### Adaptation Decision Tree **Unlocks in module 6.** Prompt, RAG, LoRA, or full fine-tune — the decision, the reasoning, and the questions that settle it. ```markdown # Adaptation Decision Tree ## Start here: what does the requester actually need? **Q1. Does the requirement involve FACTS that must be current or citable?** YES -> RETRIEVAL. Facts live in documents, not weights. Fine-tuning instils facts unreliably, cannot cite sources, and is a snapshot that goes stale. NO -> continue. **Q2. Is it about BEHAVIOUR — style, format, tone, task pattern?** YES -> continue to Q3. NO -> re-interview the requester. The need has not been stated yet. **Q3. Does a well-constructed prompt with few-shot examples already meet the bar on a held-out evaluation set?** YES -> PROMPTING. Done. Free, immediate, changeable in minutes. NO -> continue to Q4. **Q4. Is the gap consistency at volume, or token cost from long instructions?** YES -> LoRA. Small, cheap, swappable, and reversible. NO -> continue to Q5. **Q5. Is the domain genuinely far from the base model's training distribution, with substantial high-quality training data available, and a budget that survives the memory arithmetic?** YES -> FULL FINE-TUNE. Rare, deliberate, and legitimate. NO -> return to Q3 with a better prompt and a real evaluation set. ## The question that settles most arguments "What happens when this changes next month?" Facts that change -> retrieval, always Format that is stable -> fine-tuning is reasonable ## Before ANY option is chosen - [ ] A held-out evaluation set exists, built before any building starts - [ ] Success is defined numerically and agreed with the sponsor - [ ] The baseline (plain prompting) is measured, so improvement is provable - [ ] Somebody owns the retraining treadmill if a fine-tune is chosen ## Comparison table to complete | Option | Setup cost | Run cost | Hardware | Time to first result | Updates when data changes | Can cite sources | |---|---|---|---|---|---|---| | Prompting | | | | | | | | Retrieval | | | | | | | | LoRA | | | | | | | | Full fine-tune | | | | | | | ``` ### Checkpoint Pre-Download Checklist **Unlocks in module 7.** Six checks before any model reaches your infrastructure. Short enough to actually get done. ```markdown # Checkpoint Pre-Download Checklist Complete BEFORE a model touches shared infrastructure. Six checks, one page, no exceptions. ## 1. Publisher - [ ] Who published it? Official organisation account, or a re-upload? - [ ] Is this the canonical repository, or a mirror/fork/quantized derivative? - [ ] If a derivative: who made it, from what source, and can that be verified? ## 2. Format — the code-execution question - [ ] safetensors present? -> preferred, no code execution on load - [ ] .bin / .pt / .pth present? -> pickle-based, CAN execute code on load - [ ] GGUF? -> single-file, common for local inference - [ ] If only pickle-based files exist: is there a safetensors conversion, and if not, is loading it justified and isolated? NOTE: platform-side pickle scanning reduces risk. It does not transfer responsibility. ## 3. Integrity - [ ] File hashes or signed commits checked where available - [ ] Download from the canonical source over TLS, not a third-party link - [ ] Recorded: source URL, revision/commit, date, who downloaded it ## 4. Licence and acceptable use - [ ] Licence identified and the ACTUAL TEXT read (not the badge) - [ ] Any acceptable-use policy incorporated by reference — read that too - [ ] Commercial use permitted at our scale? - [ ] Field-of-use restrictions? (medical, clinical, and high-risk uses are commonly named) - [ ] Attribution, naming, or downstream obligations? - [ ] Anything requiring escalation to legal, flagged rather than interpreted ## 5. Footprint - [ ] Parameter count and precision confirmed - [ ] Disk footprint = parameters x bytes/parameter — cross-check against actual file sizes - [ ] Serving memory estimated (weights + KV cache + overhead) before it lands anywhere ## 6. Runtime completeness - [ ] config.json present - [ ] tokenizer files present - [ ] chat template present, and its expected format documented - [ ] weight index present for sharded checkpoints - [ ] every file in the repository accounted for; anything unexplained flagged ## Sign-off Downloaded by: ______ Reviewed by: ______ Date: ______ Target environment: ______ Go / No-go: ______ Reason: ______ ``` ### Deployment Sizing Worksheet **Unlocks in module 8.** Weights plus KV cache plus overhead, at real concurrency — the calculation vendor quotes skip. ```markdown # Deployment Sizing Worksheet ## Workload (fill this in FIRST — sizing without it is meaningless) - Concurrent users at peak: ______ - Typical context length: ______ tokens | Maximum: ______ tokens - Output tokens per request: ______ - Latency expectation: ______ - Hours of peak load per day: ______ ## 1. Weight memory parameters x bytes_per_parameter FP32 = 4 | FP16/BF16 = 2 | FP8/INT8 = 1 | INT4 = 0.5 = ______ GB ## 2. KV cache — the number vendors omit Per token: 2 (K and V) x num_layers x num_kv_heads x head_dim x bytes_per_element Worked example — Llama-3-8B shaped at BF16: 2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token 8,192-token context, one user = 1 GiB exactly 40 concurrent users at 8k = 40 GiB <-- larger than the 16 GB of weights Without GQA (32 KV heads) = 512 KiB/token -> 160 GiB for the same load Your model: Per token = ______ KiB x typical context = ______ per user x concurrency = ______ GB total ## 3. Runtime overhead Activations, fragmentation, engine workspace, CUDA context: allow 10-20%. = ______ GB ## 4. Total and hardware Total = weights + KV cache + overhead = ______ GB Hardware implied: ______ Headroom: ______% ## 5. Sensitivity — always include this | Scenario | KV cache | Total | Still fits? | |---|---|---|---| | Baseline | | | | | Concurrency x2 | | | | | Context x4 | | | | | Quantized to INT4 | | | | ## 6. Questions for any vendor quote 1. At what concurrency and context length was this sized? 2. Does it include KV cache, and at what precision is the cache held? 3. What headroom remains at peak, and what happens when it is exceeded? ## Red flag "The model fits in X GB" that names only weights is a single-user sizing. It is technically true and operationally useless. ``` ### Precision & Quantization Reference **Unlocks in module 7.** Bytes per parameter, what each format trades, and how to test quality honestly. ```markdown # Precision & Quantization Reference ## Bytes per parameter | Format | Bytes | Notes | |---|---|---| | FP32 | 4 | Full precision. Training master weights; rarely needed for serving. | | FP16 | 2 | 5 exponent bits, 10 mantissa. More precision, less range. Overflow risk in training. | | BF16 | 2 | 8 exponent bits (FP32 range), 7 mantissa. Robust in training; the modern default. | | FP8 | 1 | Newer hardware. Increasingly used for both weights and KV cache. | | INT8 | 1 | Usually near-lossless for weights on larger models. | | INT4 | 0.5 | Big savings, real quality cost — measure it, never assume it. | ## Fast arithmetic Model size in GB = billions of parameters x bytes per parameter 8B at BF16 = 16 GB | 8B at INT4 = 4 GB 70B at FP16 = 140 GB | 70B at INT4 = 35 GB ## BF16 vs FP16 — the distinction worth knowing Same 16 bits, split differently. BF16 keeps FP32's exponent range and sacrifices mantissa precision, so it rarely overflows during training. FP16 is more precise but narrower in range. This is why BF16 became the training default where hardware supports it. ## Method families - **Post-training quantization** — quantize an already-trained model. Fast, no retraining. (GPTQ, AWQ, bitsandbytes, GGUF k-quants.) - **Quantization-aware training** — train with quantization simulated. Better quality, much more expensive. - **Native low precision** — the model is released already quantized (e.g. MXFP4 weights), so the published size IS the quantized size. ## Honest quality assessment 1. Build an evaluation set from YOUR real tasks, before quantizing anything. 2. Define the failure threshold BEFORE you measure. This is the step people skip, and skipping it is how hardware you have already bought passes its own exam. 3. Test each candidate precision on that set. 4. Check the tails, not just the average — degradation often concentrates in exactly the long, unusual, or specialist inputs you care most about. 5. Record what you measured and on what. "It scored well on a public benchmark" is not evidence about your workload. ## What quantization buys you in serving Freed weight memory becomes KV cache capacity: more concurrent users, or longer contexts, on the same hardware. That is usually the real argument for it — not merely fitting the weights on the card. ``` ### Explainer Pack — 12 answers for non-technical stakeholders **Unlocks in module 8.** The questions boards, CFOs, clinicians and DPOs actually ask, with answers that are accurate AND repeatable. ```markdown # Explainer Pack — accurate answers that survive being repeated The test for every answer here: could the listener repeat it correctly to somebody else? If not, it was decoration. **"What are we actually buying?"** A large file of numbers, plus a small amount of code to run it. The numbers were learned from text and determine how the system responds. There is no database and no rulebook. **"Does our data go into the model?"** Not by using it — sending a prompt does not change the numbers. Whether the SERVICE keeps a copy of what you send is a separate question, and it is the one that matters. That is a contract and architecture question, not a model question. **"Why does it make things up?"** It produces the most plausible continuation of the text so far. A fake reference looks statistically identical to a real one. Nothing in it checks whether a statement is true — that has to be built around it, usually with retrieval and citation. **"Why does it not know about last month?"** The numbers were fixed when training ended. Anything after that has to be supplied at the time of asking. **"Can we look inside and see what it knows about us?"** Not straightforwardly. Meaning is spread across many numbers and each number takes part in many things. Researchers do recover some of it, with effort — nobody just reads it off. **"Why is the bigger model cheaper to run?"** Some models keep many specialists in memory but use only a few for each request. You pay memory for all of them and compute for a few. **"Why does training cost so much more than running it?"** When you run it, each number travels alone. When you train it, each number drags several bookkeeping values along — roughly eight times the memory for the same model. **"Can we fine-tune it on our documents so it knows them?"** Fine-tuning teaches it how to write, not what is true. For documents that must be quoted accurately and change over time, we let it read them at the moment of asking instead. **"Is the smaller quantized version safe to use?"** Compressing the numbers saves memory and costs some accuracy. How much depends on the task, so we measure it on our own work before committing — and we agree what "too much" means before we measure. **"The vendor says it fits on one card."** That counts the model but not the working memory, which grows with every user and every page of context. At our real concurrency the working memory exceeds the model itself. **"Is it safe to download models from the internet?"** Some model file formats can run code when opened. We require the safe format, verify the publisher, and check the licence before anything reaches our infrastructure. **"Can we legally use this one?"** Downloadable is not the same as unrestricted. Several licences specifically limit medical and other high-risk uses, so we read the licence and its acceptable-use policy before shortlisting, not after. ``` ## Capstone ### Track A — Analysis · Model Selection & Sizing Brief The board-ready recommendation: shortlist, derived parameter counts, full memory arithmetic at real concurrency, precision and licence decisions, hardware and cost. The track Elaine is expecting. ### Track A — Analysis · Self-Hosting Business Case Same arithmetic, wider frame: hosted API versus self-hosted over three years, including the KV cache reality, utilisation assumptions, and the operational cost nobody puts in the spreadsheet. ### Track A — Analysis · Model Onboarding Standard The standing control: what must be verified before any model reaches your infrastructure — format, provenance, licence, footprint, template — written so people actually complete it. ### Track B — Build · LLM Weights Explorer Build the interactive tool: parameter calculator from a real config, memory and KV cache calculator, embedding or attention visualiser, quantization comparison — with correct arithmetic and a test suite. ### Track B — Build · Deployment Sizing Calculator Narrower and deeper: a production-quality sizing tool your organisation would genuinely use, validated against real deployments, handling GQA, MoE, quantization and concurrency correctly. ### Your own · Your own decision The best capstone is a real decision you actually face. If your organisation has a live model choice, size and defend that one — the audience pushback will be free and genuine. ### Milestones - **Module 2 — Parameter audit of three real models** - **Module 4 — Annotated architecture map** - **Module 6 — Adaptation decision for a real request** - **Module 7 — Checkpoint safety & licence review** - **Module 8 — Capstone delivered and defended** ### Portfolio checklist - Sealed Week 1 explanation, plus the Week 8 rewrite alongside it - Model inventory: complete, with the Week 1 question marks now answered - Parameter audit of three models including one mixture-of-experts - Annotated architecture map with parameter distribution percentages - Tokenisation cost study for your own domain text - Training run report with three learning rates and failure signatures - Full fine-tune versus LoRA sizing for a real request - Adaptation adjudication with four-option comparison and success measures - Checkpoint safety and licence review with clauses quoted - Deployment sizing worksheet completed at real concurrency, with sensitivity analysis - Capstone (Track A brief or Track B tool), presented and defended