IITM BS Large Language Models (BSDA5004): Syllabus and Tips

Large Language Models (BSDA5004) is a 4 credit elective at the degree level of the IITM BS in Data Science. The handbook places it at level 5. You need Deep Learning (BSCS3004) before you take it. The course explains how models like GPT and BERT are built, trained, fine-tuned, tested and scaled.

The course is taught by Prof. Mitesh M. Khapra of the Department of Data Science and AI, IIT Madras. He also teaches Deep Learning and Deep Learning Practice.

Handbook check. The course table agrees on 4 credits, level 5 and the Deep Learning prerequisite. But the handbook's list of minors writes this course as BSCS5004, not BSDA5004. The name is the same, so check the code shown when you register.

What you learn in Large Language Models

The 12 weeks fall into four blocks.

• Weeks 1 and 2: the transformer. Self-attention, cross-attention, masked attention and positional encoding. Then counting parameters, working out compute cost in FLOPs, and an introduction to language modelling.
• Weeks 3 to 5: the main model families. Causal language models such as GPT, and how they are trained and used. Then masked language models such as BERT, fine-tuning, and a close look at tokenisers like BPE, WordPiece and SentencePiece. Then T5, the text-to-text idea that led to prompting, and a map of model types.
• Weeks 6 to 8: data, training and fine-tuning. Datasets and data pipelines, and why clean data matters. Attention variants and scaling methods. Optimisers such as Adam and Lion, learning rate schedules, gradient clipping and common training failures. Then prompt tuning, multi-task fine-tuning, parameter efficient fine-tuning and instruction datasets.
• Weeks 9 to 12: evaluation and scale. Benchmarks such as MMLU, BIG-bench and HELM. Training very large models with mixed precision, activation checkpointing, 3D parallelism and ZeRO, using BLOOM as a case study. Then scaling laws from Chinchilla, Gopher and PaLM 2, and recent advances.

The course also covers common training datasets such as C4, mC4, the Pile and the Stack. There is no textbook. The page lists research papers and articles instead.

How the course is assessed

The course page points to the standard course structure on the programme's Academics page. It does not list a separate assessment plan.

Where it counts

• Minor in Generative AI, with Deep Learning Practice (BSDA5013) and Mathematical Foundations of Generative AI (BSDA5002). See the Generative AI minor.
• Minor in Multimodal AI Systems, with Deep Learning for Computer Vision and Speech Technology, plus Deep Learning Practice as the prerequisite. See the Multimodal AI Systems minor.
• BS level stream rule. It is level 5 and tagged BD, which appears to mean the data science stream. See the BS degree level.

One course can count toward only one minor. If you want both minors, LLM can fill only one, so ask support what replaces it. The DS course table (updated 18 March 2026) marks LLM for May 2026, September 2026 and January 2027.

Who finds it hard and how to prepare

The course moves from ideas to engineering at scale. Weeks 2, 10 and 11 need a different kind of thinking: you reason about parameters, memory and compute, not just concepts. Learning from papers instead of a textbook also takes some getting used to.

• Revise the last weeks of Deep Learning: encoder-decoder models, attention and transformers. The course starts there.
• Count the parameters of a small transformer by hand, layer by layer. Week 2 will then be easy.
• Read the original transformer paper slowly before the term. Note every term you do not know and look it up.
• Train a tiny character level language model yourself. It makes causal language modelling concrete.
• Keep a table of model families (encoder only, decoder only, encoder-decoder) with one example each.

What to take before and after

Deep Learning (BSCS3004) is required first. Mathematical Foundations of Generative AI gives the probability side of generative models. Deep Learning Practice (BSDA5013) is the hands-on partner, with fine-tuning and instruction tuning in its first weeks.

Common questions

Is this course about using ChatGPT or APIs?

No. The syllabus is about how LLMs work inside: architecture, tokenisers, training, fine-tuning, benchmarks and scaling. If you want to build apps with LLM APIs, the Data Science and AI Lab (BSDA4001) covers LangChain, RAG and agents.

How is LLM different from i-NLP?

i-NLP (BSDA5005) covers NLP broadly, from classical linguistics and statistical methods to transformers. LLM goes deeper into transformers alone, including large scale training and scaling laws.

Do I need a powerful GPU?

The course page does not say. Much of the content is about understanding and reasoning, such as FLOPs and scaling laws. For your own practice, small models on a free cloud GPU are enough to learn the ideas.