IITM BS Reinforcement Learning (BSDA5007): Syllabus and Tips

Reinforcement Learning (BSDA5007) is a 4 credit elective at the degree level of the IITM BS in Data Science. The handbook places it at level 5. It has no prerequisite, but Deep Learning (BSCS3004) is a corequisite. So you take Deep Learning before it or in the same term. The course teaches how an agent learns to act by trial and reward, from bandits to deep RL.

The course is taught by Prof. Balaraman Ravindran, Head of the Department of Data Science and AI at IIT Madras. The course page and the handbook agree on the credits and the Deep Learning corequisite.

What you learn in Reinforcement Learning

The page says the aim is to help you spot when a problem suits RL, and to implement and test RL methods. The 12 weeks fall into four blocks.

• Weeks 1 to 3: the RL setting. A review of classification, regression, probability and optimisation. Then how RL differs from supervised learning, and the trade-off between exploring and exploiting. Then multi-armed bandits, contextual bandits and the step up to full RL.
• Weeks 4 to 6: the core theory. Markov decision processes, returns, value functions and Q-functions. Then the Bellman equation, dynamic programming, value iteration and policy iteration. Then learning from experience with Monte Carlo, TD learning, SARSA, Q-learning and eligibility traces.
• Weeks 7 to 9: bringing in neural networks. Double Q-learning, and moving from tables to learned functions. Then DQNs with replay buffers. Then policy gradients and the REINFORCE algorithm.
• Weeks 10 to 12: advanced methods. Actor-critic methods such as A3C, improved DQN variants, DDPG and Soft Actor-Critic. Then hierarchical RL and partially observable problems. The last week covers model-based RL and a case study on designing an RL solution for a real problem.

How the course is assessed

The course page lists 12 weeks of coursework, weekly online assignments, 2 in-person invigilated quizzes and 1 in-person invigilated end term exam. For how these usually work, see IITM BS Quiz 1 and Quiz 2 and the end term exam.

Where it counts

• Minor in Algorithmic Decision Making, with Algorithms for Data Science (BSDA5003) and Sequential Decision Making (BSDA6004). See the Algorithmic Decision Making minor.
• BS level stream rule. It is level 5 and tagged BD, which appears to mean the data science stream. See the BS degree level.
• Terms. The DS course table (updated 18 March 2026) marks it only for September 2026, not May 2026 or January 2027. If you want this minor, plan your terms around that.

The handbook says level 5 courses are more complex than level 3 and level 4 courses, with the same credits and fee as level 4.

Who finds it hard and how to prepare

RL has heavy notation. States, actions, rewards, policies, values and expectations all appear in one equation. If you lose track of what each symbol means in weeks 4 and 5, the rest of the course gets much harder. RL code is also hard to debug, because a bug often just makes learning slow rather than crashing.

• Revise expectation and conditional probability before week 1.
• Write out the Bellman equation for a tiny grid world with 4 states by hand. Solve it once with value iteration on paper.
• Code tabular Q-learning yourself on a simple environment before the DQN weeks. Plot the reward per episode.
• Keep a notation sheet from week 4 onward. Add each new symbol with a one line meaning.
• For each algorithm, note whether it is value based or policy based, and whether it learns from its own policy or another one.

What to take before and after

Take Deep Learning (BSCS3004) before or with this course. For the minor, Algorithms for Data Science (BSDA5003) is the other listed level 5 course. Sequential Decision Making is also in the minor, but it is not in the handbook's course table, so ask support about it.

Common questions

Can I take RL and Deep Learning in the same term?

Yes. Deep Learning is a corequisite, not a prerequisite. That allows both in one term. It is a heavy pairing, so plan fewer other courses that term.

Do I need to know game theory or control theory?

The course page does not list either. Week 1 reviews machine learning basics, probability and optimisation, and the course builds up from there.

Why is Deep Learning a corequisite?

The page does not give a reason, but from week 7 the course moves from tables to neural networks. DQNs, policy gradients and actor-critic methods all train networks inside the RL loop, so you need Deep Learning at least in the same term.