Topics in Reinforcement Learning Fall 2024
ECE 750

This advanced topics graduate course will focus on the theories, methods and applications of Reinforcement Learning (RL). RL is an Artificial Intelligence/Machine Learning (AI/ML) approach for building systems that can  learn how to make decisions through their own experiences in an environment. The domain is more difficult than supervised ML since it involves uncertainty and limited information about how the world, and its dynamics, actually function. It can also be seen the AI analogy for the Optimal Control problem, where there are no dynamics models available and the objective is not globally known. 

Expected Background

No background in Reinforcement Learning is required for this course, although we will move quite quickly through the classic concepts and background.  A background in control or decision making theory would be beneficial but not required.

Topics (roughly correct, but needs some updating)

1. Motivation and Context
   • Importance of reasoning and decision making about uncertainty.
   • Connection to Artificial Intelligence and Machine Learning.
   • Probability review.
2. Decision making under uncertainty:
   • Multi-Armed Bandit (MAB) problems, Thompson Sampling.
   • Markov Decision Processes (MDPs)
3. Solving MDPs
   • Theory, Bellman equations
   • Relation to Control Theory
   • Value Iteration, Policy Iteration
4. The Reinforcement Learning Problem
   • Approximately solving MDPs by interacting with the environment
   • SARSA algorithm
   • Q-learning algorithm
   • Other variants of these
5. Temporal Difference Learning  
   • TD(Lambda)
   • Eligibility Traces
6. State Representation
   • Value Function Approximation
   • Stochastic Gradient Descent
7. Deep Reinforcement Learning Deep Q- Networks(DQN)
   • Experience replay buffers, mini-batch training and other methods of training and architecture
8. Direct Policy Search
   • Policy Gradients methods
   • Actor-Critic methods
9. The Explosion of Algorithm Variants on Core Themes
   • A3C, A2C, DPG, DDPG, TRPO, PPO, SAC, ...
   • Decision Transformers
10. Other ways to solve (PO)MDPs:
    • Monte-Carlo Tree Search, Explaining AlphaGo, and beyond
11. Other Uses of RL Concepts
    • Human Preference Elicitation - especially for Foundation Models and Large Language Models
    • Moral Psychology - theories on human moral reasoning rely heavily on RL concept
12. Optional Topics
    • Hierarchical RL
    • Reward Machines
    • Training Spectrum from Supervised to Curriculum to Self-supervised Learning
    • Partially Observable MDPs (POMDPs)
    • Multi-Agent RL (MARL)
    • Curiosity based learning
13. Wrap-up and Review

Open AI Reference Website

This website is a great resource. It lays out concepts from start to finish. Once you get through the first half of our course, many of the concepts on this site will be familiar to you. 

• KeyPapers in Deep RL List - https://spinningup.openai.com/en/latest/spinningup/keypapers.html
• Fundamental RL Concepts Overview- The fundamentals of RL are brieflycovered here. We will go into all this and more in detail in our course. https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
• Family Tree of Algorithms - Here, a list of algorithms at the cuttingedge of RL as of 1 year ago to so, so it'sa good place to find out more. But in a fast growing field, it may be a bit out of date about the latest now. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html

Course Tools                                                                   

I know there has been app/feature/tool creep in courses as they the pandemic has worn on, we're trying to minimize that while still not holding ourselves back when a new tool does something better than an old one.

Course Outline : https://outline.uwaterloo.ca/view/nraur3

Course Website :https://markcrowley.ca/rlcourse/ 

• Course News, Details on Learning Goals
• Schedule of lectures, assignments and tests
• Links to all resources

Learn : Log in to learn.uwaterloo.ca

• Online Course management system for UWaterloo.
• Your grades will be managed here, up until the final grade submission phase of the course.
• Links, announcements and course materials will all be made available here as well.
• Only registered students can access learn.

Crowdmark : (links will be made available as needed)

• A visual grading tool for pdfs submissions of tests and assignments, allows limited online test with mark-down text entry and multiple choice questions.
• Used by the course staff for grading your tests.
• Some assignments and tests might be made available online for submission using this tool as well.

Assessment 

Assessment in the course breaks down as follows:

• 20% Theory and Programming assignment on Fundaments of RL
• 20% Programming assignment on Deep RL
• 25% Midterm Exam on Fundamentals of RL and Deep RL
• 35% Written report/paper/project on a topic covered in one or more papers from the reading list
  • Project, including the presentation, can be done alone or in pairs, research based students are encouraged to work on their own on this.
  • Papers can also be selected, with permission, from recent (past 5-10 years) at top AI/ML/RL conferences/journals such as (AAAI, IJCAI, ICLR, Neurips, AAMAS, RL-Conference, JAIR, JMLR)
• Project Components:
  • 10% Presentation of a paper from the required reading list that will be used in their project. Leading discussion with class about the paper. If working in pairs, both partners must present and discuss for roughly equal amounts of time.
  • 5% Peer Review of another presentation - for prof and the presenter. Must be constructive, balanced, give real feedback and raise questions about the topic.
  • 5% Related to at least one paper from the Reading List and do one or more of the following:
    • Coding: Implement and test the algorithm in use in that paper, or use some of it's concepts to modify some other existing RL algorithm.
    • Theory: Make some theoretical analysis, proof, or claim using one, or more, of the required papers.
    • You could also challenge/examine the paper's experiments, or combine of elements from multiple papers and carry out your own experiments.
  • 15% Written report on what you did beyond the content of the presented/reviewed paper. Summarizing the related paper(s), and explaining the work you did extending it, implementing it, combining it with another paper, or confirming it. The paper must be written in the style of a Machine Learning conference paper. A LaTeX template will be given.
    • If needed, the programming topic of Assignment 2 can be used as a platform for trying out an idea in a more recent paper.

Auditing

For students who wish to audit the course, they will need to complete the requirements of the course project to determine their Pass/Fail evaluation. This includes a presentation in class, a report on the paper and analysis done about it in terms of theory, or experiments via code using your own implementation or the original authors implementation. The presentation will happen during the normal class lectures during the term and must be scheduled. The report will be due by one week after the end of classes.

Getting Help:                                                 

 Discussion board:

 Go there there and sign up with your UWaterloo email now!

Pre-recorded Video Lectures:  These will be made available on the course youtube channel, and links from within Learn

 LEARN Website: The main course content, announcements, grade tracking and materials will be made available on Learn. All registered students should see this in their LEARN courses.

 Email the Teaching Assistant and Instructor: Office Hours will be arranged once term starts as needed.

 AccessAbility Services : http://uwaterloo.ca/accessability-services

 If you need any accommodation, assistance with exams, learning environment, assignments, then contact them for help setting you up as securely and anonymously as possible.

Exam Reference Notes

Note that the midterm exam and final exam will be entirely on paper in person, so any reliance you have on GenAI tools will not be available to you for the bulk of the course grades. For the midterm exam you will be permitted to bring in a couple pages of handwritten reference notes (a.k.a. "cheat sheets). These must be submitted with your exam and will returned afterwards if requested.

Discussion Group Protocols

• Posts on Piazza can be public or anonymous to your classmates, but they will never be anonymous to the TAs and Instructor.
• Be kind. Assume the best, not the worst. Think before you hit enter.
• Posts which are considered offensive, abusive, bullying, discriminatory to any group or person, will be made private or deleted and followed up with private discussion.
• If you feel there is inappropriate, hurtful behaviour occurring on the discussion forum, please notify the professor, TAs or department staff as you feel appropriate.

Recipe for success:                                             

• Ask questions.
• Connect with your classmates.
• Do the assignments.
• Ask questions!
• Most of all, have fun! ...yes really.

Fair Contingencies for Emergency Remote Teaching

We are facing unusual and challenging times. The course outline presents the instructor’s intentions for course assessments, their weights, and due dates in Winter 2022. As best as possible, we will keep to the specified assessments, weights, and dates. To provide contingency for unforeseen circumstances, the instructor reserves the right to modify course topics and/or assessments and/or weight and/or deadlines with due and fair notice to students. In the event of such challenges, the instructor will work with the Department/Faculty to find reasonable and fair solutions that respect rights and workloads of students, staff, and faculty.

Wellness Support and Contact Information.                        

University can be a challenging environment and it is normal to need support from time-to-time. Campus Wellness services are available to students through counselling and health services. If you are struggling or need someone to talk to you, please reach out.

To book an appointment or learn more about the services, call 519-888-4567 x 32655 or explore www.uwaterloo.ca/campus-wellness.

If you're experiencing a crisis and feel unable to cope and Campus Wellness is closed, contact any of these after-hours supports: EmpowerMe (1-833-628-5589), Good2Talk (1-866-925-5454) or Here 24/7 (1-844-437- 3247). They are available at any time of the day or night to help.

COVID-19 Policies                                            

Health and Safety

see these slides on university policy for COVID-19 safety once in-person classes begin again.

 Attendance: Students are to be instructed to attend only the section for which they are registered. If you wish to attend a different section (less people are registered for section 2) you should transfer to that section using official means.

 Absence: Students shall not attend class if they are experiencing influenza-like illness, have been in close contact with someone who is ill, or have travelled outside of Canada within the past 14 days. You will be able to engage with the course content online while reducing the risk of others becoming ill.

 Face coverings: Wearing of face-covering/mask is a requirement in all common areas on campus, including all indoor instructional spaces.

 Students who will not wear masks will be asked to leave the classroom. If the student has a medical reason why they cannot wear a mask they should contact the professor electronically and provide proof of this.

 As such, no food is allowed to be consumed in instructional space. Beverages are allowed if a straw is used or if the mask is lowered only for a brief period.

 Whena student asks or answersa question it may be difficult for them to be heard while wearing a mask. A student may briefly lower their mask to ask/answer the question and then the mask must be replaced.

  Hand hygiene: Students are expected to practice frequent hand hygiene (handwashing with soap and water or use of hand sanitizer), including immediately before coming into an instructional space

 Seating: Students are permitted to sit where they wish. Students are encouraged to sit with one seat left empty between them and other students when possible.

  Student illness: In the event of absence due to influenza-like illness or required self-isolation, students shall submit an Illness Self-declaration. Students can find the Illness Self-declaration form in the  Personal Information section of Quest. A doctor’s note for accommodation is not required.

Generative AI

This course includes the independent development and practice of specific skills, such as programming, algorithm design, machine learning training, and data analysis. Therefore, the use of Generative artificial intelligence (GenAI) trained using large language models (LLM) or other methods to produce text, images, music, or code, like Chat GPT, DALL-E, or GitHub CoPilot, is not permitted in this class for producing assignment outcomes in assessment, study notes, or exams. 

Unauthorized use in this course, such as running course materials through GenAI or using GenAI to complete a course assessment is considered a violation of Policy 71 (plagiarism or unauthorized aids or assistance). Work produced with the assistance of AI tools does not represent the author’s original work and is therefore in violation of the fundamental values of academic integrity including honesty, trust, respect, fairness, responsibility and courage (ICAI, n.d.). 

Permitted uses would include interactive use of LLMs for improving your understanding of course concepts based on your own questions about lecture and assignment material. Further discussion of permitted uses of and expectations for using GenAI will discussed in class and outlined on assignment instructions. 

Note that the midterm exam and final exam will be entirely on paper in person, so any reliance you have on GenAI tools will not be available to you for the bulk of the course grades. For both midterm and final exams you will be permitted to bring in a couple pages of handwritten reference notes only. These will be submitted with your exam and returned to the students if requested afterwards.

You should be prepared to show your work for any work done in the course. To demonstrate your learning, you should keep your rough notes, including research notes, brainstorming, and drafting notes. If the use of GenAI is suspected where not permitted, you may be asked to meet with your instructor or TA to provide explanations to support the submitted material as being your original work. Through this process, if you have not sufficiently supported your work, academic misconduct allegations may be brought to the Associate Dean. 

In addition, you should be aware that the legal/copyright status of generative AI inputs and outputs is unclear. More information is available from the Copyright Advisory Committee: https://uwaterloo.ca/copyright-at-waterloo/teaching/generative-artificial-intelligence 

Students are encouraged to reach out to campus supports if they need help with their coursework including: 

• Student Success Office for help with skills like note taking and time management
• Writing and Communication Centre for assignments with writing or presentations
• AccessAbility Services for documented accommodations
• Library for research-based assignments

Recommendations for how to cite generative AI in student work at the University of Waterloo may be found through the Library: https://subjectguides.uwaterloo.ca/chatgpt_generative_ai. Please be aware that generative AI is known to falsify references to other work and may fabricate facts and inaccurately express ideas. GenAI generates content based on the input of other human authors and may therefore contain inaccuracies or reflect biases. 

You are accountable for the content and accuracy of all work you submit in this class, including any supported by generative AI. 

Faculty of Engineering Guiding Practices 

Territorial Acknowledgement: The University of Waterloo acknowledges that much of our work takes place on the traditional territory of the Neutral, Anishinaabeg and Haudenosaunee peoples. Our main campus is situated on the Haldimand Tract, the land granted to the Six Nations that includes six miles on each side of the Grand River. Our active work toward reconciliation takes place across our campuses through research, learning, teaching, and community building, and is centralized within the Office of Indigenous Relations. 

Inclusive Teaching-Learning Spaces: The University of Waterloo values the diverse and intersectional identities of its students, faculty, and staff. The University regards equity and diversity as an integral part of academic excellence and is committed to accessibility for all. We consider our classrooms, online learning, and community spaces to be places where we all will be treated with respect, dignity, and consideration. We welcome individuals of all ages, backgrounds, beliefs, ethnicities, genders, gender identities, gender expressions, national origins, religious affiliations, sexual orientations, ability – and other visible and nonvisible differences. We are all expected to contribute to a respectful, welcoming, and inclusive teaching- learning environment. Any member of the campus community who has experienced discrimination at the University is encouraged to seek guidance from the Office of Equity, Diversity, Inclusion & Anti-racism (EDI-R) via email at equity@uwaterloo.ca. Sexual Violence Prevention & Response Office (SVPRO), supports students at UWaterloo who have experienced, or have been impacted by, sexual violence and gender-based violence. This includes those who experienced harm, those who are supporting others who experienced harm. SVPRO can be contacted at svpro@uwaterloo.ca

Religious & Spiritual Observances: The University of Waterloo has a duty to accommodate religious and spiritual observances under the Ontario Human Rights Code. Please inform the instructor at the beginning of term if special accommodation needs to be made for religious observances that are not otherwise accounted for in the scheduling of classes and assignments. Consult with your instructor(s) within two weeks of the announcement of the due date for which accommodation is being sought. 

Respectful Communication and Pronouns: Communications with Instructor(s) and teaching assistants (TAs) should be through recommended channels for the course (e.g., email, LEARN, Piazza, Teams, etc.) Please use your UWaterloo email address. Include an academic signature with your full name, program, student ID. We encourage you to include your pronouns to facilitate respectful communication (e.g., he/him; she/her; they/them). You can update your chosen/preferred name at WatIAM. You can update your pronouns in Quest. 

Mental Health and Wellbeing Resources: If you are facing challenges impacting one or more courses, contact your academic advisor, Associate Chair Undergraduate, or the Director of your academic program. Mental health is a serious issue for everyone and can affect your ability to do your best work. We encourage you to seek out mental health and wellbeing support when needed. The Faculty of Engineering Wellness Program has programming and resources for undergraduate students. For counselling (individual or group) reach out to Campus Wellness and Counselling Services. Counselling Services is an inclusive, non-judgmental, and confidential space for anyone to seek support. They offer confidential counselling for a variety of areas including anxiety, stress management, depression, grief, substance use, sexuality, relationship issues, and much more. 

Intellectual Property: Be aware that this course contains the intellectual property of their instructor, TA, and/or the University of Waterloo. Intellectual property includes items such as:

• Lecture content, spoken and written (and any audio/video recording thereof).
• Lecture handouts, presentations, and other materials prepared for the course (e.g., PowerPoint slides).
• Questions or solution sets from various types of assessments (e.g., assignments, quizzes, tests, final exams); and
• Work protected by copyright (e.g., any work authored by the instructor or TA or used by the instructor or TA with permission of the copyright owner).

Course materials and the intellectual property contained therein are used to enhance a student’s educational experience. However, sharing this intellectual property without the intellectual property owner’s permission is a violation of intellectual property rights. For this reason, it is necessary to ask the

instructor, TA and/or the University of Waterloo for permission before uploading and sharing the intellectual property of others online (e.g., to an online repository).

Permission from an instructor, TA or the University is also necessary before sharing the intellectual property of others from completed courses with students taking the same/similar courses in subsequent terms/years. In many cases, instructors might be happy to allow distribution of certain materials. However, doing so without expressed permission is considered a violation of intellectual property rights and academic integrity.

Please alert the instructor if you become aware of intellectual property belonging to others (past or present) circulating, either through the student body or online.

Continuity Plan - Fair Contingencies for Unforeseen Circumstances (e.g., resurgence of COVID-19): In the event of emergencies or highly unusual circumstances, the instructor will collaborate with the Department/Faculty to find reasonable and fair solutions that respect rights and workloads of students, staff, and faculty. This may include modifying content delivery, course topics and/or assessments and/or weight and/or deadlines with due and fair notice to students. Substantial changes after the first week of classes require the approval of the Associate Dean, Undergraduate Studies.

Declaring absences: [undergraduate students and/or courses only]  Regardless of the process used to declare an absence, students are responsible for reaching out to their instructors as soon as possible. The course instructor will determine how missed course components are accommodated. Self-declared absences (for COVID-19 and short-term absences up to 2 days) must be submitted through Quest. Absences requiring documentation (e.g., Verification of Illness Form, bereavement, etc.) are to be uploaded by completing the form on the VIF System. The UWaterloo Verification of Illness form, completed by a health professional, is the only acceptable documentation for an absence due to illness. Do not send documentation to your advisor, course instructor, teaching assistant, or lab coordinator. Submission through the VIF System, once approved, will notify your instructors of your absence. 

Rescheduling Co-op Interviews: Follow the co-op process for rescheduling co-op interviews for conflicts to graded assignments (e.g., midterms, tests, and final exams). Attendance at co-operative work-term employment interviews is not considered to be a valid reason to miss a test.

Academic integrity: In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. [Check the Office of Academic Integrity for more information.]

Grievance: A student who believes that a decision affecting some aspect of their university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4. When in doubt, please be certain to contact the department’s administrative assistant who will provide further assistance.

Discipline: A student is expected to know what constitutes academic integrity to avoid committing an academic offence, and to take responsibility for their actions. [Check the Office of Academic Integrity for more information.] A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about “rules” for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate associate dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline. For typical penalties, check Guidelines for the Assessment of Penalties.

Appeals: A decision made or penalty imposed under Policy 70, Student Petitions and Grievances (other than a petition) or Policy 71, Student Discipline may be appealed if there is a ground. A student who believes they have a ground for an appeal should refer to Policy 72, Student Appeals.

Note for students with disabilities: AccessAbility Services, located in Needles Hall, Room 1401, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with AccessAbility Services at the beginning of each academic term.

Turnitin.com: Text matching software (Turnitin®) may be used to screen assignments in this course. Turnitin® is used to verify that all materials and sources in assignments are documented. Students' submissions are stored on a U.S. server, therefore students must be given an alternative (e.g., scaffolded assignment or annotated bibliography), if they are concerned about their privacy and/or security. Students will be given due notice, in the first week of the term and/or at the time assignment details are provided, about arrangements and alternatives for the use of Turnitin in this course.

It is the responsibility of the student to notify the instructor if they, in the first week of term or at the time assignment details are provided, wish to submit alternate assignment.