Reinforcement learning has succeeded across a wide range of games such as Atari, Go, chess, Dota, and Starcraft. However, current algorithms struggle with long and complex tasks unless they are guided by expert knowledge, often in the form of frequent intermediate rewards.
In contrast, humans achieve distant goals by generating abstract plans and then sticking to them over minutes, days, weeks, and even years and adapting the plans when needed. Despite ever-increasing compute budgets, current algorithms lack this ability because they operate in the space of low-level actions for assigning credit, building models, and planning.
What kinds of structure will enable future algorithms to autonomously solve complex tasks with long horizons? What are the best mathematical tools for developing these new algorithms?
Scientific Program
The institute will be open from Friday, March 17, 2023.
The scientific program will take place from Sunday, March 19, 2023 to Thursday, March 24 inclusively.
Each day of the workshop will consist of:
- A morning session (9:30am-noon).
- An evening session (7:30pm-9pm).
- The rest of the day will be left open for discussions and collaborations.
Sessions will cover these and other questions:
- What are our algorithms missing for time, input, and action abstraction?
- What do recent language models tell us about abstraction in RL?
- What can neuroscience tell us about abstraction in RL?
- What will new algorithms for abstract planning look like?
| Day | Time | Topic |
|---|---|---|
| 1 | 9:30am−noon | Intro and identify challenges |
| 7:30pm−9pm | Learning to reach goals | |
| 2 | 9:30am−noon | Abstraction in the brain |
| 7:30pm−9pm | Abstract models through sparsity, causality, events | |
| 3 | 9:30am−noon | Long-horizon behaviors by supervising rationales |
| 7:30pm−9pm | Data, benchmarks, and ethics for RL | |
| 4 | 9:30am−noon | Language models for abstract action |
| 7:30pm−9pm | Long-term memory and credit assignment | |
| 5 | 9:30am−noon | New ideas for abstract planning |
| 7:30pm−9pm | Integrated architectures |
Background Reading
Participants are encouraged to consult the following references in advance of the workshop:
- Between MDPs and Semi-MDPs: A framework for temporal abstraction in reinforcement learning
- Feudal networks for hierarchical reinforcement learning
- WebGPT: Browser-assisted question-answering with human feedback
- Hierarchical motor control in mammals and machines
- Language models are few-shot learners
- Do as I can, not as I say: Grounding language in robotic affordances
- Deep hierarchical planning from pixels
- Optimizing agent behavior over long time scales by transporting value
Participants
Anil Ada(CMU)
David Silver(DeepMind)
Amy Zhang(UAustin/FAIR)
Danijar Hafner(Toronto)
Eszter Vertes(DeepMind)
Yann LeCun(FAIR)
Arthur Gretton(UCL)
John Schulmann(OpenAI)
Timothy Lillicrap(DeepMind)
Olivia Watkins (Berkeley)
Thomas N. Kipf(Brain/Amsterdam)
Rosemary Ke(DeepMind)
Blake Richards(Mila/McGill)
Ryan O'Donnell (CMU)
Nicolas Chapados(ServiceNow)-
Denis Therien(ServiceNow) 
Neil Lawrence(Cambridge)
Boaz Barak(Harvard)
Alexandre Piché(ServiceNow)
Doina Precup(DeepMind/MILA)
Maxime Gasse(ServiceNow)
Sylvie de Lacroix(Birmingham)
Karol Hausman (Brain/Stanford)
Mandana Samiei (McGill/MILA)
Bernhard Schölkopf (Max Planck Institute)
Dzmitry Bahdanau(ServiceNow/MILA)
Jessy Lin (Berkeley)
Jeremy Barnes (ServiceNow)
Laura Smith (Berkeley)
Liam Fedus (OpenAI)
Venue
The workshop will be held at the Bellairs Research Institute of McGill University, Holetown, St. James, Barbados.
For accommodation pricing, see the official page.
Contact
- E-Mail: manager.bellairs@caribsurf.com
- Main office: (246) 422-2087
- Dining hall: (246) 422-2034
- Fax: (246) 422-0692
The Most Important House Rules
Kitchen and Food
- Breakfast is eaten together Saturday-Friday at Bellairs.
- Lunch may be purchased from a grocery store or nearby restaurants.
- Dinner is eaten together Sunday-Thursday at Bellairs.
- We can make coffee and tea in the kitchen any time we want.
- Please leave the kitchen clean.
- There is a guest fridge in the kitchen where we can keep our own private food. Please label your food and remove any left over when you depart.
Showers and Sand
- Sand in the shower drains can cause enormous blockage problems. Please be sure to rinse off the sand from your feet before entering your rooms. There are water taps outside both blocks of rooms for this purpose.
Locked Doors and Valuables
- Barbados is a rather safe country in general but normal precautions when travelling should be taken for your money and valuables.
Telephone
- Telephones and computers are available in the main office (sort of).
Bellairs Survival Hints
Food and Snacks
- We will have a cook and the food is great but if you need anything special please bring it along. There will be a fridge where we can keep our private food items.
- The coffee there is of the instant variety. If you wish to bring your own coffee you may do so.
- Vegitarians may want to bring their favorite non-perishables, however it is not necessary since there is already a diverse selection at the local supermarket. There is also good vegetarian roti in several places near Bellairs.
Beach, Sun, Snorkeling, and SCUBA diving
- Bellairs is situated on one of the best beaches in Barbados, so don't forget your bathing suit (and skin protection) for swims before breakfast and in between work sessions.
- There is also good snorkeling right in front of Bellairs so if you have a mask and fins bring them along too. In fact, if you SCUBA dive bring your gear. There is diving right there as well and air tanks at Bellairs cost only about US$12.00 per tank!
Mosquitos
- Depending on the weather conditions and other factors, we may get some mosquitoes. You should bring some bug repellant just in case.
Travel
Flying in
Please see the Barbados Official Travel Protocols for the rules that are currently in place on the island.
As of January 10, 2023, that site said "Effective midnight, Thursday September 22, 2022, Barbados will discontinue all COVID-19 related travel protocols. Therefore, there will be no testing requirements for entering Barbados whether you are vaccinated or unvaccinated."
Details for travel from the airport will be provided by email.
Map of Bellairs
For questions please contact denis.therien@servicenow.com