Notes on Reinforcement Learning 3: Fixing RL Research

Towards Deployable RL, Exploration in Deep Reinforcement Learning and Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

Good morning everyone, this week’s notes come packed with features. As promised in last week’s post, this will include a detailed summary of Towards Deployable RL - What’s Broken with RL Research and a Potential Fix, as well as the classification of announced papers on arxiv.org and an overview of my favourites. We’ll begin with this week’s favourites, continue with today’s feature, then the sorted articles on arxiv.org and finish with some notes.

Subscribe to receive next week’s review of Reinforcement Learning from Human Feedback, and why it is important for the deployment of chatGPT.

My favourites

Of those announced on arxiv (9th-16th January), these are the ones that captured my attention . I’m sure other researchers would have chosen differently but I thought these are of great quality.

1. Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

   • How to prevent worst-case scenarios or dead-ends in Reinforcement Learning (this ties with this chapter on last week’s feature, on how to ensure agents explore the world in a safe manner)

2. Learning to Perceive in Deep Model-Free Reinforcement Learning

3. Exploration in Deep Reinforcement Learning: From Single-Agent to Multi-Agent Domain

   • This is actually from 2021 but updated on 2023.

   • A survey on the state of the art in exploration, both for single to multiple agents.

4. Decentralized model-free reinforcement learning in stochastic games with average-reward objective

5. Mastering Diverse Domains through World Models - Deepmind

6. Reducing Exploitability with Population Based Training

7. Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?

At the end of this newsletter I’ll include my notes on Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning and Exploration in Deep Reinforcement Learning: From Single-Agent to Multi-Agent Domain.

Towards Deployable RL - What’s Broken with RL Research and a Potential Fix

This is a paper written by Shie Mannor and Aviv Tamar, both at Technion, Israel’s Institute of Technology. I would characterize this paper as a Deployable RL manifesto, as its aim is to encourage research practices that close the current gap between RL and real-world applications. To them, deployable AI means RL that can work at scale, be economically feasible, and can eventually be put in the field.

They heavily criticise current focus on benchmarks, arguing that it yields very little practical value, and instead advocate for a focus on challenges, which they define as a real-world problem sponsored by a group of researchers from academia/industry. In their words: instead of comparing algorithms, solve a real world problem and bring real-world value.

They end by encouraging people to contribute challenges, frame their own research with this point of view and criticise research that is void of real-world value.

Here I offer you a schematic, heavily summarized version, but I encourage you to also read the full thing at Aviv Tamar’s Substack, link at the end.

Sections

1. Introduction

2. Generalist Agents VS Deployable RL Systems

3. Principles of Deployable RL Research

4. Actionable Next Steps

1. Introduction

• RL is not living up to our expectations.

• Here are five popular research practices that were relevant for 2015, but are currently stagnating the field:

  • Overfitting to Specific Benchmarks

    • Nearly every paper is required to show some distinction for one of the popular benchmarks (Atari, Mujoco...) which leads to tweaks that only work for those benchmarks.

    • Progress in benchmarks does not yield tangible real-world value

  • Wrong Focus

    • Current benchmarks ignore the deployable nature of situated (RL-driven) agents completely, focusing on algorithms rather than on a system/engineering view.

    • OpenAI GYM abstracts away all system-design issues

    • This hampers progress since in many practical problems, figuring out what are the useful states, actions, and rewards is a critical component of the development process

  • Detached Theory

    • Useful theory seems to be quite rare:

      • Regret minimization is overly pessimistic

      • There is a lot of prior knowledge in practical problems that is not accounted for

      • Finite states and actions are not a good model for many problems of interest

      • There is a focus on unimportant qualities

  • Uneven Playing Grounds

    • Performance is confounded by resources available to the implementer:

      • Proficiency in hyper-parameter tuning

      • Size of NN

      • Prior knowledge about the problem/solution

      • Variability of scale and the current trend at top conferences to prefer masive experimentation over conceptual novelty, can inhibit long-term progress

  • Lack of Experimental rigor

    • Impressive singular experiments give a false sense of progress

    • We need more rigorous evaluation of difficulty and success: stability, deployment time and cost, testability and life-cycles are critical

    • The publication standard is that failure cases are almost never reported, stability is impossible to tell, and software design issues are not discussed.

2. Generalist Agents vs Deployable RL Systems

• Two views:

  • Generalist/AI first: Future progress will be made by focusing attention on large-scale training of agents that solve diverse problems, with the hope that along the way a generalist agent will develop, and be a useful component in various real world problems

  • Deployable RL/RL second: Seek to design RL algorithms that solve concrete real world problems

• The five problems from the introduction are relevant to both approaches but, with the current state of the field, we should seek the second (Deployed RL) approach.

3. Principles of Deployable RL Research

• At present, RL is uneconomical to deploy.

• To change it we need:

  • to research on how to deploy it effectively

  • to understand better the gains RL brings to practical problems

• They propose a constructive model with three general principles around challenges:

  • Challenges instead of benchmarks:

    • Challenge: problem sponsored by a group of researchers from academia/industry: instead of comparing algorithms, solve a real world problem and bring real world value

    • Credit for contributing challenges: A rigorous presentation of a challenge (description, community and supporting platform) should be credited

    • Measurable progress is the main criterion for publication: Every publication should explain the limitations and issues with the proposed algorithm and how it addresses problems specifically.

    • Weight class: Computing power available should be reported to address the significance of the results

  • Theory papers shoud address specific challenges:

    • The goal of the research should be well justified in terms of its potential impact on real world problems. Should also consider problems that have to do with the life cycle of software such as data acquisition, debugging, testability and performance deterioration.

  • Design Patterns Oriented Research:

    • Real-world RL based systems should have conceptual solutions to problems where issues such as terstability, debuggability, and other system life-cycle issues are addressed.

    • We forsee that one way to make significant progress on a challenge would be by developing novel design patterns for it.

4. Actionable Next Steps:

• To us, deployable RL means RL that can work at scale, be economically feasible, and can eventually be put in the field.

• Here is what you can do:

  • Contribute challenges:

    • Requires deep understanding of the application domain, with many disciplines of impact (healthcare, engineering...)

    • Might require industry and academia joining forces

  • Frame your own research: Frame the research effort within deployable RL principles.

  • Critize others' research: Coordinate with researchers, reviewers and area chairs. Ask how a paper gets the field closer to real-world impact.

You can find find the whole article over at Aviv Tamar’s Substack or at arxiv.org.

Towards Deployable RL - What’s Broken with RL Research and a Potential Fix

Announced Papers: 9th-16th January

1. Engineering Applications

   1. From Ember to Blaze: Swift Interactive Video Adaptation via Meta-Reinforcement Learning

   2. Hierarchical Deep Q-Learning Based Handover in Wireless Networks with Dual Connectivity

   3. Traffic Steering for 5G Multi-RAT Deployments using Deep Reinforcement Learning

   4. Reinforcement Learning-based Joint Handover and Beam Tracking in Millimeter-wave Networks

   5. Learning to Control and Coordinate Hybrid Traffic Through Robot Vehicles at Complex and Unsignalized Intersections

   6. MotorFactory: A Blender Add-on for Large Dataset Generation of Small Electric Motors

   7. Long-distance migration with minimal energy consumption in a thermal turbulent environment

   8. A Decentralized Pilot Assignment Methodology for Scalable O-RAN Cell-Free Massive MIMO

   9. Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models

   10. SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning

       1. Interesting

   11. ORBIT: A Unified Simulation Framework for Interactive Robot Learning Environments

   12. Deep Reinforcement Learning for Autonomous Ground Vehicle Exploration Without A-Priori Maps

   13. Hint assisted reinforcement learning: an application in radio astronomy

   14. Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning

   15. Imbalanced Classication In Faulty Turbine Data: New Proximal Policy Optimization

   16. Learning-based Design and Control for Quadrupedal Robots with Parallel-Elastic Actuators

   17. Multi-UAV Path Learning for Age and Power Optimization in IoT with UAV Battery Recharge

   18. Tuning Path Tracking Controllers for Autonomous Cars Using Reinforcement Learning

   19. Network Slicing via Transfer Learning aided Distributed Deep Reinforcement Learning

   20. Enabling AI-Generated Content (AIGC) Services in Wireless Edge Networks

   21. Modeling Scattering Coefficients using Self-Attentive Complex Polynomials with Image-based Representation

   22. Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

   23. Fairness Guaranteed and Auction-based x-haul and Cloud Resource Allocation in Multi-tenant O-RANs

   24. Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement

   25. RL-DWA Omnidirectional Motion Planning for Person Following in Domestic Assistance and Monitoring

   26. Imitation Learning-based Implicit Semantic-aware Communication Networks: Multi-layer Representation and Collaborative Reasoning

   27. ECSAS: Exploring Critical Scenarios from Action Sequence in Autonomous Driving

   28. TarGF: Learning Target Gradient Field for Object Rearrangement

   29. MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control

   30. When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning

   31. Interpretable Hidden Markov Model-Based Deep Reinforcement Learning Hierarchical Framework for Predictive Maintenance of Turbofan Engines

   32. Verifying Learning-Based Robotic Navigation Systems

   33. DROPO: Sim-to-Real Transfer with Offline Domain Randomization

2. Healthcare Applications

   1. Multi-Target Landmark Detection with Incomplete Images via Reinforcement Learning and Shape Prior Embedding

   2. Towards AI-controlled FES-restoration of arm movements: Controlling for progressive muscular fatigue with Gaussian state-space models

   3. Towards AI-controlled FES-restoration of arm movements: neuromechanics-based reinforcement learning for 3-D reaching

3. Mathematical Theory

   1. Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm

   2. Randomization Tests for Adaptively Collected Data

   3. Asynchronous training of quantum reinforcement learning

   4. Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

   5. Safe Policy Improvement for POMDPs via Finite-State Controllers

   6. An Analysis of Quantile Temporal-Difference Learning

   7. Adversarial Online Multi-Task Reinforcement Learning

   8. Sequential Fair Resource Allocation under a Markov Decision Process Framework

   9. Minimax Weight Learning for Absorbing MDPs

   10. IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling

   11. On the Convergence of Discounted Policy Gradient Methods

   12. SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration

   13. Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification

   14. An approximate policy iteration viewpoint of actor-critic algorithms

   15. A Deep Reinforcement Learning Framework for Column Generation

   16. Investigating the Properties of Neural Network Representations in Reinforcement Learning

   17. Self-Enhancing Multi-filter Sequence-to-Sequence Model

   18. Trajectory Modeling via Random Utility Inverse Reinforcement Learning

   19. Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence

4. Reinforcement Learning Theory

   1. Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

   2. Mutation Testing of Deep Reinforcement Learning Based on Real Faults

   3. A Constrained-Optimization Approach to the Execution of Prioritized Stacks of learned Multi-Robot Tasks

   4. Predictive World Models from Real-World Partial Observations

   5. schlably: A Python Framework for Deep Reinforcement Learning Based Scheduling Experiments

   6. Actor-Director-Critic: A Novel Deep Reinforcement Learning Framework

   7. Learning to Perceive in Deep Model-Free Reinforcement Learning

   8. Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration

   9. Backward Curriculum Reinforcement Learning v2

   10. Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum

   11. Universally Expressive Communication in Multi-Agent Reinforcement Learning

   12. When does return-conditioned supervised learning work for offline reinforcement learning?

   13. A Generic Graph Sparsification Framework using Deep Reinforcement Learning

   14. Exploration in Deep Reinforcement Learning: From Single-Agent to Multi-Agent Domain

   15. Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

   16. Reinforcement Learning for Joint Optimization of Multiple Rewards

5. Financial Applications

   1. AI-driven Prices for Externalities and Sustainability in Production Markets

6. Transformer Theory

   1. TransfQMix: Transformers for Leveraging the Graph Structure of Multi-Agent Reinforcement Learning Problems

   2. Language-Informed Transfer Learning for Embodied Household Activities

7. Game Theory

   1. Decentralized model-free reinforcement learning in stochastic games with average-reward objective

   2. Switchable Lightweight Anti-symmetric Processing (SLAP) with CNN to Reduce Sample Size and Speed up Learning – Application in Gomoku Reinforcement Learning

   3. Mastering Diverse Domains through World Models

   4. On Reinforcement Learning for the Game of 2048- Phd dissertation

   5. Reducing Exploitability with Population Based Training

   6. Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?

   7. On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Games

   8. Goal Misgeneralization in Deep Reinforcement Learning

Notes on Reinforcement Learning 3.1: Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

One of the problems highlighted in The Challenge of IA Value Alignment

GREAT PAPER

• Taylor K William - University of Toronto

• Sonali Parbhoo - Imperial College London

• Marzyeh Ghassemi - Massachussets Institute of Technology

Abstract

• Identifying worst-case scenarios or dead-ends is crucial in safety-critical scenarios

• This situations are rife with uncertainity due to stochastic environments and limited offline training data

• Distributional Dead-End Discovery (DistDeD): a framework to identify worst-case decision points based on estimated distributions of the return of a decision.

• Used on a toy domain as well as assesing the risk of death severely ill patients

• Results: improves prior discovery approaches by increasing detection 20% and providing indications of the risk 10 hours earlier on average

Sections

1. Introduction

2. Related Work

   1. Safe and Risk-Sensitive RL

   2. Non-stationary and Uncertainty-Aware RL

   3. RL in safety critical domains

3. Preliminaries

   1. Distributional RL

   2. Conservatism in Offline RL

   3. Risk Estimation

   4. Dead-end Discovery (DeD)

4. Risk-sensitive Dead-end Discovery

5. Illustrative Demonstration of DistDeD

6. Assesing Medical Dead-ends with DistDeD

   1. Data

   2. State Construction

   3. D- and R- Networks

   4. Training

   5. Experimental Setup

   6. Results

      1. DistDeD Provides earlier Warning of Patient Risk

      2. DistDeD Allows for a Tunable Assesment of Risk

      3. CQL Enhances DistDeD Performance

7. Discussion

   1. Limitations

   2. Broader Impact

   3. Author Contributions

   4. Acknowledgements

1. Introduction

• In complex, safety-critical scenarios, being able to identify signs of rapid deterioration is critical: like replacing components within high value machinery or healthcare evaluation

• Quantifying worst--case outcomes is usually challenging as a result of unknown stochasticity in the environment (compunded over a sequence of decisions), potentially changing dynamics, and limited data.

• RL is a natural paradigm to address sequential decision-making tasks in safety-critical settings, focusing on maximizing the cumulative effects of decisions over time.

• Many approaches relie on a priori knowledge about which states and actions to avoid, but this is not feasible in many real-world tasks as this may be unknown due to unknown interactions between selected actions and the observed state.

• RL in high-risk settings is fully offline and off-policy due to ethical and legal reasons. As a consequence, it is very affected by the data collected and confounding information may lead to the overestimation of anticipated return, biased decisions and/or overconfident yet erroneous predictions, as well as overlooking rare but dangerous situations.

• In general, Rl has been used in risk-neutral situations.

• DeD (cite) is a framework that takes into account risk by avoiding actions proportionally to their risk of leading to dead-ends.

  • Recorded negative outcomes are leveraged to identify behaviours that should be avoided

  • Actions that lead to dead-ends are identified based on threshold point-estimates of the expected return of an action rather than considering the full distribution

    • Risk estimation in DeD is limited and too optimistic about determining which actions should be avoided.

    • By underestimating the risk associated with a particular action, we are unable to determine whether an action could be potentially dangerous.

• DistDeD:

  • Risk-sensitive decision framework positioned to serve as an early-warning system for dead-end discovery

  • Tool for thinking about risk-sensitivy in data-limited offline settings

  • Contributions:

    1. Provide distributional estimates of the return to determine whether a certain state is at risk of becoming a dead-end from the expected worst-case outcomes over available decisions.

    2. Establish DistDeD as a lower bound to DeD results -> Able to detect and provide earlier indication of high risk scenarios

    3. Modelling the full distribution -> Spectrum of risk-sensitivity when assesing dead-ends, tunable risk estimation procedures and can be customized

    4. Empirical evidence that DistDeD enables an earlier determination of high-risk areas of the state space on both a simulated environment and a real-world application.

       

       

7. Discussion

• Justification, foundational evidence and preliminary findings on DistDeD

• Limitations:

  • Discrete action spaces

  • Binary reward signal

  • Dead-ends are derived from a single condition, most real world-scenarios are more complex

  • Does not make causal claims about the impact of each action

• Broader Impact

  • Intended for assistance to domain experts, not for usage in isolation

  • It asseses high-risk situations early enough so that the human decision maker can make a decision

  • Misuse could be fatal

Notes on Reinforcement Learning 3.2: Exploration in Deep Reinforcement Learning: From Single-Agent to Multi-Agent Domain

2021 -IEEE Transactions on Neural Networks and Learning Systems

Updated 12 January 2023

Abstract

• DRL and MARL is known to be sample inefficient, preventing real-world applications

• One bottleneck is the exploration challenge: how to efficiently explore the environment collecting informative experiences

• Comprehensive survey on existing exploration methods for both single-agent and multi-agent RL.

• First: identify challenges

• Second: Survey with two categories: uncertainity-oriented exploration and motivation-oriented exploration, as well as other notable exploration methods

• Both algorithmic analysis and comparison on DRL benchmarks

• Summarization and future directions

Sections

1. Introduction

2. Preliminaries

   1. Markov Decision Process & Markov Game

   2. Reinforcement Learning Methods

      1. Value-based methods

      2. Policy Gradient Methods

      3. Actor-Critic Methods

      4. MARL Algorithms

   3. Basic Exploration Techniques

      1. Epsilon-Greedy

      2. Upper Confidence Bounds

      3. Entropy Regularization

      4. Noise Perturbation

   4. Exploration based on Bayesian Optimization

      1. Gaussian Process-Upper Confidence Bounds (GP-UCB)

      2. Thompson Sampling (TS)

3. What Makes Exploration Hard in RL

   1. Large State-action Space

   2. Sparse, Delayed Rewards

   3. White-noise Problem

   4. Multi-agent Exploration

4. Exploration in Single-agent DRL

   1. Uncertainity-oriented Exploration

      1. Exploration under Epistemic uncertainty

         1. Parametric Posterior-based Exploration

         2. Non-parametric Posterior-based Exploration

      2. Exploration under Aleatoric Uncertainty

      3. Exploration under Both Types of Uncertainty

   2. Intrinsic Motivation-oriented Exploration

      1. Prediction Error

      2. Novelty

      3. Information Gain

   3. Other Advanced Methods for Exploration

      1. Distributed Exploration

      2. Exploration with Parametric Noise

      3. Safe Exploration

5. Exploration in Multi-Agent DRL

   1. Uncertainty-oriented Exploration

   2. Intrinsic Motivation-oriented Exploration

   3. Other Methods for Multi-Agent Exploration

6. Discussion

   1. Empirical Analysis

   2. Open Problems

      1. Exploration in Large Open space

      2. Exploration in Long-horizon Environments with Extremely Sparse, Delayed Rewards

      3. Exploration with White Noise Problem

      4. Convergence

      5. Multi-Agent Exploration

      6. Safe Exploration

7. Conclusion

7. Conclusion

• Suggestions and insights:

  • Current Exploration methods are evaluated mainly in terms of cumulative rewards and sample efficient on a handful of well-known environments

  • The essential connections beteen different exploration methods are to be further revealed

  • Exploration among large action space, long horizon environments and convergence analysis are relatively lacking studies

  • Multi-Agent Exploration can be even more challenging due to complex multi-agent interactions. Coordinated exploration with decentralized execution and exploration under non-stationarity may be the key problems to address.

The end

This has been this week’s post on Notes on Reinforcement Learning. Tune next week for a review of Reinforcement Learning from Human Feedback, and why it is important for the deployment of chatGPT.