Notes on Reinforcement Learning 11: Why pay a psychologist when you have ChatGPT

Topic recommendations for psychologists, code smells and the loss of plasticity in Neural Networks

Good Morning everyone!

I hope you had a great weekend. I myself had a great week, climbing a lot… even in the nighttime! (Not many people know it, but Madrid is THE place in Europe for granite climbing).

One of our adventures this week

However, you are probably expecting more of Reinforcement Learning info and less climbing, so here a bring you very short summaries of three papers that I found interesting.

Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

In this paper, Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf propose a Reinforcement Learning Psychotherapy AI Companion that can generate topic recommendations for therapists by analyzing patient responses. The algorithm has been designed to address four distinct psychiatric conditions, namely anxiety, depression, schizophrenia, and suicidal tendencies.

The authors evaluate the effectiveness of the algorithm by measuring its ability to accomplish the task, establish a bond with the patient, and achieve the desired outcome. The results indicate that the algorithm can capture data relatively accurately but there is clearly much progress to be made before this is useful for practitioners.

Prevalence of Code Smells in Reinforcement Learning Projects

This paper has been written by Nicolás Cardozo, Ivana Dusparic and Christian Cabrera.

The authors hypothesize that the quality of RL code can often be suboptimal due to the fact that it is not exclusively developed by RL Engineers. This can result in issues related to program quality, bugs, maintainability, and evolution problems.

To investigate this matter, a study was conducted on 24 commonly used RL-based Python problems. The study utilized standard software engineering metrics to analyze the quality of the code. The findings revealed that on average, 3.95% of the code base contained code smells, indicating a suboptimal code quality.

The study measured eight different code smells:

1. Long Method

2. Large Class

3. Long Parameter List

4. Long Method Chain

5. Long Scope Chaining: multiply-nested functions

6. Long Ternary Conditional Expression

7. Multiply-Nested Container: containers inside containers

8. Long Lambda Function

The most frequent code smells identified were "long method" and "long method chain," which highlight issues with the definition and interaction of agents. Additionally, "responsibility separation" and the appropriateness of current abstractions for the definitions of RL algorithms were also identified as common code problems.

The authors conclude with some potential improvements:

1. Developing specialized software quality metrics that are specifically tailored to RL projects.

2. Creating more expressive abstractions that can accommodate the complexity of RL problems and algorithms in a simpler way. This could help to make RL code more accessible and easier to understand, which in turn could lead to better program quality and performance.

My opinion

There is an ongoing debate on the data side of the software engineering community about if some of these code smells should be categorized as such. Long parameter lists are very common in Statistics, and rightfully so, since every mathematical model is susceptible to many kinds of tuning with different parameters. Also, one of the best ways to program in pandas involves long method lists, which some people despise but are the bread and butter of the data specialist (see here for more info on that one), and far more practical and efficient than alternatives.

I therefore disagree with the smell codes proposed but I agree with the authors conclusions that there is a lot of work to be done on the software engineering side of things to improve the way code is written in our realm. For starters, I think SOLID principles should be the status quo of code analysis, and they stand true for RL software engineering, as with any object-oriented programming framework.

Loss of Plasticity in Continual Deep Reinforcement Learning

In this paper, authors Zaheer Abbas , Rosie Zhao, Joseph Modayil , Adam White and Marlos C. Machado demonstrate that deep RL agents lose their ability to learn good policies when they cycle through a sequence of Atari 2600 games.

They analyze this phenomenon (loss of plasticity) in a broad number of experiments, concluding that the culprit is that with training, the activation footprint of the network becomes sparser, contributing to diminishing gradients.

They propose a mitigation strategy (Concatenated ReLUs (CReLUs) activation function) and demonstrate its effectiveness.

Classification of papers (13th-19th March)

These are the papers on Reinforcement Learning announced from the 13th through the 19th of March on arxiv.org, classified by area of research.

1. Engineering Applications

   1. Energy

      1. Energy Management of Multi-mode Plug-in Hybrid Electric Vehicle using Multi-agent Deep Reinforcement Learning

      2. MAHTM: A Multi-Agent Framework for Hierarchical Transactive Microgrids

      3. Optimal Energy Management of Plug-in Hybrid Vehicles Through Exploration-to-Exploitation Ratio Control in Ensemble Reinforcement Learning

   2. Image

      1. FindView: Precise Target View Localization Task for Look Around Agents

   3. Industrial Control Systems

      1. Measurement Optimization under Uncertainty using Deep Reinforcement Learning

      2. Real-Time Measurement-Driven Reinforcement Learning Control Approach for Uncertain Nonlinear Systems

   4. Navigation

      1. A Data-Driven Model-Reference Adaptive Control Approach Based on Reinforcement Learning

      2. A Policy Iteration Approach for Flock Motion Control

      3. An Adaptive Fuzzy Reinforcement Learning Cooperative Approach for the Autonomous Control of Flock Systems

      4. Efficient Planning of Multi-Robot Collective Transport using Graph Reinforcement Learning with Higher Order Topological Abstraction

      5. LCS-TF: Multi-Agent Deep Reinforcement Learning-Based Intelligent Lane-Change System for Improving Traffic Flow

      6. Self-Inspection Method of Unmanned Aerial Vehicles in Power Plants Using Deep Q-Network Reinforcement Learning

      7. Sim-to-Real Deep Reinforcement Learning based Obstacle Avoidance for UAVs under Measurement Uncertainty

   5. Networks

      1. A New Intelligent Cross-Domain Routing Method in SDN Based on a Proposed Multiagent Reinforcement Learning Algorithm

      2. Mobile Edge Adversarial Detection for Digital Twinning to the Metaverse with Deep Reinforcement Learning

      3. Muti-Agent Proximal Policy Optimization For Data Freshness in UAV-assisted Networks

      4. Play to Earn in the Metaverse with Mobile Edge Computing over Wireless Networks: A Deep Reinforcement Learning Approach

      5. RL meets Multi-Link Operation in IEEE 802.11be: Multi-Headed Recurrent Soft-Actor Critic-based Traffic Allocation

      6. Reinforcement Learning-based Wavefront Sensorless Adaptive Optics Approaches for Satellite-to-Ground Laser Communication

      7. Terahertz Multiple Access: A Deep Reinforcement Learning Controlled Multihop IRS Topology

   6. Programming

      1. Prevalence of Code Smells in Reinforcement Learning Projects

   7. Robotics

      1. DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

      2. Efficient Learning of High Level Plans from Play

      3. Residual Physics Learning and System Identification for Sim-to-real Transfer of Policies on Buoyancy Assisted Legged Robots

      4. Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks

2. Reinforcement Learning Theory

   1. Actor-Critic

      1. Actor-Critic learning for mean-field control in continuous time

   2. Continual RL

      1. Loss of Plasticity in Continual Deep Reinforcement Learning

   3. Exploration Methods

      1. Fast Rates for Maximum Entropy Exploration

      2. SVDE: Scalable Value-Decomposition Exploration for Cooperative Multi-Agent Reinforcement Learning

   4. Explainable/Interpretable RL

      1. Interpretable Reinforcement Learning via Neural Additive Models for Inventory Management

   5. Graph RL

      1. Fast exploration and learning of latent graphs with aliased observations

   6. Markov Decision Processes / Deep Theory

      1. Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring

      2. Online Reinforcement Learning in Periodic MDP

      3. Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs

      4. Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

      5. Reinforcement Learning for Omega-Regular Specifications on Continuous-Time MDP

   7. Multi-Agent

      1. Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning

      2. Coordinating Fully-Cooperative Agents Using Hierarchical Learning Anticipation

   8. Multi-objective

      1. Latent-Conditioned Policy Gradient for Multi-Objective Deep Reinforcement Learning

   9. Model-based

      1. On the Benefits of Leveraging Structural Information in Planning Over the Learned Model

   10. Needs-driven

   11. Offline RL

       1. Goal-conditioned Offline Reinforcement Learning through State Space Partitioning

   12. Offline-to-Online

       1. Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

   13. Planning

       1. Path Planning using Reinforcement Learning: A Policy Iteration Approach

   14. Policy optimization

       1. Recommending the optimal policy by learning to act from temporal data

   15. Q-Learning

       1. Smoothed Q-learning

   16. Reinforcement Learning from Human Preferences/Feedback

       1. Deploying Offline Reinforcement Learning with Human Feedback

   17. Representation Learning

       1. Unsupervised Representation Learning in Partially Observable Atari Games

   18. Robust RL

       1. PoseExaminer: Automated Testing of Out-of-Distribution Robustness in Human Pose and Shape Estimation

       2. Replay Buffer With Local Forgetting for Adaptive Deep Model-Based Reinforcement Learning

   19. Reward optimization

       1. Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning

   20. World Model Learning

       1. Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting

       2. Transformer-based World Models Are Happy With 100k Interactions

3. Legal Applications

   1. A Short Survey of Viewing Large Language Models in Legal Aspect

4. Financial Applications

   1. Many learning agents interacting with an agent-based market model

5. Human-agent interaction

   1. Affective Workload Allocation for Multi-human Multi-robot Teams

   2. Bridging adaptive management and reinforcement learning for more robust decisions

   3. Multimodal Reinforcement Learning for Robots Collaborating with Humans

6. Game Theory

   1. A New Policy Iteration Algorithm For Reinforcement Learning in Zero-Sum Markov Games

   2. Recent Developments in Machine Learning Methods for Stochastic Control and Games

7. Psychology

   1. Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

8. Healthcare Applications

   1. Towards AI-controlled FES-restoration of movements: Learning cycling stimulation pattern with reinforcement learning

   2. Towards Safe Propofol Dosing during General Anesthesia Using Deep Offline Reinforcement Learning

9. Natural Language Processing

   1. Conversational Tree Search: A New Hybrid Dialog Task

   2. StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model

This is all for this week! Have a great one everyone!