Notes on Reinforcement Learning 20

Inverse RL to exploit vulnerabilities

Good morning everyone!

After a very successful crack-climbing Saturday and a very painful Sunday aftermath, I write to you this Monday to update you a bit on how my thesis is going and send you the links for all the interesting stuff that was published on RL.

A friend top-roping one of the pitches I onsighted

On my thesis: I’m now fully a doctoral student, and in the next months I’ll be selecting my subject of study. Had a meeting last week and what I’ll be researching for now will be adversarial examples and inverse reinforcement learning, for a very cool application, the exploration of vulnerabilities in machine learning models.

The things I like about this are that it is very transversal and with a wide horizon of future opportunities.

For some examples of this kind of research, I can recommend the following papers:

• SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters

• Explaining and Harnessing Adversarial Examples

• PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning

• Adversarial Machine Learning Applied to Intrusion and Malware Scenarios: A Systematic Review

And on this week’s classification of articles:

• PAGAR: Imitation Learning with Protagonist Antagonist Guided Adversarial Reward

You may receive some summaries of these papers in the following newsletters.

Also, I’m writing my second article (I’ll present my first one this July at the 9th International Conference on Time Series and Forecasting in Gran Canaria), so that’s very exciting.

Classification of articles on arxiv.org (May 29th - June 4th)

Most interesting papers

1. A Modular Test Bed for Reinforcement Learning Incorporationo Industrial Applications

2. Adaptive and Explainable Deployment of Navigation Skills via Hierarchical Deep Reinforcement Learning

3. DHRL-FNMR: Anelligent Multicast Routing Approach Based on Deep Hierarchical Reinforcement Learning in SDN

4. Simulation and Retargeting of Complex Multi-Charactereractions

5. Deep Q-Learning versus Proximal Policy Optimization: Performance Comparison in a Material Sorting Task

6. Hyperparameters in Reinforcement Learning and How To Tune Them

7. Symmetric Exploration in Combinatorial Optimization is Free!

8. Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction

9. Hierarchical Reinforcement Learning for Modeling User Novelty-Seekingent in Recommender Systems

10. Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

11. NetHack is Hard to Hack

12. PAGAR: Imitation Learning with Protagonist Antagonist Guided Adversarial Reward

13. Offline Meta Reinforcement Learning with In-Distribution Online Adaptation

14. Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

15. Safe Offline Reinforcement Learning with Real-Time Budget Constraints

16. Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity

17. Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

18. Direct Preference Optimization: Your Language Model is Secretly a Reward Model

19. Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

All papers classified

1. Engineering Applications

   1. Computing and Software Engineering

      1. BitE : Accelerating Learned Query Optimization in a Mixed-Workload Environment

   2. Energy

      1. Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility

   3. Image & Video

      1. Controllable Motion Diffusion Model

   4. Industrial Applications

      1. A Modular Test Bed for Reinforcement Learning Incorporationo Industrial Applications

      2. An Architecture for Deploying Reinforcement Learning in Industrial Environments

   5. Navigation

      1. Adaptive and Explainable Deployment of Navigation Skills via Hierarchical Deep Reinforcement Learning

      2. Efficient Learning of Urban Driving Policies Using Bird's-Eye-View State Representations

      3. Investigating Navigation Strategies in the Morris Water Maze through Deep Reinforcement Learning

      4. RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environments

   6. Networks

      1. Average AoI Minimization for Energy Harvesting Relay-aided Status Update Network Using Deep Reinforcement Learning

      2. DHRL-FNMR: Anelligent Multicast Routing Approach Based on Deep Hierarchical Reinforcement Learning in SDN

      3. Training Terahertz Wireless Systems to Battle I/Q Imbalance

   7. Traffic & Flow

      1. Perimeter Control Using Deep Reinforcement Learning: A Model-free Approach towards Homogeneous Flow Rate Optimization

   8. Robotics

      1. Active Reinforcement Learning under Limited Visual Observability

      2. Granular Gym: High Performance Simulation for Robotic Tasks with Granular Materials

      3. LIV: Language-Image Representations and Rewards for Robotic Control

      4. Multi-Robot Path Planning Combining Heuristics and Multi-Agent Reinforcement Learning

   9. UAV's

      1. A Hybrid Framework of Reinforcement Learning and Convex Optimization for UAV-Based Autonomous Metaverse Data Collection

   10. Others

       1. Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning

       2. Simulation and Retargeting of Complex Multi-Charactereractions

2. Reinforcement Learning Theory

   1. Actor-Critic

      1. ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

   2. Algorithms

      1. Deep Q-Learning versus Proximal Policy Optimization: Performance Comparison in a Material Sorting Task

      2. VA-learning as a more efficient alternative to Q-learning

   3. Computational Efficiency

      1. Handling Large Discrete Action Spaces via Dynamic Neighborhood Construction

   4. Continual learning

      1. Policy Optimization for Continuous Reinforcement Learning

   5. Control Theory

      1. Improving the performance of Learned Controllers in Behavior Trees using Value Function Estimates at Switching Boundaries

   6. Empirical Study of RL

      1. Hyperparameters in Reinforcement Learning and How To Tune Them

      2. Replicability in Reinforcement Learning

   7. Exploration Methods

      1. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration

      2. Latent Exploration for Reinforcement Learning

      3. One Objective to Rule Them All: A Maximization Objective Fusing Estimation and Planning for Exploration

      4. Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo

      5. Symmetric Exploration in Combinatorial Optimization is Free!

   8. Explainable/Interpretable Machine Learning

      1. Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction

   9. Federated Learning

      1. Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation

   10. Graph RL

       1. CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

       2. Subequivariant Graph Reinforcement Learning in 3D Environments

   11. Hierarchical RL

       1. Hierarchical Reinforcement Learning for Modeling User Novelty-Seekingent in Recommender Systems

       2. IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control

   12. Imitation / Inverse / Demonstration Reinforcement Learning

       1. Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

       2. NetHack is Hard to Hack

       3. PAGAR: Imitation Learning with Protagonist Antagonist Guided Adversarial Reward

   13. Markov Decision Processes / Deep Theory

       1. Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning

       2. Differentially Private Episodic Reinforcement Learning with Heavy-tailed Rewards

       3. Non-stationary Reinforcement Learning under General Function Approximation

   14. Meta-Learning

       1. Offline Meta Reinforcement Learning with In-Distribution Online Adaptation

   15. Model-based

       1. Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning

   16. Modular RL

       1. Augmented Modular Reinforcement Learning based on Heterogeneous Knowledge

   17. Multi-Agent RL

       1. AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-aware Prioritization

   18. Multi-Task RL

       1. Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

       2. Independent Component Alignment for Multi-Task Learning

   19. Offline RL

       1. A Convex Relaxation Approach to Bayesian Regret Minimization in Offline Bandits

       2. Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding

       3. Efficient Diffusion Policies for Offline Reinforcement Learning

       4. Improving and Benchmarking Offline Reinforcement Learning Algorithms

       5. MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL

   20. Quality Diverse RL

       1. Generating Behaviorally Diverse Policies with Latent Diffusion Models

   21. Quantum Computing

       1. Quafu-RL: The Cloud Quantum Computers based Quantum Reinforcement Learning

   22. Reinforcement Learning from Human Preferences/Feedback

       1. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

       2. How to Query Human Feedback Efficiently in RL?

   23. Representation Learning

       1. Representation-Driven Reinforcement Learning

       2. Towards a Better Understanding of Representation Dynamics under TD-learning

   24. Risk-sensitive/safe/constrained RL

       1. Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning

       2. ROSARL: Reward-Only Safe Reinforcement Learning

       3. Safe Offline Reinforcement Learning with Real-Time Budget Constraints

   25. Robust RL

       1. Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

       2. Learning for Edge-Weighted Online Bipartite Matching with Robustness Guarantees

       3. Solving Robust MDPs through No-Regret Dynamics

   26. Sim-to-real

       1. Privileged Knowledge Distillation for Sim-to-Real Policy Generalization

   27. Transformers

       1. Blockwise Parallel Transformer for Long Context Large Models

   28. Vision-Language Models

       1. Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models

   29. Visual RL

       1. Normalization Enhances Generalization in Visual Reinforcement Learning

   30. Others

       1. TorchRL: A data-driven decision-making library for PyTorch

3. Recommender Systems

   1. Robust Reinforcement Learning Objectives for Sequential Recommender Systems

4. Games and Game Theory

   1. Score-Based Equilibrium Learning in Multi-Player Finite Games with Imperfect Information

   2. What model does MuZero learn?

5. Philosophy and Ethics

   1. Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity

   2. Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

6. Healthcare Applications

   1. Deep Reinforcement Learning Framework for Thoracic Diseases Classification via Prior Knowledge Guidance

   2. FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection

   3. Multi-environment lifelong deep reinforcement learning for medical imaging

7. Natural Language Processing

   1. Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting

   2. Adversarial learning of neural user simulators for dialogue policy optimisation

   3. Chatting Makes Perfect -- Chat-based Image Retrieval

   4. Direct Preference Optimization: Your Language Model is Secretly a Reward Model

   5. Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

   6. Thought Cloning: Learning to Think while Acting by Imitating Human Thinking

See you next week!!