Notes on Reinforcement Learning 19

Replicability crisis and Language Models

Good Morning Everyone!

One of the most important research topics in the RL community has been lately the study of how to make our research more scientific, applicable, and replicable.

On a more global scale, it is no secret that the scientific community has been suffering a replicability crisis, with lots of research being questioned because of the impossibility to repeat it with similar results. It has mostly been tackled in medicine and psychology, but ML researchers have the same concerns, as our research is heavily dependent on the amount of compute and resources, as well as random seeds and methodology.

The paper Towards Deployable RL - What’s Broken with RL Research and a Potential Fix highlights the problem (NRL 3), Empirical Design in Reinforcement Learning tackles it through an educational perspective on good practices (Currently in chapter 4, we will have that summary!!), and this week’s publication, Replicable Reinforcement Learning tries to fix it by proposing an easily replicable algorithm.

In other news, LLM’s clearly are in vogue, and this week’s classification of articles reflects that, with more than 15 articles addressing the topic. At the end of this newsletter some very raw notes on last week’s highlighted article SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters.

On this week’s articles on LLM’s I found these two very interesting:

Mindstorms in Natural Language-Based Societies of Mind

• Minsky's "society of mind" and Schmidhuber's "learning to think" inspire diverse societies of large multimodal neural networks that engage in "Mindstorm" interviews to solve problems.

• Recent implementations of these societies of minds involve large language models (LLMs) and other neural network-based experts communicating through a natural language interface, overcoming the limitations of single LLMs and improving multimodal zero-shot reasoning.

• Natural language-based societies of mind (NLSOMs) allow for the easy addition of new agents using a universal symbolic language, facilitating modular expansion.

• NLSOMs, with up to 129 members, have been assembled and experimented with, successfully addressing practical AI tasks such as visual question answering, image captioning, text-to-image synthesis, 3D generation, egocentric retrieval, embodied AI, and general language-based task solving.

• This serves as a foundation for future NLSOMs with billions of agents, potentially including humans, raising crucial research questions about social structures, the (dis)advantages of different governance models, and the application of neural network economies to maximize rewards in reinforcement learning NLSOMs.

Language Model Self-improvement by Reinforcement Learning Contemplation

• Large Language Models (LLMs) have shown impressive performance in NLP tasks, but fine-tuning them requires significant supervision, which is costly and time-consuming.

• A new unsupervised method called Language Model Self-Improvement by Reinforcement Learning Contemplation (SIRLC) is introduced to enhance LLMs without external labels.

• SIRLC leverages the LLM's ability to assess text quality and assigns it dual roles as a student and teacher. The LLM generates text and evaluates it, with reinforcement learning used to update the model parameters.

• SIRLC demonstrates its effectiveness across various NLP tasks, including reasoning, text generation, and machine translation. Experimental results show improved performance without the need for external supervision, with increased answering accuracy for reasoning tasks and higher BERTScore for translation tasks. SIRLC is also applicable to models of different sizes.

Classification of articles (22nd-28th May)

1. Engineering Applications

   1. Energy

      1. Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty

   2. Image & Video

      1. Collaborative Multi-Agent Video Fast-Forwarding

   3. Industrial Applications

      1. A Mini Review on the utilization of Reinforcement Learning with OPC UA

      2. INVICTUS: Optimizing Boolean Logic Circuit Synthesis via Synergistic Learning and Search

      3. IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Reality

   4. Navigation

      1. Deep Reinforcement Learning-based Multi-objective Path Planning on the Off-road Terrain Environment for Ground Vehicles

      2. How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers

      3. KARNet: Kalman Filter Augmented Recurrent Neural Network for Learning World Models in Autonomous Driving Tasks

      4. Spatio-Temporal Transformer-Based Reinforcement Learning for Robot Crowd Navigation

   5. Networks

      1. Distributed Online Rollout for Multivehicle Routing in Unmapped Environments

      2. Road Planning for Slums via Deep Reinforcement Learning

      3. Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks

      4. Scaling Serverless Functions in Edge Networks: A Reinforcement Learning Approach

      5. Semantic-aware Transmission Scheduling: a Monotonicity-driven Deep Reinforcement Learning Approach

      6. XRoute Environment: A Novel Reinforcement Learning Environment for Routingç

   6. Robotics

      1. Adaptive PD Control using Deep Reinforcement Learning for Local-Remote Teleoperation with Stochastic Time Delays

      2. Aerial Gym -- Isaac Gym Simulator for Aerial Robots

      3. Barkour: Benchmarking Animal-level Agility with Quadruped Robots

      4. Communication-Efficient Reinforcement Learning in Swarm Robotic Networks for Maze Exploration

      5. Constrained Reinforcement Learning for Dynamic Material Handling

      6. Formal Modelling for Multi-Robot Systems Under Uncertainty

      7. FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation

      8. Learning from demonstrations: An intuitive VR environment for imitation learning of construction robots

      9. MARC: A multi-agent robots control framework for enhancing reinforcement learning in construction tasks

      10. M-EMBER: Tackling Long-Horizon Mobile Manipulation via Factorized Domain Transfer

      11. Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning

2. Reinforcement Learning Theory

   1. Actor Critic

      1. Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees

      2. Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities

   2. Attacks

      1. Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in Multi-Agent RL

   3. Control Theory

      1. Reinforcement Learning based optimal control with a probabilistic risk constraint

      2. Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning

   4. Curriculum RL

      1. Reward-Machine-Guided, Self-Paced Reinforcement Learning

   5. Diffussion Models

      1. Training Diffusion Models with Reinforcement Learning

   6. Deep Reinforcement Learning

      1. RLBoost: Boosting Supervised Models using Deep Reinforcement Learning

   7. Distributed RL

      1. The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning

      2. The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model

   8. Empirical Study of RL

      1. Replicable Reinforcement Learning

      2. Testing of Deep Reinforcement Learning Agents with Surrogate Models

   9. Evolutionary Learning

      1. Learning to Act through Evolution of Neural Diversity in Random Neural Networks

   10. Exploration Methods

       1. L-SA: Learning Under-Explored Targets in Multi-Target Reinforcement Learning

       2. Successor-Predecessor Intrinsic Exploration

   11. Explainable/Interpretable Machine Learning

       1. Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation

   12. Feature Engineering

       1. Conditional Mutual Information for Disentangled Representations in Reinforcement Learning

       2. Self-Supervised Reinforcement Learning that Transfers using Random Features

   13. Generalization

       1. Cross-Domain Policy Adaptation via Value-Guided Data Filtering

       2. NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios

   14. Imitation / Inverse / Demonstration Reinforcement Learning

       1. Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations

       2. Coherent Soft Imitation Learning

       3. "GAME-UP: Game-Aware Mode Enumeration and Understanding for Trajectory Prediction

       4. Learning Safety Constraints from Demonstrations with Unknown Rewards

   15. Markov Decision Processes / Deep Theory

       1. Accelerating Value Iteration with Anchoring

       2. Distributional Reinforcement Learning with Dual Expectile-Quantile Regression

       3. Koopman Kernel Regression

       4. Markov Decision Process with an External Temporal Process

       5. Policy Synthesis and Reinforcement Learning for Discounted LTL

       6. Regret-Optimal Model-Free Reinforcement Learning for Discounted MDPs with Short Burn-In Time

       7. Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice

       8. Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning

   16. Multi-Agent RL

       1. A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

       2. Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

       3. Research on Multi-Agent Communication and Collaborative Decision-Making Based on Deep Reinforcement Learning

       4. Towards Efficient Multi-Agent Learning Systems

   17. Neural Networks

       1. Attention Schema in Neural Agents

       2. Augmented Random Search for Multi-Objective Bayesian Optimization of Neural Networks

       3. Deep Reinforcement Learning with Plasticity Injection

       4. Non-parametric, Nearest-neighbor-assisted Fine-tuning for Neural Machine Translation

   18. Offline RL

       1. MADiff: Offline Multi-agent Learning with Diffusion Models

       2. Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure

       3. Offline Experience Replay for Continual Offline Reinforcement Learning

       4. Offline Primal-Dual Reinforcement Learning for Linear MDPs

       5. Sequence Modeling is a Robust Contender for Offline Reinforcement Learning

   19. Online RL

       1. No-Regret Online Reinforcement Learning with Adversarial Losses and Transitions

       2. Online Nonstochastic Model-Free Reinforcement Learning

   20. Optimization

       1. Distributionally Robust Optimization Efficiently Solves Offline Reinforcement Learning

   21. Policy/Value Optimization

       1. Constrained Proximal Policy Optimization

       2. On the Value of Myopic Behavior in Policy Reuse

       3. Policy Representation via Diffusion Probability Model for Reinforcement Learning

   22. Reinforcement Learning from Human Preferences/Feedback

       1. Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback

       2. Provable Offline Reinforcement Learning with Human Feedback

       3. Query-Policy Misalignment in Preference-Based Reinforcement Learning

   23. Reward Optimization

       1. Video Prediction Models as Rewards for Reinforcement Learning

   24. Risk-sensitive/safe/constrained RL

       1. C-MCTS: Safe Planning with Monte Carlo Tree Search

       2. Control invariant set enhanced safe reinforcement learning: improved sampling efficiency, guaranteed stability and robustness

       3. GUARD: A Safe Reinforcement Learning Benchmark

       4. HJB based online safe reinforcement learning for state-constrained systems

       5. Inverse Preference Learning: Preference-based RL without a Reward Function

       6. Inverse Reinforcement Learning with the Average Reward Criterion

   25. Sample-efficiency

       1. Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning

   26. Teacher-Student Framework

       1. Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers

   27. Text-to-image

       1. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models

   28. Theory of Information

       1. Reinforcement Learning with Simple Sequence Priors

   29. Transfer RL

       1. Collaborative World Models: An Online-Offline Transfer RL Approach

       2. PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning

   30. Transformers

       1. Emergent Agentic Transformer from Chain of Hindsight Experience

       2. End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes

       3. Future-conditioned Unsupervised Pretraining for Decision Transformer

       4. Reinforcement Learning finetuned Vision-Code Transformer for UI-to-Code Generation

3. Recommender Systems

   1. Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning

4. Financial Applications

   1. A Comparative Analysis of Portfolio Optimization Using Mean-Variance, Hierarchical Risk Parity, and Reinforcement Learning Approaches on the Indian Stock Market

   2. Market Making with Deep Reinforcement Learning from Limit Order Books

5. Human-agent interaction

   1. A Hierarchical Approach to Population Training for Human-AI Collaboration

6. Games and Game Theory

   1. Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game

   2. Deterministic Algorithmic Approaches to Solve Generalised Wordle

   3. Ghost in the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory

   4. Know your Enemy: Investigating Monte-Carlo Tree Search with Opponent Models in Pommerman

   5. Lucy-SKG: Learning to Play Rocket League Efficiently Using Deep Reinforcement Learning

   6. Reinforcement Learning With Reward Machines in Stochastic Games

7. Physics

   1. Physical Deep Reinforcement Learning: Safety and Unknown Unknowns"

8. Chemistry

   1. ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry

   2. Probing reaction channels via reinforcement learning

9. Healthcare Applications

   1. Control of a simulated MRI scanner with deep reinforcement learning

   2. HuatuoGPT, towards Taming Language Model to Be a Doctor

10. Natural Language Processing

    1. A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agents

    2. Aligning Large Language Models through Synthetic Feedback

    3. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

    4. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks

    5. Gender Biases in Automatic Evaluation Metrics: A Case Study on Image Captioning

    6. Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation

    7. Improving Language Models with Advantage-based Offline Policy Gradients

    8. Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning

    9. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

    10. Language Model Self-improvement by Reinforcement Learning Contemplation

    11. Mindstorms in Natural Language-Based Societies of Mind

    12. Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making

    13. On the Correspondence between Compositionality and Imitation in Emergent Neural Communication

    14. Query Rewriting for Retrieval-Augmented Large Language Models

    15. SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning

11. Mathematics

    1. Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Proving

    2. RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning

SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters

Abstract

• One challenging problem of text-to-image generative models is the generation of Not-Safe-For-Work (NSFW) content

• SneakyPrompt

  • First adversarial attack framework to evaluate the robustness of real-world safety filters in state-of-the-art generative models.

  • Searches for alternate tokens in a prompt that generates NSFW images so that the generated prompt (advesarial prompt) bypasses existing safety filters

  • Uses RL to guide an agent with positive rewardson semantic similarity and bypass success.

• Evaluation

  • SneakyPrompt successfully generatess NSFW content on default closed-box safety filter DALL-E 2

  • Bypasses also several state-of-the-art safety filters on a StableDifussion model

  • Succesfully generates NSFW content and outperforms existing adversarial attacks

Sections

1. Introduction

2. Related Work and Preliminary

3. Problem Formulation

   1. Definitions

   2. Threat Model

4. Methodology

   1. Overview

   2. Baseline Search with Heuristics

   3. Guided Search via Reinforcement Learning

5. Experimental Setup

6. Evaluation

   1. RQ1: Effectiveness in bypassing safety filters

   2. RQ2: Performance compared to baselines

   3. RQ3: Study of different parameter selection

   4. RQ4: Explanation of bypassing

7. Possible Defenses

   1. Defense Type I: Input Filtering

   2. Defense Type II: Training Improvement

8. Conclusion

1. Introduction

• Text-to-image models may generate NSFW content

  • They adopt safety filters, that are bypassable because of their complexity

  • There is a need for a through study of the robustness of these filters

• Attempts:

  • Treat them as a closed box and launcha a text-based adversarial attack

    • TextBugger, Textfooler, BAE

    • TextBugger

      • Generates utility-preserving adversarial texts against text classfication algorithms

    • A text-based attack focuses only on the bypass but not the quality of the generated images, because they are not designed for text-to-image models

    • Eg. When a text found bypasses the safety filter, the NSFW semantics may be lost as well

    • More over: may need large and numerous queries, very costly

  • Rando et al.

    • Reverse engineer StableDiffussion safety filter

    • Propose a manual strategy to bypass the safety filter with extra unrelated content to surround a target prompt

    • 24% success rate

  • Sneaky Prompt

    • First automated framework

    • First adversarial prompt attack to evaluate safety filters on text-to-image models using different search strategies with RL and baselines such as beam, greedy and brute-force.

    • Available on the repository

    • Successfully finds adversarial prompts for SOTA models, including DALL-E and StableDiffusion

    • Outperforms existing adversarial models

8. Conclusion

• First automated framework to evaluate the robustness of existing safety filters via searching the prompt space to find adversarial prompts that bypass safety filter but preserve the semantics

• Categorize safety filters into three categories:

  • Text-based

  • Image-based

  • image-text-based

• Evaluation

  • All existing safety filters are vulnerable to SneakyPrompt

  • Dall-E closed box safety filter is also vulnerable to SneakyPrompt, as opposed to all other existing attacks

  • SneakyPrompt outperforms all other algorithms in terms of bypass rate, FID score and number of queries

• Defenses

  • Proposes possible defenses such as:

    • Input filtering

    • Training improvement

• Expects text-to-image mantainers to improve their safety filters based on the findings of SneakyPrompt

This is all! Have a great week, see you next Monday.