Shielding is a safe reinforcement learning technique that provides safety guarantees while learning, by blocking unsafe actions. Traditional shielding relies on a complete model of the environment to provide long-term guarantees. Probabilistic logic shielding (PLS) lifts the requirement of having such model, but only considers safety in the short term, typically looking only one or few steps ahead. We propose Probabilistic Option Shielding (POS), which extends PLS to the option level, recovering long-horizon safety without reintroducing the need for a complete environment model. POS learns a model-free critic directly from one-step PLS signals, to estimate the probability that an entire option will complete safely under uncertainty. We evaluate POS on safe navigation tasks in the LogiCity environment, and show that it achieves considerably safer training while matching the task performance of standard step-level PLS and other safety related reinforcement learning baselines.
Key ideas
- Traditional shielding requires a complete environment model to ensure safety over long horizons.
- Probabilistic logic shielding (PLS) removes the need for a complete model but only addresses short-term safety.
- Probabilistic Option Shielding (POS) extends PLS to the option level, enabling long-horizon safety without requiring a full environment model.
- POS learns a model-free critic from one-step PLS signals to estimate the likelihood that an entire option completes safely under uncertainty.
- Evaluations in LogiCity safe navigation demonstrate safer training with comparable task performance to step-level PLS and other safety-related RL baselines.
Method overview
- Safety signal: Use PLS to provide short-horizon safety guarantees and block unsafe actions at each step.
- Option-level safety estimate: Train a model-free critic using one-step PLS signals as supervision to predict option completion safety.
- Training-time shielding: During learning, POS blocks options whose predicted safe-completion probability falls below a threshold, while allowing low-risk options to proceed.
- Deployment considerations: At execution time, policy outputs can be constrained to options with sufficiently high predicted safety, aligning with PLS’s role as a safety layer rather than a controller.
Evaluation setup
- Environment: Safe navigation tasks in LogiCity.
- Compared approaches: Standard step-level PLS and other safety-related RL baselines.
- Metrics: Safety during training and task performance relative to baselines.
Model-free critic example
# Inputs:
# - s_t: current state
# - o: selected option (parameterized policy/controller)
# - PLS_step(s_t, a): one-step safety signal for immediate action a
# Goal:
# - Predict P_safe(o | s_t): probability that option o completes safely from s_t
# Training data:
# - (s_t, o, label) where label is derived from PLS-guided rollout outcomes over the option’s horizon
# Architecture (example):
# - Shared feature encoder f_theta(s_t)
# - Gating g_phi(f_theta(s_t), o_embedding)
# - Safety head h_psi(g_phi) -> scalar P_safe(o | s_t) in [0, 1]
# Loss:
# - MSE or BCE between predicted P_safe and observed safe completion labels
How POS uses PLS signals
- At each step within an option’s execution, capture the one-step PLS safety evaluation.
- Aggregate these step-level signals to form a supervision target for the option’s overall safe-completion outcome.
- Update the critic to better estimate the option-level safety probability, enabling long-horizon decisions without a complete environment model.
Safety and performance characteristics
POS achieves considerably safer training while matching the task performance of standard step-level PLS and other safety-related RL baselines.
This indicates that option-level shielding can reduce unsafe exploration over extended horizons without sacrificing task efficacy relative to step-level safety constraints.
Experimental results summary
- Safety: Markedly safer training dynamics compared to non-option shielding approaches.
- Performance parity: Task performance matches step-level PLS and other safety-related baselines.
- Generalization: Effective in LogiCity navigation scenarios, suggesting viability for model-free, long-horizon safety.