Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

Quick Highlights

Abstract

The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts. While effective at reducing immediate toxicity, this approach fundamentally constricts the model's epistemological scope, resulting in over-cautious systems that output uninformative blanket refusals to sensitive yet benign queries. In this work, we challenge the orthodoxy that unsafe data must be discarded. We propose a dialectical approach to alignment, positing that unsafe data encodes rich, domain-specific knowledge critical for nuanced, safe, and informative generation. To operationalize this, we introduce SafeMoE, a Mixture-of-Experts (MoE) framework that isolates unsafe knowledge into domain-specific Low-Rank Adapters (LoRA experts) trained exclusively on harmful corpora. To synthesize safety from these unsafe primitives, we train a lightweight gating network using a minimal, highly curated set of safe-informative responses. During inference, this router dynamically orchestrates the unsafe experts, effectively steering the generation trajectory to harness their deep domain knowledge while strictly enforcing safety constraints. Extensive empirical evaluations across stringent safety benchmarks demonstrate that SafeMoE is not only safer, achieving over a 20% relative improvement in safe response rate (more than a 15% absolute gain), but also produces more informative responses when safety and harmfulness are of paramount concern. Furthermore, the routing mechanism exhibits strong zero-shot generalization to unseen domains and broader safety tasks without domain-specific supervision. Our findings suggest a paradigm shift in alignment: true safety requires not the masking of unsafe knowledge, but its controlled integration.

💡 The Problem: "Safety-by-Erasure"

Recent efforts in LLM safety have focused on preventing the generation of harmful content. However, the prevailing alignment paradigm often over-corrects, causing models to strictly refuse prompts that exhibit even tangential hints of suspicious intent.

The Hidden Danger: In contexts involving psychological distress (e.g., inquiries related to self-harm), uninformative blanket refusals can inadvertently drive well-intentioned users toward adversarial circumvention or migration to less-restrictive, unmoderated platforms, thereby exacerbating the likelihood of harmful outcomes.

🚀 Our Approach: The SafeMoE Framework

By isolating unsafe knowledge into specialized components, we hypothesize that its informative value can be harnessed while steering models toward safe outcomes using a minimal set of aligned examples. We introduce SafeMoE, a Mixture-of-Experts (MoE) framework that utilizes domain-specific Low-Rank Adapters (LoRAs) to "bottle" unsafe knowledge. Uniquely, these experts encapsulate rich, domain-specific primitives that, despite their unsafe nature, are critical for detailed generation.

Figure 1: (1) Unsafe experts are trained using a large set of unsafe response data that can be split into domains. (2) Experts are used to train a router with a smaller set of safe response data. In the second stage, the experts are frozen, while only the router is trainable.

Training Stages

• Stage 1 - Expert Training: SafeMoE first uses unsafe data to train unsafe experts. For efficiency, we use low-rank adapters to train different experts on each domain.
• Stage 2 - Router Training: To leverage the expressive power of unsafe datasets to generate safe responses, our method next tunes the router parameters using a cross-entropy loss such that, for any given example, the model selects a subset, top-k, of the trained expert adapters to use.

📊 Main Results: Safety Across Benchmarks

To assess generalization, we evaluated our method on several standard safety benchmarks: AdvBench, HarmBench, BeaverTails, HarmfulQA, and PKU-Safe. We compared performance against the baselines using raw behavioral prompt sets, without any additional training.

Our models achieve high safety scores even without overlapping unsafe experts, consistently outperforming baselines like SN-Tune and SafeLoRA across all benchmarks. Notably, SafeMoE-XL reaches 97.2% safety on AdvBench while remaining highly informative. While Oyster-I matches SafeMoE-XL's performance, it requires costly reasoning and the scale of 14B parameters.

Safety Rate & Informativeness Comparison

📈 Additional Evaluations

Adversarial Robustness & Data Efficiency

We assess the adversarial resilience of SafeMoE-XL using PAIR with Claude-3.5-Sonnet as the red-teaming agent. As shown below, SafeMoE achieves a near-optimal 1% Attack Success Rate (ASR) on HarmBench, a 41% absolute improvement over the baseline, demonstrating robust generalization against iterative attacks.

Model	HarmBench		PKU-Safe
	ASR (↓)	MJS (↓)	ASR (↓)	MJS (↓)
Mistral-SFT	42%	3.15	62%	5.60
SafeMoE-XL (Ours)	1%	1.10	25%	2.78
Qwen3-4B-Instruct	24%	2.70	48%	3.60
SafeMoE-Qwen (Ours)	20%	2.10	32%	3.10

Table 2: Attack Success Rate (ASR) and Mean Judge Score (MJS) for adversarial robustness evaluation.

Ablating the volume of safe training data reveals remarkable few-shot efficiency. SafeMoE matches the RealSafe-R1 safety baseline with just 20 safe samples per domain, and matches its informativeness with 100 samples.

Figure 2: SafeMoE exhibits strong few-shot efficiency as the number of safe samples per domain scales.

Over-Refusal Analysis

Evaluated on OrBench, SafeMoE maintains high safety on toxic prompts while effectively minimizing over-refusals on hard, benign prompts.

Figure 3: SafeMoE models maintain high performance on hard categories and improve safety on toxic categories without collapsing into blanket refusals.

📝 Citation

If you find our work useful, please consider citing our paper:

@article{hashemzadeh2026dialectics,
  title={Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing},
  author={Hashemzadeh, Maryam and Huang, Jerry and Kim, Minseon and C{\^o}t{\'e}, Marc-Alexandre and Chandar, Sarath},
  journal={arXiv preprint arXiv:2606.00686},
  year={2026}
}