SlotKeeper Fleet.
1.5B Qwen warehouse agent · RL: SFT→GRPO→DPO · matches a 31B teacher
“LRU is optimal when eviction is free. It isn't, in a warehouse — and the moment you account for physical cost, the classical cache textbook stops being right.”
Quick-commerce warehouses (Blinkit, Zepto, Amazon dark stores) all face the same question: which SKUs should live in the hot zone near dispatch? Existing literature treats this as a cache replacement problem and reaches for LRU / LFU. That's correct in software where eviction costs zero clock cycles. In a warehouse, moving an SKU takes 8–14 worker steps, blocks a picker, and incurs a -0.15 reorganization penalty. The optimal physical policy needs to predict future demand and weigh the expected payoff against the eviction cost. That's reasoning under uncertainty — what LLMs are supposed to be good at.
I built an OpenEnv environment that frames this precisely: 8×8 grid, 20 SKUs, 3 robots (2 pickers + 1 mover), Zipfian demand, 200-step episodes, 4 task variants stressing static / shifting / constrained / surge demand. The manager is a pure-prompt LLM that emits structured task assignments each step. Worker pathfinding is deterministic A* — I separated *execution* (solved) from *planning* (learned), so the only thing the LLM does is decide what each worker should do, not how. The reward is a 9-component dense signal where the prefetch bonus only pays out 5–20 steps after the manager's decision, creating a genuine long-horizon credit-assignment problem.
The choices that mattered, with the reasoning at the time.
- Workers deterministic, manager learns
If the LLM had to do pathfinding too, every failure mode would be ambiguous. By making A* deterministic and giving the manager only strategic decisions, every score difference between models maps cleanly to planning quality — not navigation noise.
decision · 01 - 9-component dense reward, not sparse
Sparse 'did you fulfill the order' feedback won't teach prefetching — the credit assignment is too far. The 9-component signal (slot alignment, parallel utilization, idle penalty, reorg cost, …) gives signal every single step, and the prefetch bonus carries the delayed reward that makes the problem genuinely hard.
decision · 02 - Benchmark frontier models before training
I ran 9 models against the env first. Gemma 4 31B scored 0.9412 multi-task average from prompting alone — 2× the heuristic, 26% over LRU. That established the ceiling and gave me a teacher to distill from — the distilled 1.5B Qwen student then matched it.
decision · 03
One code surface that captures the structural decision.
def compute_reward(prev: FleetState, action: Action, next_: FleetState) -> RewardBreakdown:
"""9-component dense reward — every step gets learning signal."""
delivered = next_.deliveries_this_step
return RewardBreakdown(
delivery = sum(1.5 + 1.0 * (1 - d.dispatch_distance / GRID_W) for d in delivered),
prefetch_bonus = 0.5 if any(_is_prefetch_payoff(d, prev) for d in delivered) else 0.0,
slot_alignment = _zone_alignment_score(next_) * 0.02,
parallel_bonus = 0.01 if (next_.p1_busy and next_.p2_busy) else 0.0,
step_cost = -0.02,
reorg_cost = -0.15 * sum(1 for w in next_.workers if w.just_completed_reorg),
idle_penalty = -0.01 * sum(1 for w in next_.workers if w.role == "picker" and w.idle),
invalid = -0.30 * action.invalid_count,
interrupt = -0.05 if action.interrupted_in_progress else 0.0,
)
The case study is more useful when you can poke it.
try moving SKU A into the hot zone (cols 1–2) and watch the prefetch bonus appear. moving a cold SKU costs reorg points.
The trained Qwen 2.5-1.5B + LoRA student is the headline — it reaches 0.9412 multi-task average across all 4 task variants (+26% over LRU), matching the 31B teacher at ~20× fewer parameters, and is published on HuggingFace. The 9 frontier/open models served as baselines (three of four clear LRU). The full pipeline (SFT → GRPO → DPO from 96 Gemma preference pairs → Best-of-N) ran end-to-end on Kaggle 2× T4 free tier. Live env + interactive HTML5 visualizer + 14 trajectory replays + per-step inference logs all open-source.
- trained Qwen-1.5B multi-task avg
- 0.9412
- over LRU baseline
- +26%
- frontier baselines
- 9
- training compute
- Kaggle 2× T4
GRPO from the SFT checkpoint flatlined to 0.0167 — the SFT eval-loss of 0.0525 left the policy near-deterministic, zeroing the GRPO gradient. I documented the negative result transparently and built the DPO + Best-of-N path that recovered the score to 0.9412. Lesson: don't stack RL stages without checking entropy along the way. Next time I'd run rejection sampling against frontier trajectories first, then SFT on a higher-entropy distribution.
- Environment: 8×8 grid · 20 SKUs · 3 robots (2 pickers + 1 mover) · 200-step episodes · Zipf demand
- 9-component dense reward: delivery efficiency, prefetch bonus, slot alignment, parallel bonus, step cost, reorg cost, idle penalty, invalid penalty, interrupt penalty
- 4 task variants: BASIC (static Zipf), ADAPTIVE (shifts at 67/134), CONSTRAINED (6-slot hot zone), SURGE (α=3 spike at 100-130)
- Distillation: Qwen 2.5-1.5B + LoRA r=16 (4-bit NF4) · SFT (Kaggle T4 + Unsloth) → GRPO (50 ep × 100 steps, 2× T4) → DPO (96 pairs mined from Gemma) → Best-of-N (N=4)