Skip to content
PR Quorum
Sign inStart free
← Review gallery
sgl-project/sglang #31674openreviewed 5h ago

fix: preserve speculative draft weights across release/resume_memory_occupation

by acmore+292 02 filesreviewed in 1m 11sView PR on GitHub
minor notesThe panel surfaced minor issues worth a look before merge.3 findings

Reviewer panel

Correctness
deepseek/deepseek-v4-pro
3 found
Security
deepseek/deepseek-v4-pro
0 found
Architecture
deepseek/deepseek-v4-pro
0 found

Findings

01Mmedconf 85%

_export_full_state silently skips tensors that fail to clone, potentially losing draft weights

python/sglang/srt/managers/scheduler_components/weight_updater.py:370

The _export_full_state function catches all exceptions when cloning tensors to CPU and silently skips them. While this is intentional for KV-cache region-tied buffers that become unclonable after the region is paused, the stash happens BEFORE any region is paused (line 200-208). At that point, all tensors should be clonable. If a tensor fails to clone for any other reason (e.g., OOM, device mismatch, corrupted tensor), it is silently dropped from the stash with only an info log. This means the draft model could be partially restored after resume, leading to subtle correctness issues. The function should distinguish between expected failures (region-tied tensors that are genuinely unclonable) and unexpected failures, or at minimum warn at a higher severity when tensors are skipped during the pre-pause stash phase.

02Mmedconf 80%

_import_full_state may leave draft model in inconsistent state if restore partially fails

python/sglang/srt/managers/scheduler_components/weight_updater.py:375

The _import_full_state function restores tensors one-by-one and collects failures. If some tensors succeed and others fail, the draft model is left in a partially-restored state (some weights from the stash, some garbage from the paused region). The two-phase retry mechanism only helps if the failed tensors become restorable after KV_CACHE resume. However, if a tensor fails for a reason that persists (e.g., shape mismatch due to a code change, or a tensor that was skipped during export), the model remains partially corrupted. The warning log on line 287-292 fires but the model continues operating with corrupted weights. This could lead to silent degradation of speculative decoding quality.

03Llowconf 75%

draft stash occurs before WEIGHTS region is paused, but comment says 'BEFORE any region is paused'

python/sglang/srt/managers/scheduler_components/weight_updater.py:199

The comment on line 200 says 'Stash the draft model BEFORE any region is paused', but the code at line 199 checks if GPU_MEMORY_TYPE_WEIGHTS in tags and the stash happens before the WEIGHTS region is paused (line 215+). However, if GPU_MEMORY_TYPE_KV_CACHE is also in tags, the KV_CACHE pause happens at line 203-213 BEFORE the WEIGHTS pause. The comment is slightly misleading since KV_CACHE may be paused before the stash if both tags are present. This is not a bug in practice because the stash targets the draft model's parameters/buffers, not KV cache, but the ordering could matter for FrozenKVMTP drafts where draft buffers are tied to the KV region. The code handles this via the per-tensor guard in _export_full_state, but the comment doesn't accurately reflect the actual execution order when both tags are present.

Want this on your own pull requests?
The same reviewer panel runs on every PR — one focused review, no noise.
Start freeLive demo