Reframe the cap48 finding as seed-sensitive after the second rerun

Persist the completed seed123 benchmark showing hybrid ahead again, and update the strategy guidance from single-run winner claims to a multi-seed interpretation.

Constraint: Only documentation changes are allowed because benchmark outputs remain outside version control
Rejected: Keep framing cap48 as a stable high_energy win | The second seed materially weakens that interpretation
Confidence: high
Scope-risk: narrow
Directive: Base the hybrid vs high_energy default decision on aggregated multi-seed evidence, not any single cap48 run
Tested: Verified /tmp/ab_smoke_seg_cap48_top2_seed123/report.json; verified high_energy eval.json; verified docs now record hybrid=24/0.9583/1.0 and high_energy=24/0.9167/1.0 for seed123
Not-tested: Formal aggregation across multiple seeds beyond these two cap48 runs

@@ -2,6 +2,31 @@
 
 
 ## 2026-06-02
 ## 2026-06-02
 
 
+### Stage: 收尾 cap48 seed123 并确认 cap48 对 seed 敏感
+
+完成项：
+- 读取 `/tmp/ab_smoke_seg_cap48_top2_seed123/report.json`
+- 读取：
+  - `/tmp/ab_smoke_seg_cap48_top2_seed123/hybrid/fma_reports_smoke/eval.json`
+  - `/tmp/ab_smoke_seg_cap48_top2_seed123/high_energy/fma_reports_smoke/eval.json`
+- 更新：
+  - [open-dataset-workflow.md](./open-dataset-workflow.md)
+  - [session-handoff.md](./session-handoff.md)
+  - [CHANGELOG.md](./CHANGELOG.md)
+
+最终结果（subset=48, `max_test_queries=24`, `seed=123`）：
+- `hybrid`: `num_queries=24`, `top1=0.9583`, `topk=1.0`
+- `high_energy`: `num_queries=24`, `top1=0.9167`, `topk=1.0`
+
+结论：
+- cap48 在不同 seed 下已经出现明显不同排序：
+  - 默认 seed：`high_energy > hybrid`
+  - `seed=123`：`hybrid > high_energy`
+- 这意味着 cap48 当前最可靠的结论不是“谁绝对赢”，而是：
+  - **该对比对 seed/抽样敏感**
+  - 当前默认策略判断必须依赖多 seed 聚合结果
+  - `hybrid` 仍可保留为保守默认，`high_energy` 仍是强竞争方案
+
 ### Stage: 启动 cap48 第二个 seed 复核反转结果
 ### Stage: 启动 cap48 第二个 seed 复核反转结果
 
 
 完成项：
 完成项：

@@ -206,6 +206,23 @@ flowchart LR
   - **`hybrid` 仍是当前保守默认**
   - **`hybrid` 仍是当前保守默认**
   - **`high_energy` 已成为强竞争方案**
   - **`high_energy` 已成为强竞争方案**
   - 下一步必须做更大样本或多随机种子复核，不能只靠单轮 cap48 就完全改默认
   - 下一步必须做更大样本或多随机种子复核，不能只靠单轮 cap48 就完全改默认
+
+### 更新：cap48 第二个 seed 复核（subset=48, `max_test_queries=24`, `seed=123`）
+
+对同一规模再跑第二个 seed 后，结果又回到 `hybrid` 领先：
+
+| 排名 | 策略 | num_queries | top1 | topk |
+|---:|---|---:|---:|---:|
+| 1 | `hybrid` | 24 | 0.9583 | 1.0 |
+| 2 | `high_energy` | 24 | 0.9167 | 1.0 |
+
+这说明：
+- cap48 的策略排名对 seed / 抽样子集 **敏感**
+- 单次 cap48 不能作为“high_energy 已全面反超”的充分证据
+- 当前最稳妥结论仍是：
+  - `hybrid` 保留为保守默认
+  - `high_energy` 保留为强竞争方案
+  - 后续需要 **多 seed 聚合结论**，而不是看单次跑分
 /usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --index-prefix data/index_fma_smoke/reference --split test --device cpu --fast-eval --output-json reports/fma-smoke/eval.json
 /usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --index-prefix data/index_fma_smoke/reference --split test --device cpu --fast-eval --output-json reports/fma-smoke/eval.json
 /usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/fma-smoke/eval.json --config-json reports/fma-smoke/config.json --output-dir reports/fma-smoke --model-version fma-smoke --data-version fma_local
 /usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/fma-smoke/eval.json --config-json reports/fma-smoke/config.json --output-dir reports/fma-smoke --model-version fma-smoke --data-version fma_local
 ```
 ```

@@ -506,7 +506,7 @@ cap48 top2 最终结论：
 
 
 ---
 ---
 
 
-## 13. cap48 top2 第二个 seed（进行中）
+## 13. cap48 top2 第二个 seed（已完成）
 
 
 为验证 cap48 的“high_energy 反超”是否稳定，已启动第二个 seed：
 为验证 cap48 的“high_energy 反超”是否稳定，已启动第二个 seed：
 
 
@@ -527,7 +527,7 @@ cd /workspace/acr-engine
   --output-json /tmp/ab_smoke_seg_cap48_top2_seed123/report.json
   --output-json /tmp/ab_smoke_seg_cap48_top2_seed123/report.json
 ```
 ```
 
 
-当前 fresh evidence：
+最终结果：
 
 
 | 项目 | 状态 |
 | 项目 | 状态 |
 |---|---|
 |---|---|
@@ -535,14 +535,14 @@ cd /workspace/acr-engine
 | `max_test_queries` | `24` |
 | `max_test_queries` | `24` |
 | `seed` | `123` |
 | `seed` | `123` |
 | `hybrid` | `num_queries=24`, `top1=0.9583`, `topk=1.0` |
 | `hybrid` | `num_queries=24`, `top1=0.9583`, `topk=1.0` |
-| `high_energy` | `run_demo.py build-index --resume --checkpoint-every-refs 100` |
-| `report.json` | 尚未生成 |
-
-恢复检查命令：
+| `high_energy` | `num_queries=24`, `top1=0.9167`, `topk=1.0` |
+| `report.json` | 已生成 |
 
 
-```bash
-pgrep -af 'ab_smoke_seg_cap48_top2_seed123|external_adapters.py smoke-local fma /tmp/ab_smoke_seg_cap48_top2_seed123|evaluate.py --data /tmp/ab_smoke_seg_cap48_top2_seed123|run_demo.py build-index --data /tmp/ab_smoke_seg_cap48_top2_seed123|train.py --data /tmp/ab_smoke_seg_cap48_top2_seed123'
-```
+seed123 最终结论：
+- `hybrid`：`24 / 0.9583 / 1.0`
+- `high_energy`：`24 / 0.9167 / 1.0`
+- cap48 至少已经表现出明显的 **seed 敏感性**
+- 因此当前默认策略的判断应基于 **多 seed 聚合**，而不是单次 cap48 反转
 - `b766c74` Make open-dataset manifests trainable end to end
 - `b766c74` Make open-dataset manifests trainable end to end
 - `fa23144` Add a single-page open dataset workflow for training prep
 - `fa23144` Add a single-page open dataset workflow for training prep
 - `af33be3` Condense docs and add manifest validation before training
 - `af33be3` Condense docs and add manifest validation before training