Skip to content

wanghai-tech / hikoon-ACR

Go to a project
Toggle navigation
Toggle navigation pinning
Commit e519dab7 e519dab7471810641056e3d9fc75354069b9e5c3 by cnb.bofCdSsphPA
Options

Reframe the cap48 finding as seed-sensitive after the second rerun 

Persist the completed seed123 benchmark showing hybrid ahead again, and update the strategy guidance from single-run winner claims to a multi-seed interpretation.

Constraint: Only documentation changes are allowed because benchmark outputs remain outside version control
Rejected: Keep framing cap48 as a stable high_energy win | The second seed materially weakens that interpretation
Confidence: high
Scope-risk: narrow
Directive: Base the hybrid vs high_energy default decision on aggregated multi-seed evidence, not any single cap48 run
Tested: Verified /tmp/ab_smoke_seg_cap48_top2_seed123/report.json; verified high_energy eval.json; verified docs now record hybrid=24/0.9583/1.0 and high_energy=24/0.9167/1.0 for seed123
Not-tested: Formal aggregation across multiple seeds beyond these two cap48 runs
1 parent a3a5303f
Inline Side-by-side
Showing 3 changed files with 51 additions and 9 deletions
......@@ -2,6 +2,31 @@
## 2026-06-02
### Stage: 收尾 cap48 seed123 并确认 cap48 对 seed 敏感
完成项：
- 读取 `/tmp/ab_smoke_seg_cap48_top2_seed123/report.json`
- 读取：
- `/tmp/ab_smoke_seg_cap48_top2_seed123/hybrid/fma_reports_smoke/eval.json`
- `/tmp/ab_smoke_seg_cap48_top2_seed123/high_energy/fma_reports_smoke/eval.json`
- 更新：
- [open-dataset-workflow.md](./open-dataset-workflow.md)
- [session-handoff.md](./session-handoff.md)
- [CHANGELOG.md](./CHANGELOG.md)
最终结果（subset=48, `max_test_queries=24`, `seed=123`）：
- `hybrid`: `num_queries=24`, `top1=0.9583`, `topk=1.0`
- `high_energy`: `num_queries=24`, `top1=0.9167`, `topk=1.0`
结论：
- cap48 在不同 seed 下已经出现明显不同排序：
- 默认 seed：`high_energy > hybrid`
- `seed=123`：`hybrid > high_energy`
- 这意味着 cap48 当前最可靠的结论不是“谁绝对赢”，而是：
- **该对比对 seed/抽样敏感**
- 当前默认策略判断必须依赖多 seed 聚合结果
- `hybrid` 仍可保留为保守默认，`high_energy` 仍是强竞争方案
### Stage: 启动 cap48 第二个 seed 复核反转结果
完成项：
......
......@@ -206,6 +206,23 @@ flowchart LR
- **`hybrid` 仍是当前保守默认**
- **`high_energy` 已成为强竞争方案**
- 下一步必须做更大样本或多随机种子复核，不能只靠单轮 cap48 就完全改默认
### 更新：cap48 第二个 seed 复核（subset=48, `max_test_queries=24`, `seed=123`）
对同一规模再跑第二个 seed 后，结果又回到 `hybrid` 领先：
| 排名 | 策略 | num_queries | top1 | topk |
|---:|---|---:|---:|---:|
| 1 | `hybrid` | 24 | 0.9583 | 1.0 |
| 2 | `high_energy` | 24 | 0.9167 | 1.0 |
这说明：
- cap48 的策略排名对 seed / 抽样子集 **敏感**
- 单次 cap48 不能作为“high_energy 已全面反超”的充分证据
- 当前最稳妥结论仍是：
- `hybrid` 保留为保守默认
- `high_energy` 保留为强竞争方案
- 后续需要 **多 seed 聚合结论**，而不是看单次跑分
/usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --index-prefix data/index_fma_smoke/reference --split test --device cpu --fast-eval --output-json reports/fma-smoke/eval.json
/usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/fma-smoke/eval.json --config-json reports/fma-smoke/config.json --output-dir reports/fma-smoke --model-version fma-smoke --data-version fma_local
```
......
......@@ -506,7 +506,7 @@ cap48 top2 最终结论：
---
## 13. cap48 top2 第二个 seed（进行中）
## 13. cap48 top2 第二个 seed（已完成）
为验证 cap48 的“high_energy 反超”是否稳定，已启动第二个 seed：
......@@ -527,7 +527,7 @@ cd /workspace/acr-engine
--output-json /tmp/ab_smoke_seg_cap48_top2_seed123/report.json
```
当前 fresh evidence：
最终结果：
| 项目 | 状态 |
|---|---|
......@@ -535,14 +535,14 @@ cd /workspace/acr-engine
| `max_test_queries` | `24` |
| `seed` | `123` |
| `hybrid` | `num_queries=24`, `top1=0.9583`, `topk=1.0` |
| `high_energy` | `run_demo.py build-index --resume --checkpoint-every-refs 100` |
| `report.json` | 尚未生成 |
恢复检查命令：
| `high_energy` | `num_queries=24`, `top1=0.9167`, `topk=1.0` |
| `report.json` | 已生成 |
```bash
pgrep -af 'ab_smoke_seg_cap48_top2_seed123|external_adapters.py smoke-local fma /tmp/ab_smoke_seg_cap48_top2_seed123|evaluate.py --data /tmp/ab_smoke_seg_cap48_top2_seed123|run_demo.py build-index --data /tmp/ab_smoke_seg_cap48_top2_seed123|train.py --data /tmp/ab_smoke_seg_cap48_top2_seed123'
```
seed123 最终结论：
- `hybrid`：`24 / 0.9583 / 1.0`
- `high_energy`：`24 / 0.9167 / 1.0`
- cap48 至少已经表现出明显的 **seed 敏感性**
- 因此当前默认策略的判断应基于 **多 seed 聚合**，而不是单次 cap48 反转
- `b766c74` Make open-dataset manifests trainable end to end
- `fa23144` Add a single-page open dataset workflow for training prep
- `af33be3` Condense docs and add manifest validation before training
......