Skip to content

wanghai-tech / hikoon-ACR

Go to a project
Toggle navigation
Toggle navigation pinning
Commit 87959076 879590762a309cd1430042677d51e90bdf53c68b by cnb.bofCdSsphPA
Options

Generate release artifacts for the open-dataset smoke path 

Constraint: Open-dataset workflow needed the same reporting/release outputs as the synthetic baseline to be operationally useful
Rejected: Treat open-data smoke as a one-off test only | Leaves no reusable benchmark or release documentation trail
Confidence: high
Scope-risk: narrow
Directive: Every future real-dataset smoke run should emit eval JSON plus artifact bundle in the same directory
Tested: /usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/open-smoke-fixed/fma/eval.json --config-json reports/open-smoke-fixed/fma/config.json --output-dir reports/open-smoke-fixed/fma --model-version open-smoke-fixed --data-version synthetic_as_open_fixed_fma
Not-tested: Artifact generation on a larger real downloaded corpus with multiple hard-case buckets
1 parent dc9ef1b8
Inline Side-by-side
Showing 7 changed files with 166 additions and 0 deletions
1 {
2 "generated_at": "2026-06-02T05:01:12Z",
3 "model_version": "open-smoke-fixed",
4 "data_version": "synthetic_as_open_fixed_fma",
5 "files": {
6 "benchmark_report": "reports/open-smoke-fixed/fma/benchmark-report.md",
7 "model_card": "reports/open-smoke-fixed/fma/model-card.md",
8 "release_checklist": "reports/open-smoke-fixed/fma/release-checklist.md"
9 }
10 }
...\ No newline at end of file ...\ No newline at end of file
1 # Benchmark Report
2
3 ## 一页结论
4 - 模型版本：open-smoke-fixed
5 - 数据版本：synthetic_as_open_fixed_fma
6 - 核心结论：top1=1.0 top5=1.0
7 - 是否通过上线门禁：TBD
8
9 ## 1. 评测范围图
10
11 ```mermaid
12 flowchart LR
13 A[open-smoke-fixed] --> B[synthetic_as_open_fixed_fma]
14 A --> C[Scenario Buckets]
15 A --> D[Latency / Ops]
16 ```
17
18 ## 2. 指标表
19
20 | Bucket | top1 | top5 | MRR | FAR | Notes |
21 |---|---:|---:|---:|---:|---|
22 | clean | 1.0 | 1.0 | | | |
23
24 ## 3. 文字分析
25 - 最强项：clean/augmented buckets if present
26 - 最弱项：see hard-case summary
27 - 与上一版本对比：TBD
28
29 ## 4. 细节附录
30 - 原始 JSON 报告：embedded source
31
32 ## Sources
33 - docs/industrial-benchmark-spec.md
1 {
2 "model": {
3 "embed_dim": 192,
4 "channels": 512,
5 "n_mels": 128,
6 "use_band_split": true
7 },
8 "data": {
9 "source_dataset": "fma",
10 "manifests_dir": "data/external_ingested/synthetic_as_open_fixed/fma/manifests",
11 "query_duration": 5.0
12 },
13 "run": {
14 "train_command": "/usr/local/miniconda3/bin/python train.py --data data/external_ingested/synthetic_as_open_fixed/fma/manifests --output data/models_open_smoke_fixed --device cpu --epochs 1 --batch-size 2",
15 "index_command": "/usr/local/miniconda3/bin/python run_demo.py build-index --data data/external_ingested/synthetic_as_open_fixed/fma/manifests --model data/models_open_smoke_fixed/best_model.pt --output data/index_open_smoke_fixed --device cpu",
16 "eval_command": "/usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/synthetic_as_open_fixed/fma/manifests --model data/models_open_smoke_fixed/best_model.pt --index-prefix data/index_open_smoke_fixed/reference --split test --device cpu --fast-eval --output-json reports/open-smoke-fixed/fma/eval.json"
17 }
18 }
1 # Model Card
2
3 ## 一页结论
4 - 模型名称：ACR Hybrid Encoder
5 - 版本：open-smoke-fixed
6 - 适用场景：music ACR prototype / retrieval
7 - 不适用场景：未经白名单数据验证的生产商用全量上线
8
9 ## 1. 模型结构图
10
11 ```mermaid
12 flowchart LR
13 A[Input Audio] --> B[128 Mel + BandSplit]
14 B --> C[Encoder]
15 C --> D[Embedding]
16 D --> E[Hybrid Retrieval]
17 ```
18
19 ## 2. 关键信息表
20
21 | 项 | 内容 |
22 |---|---|
23 | embed_dim | 192 |
24 | channels | 512 |
25 | n_mels | 128 |
26 | use_band_split | True |
27 | benchmark report | reports/open-smoke-fixed/fma/benchmark-report.md |
28
29 ## 3. 文字说明
30 - 训练方式：retrieval-oriented pair training
31 - 模型限制：hard-case accuracy still evolving
32 - 风险提示：requires whitelist-reviewed datasets for commercial deployment
33
34 ## 4. 细节附录
35 - config embedded from source JSON
36
37 ## Sources
38 - docs/dataset-spec.md
39 - docs/benchmark-report-template.md
1 # Release Checklist
2
3 ## 一页结论
4 发布前必须同时满足：质量通过、合规通过、服务通过、文档齐全。
5
6 ## 1. 发布门禁图
7
8 ```mermaid
9 flowchart TD
10 A[open-smoke-fixed] --> B[Benchmark Pass]
11 A --> C[License Review Pass]
12 A --> D[Service Smoke Pass]
13 A --> E[Docs Complete]
14 ```
15
16 ## 2. Checklist 表
17
18 | 项目 | 状态 |
19 |---|---|
20 | benchmark report 已生成 | yes |
21 | model card 已生成 | yes |
22 | license registry 已更新 | pending |
23 | service smoke test 通过 | yes |
24 | dataset whitelist 已确认 | pending |
25 | changelog 已更新 | pending |
26
27 ## 3. 文字说明
28 - 当前用于工程治理与预发布检查，不代表已满足商用法律门槛。
29
30 ## 4. 细节附录
31 - benchmark 报告路径：reports/open-smoke-fixed/fma/benchmark-report.md
32 - model card 路径：reports/open-smoke-fixed/fma/model-card.md
33
34 ## Sources
35 - docs/dataset-sources-and-licensing.md
36 - docs/industrial-benchmark-spec.md
...@@ -93,6 +93,28 @@ ...@@ -93,6 +93,28 @@
93 - 开放数据接入链路现在已经完整闭环 93 - 开放数据接入链路现在已经完整闭环
94 - 真实 FMA / MTG-Jamendo 本地目录接入时，可直接复用同一流程 94 - 真实 FMA / MTG-Jamendo 本地目录接入时，可直接复用同一流程
95 95
96 ### Stage: 开放数据 smoke 发布制品
97
98 完成项：
99 - 为 `reports/open-smoke-fixed/fma/` 补充 `config.json`
100 - 用 `scripts/generate_artifacts.py` 生成开放数据 smoke 的：
101 - benchmark report
102 - model card
103 - release checklist
104 - artifact manifest
105
106 验证结果：
107 - `/usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/open-smoke-fixed/fma/eval.json --config-json reports/open-smoke-fixed/fma/config.json --output-dir reports/open-smoke-fixed/fma --model-version open-smoke-fixed --data-version synthetic_as_open_fixed_fma` 成功
108 - 产物存在：
109 - `reports/open-smoke-fixed/fma/benchmark-report.md`
110 - `reports/open-smoke-fixed/fma/model-card.md`
111 - `reports/open-smoke-fixed/fma/release-checklist.md`
112 - `reports/open-smoke-fixed/fma/artifact-manifest.json`
113
114 结论：
115 - 现在开放数据链路已经不只是“能跑”，还具备基础发布/汇报产物
116 - 下一步替换成真实 FMA / MTG-Jamendo 本地目录后，可直接复用同一 release 流程
117
96 ### Stage: confused 定向优化 v6（sample-level weighting） 118 ### Stage: confused 定向优化 v6（sample-level weighting）
97 119
98 完成项： 120 完成项：
......
...@@ -10,6 +10,7 @@ ...@@ -10,6 +10,7 @@
10 2. **prepare-local** 10 2. **prepare-local**
11 3. **validate-local** 11 3. **validate-local**
12 4. 再进入训练与评估 12 4. 再进入训练与评估
13 5. 生成 benchmark / model card / release artifacts
13 14
14 --- 15 ---
15 16
...@@ -22,6 +23,7 @@ flowchart LR ...@@ -22,6 +23,7 @@ flowchart LR
22 C --> D[validate-local] 23 C --> D[validate-local]
23 D --> E[train.py] 24 D --> E[train.py]
24 D --> F[evaluate.py] 25 D --> F[evaluate.py]
26 F --> G[artifact bundle]
25 ``` 27 ```
26 28
27 --- 29 ---
...@@ -35,6 +37,7 @@ flowchart LR ...@@ -35,6 +37,7 @@ flowchart LR
35 | 生成清单 | [`src/data/external_adapters.py`](../acr-engine/src/data/external_adapters.py) `prepare-local ...` | 产出 train/test/catalog | 37 | 生成清单 | [`src/data/external_adapters.py`](../acr-engine/src/data/external_adapters.py) `prepare-local ...` | 产出 train/test/catalog |
36 | 训练前校验 | [`src/data/external_adapters.py`](../acr-engine/src/data/external_adapters.py) `validate-local ...` | 确认结构正确 | 38 | 训练前校验 | [`src/data/external_adapters.py`](../acr-engine/src/data/external_adapters.py) `validate-local ...` | 确认结构正确 |
37 | 训练 smoke | [`train.py`](../acr-engine/train.py) `--data ... --dry-run` | 验证 manifests 可直接进入训练 | 39 | 训练 smoke | [`train.py`](../acr-engine/train.py) `--data ... --dry-run` | 验证 manifests 可直接进入训练 |
40 | 发布制品 | [`scripts/generate_artifacts.py`](../acr-engine/scripts/generate_artifacts.py) | 生成 benchmark/model-card/release-checklist |
38 41
39 --- 42 ---
40 43
...@@ -49,6 +52,7 @@ flowchart LR ...@@ -49,6 +52,7 @@ flowchart LR
49 /usr/local/miniconda3/bin/python train.py --data data/external_ingested/fma/manifests --output data/models_fma_smoke --device cpu --epochs 1 --batch-size 2 --dry-run 52 /usr/local/miniconda3/bin/python train.py --data data/external_ingested/fma/manifests --output data/models_fma_smoke --device cpu --epochs 1 --batch-size 2 --dry-run
50 /usr/local/miniconda3/bin/python run_demo.py build-index --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --output data/index_fma_smoke --device cpu 53 /usr/local/miniconda3/bin/python run_demo.py build-index --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --output data/index_fma_smoke --device cpu
51 /usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --index-prefix data/index_fma_smoke/reference --split test --device cpu --fast-eval --output-json reports/fma-smoke/eval.json 54 /usr/local/miniconda3/bin/python evaluate.py --data data/external_ingested/fma/manifests --model data/models_fma_smoke/best_model.pt --index-prefix data/index_fma_smoke/reference --split test --device cpu --fast-eval --output-json reports/fma-smoke/eval.json
55 /usr/local/miniconda3/bin/python scripts/generate_artifacts.py --eval-json reports/fma-smoke/eval.json --config-json reports/fma-smoke/config.json --output-dir reports/fma-smoke --model-version fma-smoke --data-version fma_local
52 ``` 56 ```
53 57
54 ### 3.2 多目录比较 58 ### 3.2 多目录比较
...@@ -87,6 +91,10 @@ flowchart LR ...@@ -87,6 +91,10 @@ flowchart LR
87 - `build-index + evaluate`： 91 - `build-index + evaluate`：
88 - `top1=1.0` 92 - `top1=1.0`
89 - `topk=1.0` 93 - `topk=1.0`
94 - `generate_artifacts`：
95 - `benchmark-report.md`
96 - `model-card.md`
97 - `release-checklist.md`
90 98
91 --- 99 ---
92 100
......