Commit 1b812bea 1b812bea5acc67d01bba4452f2bbb1a6f45cbabd by cnb.bofCdSsphPA

Connect real evaluation outputs to release artifacts

Make the benchmark pipeline produce reusable release artifacts from actual evaluation results so model iterations can be tracked, reviewed, and shipped with evidence.

Constraint: Continuous training only helps if each stage emits durable reports and release metadata
Rejected: Keep artifact generation as a disconnected smoke utility | would block repeatable release discipline
Confidence: high
Scope-risk: moderate
Directive: Next iterations should improve hard-case metrics on real/whitelisted datasets and keep artifact generation on every training milestone
Tested: synthetic_v2 data regeneration; 2-epoch CPU training; index build; fast evaluation JSON export; artifact generation to reports/smoke-v2/synthetic_v2
Not-tested: full melody-aware slow evaluation as release default; real external dataset benchmark generation
1 parent dc00d026
Showing 173 changed files with 1827 additions and 4 deletions
1 {
2 "generated_at": "2026-06-02T03:59:45Z",
3 "model_version": "smoke-v1",
4 "data_version": "synthetic_v2",
5 "files": {
6 "benchmark_report": "data/artifact_smoke/benchmark-report.md",
7 "model_card": "data/artifact_smoke/model-card.md",
8 "release_checklist": "data/artifact_smoke/release-checklist.md"
9 }
10 }
...\ No newline at end of file ...\ No newline at end of file
1 # Benchmark Report
2
3 ## 一页结论
4 - 模型版本:smoke-v1
5 - 数据版本:synthetic_v2
6 - 核心结论:top1=0.65 top5=0.95
7 - 是否通过上线门禁:TBD
8
9 ## 1. 评测范围图
10
11 ```mermaid
12 flowchart LR
13 A[smoke-v1] --> B[synthetic_v2]
14 A --> C[Scenario Buckets]
15 A --> D[Latency / Ops]
16 ```
17
18 ## 2. 指标表
19
20 | Bucket | top1 | top5 | MRR | FAR | Notes |
21 |---|---:|---:|---:|---:|---|
22 | clean | 1.0 | 1.0 | | | |
23 | humming_like | 0.25 | 0.75 | | | |
24 | confused | 0.25 | 1.0 | | | |
25
26 ## 3. 文字分析
27 - 最强项:clean/augmented buckets if present
28 - 最弱项:see hard-case summary
29 - 与上一版本对比:TBD
30
31 ## 4. 细节附录
32 - 原始 JSON 报告:embedded source
33
34 ## Sources
35 - docs/industrial-benchmark-spec.md
1 # Model Card
2
3 ## 一页结论
4 - 模型名称:ACR Hybrid Encoder
5 - 版本:smoke-v1
6 - 适用场景:music ACR prototype / retrieval
7 - 不适用场景:未经白名单数据验证的生产商用全量上线
8
9 ## 1. 模型结构图
10
11 ```mermaid
12 flowchart LR
13 A[Input Audio] --> B[128 Mel + BandSplit]
14 B --> C[Encoder]
15 C --> D[Embedding]
16 D --> E[Hybrid Retrieval]
17 ```
18
19 ## 2. 关键信息表
20
21 | 项 | 内容 |
22 |---|---|
23 | embed_dim | 192 |
24 | channels | 512 |
25 | n_mels | 128 |
26 | use_band_split | True |
27 | benchmark report | data/artifact_smoke/benchmark-report.md |
28
29 ## 3. 文字说明
30 - 训练方式:retrieval-oriented pair training
31 - 模型限制:hard-case accuracy still evolving
32 - 风险提示:requires whitelist-reviewed datasets for commercial deployment
33
34 ## 4. 细节附录
35 - config embedded from source JSON
36
37 ## Sources
38 - docs/dataset-spec.md
39 - docs/benchmark-report-template.md
1 # Release Checklist
2
3 ## 一页结论
4 发布前必须同时满足:质量通过、合规通过、服务通过、文档齐全。
5
6 ## 1. 发布门禁图
7
8 ```mermaid
9 flowchart TD
10 A[smoke-v1] --> B[Benchmark Pass]
11 A --> C[License Review Pass]
12 A --> D[Service Smoke Pass]
13 A --> E[Docs Complete]
14 ```
15
16 ## 2. Checklist 表
17
18 | 项目 | 状态 |
19 |---|---|
20 | benchmark report 已生成 | yes |
21 | model card 已生成 | yes |
22 | license registry 已更新 | pending |
23 | service smoke test 通过 | yes |
24 | dataset whitelist 已确认 | pending |
25 | changelog 已更新 | pending |
26
27 ## 3. 文字说明
28 - 当前用于工程治理与预发布检查,不代表已满足商用法律门槛。
29
30 ## 4. 细节附录
31 - benchmark 报告路径:data/artifact_smoke/benchmark-report.md
32 - model card 路径:data/artifact_smoke/model-card.md
33
34 ## Sources
35 - docs/dataset-sources-and-licensing.md
36 - docs/industrial-benchmark-spec.md
No preview for this file type
No preview for this file type
No preview for this file type
This file is too large to display.
1 {
2 "song_0000": 0,
3 "song_0001": 1,
4 "song_0002": 2,
5 "song_0003": 3,
6 "song_0004": 4,
7 "song_0005": 5,
8 "song_0006": 6,
9 "song_0007": 7,
10 "song_0008": 8,
11 "song_0009": 9,
12 "song_0010": 10,
13 "song_0011": 11,
14 "song_0012": 12,
15 "song_0013": 13,
16 "song_0014": 14,
17 "song_0015": 15
18 }
...\ No newline at end of file ...\ No newline at end of file
1 [
2 {
3 "song_id": "song_0000",
4 "audio_path": "songs/song_0000.wav",
5 "duration": 15.0,
6 "base_freq": 130.81,
7 "type": "reference"
8 },
9 {
10 "song_id": "song_0001",
11 "audio_path": "songs/song_0001.wav",
12 "duration": 15.0,
13 "base_freq": 146.83,
14 "type": "reference"
15 },
16 {
17 "song_id": "song_0002",
18 "audio_path": "songs/song_0002.wav",
19 "duration": 15.0,
20 "base_freq": 164.81,
21 "type": "reference"
22 },
23 {
24 "song_id": "song_0003",
25 "audio_path": "songs/song_0003.wav",
26 "duration": 15.0,
27 "base_freq": 174.61,
28 "type": "reference"
29 },
30 {
31 "song_id": "song_0004",
32 "audio_path": "songs/song_0004.wav",
33 "duration": 15.0,
34 "base_freq": 196.0,
35 "type": "reference"
36 },
37 {
38 "song_id": "song_0005",
39 "audio_path": "songs/song_0005.wav",
40 "duration": 15.0,
41 "base_freq": 220.0,
42 "type": "reference"
43 },
44 {
45 "song_id": "song_0006",
46 "audio_path": "songs/song_0006.wav",
47 "duration": 15.0,
48 "base_freq": 246.94,
49 "type": "reference"
50 },
51 {
52 "song_id": "song_0007",
53 "audio_path": "songs/song_0007.wav",
54 "duration": 15.0,
55 "base_freq": 261.63,
56 "type": "reference"
57 },
58 {
59 "song_id": "song_0008",
60 "audio_path": "songs/song_0008.wav",
61 "duration": 15.0,
62 "base_freq": 293.66,
63 "type": "reference"
64 },
65 {
66 "song_id": "song_0009",
67 "audio_path": "songs/song_0009.wav",
68 "duration": 15.0,
69 "base_freq": 329.63,
70 "type": "reference"
71 },
72 {
73 "song_id": "song_0010",
74 "audio_path": "songs/song_0010.wav",
75 "duration": 15.0,
76 "base_freq": 349.23,
77 "type": "reference"
78 },
79 {
80 "song_id": "song_0011",
81 "audio_path": "songs/song_0011.wav",
82 "duration": 15.0,
83 "base_freq": 392.0,
84 "type": "reference"
85 },
86 {
87 "song_id": "song_0012",
88 "audio_path": "songs/song_0012.wav",
89 "duration": 15.0,
90 "base_freq": 440.0,
91 "type": "reference"
92 },
93 {
94 "song_id": "song_0013",
95 "audio_path": "songs/song_0013.wav",
96 "duration": 15.0,
97 "base_freq": 493.88,
98 "type": "reference"
99 },
100 {
101 "song_id": "song_0014",
102 "audio_path": "songs/song_0014.wav",
103 "duration": 15.0,
104 "base_freq": 523.25,
105 "type": "reference"
106 },
107 {
108 "song_id": "song_0015",
109 "audio_path": "songs/song_0015.wav",
110 "duration": 15.0,
111 "base_freq": 587.33,
112 "type": "reference"
113 },
114 {
115 "song_id": "song_0016",
116 "audio_path": "songs/song_0016.wav",
117 "duration": 15.0,
118 "base_freq": 659.25,
119 "type": "reference"
120 },
121 {
122 "song_id": "song_0017",
123 "audio_path": "songs/song_0017.wav",
124 "duration": 15.0,
125 "base_freq": 698.46,
126 "type": "reference"
127 },
128 {
129 "song_id": "song_0018",
130 "audio_path": "songs/song_0018.wav",
131 "duration": 15.0,
132 "base_freq": 783.99,
133 "type": "reference"
134 },
135 {
136 "song_id": "song_0019",
137 "audio_path": "songs/song_0019.wav",
138 "duration": 15.0,
139 "base_freq": 880.0,
140 "type": "reference"
141 },
142 {
143 "song_id": "song_0020",
144 "audio_path": "songs/song_0020.wav",
145 "duration": 15.0,
146 "base_freq": 987.77,
147 "type": "reference"
148 },
149 {
150 "song_id": "song_0021",
151 "audio_path": "songs/song_0021.wav",
152 "duration": 15.0,
153 "base_freq": 146.8292605393491,
154 "type": "reference"
155 },
156 {
157 "song_id": "song_0022",
158 "audio_path": "songs/song_0022.wav",
159 "duration": 15.0,
160 "base_freq": 164.81110255326524,
161 "type": "reference"
162 },
163 {
164 "song_id": "song_0023",
165 "audio_path": "songs/song_0023.wav",
166 "duration": 15.0,
167 "base_freq": 184.99297018186778,
168 "type": "reference"
169 }
170 ]
...\ No newline at end of file ...\ No newline at end of file
1 [
2 {
3 "song_id": "song_0020",
4 "audio_path": "segments/song_0020_seg_00.wav",
5 "duration": 5.0,
6 "type": "clean",
7 "offset": 4.349828784349853,
8 "segment_type": "mid"
9 },
10 {
11 "song_id": "song_0020",
12 "audio_path": "segments/song_0020_seg_01.wav",
13 "duration": 5.0,
14 "type": "clean",
15 "offset": 9.642182747327407,
16 "segment_type": "mid"
17 },
18 {
19 "song_id": "song_0020",
20 "audio_path": "segments/song_0020_seg_02_augmented.wav",
21 "duration": 5.0,
22 "type": "augmented",
23 "offset": 2.367717347418965,
24 "segment_type": "intro"
25 },
26 {
27 "song_id": "song_0020",
28 "audio_path": "segments/song_0020_seg_03_humming_like.wav",
29 "duration": 5.0,
30 "type": "humming_like",
31 "offset": 3.180577192661006,
32 "segment_type": "mid"
33 },
34 {
35 "song_id": "song_0020",
36 "audio_path": "segments/song_0020_seg_04_confused.wav",
37 "duration": 5.0,
38 "type": "confused",
39 "offset": 4.660551124366617,
40 "segment_type": "mid"
41 },
42 {
43 "song_id": "song_0020",
44 "audio_path": "songs/song_0020.wav",
45 "duration": 15.0,
46 "base_freq": 987.77,
47 "type": "reference"
48 },
49 {
50 "song_id": "song_0021",
51 "audio_path": "segments/song_0021_seg_00.wav",
52 "duration": 5.0,
53 "type": "clean",
54 "offset": 5.631088908640184,
55 "segment_type": "mid"
56 },
57 {
58 "song_id": "song_0021",
59 "audio_path": "segments/song_0021_seg_01.wav",
60 "duration": 5.0,
61 "type": "clean",
62 "offset": 1.8823366490525628,
63 "segment_type": "intro"
64 },
65 {
66 "song_id": "song_0021",
67 "audio_path": "segments/song_0021_seg_02_augmented.wav",
68 "duration": 5.0,
69 "type": "augmented",
70 "offset": 9.88006210404643,
71 "segment_type": "mid"
72 },
73 {
74 "song_id": "song_0021",
75 "audio_path": "segments/song_0021_seg_03_humming_like.wav",
76 "duration": 5.0,
77 "type": "humming_like",
78 "offset": 0.9025737685090285,
79 "segment_type": "intro"
80 },
81 {
82 "song_id": "song_0021",
83 "audio_path": "segments/song_0021_seg_04_confused.wav",
84 "duration": 5.0,
85 "type": "confused",
86 "offset": 1.3048954561918258,
87 "segment_type": "intro"
88 },
89 {
90 "song_id": "song_0021",
91 "audio_path": "songs/song_0021.wav",
92 "duration": 15.0,
93 "base_freq": 146.8292605393491,
94 "type": "reference"
95 },
96 {
97 "song_id": "song_0022",
98 "audio_path": "segments/song_0022_seg_00.wav",
99 "duration": 5.0,
100 "type": "clean",
101 "offset": 3.9746734850812295,
102 "segment_type": "mid"
103 },
104 {
105 "song_id": "song_0022",
106 "audio_path": "segments/song_0022_seg_01.wav",
107 "duration": 5.0,
108 "type": "clean",
109 "offset": 4.890968121206573,
110 "segment_type": "mid"
111 },
112 {
113 "song_id": "song_0022",
114 "audio_path": "segments/song_0022_seg_02_augmented.wav",
115 "duration": 5.0,
116 "type": "augmented",
117 "offset": 6.610400547460049,
118 "segment_type": "mid"
119 },
120 {
121 "song_id": "song_0022",
122 "audio_path": "segments/song_0022_seg_03_humming_like.wav",
123 "duration": 5.0,
124 "type": "humming_like",
125 "offset": 2.6329596668288424,
126 "segment_type": "intro"
127 },
128 {
129 "song_id": "song_0022",
130 "audio_path": "segments/song_0022_seg_04_confused.wav",
131 "duration": 5.0,
132 "type": "confused",
133 "offset": 0.8570731183991709,
134 "segment_type": "intro"
135 },
136 {
137 "song_id": "song_0022",
138 "audio_path": "songs/song_0022.wav",
139 "duration": 15.0,
140 "base_freq": 164.81110255326524,
141 "type": "reference"
142 },
143 {
144 "song_id": "song_0023",
145 "audio_path": "segments/song_0023_seg_00.wav",
146 "duration": 5.0,
147 "type": "clean",
148 "offset": 4.461034326075292,
149 "segment_type": "mid"
150 },
151 {
152 "song_id": "song_0023",
153 "audio_path": "segments/song_0023_seg_01.wav",
154 "duration": 5.0,
155 "type": "clean",
156 "offset": 9.605203782802876,
157 "segment_type": "mid"
158 },
159 {
160 "song_id": "song_0023",
161 "audio_path": "segments/song_0023_seg_02_augmented.wav",
162 "duration": 5.0,
163 "type": "augmented",
164 "offset": 4.7458228906154805,
165 "segment_type": "mid"
166 },
167 {
168 "song_id": "song_0023",
169 "audio_path": "segments/song_0023_seg_03_humming_like.wav",
170 "duration": 5.0,
171 "type": "humming_like",
172 "offset": 8.308702013555955,
173 "segment_type": "mid"
174 },
175 {
176 "song_id": "song_0023",
177 "audio_path": "segments/song_0023_seg_04_confused.wav",
178 "duration": 5.0,
179 "type": "confused",
180 "offset": 2.213510770155481,
181 "segment_type": "intro"
182 },
183 {
184 "song_id": "song_0023",
185 "audio_path": "songs/song_0023.wav",
186 "duration": 15.0,
187 "base_freq": 184.99297018186778,
188 "type": "reference"
189 }
190 ]
...\ No newline at end of file ...\ No newline at end of file
1 [
2 {
3 "song_id": "song_0000",
4 "audio_path": "segments/song_0000_seg_00.wav",
5 "duration": 5.0,
6 "type": "clean",
7 "offset": 9.538159275210802,
8 "segment_type": "mid"
9 },
10 {
11 "song_id": "song_0000",
12 "audio_path": "segments/song_0000_seg_01.wav",
13 "duration": 5.0,
14 "type": "clean",
15 "offset": 8.75852940378194,
16 "segment_type": "mid"
17 },
18 {
19 "song_id": "song_0000",
20 "audio_path": "segments/song_0000_seg_02_augmented.wav",
21 "duration": 5.0,
22 "type": "augmented",
23 "offset": 2.6338905075109076,
24 "segment_type": "intro"
25 },
26 {
27 "song_id": "song_0000",
28 "audio_path": "segments/song_0000_seg_03_humming_like.wav",
29 "duration": 5.0,
30 "type": "humming_like",
31 "offset": 6.389494948660052,
32 "segment_type": "mid"
33 },
34 {
35 "song_id": "song_0000",
36 "audio_path": "segments/song_0000_seg_04_confused.wav",
37 "duration": 5.0,
38 "type": "confused",
39 "offset": 5.303536721951775,
40 "segment_type": "mid"
41 },
42 {
43 "song_id": "song_0000",
44 "audio_path": "songs/song_0000.wav",
45 "duration": 15.0,
46 "base_freq": 130.81,
47 "type": "reference"
48 },
49 {
50 "song_id": "song_0001",
51 "audio_path": "segments/song_0001_seg_00.wav",
52 "duration": 5.0,
53 "type": "clean",
54 "offset": 5.227827155319589,
55 "segment_type": "mid"
56 },
57 {
58 "song_id": "song_0001",
59 "audio_path": "segments/song_0001_seg_01.wav",
60 "duration": 5.0,
61 "type": "clean",
62 "offset": 9.347062577364273,
63 "segment_type": "mid"
64 },
65 {
66 "song_id": "song_0001",
67 "audio_path": "segments/song_0001_seg_02_augmented.wav",
68 "duration": 5.0,
69 "type": "augmented",
70 "offset": 2.042591994235364,
71 "segment_type": "intro"
72 },
73 {
74 "song_id": "song_0001",
75 "audio_path": "segments/song_0001_seg_03_humming_like.wav",
76 "duration": 5.0,
77 "type": "humming_like",
78 "offset": 3.1617719627185403,
79 "segment_type": "mid"
80 },
81 {
82 "song_id": "song_0001",
83 "audio_path": "segments/song_0001_seg_04_confused.wav",
84 "duration": 5.0,
85 "type": "confused",
86 "offset": 0.73260721099633,
87 "segment_type": "intro"
88 },
89 {
90 "song_id": "song_0001",
91 "audio_path": "songs/song_0001.wav",
92 "duration": 15.0,
93 "base_freq": 146.83,
94 "type": "reference"
95 },
96 {
97 "song_id": "song_0002",
98 "audio_path": "segments/song_0002_seg_00.wav",
99 "duration": 5.0,
100 "type": "clean",
101 "offset": 3.0928466220865323,
102 "segment_type": "mid"
103 },
104 {
105 "song_id": "song_0002",
106 "audio_path": "segments/song_0002_seg_01.wav",
107 "duration": 5.0,
108 "type": "clean",
109 "offset": 4.083929086192168,
110 "segment_type": "mid"
111 },
112 {
113 "song_id": "song_0002",
114 "audio_path": "segments/song_0002_seg_02_augmented.wav",
115 "duration": 5.0,
116 "type": "augmented",
117 "offset": 4.024003870577246,
118 "segment_type": "mid"
119 },
120 {
121 "song_id": "song_0002",
122 "audio_path": "segments/song_0002_seg_03_humming_like.wav",
123 "duration": 5.0,
124 "type": "humming_like",
125 "offset": 9.028055457325827,
126 "segment_type": "mid"
127 },
128 {
129 "song_id": "song_0002",
130 "audio_path": "segments/song_0002_seg_04_confused.wav",
131 "duration": 5.0,
132 "type": "confused",
133 "offset": 4.2988814998983464,
134 "segment_type": "mid"
135 },
136 {
137 "song_id": "song_0002",
138 "audio_path": "songs/song_0002.wav",
139 "duration": 15.0,
140 "base_freq": 164.81,
141 "type": "reference"
142 },
143 {
144 "song_id": "song_0003",
145 "audio_path": "segments/song_0003_seg_00.wav",
146 "duration": 5.0,
147 "type": "clean",
148 "offset": 0.1938328705001069,
149 "segment_type": "intro"
150 },
151 {
152 "song_id": "song_0003",
153 "audio_path": "segments/song_0003_seg_01.wav",
154 "duration": 5.0,
155 "type": "clean",
156 "offset": 5.394190479225337,
157 "segment_type": "mid"
158 },
159 {
160 "song_id": "song_0003",
161 "audio_path": "segments/song_0003_seg_02_augmented.wav",
162 "duration": 5.0,
163 "type": "augmented",
164 "offset": 9.999078285092093,
165 "segment_type": "mid"
166 },
167 {
168 "song_id": "song_0003",
169 "audio_path": "segments/song_0003_seg_03_humming_like.wav",
170 "duration": 5.0,
171 "type": "humming_like",
172 "offset": 9.496117327159888,
173 "segment_type": "mid"
174 },
175 {
176 "song_id": "song_0003",
177 "audio_path": "segments/song_0003_seg_04_confused.wav",
178 "duration": 5.0,
179 "type": "confused",
180 "offset": 2.1796454090650363,
181 "segment_type": "intro"
182 },
183 {
184 "song_id": "song_0003",
185 "audio_path": "songs/song_0003.wav",
186 "duration": 15.0,
187 "base_freq": 174.61,
188 "type": "reference"
189 },
190 {
191 "song_id": "song_0004",
192 "audio_path": "segments/song_0004_seg_00.wav",
193 "duration": 5.0,
194 "type": "clean",
195 "offset": 9.654976431382948,
196 "segment_type": "mid"
197 },
198 {
199 "song_id": "song_0004",
200 "audio_path": "segments/song_0004_seg_01.wav",
201 "duration": 5.0,
202 "type": "clean",
203 "offset": 2.524783904929726,
204 "segment_type": "intro"
205 },
206 {
207 "song_id": "song_0004",
208 "audio_path": "segments/song_0004_seg_02_augmented.wav",
209 "duration": 5.0,
210 "type": "augmented",
211 "offset": 8.617229646275131,
212 "segment_type": "mid"
213 },
214 {
215 "song_id": "song_0004",
216 "audio_path": "segments/song_0004_seg_03_humming_like.wav",
217 "duration": 5.0,
218 "type": "humming_like",
219 "offset": 1.5172700695095642,
220 "segment_type": "intro"
221 },
222 {
223 "song_id": "song_0004",
224 "audio_path": "segments/song_0004_seg_04_confused.wav",
225 "duration": 5.0,
226 "type": "confused",
227 "offset": 4.161740214103284,
228 "segment_type": "mid"
229 },
230 {
231 "song_id": "song_0004",
232 "audio_path": "songs/song_0004.wav",
233 "duration": 15.0,
234 "base_freq": 196.0,
235 "type": "reference"
236 },
237 {
238 "song_id": "song_0005",
239 "audio_path": "segments/song_0005_seg_00.wav",
240 "duration": 5.0,
241 "type": "clean",
242 "offset": 5.088720150695117,
243 "segment_type": "mid"
244 },
245 {
246 "song_id": "song_0005",
247 "audio_path": "segments/song_0005_seg_01.wav",
248 "duration": 5.0,
249 "type": "clean",
250 "offset": 2.734248967132742,
251 "segment_type": "intro"
252 },
253 {
254 "song_id": "song_0005",
255 "audio_path": "segments/song_0005_seg_02_augmented.wav",
256 "duration": 5.0,
257 "type": "augmented",
258 "offset": 8.347239455766944,
259 "segment_type": "mid"
260 },
261 {
262 "song_id": "song_0005",
263 "audio_path": "segments/song_0005_seg_03_humming_like.wav",
264 "duration": 5.0,
265 "type": "humming_like",
266 "offset": 5.08240891592894,
267 "segment_type": "mid"
268 },
269 {
270 "song_id": "song_0005",
271 "audio_path": "segments/song_0005_seg_04_confused.wav",
272 "duration": 5.0,
273 "type": "confused",
274 "offset": 9.3424839368252,
275 "segment_type": "mid"
276 },
277 {
278 "song_id": "song_0005",
279 "audio_path": "songs/song_0005.wav",
280 "duration": 15.0,
281 "base_freq": 220.0,
282 "type": "reference"
283 },
284 {
285 "song_id": "song_0006",
286 "audio_path": "segments/song_0006_seg_00.wav",
287 "duration": 5.0,
288 "type": "clean",
289 "offset": 2.5062680004361604,
290 "segment_type": "intro"
291 },
292 {
293 "song_id": "song_0006",
294 "audio_path": "segments/song_0006_seg_01.wav",
295 "duration": 5.0,
296 "type": "clean",
297 "offset": 7.555773237416772,
298 "segment_type": "mid"
299 },
300 {
301 "song_id": "song_0006",
302 "audio_path": "segments/song_0006_seg_02_augmented.wav",
303 "duration": 5.0,
304 "type": "augmented",
305 "offset": 7.674707744954641,
306 "segment_type": "mid"
307 },
308 {
309 "song_id": "song_0006",
310 "audio_path": "segments/song_0006_seg_03_humming_like.wav",
311 "duration": 5.0,
312 "type": "humming_like",
313 "offset": 0.33364531245632434,
314 "segment_type": "intro"
315 },
316 {
317 "song_id": "song_0006",
318 "audio_path": "segments/song_0006_seg_04_confused.wav",
319 "duration": 5.0,
320 "type": "confused",
321 "offset": 2.007947946500762,
322 "segment_type": "intro"
323 },
324 {
325 "song_id": "song_0006",
326 "audio_path": "songs/song_0006.wav",
327 "duration": 15.0,
328 "base_freq": 246.94,
329 "type": "reference"
330 },
331 {
332 "song_id": "song_0007",
333 "audio_path": "segments/song_0007_seg_00.wav",
334 "duration": 5.0,
335 "type": "clean",
336 "offset": 6.589030736792923,
337 "segment_type": "mid"
338 },
339 {
340 "song_id": "song_0007",
341 "audio_path": "segments/song_0007_seg_01.wav",
342 "duration": 5.0,
343 "type": "clean",
344 "offset": 3.016303290280887,
345 "segment_type": "mid"
346 },
347 {
348 "song_id": "song_0007",
349 "audio_path": "segments/song_0007_seg_02_augmented.wav",
350 "duration": 5.0,
351 "type": "augmented",
352 "offset": 6.433406842054888,
353 "segment_type": "mid"
354 },
355 {
356 "song_id": "song_0007",
357 "audio_path": "segments/song_0007_seg_03_humming_like.wav",
358 "duration": 5.0,
359 "type": "humming_like",
360 "offset": 4.435623293630087,
361 "segment_type": "mid"
362 },
363 {
364 "song_id": "song_0007",
365 "audio_path": "segments/song_0007_seg_04_confused.wav",
366 "duration": 5.0,
367 "type": "confused",
368 "offset": 5.8536468854812105,
369 "segment_type": "mid"
370 },
371 {
372 "song_id": "song_0007",
373 "audio_path": "songs/song_0007.wav",
374 "duration": 15.0,
375 "base_freq": 261.63,
376 "type": "reference"
377 },
378 {
379 "song_id": "song_0008",
380 "audio_path": "segments/song_0008_seg_00.wav",
381 "duration": 5.0,
382 "type": "clean",
383 "offset": 0.42302261562791377,
384 "segment_type": "intro"
385 },
386 {
387 "song_id": "song_0008",
388 "audio_path": "segments/song_0008_seg_01.wav",
389 "duration": 5.0,
390 "type": "clean",
391 "offset": 0.18741536585645702,
392 "segment_type": "intro"
393 },
394 {
395 "song_id": "song_0008",
396 "audio_path": "segments/song_0008_seg_02_augmented.wav",
397 "duration": 5.0,
398 "type": "augmented",
399 "offset": 9.211624345024124,
400 "segment_type": "mid"
401 },
402 {
403 "song_id": "song_0008",
404 "audio_path": "segments/song_0008_seg_03_humming_like.wav",
405 "duration": 5.0,
406 "type": "humming_like",
407 "offset": 4.176939598434806,
408 "segment_type": "mid"
409 },
410 {
411 "song_id": "song_0008",
412 "audio_path": "segments/song_0008_seg_04_confused.wav",
413 "duration": 5.0,
414 "type": "confused",
415 "offset": 8.320259130717071,
416 "segment_type": "mid"
417 },
418 {
419 "song_id": "song_0008",
420 "audio_path": "songs/song_0008.wav",
421 "duration": 15.0,
422 "base_freq": 293.66,
423 "type": "reference"
424 },
425 {
426 "song_id": "song_0009",
427 "audio_path": "segments/song_0009_seg_00.wav",
428 "duration": 5.0,
429 "type": "clean",
430 "offset": 5.076897127246463,
431 "segment_type": "mid"
432 },
433 {
434 "song_id": "song_0009",
435 "audio_path": "segments/song_0009_seg_01.wav",
436 "duration": 5.0,
437 "type": "clean",
438 "offset": 5.397707584136711,
439 "segment_type": "mid"
440 },
441 {
442 "song_id": "song_0009",
443 "audio_path": "segments/song_0009_seg_02_augmented.wav",
444 "duration": 5.0,
445 "type": "augmented",
446 "offset": 7.3864400300146755,
447 "segment_type": "mid"
448 },
449 {
450 "song_id": "song_0009",
451 "audio_path": "segments/song_0009_seg_03_humming_like.wav",
452 "duration": 5.0,
453 "type": "humming_like",
454 "offset": 5.9724644107162845,
455 "segment_type": "mid"
456 },
457 {
458 "song_id": "song_0009",
459 "audio_path": "segments/song_0009_seg_04_confused.wav",
460 "duration": 5.0,
461 "type": "confused",
462 "offset": 7.21182997805427,
463 "segment_type": "mid"
464 },
465 {
466 "song_id": "song_0009",
467 "audio_path": "songs/song_0009.wav",
468 "duration": 15.0,
469 "base_freq": 329.63,
470 "type": "reference"
471 },
472 {
473 "song_id": "song_0010",
474 "audio_path": "segments/song_0010_seg_00.wav",
475 "duration": 5.0,
476 "type": "clean",
477 "offset": 3.1007588293689183,
478 "segment_type": "mid"
479 },
480 {
481 "song_id": "song_0010",
482 "audio_path": "segments/song_0010_seg_01.wav",
483 "duration": 5.0,
484 "type": "clean",
485 "offset": 3.9822405568601704,
486 "segment_type": "mid"
487 },
488 {
489 "song_id": "song_0010",
490 "audio_path": "segments/song_0010_seg_02_augmented.wav",
491 "duration": 5.0,
492 "type": "augmented",
493 "offset": 8.154060806559823,
494 "segment_type": "mid"
495 },
496 {
497 "song_id": "song_0010",
498 "audio_path": "segments/song_0010_seg_03_humming_like.wav",
499 "duration": 5.0,
500 "type": "humming_like",
501 "offset": 2.7321660611387344,
502 "segment_type": "intro"
503 },
504 {
505 "song_id": "song_0010",
506 "audio_path": "segments/song_0010_seg_04_confused.wav",
507 "duration": 5.0,
508 "type": "confused",
509 "offset": 9.564787178236601,
510 "segment_type": "mid"
511 },
512 {
513 "song_id": "song_0010",
514 "audio_path": "songs/song_0010.wav",
515 "duration": 15.0,
516 "base_freq": 349.23,
517 "type": "reference"
518 },
519 {
520 "song_id": "song_0011",
521 "audio_path": "segments/song_0011_seg_00.wav",
522 "duration": 5.0,
523 "type": "clean",
524 "offset": 8.949259168211244,
525 "segment_type": "mid"
526 },
527 {
528 "song_id": "song_0011",
529 "audio_path": "segments/song_0011_seg_01.wav",
530 "duration": 5.0,
531 "type": "clean",
532 "offset": 8.459337061558657,
533 "segment_type": "mid"
534 },
535 {
536 "song_id": "song_0011",
537 "audio_path": "segments/song_0011_seg_02_augmented.wav",
538 "duration": 5.0,
539 "type": "augmented",
540 "offset": 2.5060530898199906,
541 "segment_type": "intro"
542 },
543 {
544 "song_id": "song_0011",
545 "audio_path": "segments/song_0011_seg_03_humming_like.wav",
546 "duration": 5.0,
547 "type": "humming_like",
548 "offset": 5.0257314474126265,
549 "segment_type": "mid"
550 },
551 {
552 "song_id": "song_0011",
553 "audio_path": "segments/song_0011_seg_04_confused.wav",
554 "duration": 5.0,
555 "type": "confused",
556 "offset": 8.42530004113389,
557 "segment_type": "mid"
558 },
559 {
560 "song_id": "song_0011",
561 "audio_path": "songs/song_0011.wav",
562 "duration": 15.0,
563 "base_freq": 392.0,
564 "type": "reference"
565 },
566 {
567 "song_id": "song_0012",
568 "audio_path": "segments/song_0012_seg_00.wav",
569 "duration": 5.0,
570 "type": "clean",
571 "offset": 7.253242125518553,
572 "segment_type": "mid"
573 },
574 {
575 "song_id": "song_0012",
576 "audio_path": "segments/song_0012_seg_01.wav",
577 "duration": 5.0,
578 "type": "clean",
579 "offset": 6.880436512027717,
580 "segment_type": "mid"
581 },
582 {
583 "song_id": "song_0012",
584 "audio_path": "segments/song_0012_seg_02_augmented.wav",
585 "duration": 5.0,
586 "type": "augmented",
587 "offset": 0.26647154963833186,
588 "segment_type": "intro"
589 },
590 {
591 "song_id": "song_0012",
592 "audio_path": "segments/song_0012_seg_03_humming_like.wav",
593 "duration": 5.0,
594 "type": "humming_like",
595 "offset": 7.214001122963067,
596 "segment_type": "mid"
597 },
598 {
599 "song_id": "song_0012",
600 "audio_path": "segments/song_0012_seg_04_confused.wav",
601 "duration": 5.0,
602 "type": "confused",
603 "offset": 1.4777570830033182,
604 "segment_type": "intro"
605 },
606 {
607 "song_id": "song_0012",
608 "audio_path": "songs/song_0012.wav",
609 "duration": 15.0,
610 "base_freq": 440.0,
611 "type": "reference"
612 },
613 {
614 "song_id": "song_0013",
615 "audio_path": "segments/song_0013_seg_00.wav",
616 "duration": 5.0,
617 "type": "clean",
618 "offset": 3.3711217932975037,
619 "segment_type": "mid"
620 },
621 {
622 "song_id": "song_0013",
623 "audio_path": "segments/song_0013_seg_01.wav",
624 "duration": 5.0,
625 "type": "clean",
626 "offset": 2.95024257658282,
627 "segment_type": "intro"
628 },
629 {
630 "song_id": "song_0013",
631 "audio_path": "segments/song_0013_seg_02_augmented.wav",
632 "duration": 5.0,
633 "type": "augmented",
634 "offset": 6.7440113989474435,
635 "segment_type": "mid"
636 },
637 {
638 "song_id": "song_0013",
639 "audio_path": "segments/song_0013_seg_03_humming_like.wav",
640 "duration": 5.0,
641 "type": "humming_like",
642 "offset": 3.27926658740176,
643 "segment_type": "mid"
644 },
645 {
646 "song_id": "song_0013",
647 "audio_path": "segments/song_0013_seg_04_confused.wav",
648 "duration": 5.0,
649 "type": "confused",
650 "offset": 0.06830120539555451,
651 "segment_type": "intro"
652 },
653 {
654 "song_id": "song_0013",
655 "audio_path": "songs/song_0013.wav",
656 "duration": 15.0,
657 "base_freq": 493.88,
658 "type": "reference"
659 },
660 {
661 "song_id": "song_0014",
662 "audio_path": "segments/song_0014_seg_00.wav",
663 "duration": 5.0,
664 "type": "clean",
665 "offset": 4.389628114874606,
666 "segment_type": "mid"
667 },
668 {
669 "song_id": "song_0014",
670 "audio_path": "segments/song_0014_seg_01.wav",
671 "duration": 5.0,
672 "type": "clean",
673 "offset": 5.397598089074283,
674 "segment_type": "mid"
675 },
676 {
677 "song_id": "song_0014",
678 "audio_path": "segments/song_0014_seg_02_augmented.wav",
679 "duration": 5.0,
680 "type": "augmented",
681 "offset": 7.543857087472844,
682 "segment_type": "mid"
683 },
684 {
685 "song_id": "song_0014",
686 "audio_path": "segments/song_0014_seg_03_humming_like.wav",
687 "duration": 5.0,
688 "type": "humming_like",
689 "offset": 5.77474814637882,
690 "segment_type": "mid"
691 },
692 {
693 "song_id": "song_0014",
694 "audio_path": "segments/song_0014_seg_04_confused.wav",
695 "duration": 5.0,
696 "type": "confused",
697 "offset": 5.212510542649235,
698 "segment_type": "mid"
699 },
700 {
701 "song_id": "song_0014",
702 "audio_path": "songs/song_0014.wav",
703 "duration": 15.0,
704 "base_freq": 523.25,
705 "type": "reference"
706 },
707 {
708 "song_id": "song_0015",
709 "audio_path": "segments/song_0015_seg_00.wav",
710 "duration": 5.0,
711 "type": "clean",
712 "offset": 5.3221248501273655,
713 "segment_type": "mid"
714 },
715 {
716 "song_id": "song_0015",
717 "audio_path": "segments/song_0015_seg_01.wav",
718 "duration": 5.0,
719 "type": "clean",
720 "offset": 4.113385082174164,
721 "segment_type": "mid"
722 },
723 {
724 "song_id": "song_0015",
725 "audio_path": "segments/song_0015_seg_02_augmented.wav",
726 "duration": 5.0,
727 "type": "augmented",
728 "offset": 0.16726147602629915,
729 "segment_type": "intro"
730 },
731 {
732 "song_id": "song_0015",
733 "audio_path": "segments/song_0015_seg_03_humming_like.wav",
734 "duration": 5.0,
735 "type": "humming_like",
736 "offset": 4.305732086760379,
737 "segment_type": "mid"
738 },
739 {
740 "song_id": "song_0015",
741 "audio_path": "segments/song_0015_seg_04_confused.wav",
742 "duration": 5.0,
743 "type": "confused",
744 "offset": 6.197808424119352,
745 "segment_type": "mid"
746 },
747 {
748 "song_id": "song_0015",
749 "audio_path": "songs/song_0015.wav",
750 "duration": 15.0,
751 "base_freq": 587.33,
752 "type": "reference"
753 }
754 ]
...\ No newline at end of file ...\ No newline at end of file
1 [
2 {
3 "song_id": "song_0016",
4 "audio_path": "segments/song_0016_seg_00.wav",
5 "duration": 5.0,
6 "type": "clean",
7 "offset": 7.208994524555927,
8 "segment_type": "mid"
9 },
10 {
11 "song_id": "song_0016",
12 "audio_path": "segments/song_0016_seg_01.wav",
13 "duration": 5.0,
14 "type": "clean",
15 "offset": 4.958024367228626,
16 "segment_type": "mid"
17 },
18 {
19 "song_id": "song_0016",
20 "audio_path": "segments/song_0016_seg_02_augmented.wav",
21 "duration": 5.0,
22 "type": "augmented",
23 "offset": 6.1666879203579,
24 "segment_type": "mid"
25 },
26 {
27 "song_id": "song_0016",
28 "audio_path": "segments/song_0016_seg_03_humming_like.wav",
29 "duration": 5.0,
30 "type": "humming_like",
31 "offset": 8.621983105655142,
32 "segment_type": "mid"
33 },
34 {
35 "song_id": "song_0016",
36 "audio_path": "segments/song_0016_seg_04_confused.wav",
37 "duration": 5.0,
38 "type": "confused",
39 "offset": 3.004352846791234,
40 "segment_type": "mid"
41 },
42 {
43 "song_id": "song_0016",
44 "audio_path": "songs/song_0016.wav",
45 "duration": 15.0,
46 "base_freq": 659.25,
47 "type": "reference"
48 },
49 {
50 "song_id": "song_0017",
51 "audio_path": "segments/song_0017_seg_00.wav",
52 "duration": 5.0,
53 "type": "clean",
54 "offset": 5.277150196277827,
55 "segment_type": "mid"
56 },
57 {
58 "song_id": "song_0017",
59 "audio_path": "segments/song_0017_seg_01.wav",
60 "duration": 5.0,
61 "type": "clean",
62 "offset": 6.391085856661506,
63 "segment_type": "mid"
64 },
65 {
66 "song_id": "song_0017",
67 "audio_path": "segments/song_0017_seg_02_augmented.wav",
68 "duration": 5.0,
69 "type": "augmented",
70 "offset": 5.969708292829935,
71 "segment_type": "mid"
72 },
73 {
74 "song_id": "song_0017",
75 "audio_path": "segments/song_0017_seg_03_humming_like.wav",
76 "duration": 5.0,
77 "type": "humming_like",
78 "offset": 6.1736267933642495,
79 "segment_type": "mid"
80 },
81 {
82 "song_id": "song_0017",
83 "audio_path": "segments/song_0017_seg_04_confused.wav",
84 "duration": 5.0,
85 "type": "confused",
86 "offset": 1.1786165266165671,
87 "segment_type": "intro"
88 },
89 {
90 "song_id": "song_0017",
91 "audio_path": "songs/song_0017.wav",
92 "duration": 15.0,
93 "base_freq": 698.46,
94 "type": "reference"
95 },
96 {
97 "song_id": "song_0018",
98 "audio_path": "segments/song_0018_seg_00.wav",
99 "duration": 5.0,
100 "type": "clean",
101 "offset": 6.641438208318426,
102 "segment_type": "mid"
103 },
104 {
105 "song_id": "song_0018",
106 "audio_path": "segments/song_0018_seg_01.wav",
107 "duration": 5.0,
108 "type": "clean",
109 "offset": 3.582227293409872,
110 "segment_type": "mid"
111 },
112 {
113 "song_id": "song_0018",
114 "audio_path": "segments/song_0018_seg_02_augmented.wav",
115 "duration": 5.0,
116 "type": "augmented",
117 "offset": 0.6333068606017467,
118 "segment_type": "intro"
119 },
120 {
121 "song_id": "song_0018",
122 "audio_path": "segments/song_0018_seg_03_humming_like.wav",
123 "duration": 5.0,
124 "type": "humming_like",
125 "offset": 3.3775515517078736,
126 "segment_type": "mid"
127 },
128 {
129 "song_id": "song_0018",
130 "audio_path": "segments/song_0018_seg_04_confused.wav",
131 "duration": 5.0,
132 "type": "confused",
133 "offset": 6.825519260932059,
134 "segment_type": "mid"
135 },
136 {
137 "song_id": "song_0018",
138 "audio_path": "songs/song_0018.wav",
139 "duration": 15.0,
140 "base_freq": 783.99,
141 "type": "reference"
142 },
143 {
144 "song_id": "song_0019",
145 "audio_path": "segments/song_0019_seg_00.wav",
146 "duration": 5.0,
147 "type": "clean",
148 "offset": 6.405372883123518,
149 "segment_type": "mid"
150 },
151 {
152 "song_id": "song_0019",
153 "audio_path": "segments/song_0019_seg_01.wav",
154 "duration": 5.0,
155 "type": "clean",
156 "offset": 5.376553581360508,
157 "segment_type": "mid"
158 },
159 {
160 "song_id": "song_0019",
161 "audio_path": "segments/song_0019_seg_02_augmented.wav",
162 "duration": 5.0,
163 "type": "augmented",
164 "offset": 1.5268044380447066,
165 "segment_type": "intro"
166 },
167 {
168 "song_id": "song_0019",
169 "audio_path": "segments/song_0019_seg_03_humming_like.wav",
170 "duration": 5.0,
171 "type": "humming_like",
172 "offset": 5.864371630124319,
173 "segment_type": "mid"
174 },
175 {
176 "song_id": "song_0019",
177 "audio_path": "segments/song_0019_seg_04_confused.wav",
178 "duration": 5.0,
179 "type": "confused",
180 "offset": 4.37486043050575,
181 "segment_type": "mid"
182 },
183 {
184 "song_id": "song_0019",
185 "audio_path": "songs/song_0019.wav",
186 "duration": 15.0,
187 "base_freq": 880.0,
188 "type": "reference"
189 }
190 ]
...\ No newline at end of file ...\ No newline at end of file
...@@ -23,6 +23,8 @@ def main(): ...@@ -23,6 +23,8 @@ def main():
23 parser.add_argument("--split", default="test") 23 parser.add_argument("--split", default="test")
24 parser.add_argument("--top-k", type=int, default=5) 24 parser.add_argument("--top-k", type=int, default=5)
25 parser.add_argument("--device", default="cpu") 25 parser.add_argument("--device", default="cpu")
26 parser.add_argument("--output-json", default=None)
27 parser.add_argument("--fast-eval", action="store_true")
26 args = parser.parse_args() 28 args = parser.parse_args()
27 29
28 data_dir = Path(args.data) 30 data_dir = Path(args.data)
...@@ -32,7 +34,7 @@ def main(): ...@@ -32,7 +34,7 @@ def main():
32 ref_embs = np.load(f"{args.index_prefix}_embs.npy") 34 ref_embs = np.load(f"{args.index_prefix}_embs.npy")
33 ref_ids = np.load(f"{args.index_prefix}_ids.npy", allow_pickle=True).tolist() 35 ref_ids = np.load(f"{args.index_prefix}_ids.npy", allow_pickle=True).tolist()
34 36
35 engine = HybridEngine(matcher, embedder, ref_embs, ref_ids) 37 engine = HybridEngine(matcher, embedder, ref_embs, ref_ids, disable_melody=args.fast_eval)
36 for split in ["train.json", "val.json", "test.json"]: 38 for split in ["train.json", "val.json", "test.json"]:
37 p = data_dir / split 39 p = data_dir / split
38 if p.exists(): 40 if p.exists():
...@@ -91,7 +93,12 @@ def main(): ...@@ -91,7 +93,12 @@ def main():
91 }, 93 },
92 "sample_failures": failures[:10], 94 "sample_failures": failures[:10],
93 } 95 }
94 print(json.dumps(report, ensure_ascii=False, indent=2)) 96 output = json.dumps(report, ensure_ascii=False, indent=2)
97 print(output)
98 if args.output_json:
99 out = Path(args.output_json)
100 out.parent.mkdir(parents=True, exist_ok=True)
101 out.write_text(output)
95 102
96 103
97 if __name__ == "__main__": 104 if __name__ == "__main__":
......
1 {
2 "generated_at": "2026-06-02T04:08:01Z",
3 "model_version": "smoke-v2",
4 "data_version": "synthetic_v2",
5 "files": {
6 "benchmark_report": "reports/smoke-v2/synthetic_v2/benchmark-report.md",
7 "model_card": "reports/smoke-v2/synthetic_v2/model-card.md",
8 "release_checklist": "reports/smoke-v2/synthetic_v2/release-checklist.md"
9 }
10 }
...\ No newline at end of file ...\ No newline at end of file
1 # Benchmark Report
2
3 ## 一页结论
4 - 模型版本:smoke-v2
5 - 数据版本:synthetic_v2
6 - 核心结论:top1=0.6 top5=0.75
7 - 是否通过上线门禁:TBD
8
9 ## 1. 评测范围图
10
11 ```mermaid
12 flowchart LR
13 A[smoke-v2] --> B[synthetic_v2]
14 A --> C[Scenario Buckets]
15 A --> D[Latency / Ops]
16 ```
17
18 ## 2. 指标表
19
20 | Bucket | top1 | top5 | MRR | FAR | Notes |
21 |---|---:|---:|---:|---:|---|
22 | clean | 1.0 | 1.0 | | | |
23 | augmented | 0.75 | 0.75 | | | |
24 | humming_like | 0.0 | 0.75 | | | |
25 | confused | 0.25 | 0.25 | | | |
26
27 ## 3. 文字分析
28 - 最强项:clean/augmented buckets if present
29 - 最弱项:see hard-case summary
30 - 与上一版本对比:TBD
31
32 ## 4. 细节附录
33 - 原始 JSON 报告:embedded source
34
35 ## Sources
36 - docs/industrial-benchmark-spec.md
1 {
2 "split": "test",
3 "num_queries": 20,
4 "top1": 0.6,
5 "topk": 0.75,
6 "by_type": {
7 "clean": {
8 "n": 8,
9 "top1": 1.0,
10 "topk": 1.0
11 },
12 "augmented": {
13 "n": 4,
14 "top1": 0.75,
15 "topk": 0.75
16 },
17 "humming_like": {
18 "n": 4,
19 "top1": 0.0,
20 "topk": 0.75
21 },
22 "confused": {
23 "n": 4,
24 "top1": 0.25,
25 "topk": 0.25
26 }
27 },
28 "hard_case_summary": {
29 "humming_like": {
30 "n": 4,
31 "top1": 0.0,
32 "topk": 0.75
33 },
34 "confused": {
35 "n": 4,
36 "top1": 0.25,
37 "topk": 0.25
38 }
39 },
40 "sample_failures": [
41 {
42 "truth": "song_0020",
43 "query": "segments/song_0020_seg_04_confused.wav",
44 "type": "confused",
45 "preds": [
46 "song_0005",
47 "song_0022",
48 "song_0001",
49 "song_0002",
50 "song_0008"
51 ]
52 },
53 {
54 "truth": "song_0021",
55 "query": "segments/song_0021_seg_04_confused.wav",
56 "type": "confused",
57 "preds": [
58 "song_0005",
59 "song_0022",
60 "song_0000",
61 "song_0002",
62 "song_0001"
63 ]
64 },
65 {
66 "truth": "song_0022",
67 "query": "segments/song_0022_seg_02_augmented.wav",
68 "type": "augmented",
69 "preds": [
70 "song_0009",
71 "song_0010",
72 "song_0006",
73 "song_0008",
74 "song_0002"
75 ]
76 },
77 {
78 "truth": "song_0022",
79 "query": "segments/song_0022_seg_03_humming_like.wav",
80 "type": "humming_like",
81 "preds": [
82 "song_0008",
83 "song_0007",
84 "song_0021",
85 "song_0001",
86 "song_0004"
87 ]
88 },
89 {
90 "truth": "song_0023",
91 "query": "segments/song_0023_seg_04_confused.wav",
92 "type": "confused",
93 "preds": [
94 "song_0022",
95 "song_0002",
96 "song_0021",
97 "song_0001",
98 "song_0005"
99 ]
100 }
101 ]
102 }
...\ No newline at end of file ...\ No newline at end of file
1 # Model Card
2
3 ## 一页结论
4 - 模型名称:ACR Hybrid Encoder
5 - 版本:smoke-v2
6 - 适用场景:music ACR prototype / retrieval
7 - 不适用场景:未经白名单数据验证的生产商用全量上线
8
9 ## 1. 模型结构图
10
11 ```mermaid
12 flowchart LR
13 A[Input Audio] --> B[128 Mel + BandSplit]
14 B --> C[Encoder]
15 C --> D[Embedding]
16 D --> E[Hybrid Retrieval]
17 ```
18
19 ## 2. 关键信息表
20
21 | 项 | 内容 |
22 |---|---|
23 | embed_dim | 192 |
24 | channels | 512 |
25 | n_mels | 128 |
26 | use_band_split | True |
27 | benchmark report | reports/smoke-v2/synthetic_v2/benchmark-report.md |
28
29 ## 3. 文字说明
30 - 训练方式:retrieval-oriented pair training
31 - 模型限制:hard-case accuracy still evolving
32 - 风险提示:requires whitelist-reviewed datasets for commercial deployment
33
34 ## 4. 细节附录
35 - config embedded from source JSON
36
37 ## Sources
38 - docs/dataset-spec.md
39 - docs/benchmark-report-template.md
1 # Release Checklist
2
3 ## 一页结论
4 发布前必须同时满足:质量通过、合规通过、服务通过、文档齐全。
5
6 ## 1. 发布门禁图
7
8 ```mermaid
9 flowchart TD
10 A[smoke-v2] --> B[Benchmark Pass]
11 A --> C[License Review Pass]
12 A --> D[Service Smoke Pass]
13 A --> E[Docs Complete]
14 ```
15
16 ## 2. Checklist 表
17
18 | 项目 | 状态 |
19 |---|---|
20 | benchmark report 已生成 | yes |
21 | model card 已生成 | yes |
22 | license registry 已更新 | pending |
23 | service smoke test 通过 | yes |
24 | dataset whitelist 已确认 | pending |
25 | changelog 已更新 | pending |
26
27 ## 3. 文字说明
28 - 当前用于工程治理与预发布检查,不代表已满足商用法律门槛。
29
30 ## 4. 细节附录
31 - benchmark 报告路径:reports/smoke-v2/synthetic_v2/benchmark-report.md
32 - model card 路径:reports/smoke-v2/synthetic_v2/model-card.md
33
34 ## Sources
35 - docs/dataset-sources-and-licensing.md
36 - docs/industrial-benchmark-spec.md
1 #!/usr/bin/env python3
2 """Generate benchmark report, model card, and release bundle artifacts."""
3
4 from __future__ import annotations
5
6 import argparse
7 import json
8 from datetime import datetime, timezone
9 from pathlib import Path
10
11
12 def utc_now():
13 return datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
14
15
16 def load_json(path: str):
17 with open(path) as f:
18 return json.load(f)
19
20
21 def write_text(path: Path, text: str):
22 path.parent.mkdir(parents=True, exist_ok=True)
23 path.write_text(text)
24
25
26 def benchmark_md(model_version: str, data_version: str, report: dict) -> str:
27 by_type = report.get('by_type', {})
28 rows = []
29 for k, v in by_type.items():
30 rows.append(f"| {k} | {v.get('top1','')} | {v.get('topk','')} | | | |")
31 rows_text = '\n'.join(rows) if rows else '| n/a | | | | | |'
32 return f'''# Benchmark Report\n\n## 一页结论\n- 模型版本:{model_version}\n- 数据版本:{data_version}\n- 核心结论:top1={report.get('top1')} top5={report.get('topk')}\n- 是否通过上线门禁:TBD\n\n## 1. 评测范围图\n\n```mermaid\nflowchart LR\n A[{model_version}] --> B[{data_version}]\n A --> C[Scenario Buckets]\n A --> D[Latency / Ops]\n```\n\n## 2. 指标表\n\n| Bucket | top1 | top5 | MRR | FAR | Notes |\n|---|---:|---:|---:|---:|---|\n{rows_text}\n\n## 3. 文字分析\n- 最强项:clean/augmented buckets if present\n- 最弱项:see hard-case summary\n- 与上一版本对比:TBD\n\n## 4. 细节附录\n- 原始 JSON 报告:embedded source\n\n## Sources\n- docs/industrial-benchmark-spec.md\n'''
33
34
35 def model_card_md(model_version: str, config: dict, benchmark_path: str) -> str:
36 model_cfg = config.get('model', {})
37 return f'''# Model Card\n\n## 一页结论\n- 模型名称:ACR Hybrid Encoder\n- 版本:{model_version}\n- 适用场景:music ACR prototype / retrieval\n- 不适用场景:未经白名单数据验证的生产商用全量上线\n\n## 1. 模型结构图\n\n```mermaid\nflowchart LR\n A[Input Audio] --> B[128 Mel + BandSplit]\n B --> C[Encoder]\n C --> D[Embedding]\n D --> E[Hybrid Retrieval]\n```\n\n## 2. 关键信息表\n\n| 项 | 内容 |\n|---|---|\n| embed_dim | {model_cfg.get('embed_dim')} |\n| channels | {model_cfg.get('channels')} |\n| n_mels | {model_cfg.get('n_mels')} |\n| use_band_split | {model_cfg.get('use_band_split')} |\n| benchmark report | {benchmark_path} |\n\n## 3. 文字说明\n- 训练方式:retrieval-oriented pair training\n- 模型限制:hard-case accuracy still evolving\n- 风险提示:requires whitelist-reviewed datasets for commercial deployment\n\n## 4. 细节附录\n- config embedded from source JSON\n\n## Sources\n- docs/dataset-spec.md\n- docs/benchmark-report-template.md\n'''
38
39
40 def release_checklist_md(model_version: str, benchmark_path: str, model_card_path: str) -> str:
41 return f'''# Release Checklist\n\n## 一页结论\n发布前必须同时满足:质量通过、合规通过、服务通过、文档齐全。\n\n## 1. 发布门禁图\n\n```mermaid\nflowchart TD\n A[{model_version}] --> B[Benchmark Pass]\n A --> C[License Review Pass]\n A --> D[Service Smoke Pass]\n A --> E[Docs Complete]\n```\n\n## 2. Checklist 表\n\n| 项目 | 状态 |\n|---|---|\n| benchmark report 已生成 | yes |\n| model card 已生成 | yes |\n| license registry 已更新 | pending |\n| service smoke test 通过 | yes |\n| dataset whitelist 已确认 | pending |\n| changelog 已更新 | pending |\n\n## 3. 文字说明\n- 当前用于工程治理与预发布检查,不代表已满足商用法律门槛。\n\n## 4. 细节附录\n- benchmark 报告路径:{benchmark_path}\n- model card 路径:{model_card_path}\n\n## Sources\n- docs/dataset-sources-and-licensing.md\n- docs/industrial-benchmark-spec.md\n'''
42
43
44 def main():
45 parser = argparse.ArgumentParser()
46 parser.add_argument('--eval-json', required=True)
47 parser.add_argument('--config-json', required=True)
48 parser.add_argument('--output-dir', required=True)
49 parser.add_argument('--model-version', default='dev')
50 parser.add_argument('--data-version', default='synthetic')
51 args = parser.parse_args()
52
53 out_dir = Path(args.output_dir)
54 out_dir.mkdir(parents=True, exist_ok=True)
55 eval_report = load_json(args.eval_json)
56 config = load_json(args.config_json)
57
58 bench_path = out_dir / 'benchmark-report.md'
59 card_path = out_dir / 'model-card.md'
60 checklist_path = out_dir / 'release-checklist.md'
61 manifest_path = out_dir / 'artifact-manifest.json'
62
63 write_text(bench_path, benchmark_md(args.model_version, args.data_version, eval_report))
64 write_text(card_path, model_card_md(args.model_version, config, str(bench_path)))
65 write_text(checklist_path, release_checklist_md(args.model_version, str(bench_path), str(card_path)))
66
67 manifest = {
68 'generated_at': utc_now(),
69 'model_version': args.model_version,
70 'data_version': args.data_version,
71 'files': {
72 'benchmark_report': str(bench_path),
73 'model_card': str(card_path),
74 'release_checklist': str(checklist_path),
75 },
76 }
77 manifest_path.write_text(json.dumps(manifest, indent=2))
78 print(json.dumps(manifest, indent=2))
79
80
81 if __name__ == '__main__':
82 main()
...@@ -39,6 +39,7 @@ class HybridEngine: ...@@ -39,6 +39,7 @@ class HybridEngine:
39 ecapa_weight: float = 0.5, 39 ecapa_weight: float = 0.5,
40 melody_weight: float = 0.25, 40 melody_weight: float = 0.25,
41 reject_threshold: float = 0.35, 41 reject_threshold: float = 0.35,
42 disable_melody: bool = False,
42 ): 43 ):
43 self.chroma = chroma_matcher 44 self.chroma = chroma_matcher
44 self.ecapa = ecapa_embedder 45 self.ecapa = ecapa_embedder
...@@ -49,6 +50,7 @@ class HybridEngine: ...@@ -49,6 +50,7 @@ class HybridEngine:
49 self.ecapa_weight = ecapa_weight 50 self.ecapa_weight = ecapa_weight
50 self.melody_weight = melody_weight 51 self.melody_weight = melody_weight
51 self.reject_threshold = reject_threshold 52 self.reject_threshold = reject_threshold
53 self.disable_melody = disable_melody
52 self.song_metadata: Dict[str, Dict] = {} 54 self.song_metadata: Dict[str, Dict] = {}
53 self.song_audio_paths: Dict[str, str] = {} 55 self.song_audio_paths: Dict[str, str] = {}
54 self.audio = AudioProcessor(sr=sr) 56 self.audio = AudioProcessor(sr=sr)
...@@ -114,8 +116,8 @@ class HybridEngine: ...@@ -114,8 +116,8 @@ class HybridEngine:
114 ecapa_matches = [(self.ref_ids[idx], float(scores[idx])) for idx in top_indices] 116 ecapa_matches = [(self.ref_ids[idx], float(scores[idx])) for idx in top_indices]
115 ecapa_norm = self._normalize_scores(ecapa_matches) 117 ecapa_norm = self._normalize_scores(ecapa_matches)
116 118
117 candidate_pool = list(set(list(chroma_norm.keys())[: top_n * 8] + list(ecapa_norm.keys())[: top_n * 8])) 119 candidate_pool = list(set(list(chroma_norm.keys())[: top_n * 4] + list(ecapa_norm.keys())[: top_n * 4]))
118 melody_norm = self._melody_scores(y, candidate_pool) 120 melody_norm = {} if self.disable_melody else self._melody_scores(y, candidate_pool)
119 121
120 all_song_ids = set(candidate_pool) | set(melody_norm) 122 all_song_ids = set(candidate_pool) | set(melody_norm)
121 combined: List[Candidate] = [] 123 combined: List[Candidate] = []
......
...@@ -95,3 +95,21 @@ ...@@ -95,3 +95,21 @@
95 - 核心 docs 存在性检查通过 95 - 核心 docs 存在性检查通过
96 - benchmark/model/release 模板结构检查通过 96 - benchmark/model/release 模板结构检查通过
97 - 所有核心文档均具备 Sources;SOTA 文档已补齐 Mermaid 图 97 - 所有核心文档均具备 Sources;SOTA 文档已补齐 Mermaid 图
98
99 ## 2026-06-02
100
101 ### Stage: 真实评测到发布产物链路打通
102
103 完成项:
104 - `evaluate.py` 支持 `--output-json`
105 - 新增 `docs/report-layout.md`
106 - 新增 `scripts/generate_artifacts.py`
107 - 打通 `eval.json -> benchmark-report.md / model-card.md / release-checklist.md / artifact-manifest.json`
108 - 为快速发布链路新增 `--fast-eval`(关闭 melody 重排以加快报告生成)
109
110 验证结果:
111 - synthetic_v2 重建、训练、建索引成功
112 - `evaluate.py --fast-eval --output-json ...` 成功输出 JSON
113 - artifact generator 成功输出 4 类发布产物
114 - `reports/smoke-v2/synthetic_v2/` 目录产物存在性检查通过
115 - 当前 fast-eval 指标:top1=0.60, top5=0.75,hard-case 仍需继续优化
......
1 # Report Layout Convention
2
3 ## 一页结论
4
5 所有评测与发布产物统一放入:
6
7 - `reports/<model-version>/<data-version>/eval.json`
8 - `reports/<model-version>/<data-version>/benchmark-report.md`
9 - `reports/<model-version>/<data-version>/model-card.md`
10 - `reports/<model-version>/<data-version>/release-checklist.md`
11 - `reports/<model-version>/<data-version>/artifact-manifest.json`
12
13 ---
14
15 ## 1. 布局图
16
17 ```mermaid
18 flowchart TD
19 A[reports/] --> B[model-version]
20 B --> C[data-version]
21 C --> D[eval.json]
22 C --> E[benchmark-report.md]
23 C --> F[model-card.md]
24 C --> G[release-checklist.md]
25 C --> H[artifact-manifest.json]
26 ```
27
28 ---
29
30 ## 2. 约定表
31
32 | 文件 | 用途 |
33 |---|---|
34 | eval.json | 机器可读评测输出 |
35 | benchmark-report.md | 人类可读 benchmark 摘要 |
36 | model-card.md | 模型说明 |
37 | release-checklist.md | 发布门禁 |
38 | artifact-manifest.json | 产物索引 |
39
40 ---
41
42 ## 3. 文字说明
43 - 所有 release 候选都应有独立目录
44 - 不要把临时 smoke 文件与正式 release 报告混放
45
46 ## Sources
47 - docs/benchmark-report-template.md
48 - docs/model-card-template.md
49 - docs/release-checklist.md