Skip to content

wanghai-tech / hikoon-ACR

Go to a project
Toggle navigation
Toggle navigation pinning
Commit 2898ef26 2898ef26ef8c15a9eacad47c63db03ff172ed1b3 by cnb.bofCdSsphPA
Options

Add the song_id pgvector evaluation scaffolding 

Constraint: we need a song-level evaluation path that matches the future pgvector production shape before moving off the local FAISS proving lane
Rejected: jumping straight to a live pgvector-only implementation | we still need a reproducible repo-local evaluation harness and artifact trail first
Confidence: high
Scope-risk: moderate
Directive: keep future pgvector work song_id-first and measure each query type separately before aggregating product claims
Tested: /usr/local/miniconda3/bin/python -m unittest discover -s acr-engine/tests -v; /usr/local/miniconda3/bin/python acr-engine/scripts/export_workspace_music20_embeddings_jsonl.py --downloads-dir /workspace/downloads --song-limit 20 --out-dir acr-engine/data/pgvector_eval/music20; /usr/local/miniconda3/bin/python acr-engine/scripts/evaluate_songid_pgvector_path.py --reference-embeddings-jsonl acr-engine/data/pgvector_eval/music20/reference_embeddings.jsonl --query-embeddings-jsonl acr-engine/data/pgvector_eval/music20/query_embeddings.jsonl --output acr-engine/data/pgvector_eval/music20/songid_eval_report.json
Not-tested: live PostgreSQL/pgvector online retrieval path
1 parent a0ceb991
Inline Side-by-side
Showing 9 changed files with 475 additions and 0 deletions
{"song_id": "100", "audio_path": "/workspace/downloads/100/type_1/BQ0000095-1.mp3", "query_type": 1, "embedding": [0.21024104952812195, 0.15962393581867218, 0.2241452932357788, 0.3148674964904785, 0.4444425404071808, 0.31021809577941895, 0.18650107085704803, 0.12177353352308273, 0.11860250681638718, 0.10217300802469254, 0.14589598774909973, 0.2586447596549988, 0.1861090511083603, 0.15307486057281494, 0.1623416393995285, 0.1619385927915573, 0.18931016325950623, 0.1651846319437027, 0.13936662673950195, 0.1476472020149231, 0.1647593230009079, 0.16297665238380432, 0.1492851972579956, 0.18407653272151947]}
{"song_id": "101", "audio_path": "/workspace/downloads/101/type_1/7679087a-c16c-4d0f-851f-b10f61b2032d1650364264.mp3", "query_type": 1, "embedding": [0.24612194299697876, 0.23341815173625946, 0.3381713032722473, 0.27765655517578125, 0.2393040955066681, 0.1684470772743225, 0.1554522067308426, 0.23536475002765656, 0.17160944640636444, 0.15149782598018646, 0.13447199761867523, 0.1996358335018158, 0.21709036827087402, 0.15920977294445038, 0.24987798929214478, 0.20510177314281464, 0.2138374298810959, 0.16478939354419708, 0.1490156501531601, 0.19698557257652283, 0.16875366866588593, 0.17671868205070496, 0.13250137865543365, 0.17265257239341736]}
{"song_id": "102", "audio_path": "/workspace/downloads/102/type_1/8fd5d3e8-e214-4d07-a283-1c5f6ac061f21650376142.mp3", "query_type": 1, "embedding": [0.36753541231155396, 0.20866069197654724, 0.3172970116138458, 0.14963465929031372, 0.12173042446374893, 0.3095773458480835, 0.1355847865343094, 0.17845334112644196, 0.1055484488606453, 0.11435747891664505, 0.12254239618778229, 0.17255263030529022, 0.30091187357902527, 0.17819836735725403, 0.25678935647010803, 0.16389189660549164, 0.11638917773962021, 0.28036215901374817, 0.11843205988407135, 0.2092035710811615, 0.14607305824756622, 0.145400270819664, 0.16659753024578094, 0.1820361614227295]}
{"song_id": "103", "audio_path": "/workspace/downloads/103/type_1/BQ0000092-1.mp3", "query_type": 1, "embedding": [0.17971792817115784, 0.14903388917446136, 0.22048214077949524, 0.29388782382011414, 0.3075105547904968, 0.19851748645305634, 0.24967841804027557, 0.25506410002708435, 0.1467844843864441, 0.17572614550590515, 0.187843918800354, 0.32441458106040955, 0.16686667501926422, 0.18699677288532257, 0.15676064789295197, 0.1528501808643341, 0.18341246247291565, 0.1555844247341156, 0.15386150777339935, 0.2026023119688034, 0.14696024358272552, 0.20919732749462128, 0.14691756665706635, 0.1895277053117752]}
{"song_id": "104", "audio_path": "/workspace/downloads/104/type_1/BQ0000091-1.mp3", "query_type": 1, "embedding": [0.12596824765205383, 0.10831266641616821, 0.10855133086442947, 0.2234056293964386, 0.3909367322921753, 0.17719240486621857, 0.13645866513252258, 0.16980375349521637, 0.37951889634132385, 0.18250399827957153, 0.1659088283777237, 0.19229331612586975, 0.1396591067314148, 0.14381667971611023, 0.14550378918647766, 0.22431398928165436, 0.27914959192276, 0.15777285397052765, 0.17921942472457886, 0.16336433589458466, 0.26766735315322876, 0.15884055197238922, 0.1861536204814911, 0.1798599511384964]}
{"song_id": "105", "audio_path": "/workspace/downloads/105/type_1/BQ0000090-1.mp3", "query_type": 1, "embedding": [0.13233302533626556, 0.21230916678905487, 0.16173411905765533, 0.29429376125335693, 0.15579722821712494, 0.12786518037319183, 0.3184970021247864, 0.1479405164718628, 0.2234833836555481, 0.10799010097980499, 0.17743690311908722, 0.2153404951095581, 0.16153615713119507, 0.2584902048110962, 0.1614491492509842, 0.26231157779693604, 0.17039120197296143, 0.1341170221567154, 0.2790423333644867, 0.1368902623653412, 0.2508905529975891, 0.15243247151374817, 0.1892421543598175, 0.2604464292526245]}
{"song_id": "106", "audio_path": "/workspace/downloads/106/type_1/BQ0000089-1.mp3", "query_type": 1, "embedding": [0.13990773260593414, 0.17645616829395294, 0.12955829501152039, 0.256458044052124, 0.10982277244329453, 0.2533506751060486, 0.21405254304409027, 0.11095551401376724, 0.2535003125667572, 0.10436020791530609, 0.20970474183559418, 0.1948678195476532, 0.16659295558929443, 0.26834505796432495, 0.11324425786733627, 0.284646213054657, 0.10309609770774841, 0.27712053060531616, 0.2610929310321808, 0.08919882774353027, 0.30467721819877625, 0.07828068733215332, 0.24599772691726685, 0.24211832880973816]}
{"song_id": "107", "audio_path": "/workspace/downloads/107/type_1/BQ0000088-1.mp3", "query_type": 1, "embedding": [0.11802207678556442, 0.14734995365142822, 0.31192946434020996, 0.20929190516471863, 0.17628973722457886, 0.16828076541423798, 0.2857673168182373, 0.25306567549705505, 0.11196132004261017, 0.1483144760131836, 0.14235477149486542, 0.23151107132434845, 0.17207719385623932, 0.16520583629608154, 0.26492542028427124, 0.16919656097888947, 0.2470686435699463, 0.16162309050559998, 0.2355172336101532, 0.215665802359581, 0.15730924904346466, 0.2094884067773819, 0.19077913463115692, 0.24468235671520233]}
{"song_id": "108", "audio_path": "/workspace/downloads/108/type_1/BQ0000087-1.mp3", "query_type": 1, "embedding": [0.24093328416347504, 0.43462124466896057, 0.23172585666179657, 0.14119377732276917, 0.13906128704547882, 0.1968367099761963, 0.27289819717407227, 0.24881495535373688, 0.3518664240837097, 0.17927215993404388, 0.1852816492319107, 0.14982320368289948, 0.13182325661182404, 0.15434366464614868, 0.11594922095537186, 0.1461736112833023, 0.14101016521453857, 0.1461103856563568, 0.17555758357048035, 0.15907242894172668, 0.19687418639659882, 0.15102709829807281, 0.14507214725017548, 0.12748195230960846]}
{"song_id": "109", "audio_path": "/workspace/downloads/109/type_1/d47d92f7-a39d-4e83-b5a2-05f98f3b74b3.mp3", "query_type": 1, "embedding": [0.22293241322040558, 0.2370966076850891, 0.17380686104297638, 0.1854366958141327, 0.25017234683036804, 0.17848452925682068, 0.19179458916187286, 0.15568006038665771, 0.15994977951049805, 0.18806500732898712, 0.1951633095741272, 0.32549193501472473, 0.2025820016860962, 0.22612814605236053, 0.17814336717128754, 0.17543891072273254, 0.23300780355930328, 0.165217325091362, 0.20080013573169708, 0.17236775159835815, 0.16630442440509796, 0.20452821254730225, 0.17076167464256287, 0.25314784049987793]}
{"song_id": "110", "audio_path": "/workspace/downloads/110/type_1/BQ0000085-1.mp3", "query_type": 1, "embedding": [0.0923147201538086, 0.14002178609371185, 0.18535864353179932, 0.18024131655693054, 0.28263425827026367, 0.14943769574165344, 0.35220447182655334, 0.1267814040184021, 0.11486991494894028, 0.2268853783607483, 0.12102863192558289, 0.08934400975704193, 0.1680677980184555, 0.18605725467205048, 0.2470562607049942, 0.22369641065597534, 0.29958194494247437, 0.1330440640449524, 0.3051621615886688, 0.15444040298461914, 0.1783791482448578, 0.283684104681015, 0.17982763051986694, 0.17996935546398163]}
{"song_id": "111", "audio_path": "/workspace/downloads/111/type_1/BQ0000084-1.mp3", "query_type": 1, "embedding": [0.12951412796974182, 0.138917937874794, 0.15827377140522003, 0.12580843269824982, 0.13527721166610718, 0.2000921666622162, 0.5906922221183777, 0.2232024222612381, 0.1713859736919403, 0.19817686080932617, 0.14916658401489258, 0.1418183296918869, 0.18095234036445618, 0.18046993017196655, 0.20480632781982422, 0.18179096281528473, 0.16934867203235626, 0.12221252173185349, 0.16200245916843414, 0.14478924870491028, 0.1736854612827301, 0.19692882895469666, 0.15684448182582855, 0.17403315007686615]}
{"song_id": "112", "audio_path": "/workspace/downloads/112/type_1/BQ0000083-1.mp3", "query_type": 1, "embedding": [0.3081006407737732, 0.1967841386795044, 0.18962736427783966, 0.12100760638713837, 0.12793976068496704, 0.08483117073774338, 0.14855912327766418, 0.4079069495201111, 0.19931305944919586, 0.12627440690994263, 0.13576605916023254, 0.25072163343429565, 0.26618653535842896, 0.1772749274969101, 0.22957532107830048, 0.1754845678806305, 0.195137619972229, 0.15984870493412018, 0.13231512904167175, 0.24873186647891998, 0.14508719742298126, 0.18510107696056366, 0.17469771206378937, 0.22834768891334534]}
{"song_id": "113", "audio_path": "/workspace/downloads/113/type_1/BQ0000082-1.mp3", "query_type": 1, "embedding": [0.3559301197528839, 0.2378990650177002, 0.24381083250045776, 0.20495976507663727, 0.23166902363300323, 0.41951555013656616, 0.17573870718479156, 0.11058478057384491, 0.10917072743177414, 0.1251496970653534, 0.13918432593345642, 0.1889239251613617, 0.1769564151763916, 0.17711122334003448, 0.2307208627462387, 0.18921402096748352, 0.16688917577266693, 0.2029678225517273, 0.1305846869945526, 0.14225593209266663, 0.1549311876296997, 0.15611770749092102, 0.1664435714483261, 0.1603447049856186]}
{"song_id": "114", "audio_path": "/workspace/downloads/114/type_1/BQ0000081-1.mp3", "query_type": 1, "embedding": [0.12785586714744568, 0.1019480973482132, 0.13021135330200195, 0.1881323754787445, 0.20822808146476746, 0.32445526123046875, 0.23042353987693787, 0.14838816225528717, 0.11805569380521774, 0.22954104840755463, 0.27340465784072876, 0.13345690071582794, 0.2090177834033966, 0.17831243574619293, 0.1789405643939972, 0.214960515499115, 0.1985044628381729, 0.2759980261325836, 0.22153766453266144, 0.19922029972076416, 0.16044244170188904, 0.2330169826745987, 0.26874884963035583, 0.16630280017852783]}
{"song_id": "115", "audio_path": "/workspace/downloads/115/type_1/BQ0000080-1.mp3", "query_type": 1, "embedding": [0.18009409308433533, 0.17446956038475037, 0.1464615762233734, 0.15856200456619263, 0.22644387185573578, 0.2851681709289551, 0.29514625668525696, 0.2613151967525482, 0.28195515275001526, 0.35818907618522644, 0.28291913866996765, 0.22453011572360992, 0.12906445562839508, 0.14289046823978424, 0.1263301819562912, 0.12684310972690582, 0.1259937286376953, 0.15661774575710297, 0.16607502102851868, 0.15889810025691986, 0.16185706853866577, 0.18293358385562897, 0.12912818789482117, 0.15997016429901123]}
{"song_id": "116", "audio_path": "/workspace/downloads/116/type_1/BQ0000079-1.mp3", "query_type": 1, "embedding": [0.19200274348258972, 0.22180412709712982, 0.30858558416366577, 0.2188439816236496, 0.26820650696754456, 0.200646311044693, 0.23710240423679352, 0.18338042497634888, 0.17693226039409637, 0.20030581951141357, 0.21940822899341583, 0.21812374889850616, 0.1901625245809555, 0.1764918565750122, 0.1931091547012329, 0.16774535179138184, 0.19387923181056976, 0.1637924462556839, 0.18354196846485138, 0.17757819592952728, 0.17959898710250854, 0.1894761472940445, 0.18745748698711395, 0.18962609767913818]}
{"song_id": "117", "audio_path": "/workspace/downloads/117/type_1/BQ0000078-1.mp3", "query_type": 1, "embedding": [0.29274022579193115, 0.18653437495231628, 0.15948711335659027, 0.22883015871047974, 0.19988803565502167, 0.3155744969844818, 0.18691709637641907, 0.3038600981235504, 0.35585933923721313, 0.1658044308423996, 0.1274653673171997, 0.16402046382427216, 0.19194376468658447, 0.16530011594295502, 0.18721053004264832, 0.20231953263282776, 0.18095727264881134, 0.21002627909183502, 0.14688792824745178, 0.15753668546676636, 0.128390833735466, 0.11982294917106628, 0.14914608001708984, 0.14538182318210602]}
{"song_id": "118", "audio_path": "/workspace/downloads/118/type_1/BQ0000077-1.mp3", "query_type": 1, "embedding": [0.3344380855560303, 0.19755131006240845, 0.2527172863483429, 0.1747017204761505, 0.22885045409202576, 0.16794882714748383, 0.14696718752384186, 0.2494439035654068, 0.15880323946475983, 0.18371231853961945, 0.14517156779766083, 0.18769191205501556, 0.2267385721206665, 0.18207453191280365, 0.24127615988254547, 0.18953460454940796, 0.20604699850082397, 0.1852400302886963, 0.15550993382930756, 0.21095797419548035, 0.17139367759227753, 0.2163010835647583, 0.20236177742481232, 0.1838425248861313]}
{"song_id": "119", "audio_path": "/workspace/downloads/119/type_1/BQ0000076-1.mp3", "query_type": 1, "embedding": [0.15240950882434845, 0.13699837028980255, 0.1516738384962082, 0.13811202347278595, 0.19630475342273712, 0.15767088532447815, 0.14158205687999725, 0.2548941969871521, 0.20330825448036194, 0.3957462012767792, 0.17246221005916595, 0.16122062504291534, 0.20773783326148987, 0.20444484055042267, 0.21932820975780487, 0.1872025728225708, 0.23045019805431366, 0.19813372194766998, 0.16469071805477142, 0.2484082579612732, 0.16560359299182892, 0.24336668848991394, 0.17760069668293, 0.21525786817073822]}
{"song_id": "111", "audio_path": "/workspace/downloads/111/type_7/75cd601b-7604-4b37-8132-cfab39e7c644.mp3", "query_type": 7, "embedding": [0.20359934866428375, 0.2908914089202881, 0.2567686140537262, 0.11649510264396667, 0.11678016930818558, 0.1253930926322937, 0.19685228168964386, 0.16798095405101776, 0.19045566022396088, 0.2650298774242401, 0.21683940291404724, 0.2816224694252014, 0.1613050252199173, 0.23058943450450897, 0.21068745851516724, 0.1206851378083229, 0.15726202726364136, 0.15407270193099976, 0.2286389023065567, 0.18348382413387299, 0.1911710798740387, 0.24720506370067596, 0.1722482144832611, 0.25567784905433655]}
{"song_id": "116", "audio_path": "/workspace/downloads/116/type_7/d7098db6-fed3-485f-906e-e55e42187f0c.mp3", "query_type": 7, "embedding": [0.11523015052080154, 0.10058188438415527, 0.27022916078567505, 0.20508715510368347, 0.3842238783836365, 0.14439290761947632, 0.14340370893478394, 0.19648931920528412, 0.13081400096416473, 0.29764896631240845, 0.1970096230506897, 0.2305494099855423, 0.1241917833685875, 0.11157573014497757, 0.2397569864988327, 0.14335812628269196, 0.27947866916656494, 0.12740325927734375, 0.17152757942676544, 0.23974281549453735, 0.10176216810941696, 0.23153623938560486, 0.16954459249973297, 0.23736445605754852]}
{"song_id": "100", "audio_path": "/workspace/downloads/100/type_11/93dfdeb0-7da5-42a8-9c71-cf12af57dd191650256918.wav", "type": "reference", "embedding": [0.21065084636211395, 0.1609223634004593, 0.22430843114852905, 0.3134746849536896, 0.4447475075721741, 0.309840589761734, 0.1858956664800644, 0.12051492184400558, 0.11787121742963791, 0.1021094024181366, 0.14646515250205994, 0.2596478760242462, 0.1859695017337799, 0.15549223124980927, 0.16317465901374817, 0.1605590581893921, 0.1895274519920349, 0.1652381271123886, 0.13907086849212646, 0.1450473815202713, 0.16308185458183289, 0.16401565074920654, 0.15024326741695404, 0.18503132462501526]}
{"song_id": "101", "audio_path": "/workspace/downloads/101/type_11/83c0c07f-4f96-4ff4-998c-58db910f3cfa1650256915.wav", "type": "reference", "embedding": [0.246183842420578, 0.2334545999765396, 0.33816978335380554, 0.2775076627731323, 0.23908916115760803, 0.16827492415905, 0.15532171726226807, 0.2354569435119629, 0.17172180116176605, 0.15155136585235596, 0.13451719284057617, 0.1995375007390976, 0.2173372507095337, 0.15937583148479462, 0.24978476762771606, 0.20494309067726135, 0.21382376551628113, 0.16457633674144745, 0.1487995833158493, 0.19715313613414764, 0.16892170906066895, 0.17687413096427917, 0.13278792798519135, 0.1726655513048172]}
{"song_id": "102", "audio_path": "/workspace/downloads/102/type_11/43440ec5-70b4-4d50-8683-d3e41cad29411650256908.wav", "type": "reference", "embedding": [0.36811912059783936, 0.20892228186130524, 0.3171277642250061, 0.14897309243679047, 0.1218678429722786, 0.30987393856048584, 0.1353084295988083, 0.17863525450229645, 0.10566563904285431, 0.11418912559747696, 0.12242863327264786, 0.1722874641418457, 0.30137741565704346, 0.17853644490242004, 0.25696876645088196, 0.16238892078399658, 0.11582915484905243, 0.2806650996208191, 0.117375947535038, 0.2091892957687378, 0.14633587002754211, 0.145274356007576, 0.1666344702243805, 0.18162840604782104]}
{"song_id": "103", "audio_path": "/workspace/downloads/103/type_11/19876dbb-fffc-40f8-9530-9322c9ed77681650256912.wav", "type": "reference", "embedding": [0.18311451375484467, 0.1384681761264801, 0.20633022487163544, 0.28072771430015564, 0.29115739464759827, 0.18233196437358856, 0.24884840846061707, 0.25734663009643555, 0.15278112888336182, 0.19237768650054932, 0.20607224106788635, 0.3431529700756073, 0.1700255274772644, 0.17014040052890778, 0.13196253776550293, 0.14030981063842773, 0.18003013730049133, 0.13703177869319916, 0.15393109619617462, 0.20614038407802582, 0.15582633018493652, 0.22947219014167786, 0.17079930007457733, 0.20181122422218323]}
{"song_id": "104", "audio_path": "/workspace/downloads/104/type_11/4c1d3e22-045f-445b-ab87-ba1ae3ee09b31650256912.wav", "type": "reference", "embedding": [0.12549766898155212, 0.10742286592721939, 0.10937031358480453, 0.22615201771259308, 0.390522837638855, 0.1752963364124298, 0.1364704817533493, 0.17149178683757782, 0.37918031215667725, 0.1808341145515442, 0.16585159301757812, 0.19238241016864777, 0.13994762301445007, 0.14246968924999237, 0.14677493274211884, 0.22480365633964539, 0.27884525060653687, 0.15748341381549835, 0.17947140336036682, 0.16379496455192566, 0.2672381103038788, 0.15885670483112335, 0.18555191159248352, 0.18082855641841888]}
{"song_id": "105", "audio_path": "/workspace/downloads/105/type_11/57e61cde-4410-4751-93e9-d7a4ecece5791650256910.wav", "type": "reference", "embedding": [0.13152149319648743, 0.21088537573814392, 0.16159486770629883, 0.29413774609565735, 0.15677160024642944, 0.1292632818222046, 0.3177519142627716, 0.146414116024971, 0.22267457842826843, 0.10971412807703018, 0.17797042429447174, 0.21500560641288757, 0.16037365794181824, 0.25689616799354553, 0.16245312988758087, 0.2621445059776306, 0.17364665865898132, 0.13856162130832672, 0.27744248509407043, 0.13332879543304443, 0.2501125633716583, 0.15843498706817627, 0.19032709300518036, 0.2599537670612335]}
{"song_id": "106", "audio_path": "/workspace/downloads/106/type_11/bf61426c-67b7-4cf1-a9e7-f78cf519a0021650256910.wav", "type": "reference", "embedding": [0.14020437002182007, 0.17636266350746155, 0.12965762615203857, 0.25691142678260803, 0.10978095233440399, 0.2533515393733978, 0.21380609273910522, 0.11107411980628967, 0.2536095976829529, 0.10436597466468811, 0.209770068526268, 0.19501900672912598, 0.1669294834136963, 0.2681954503059387, 0.11313167214393616, 0.28466466069221497, 0.10289677232503891, 0.27697089314460754, 0.26080024242401123, 0.08941096067428589, 0.30462178587913513, 0.07827121764421463, 0.2455075979232788, 0.24239546060562134]}
{"song_id": "107", "audio_path": "/workspace/downloads/107/type_11/296bbc25-617c-4368-9a69-357aeec394381650256910.wav", "type": "reference", "embedding": [0.11817862093448639, 0.11974938958883286, 0.3062351644039154, 0.208195298910141, 0.1705242097377777, 0.15676525235176086, 0.2992258071899414, 0.27951300144195557, 0.11556538939476013, 0.14152923226356506, 0.13331057131290436, 0.22344675660133362, 0.159936785697937, 0.13560564815998077, 0.2744443416595459, 0.1467897891998291, 0.24812902510166168, 0.17083309590816498, 0.2584320902824402, 0.21770411729812622, 0.15618552267551422, 0.2125440090894699, 0.18987581133842468, 0.24652305245399475]}
{"song_id": "108", "audio_path": "/workspace/downloads/108/type_11/d7e28fe6-4ad6-4243-b66b-d90ff5ca1e491650256909.wav", "type": "reference", "embedding": [0.2387375384569168, 0.4361359775066376, 0.2251766175031662, 0.13100668787956238, 0.1407063752412796, 0.20391406118869781, 0.28039926290512085, 0.25538650155067444, 0.36026281118392944, 0.17519590258598328, 0.17512458562850952, 0.1432817578315735, 0.12832649052143097, 0.16832631826400757, 0.10479722172021866, 0.12569355964660645, 0.1445886343717575, 0.15490975975990295, 0.18144796788692474, 0.16444800794124603, 0.2054930180311203, 0.13935047388076782, 0.1285625696182251, 0.11571376770734787]}
{"song_id": "109", "audio_path": "/workspace/downloads/109/type_11/84acef9b-2a74-44bc-9eff-5ca7969ac9b61650256909.wav", "type": "reference", "embedding": [0.17926231026649475, 0.1749405562877655, 0.1743948757648468, 0.18779875338077545, 0.2716197669506073, 0.18236830830574036, 0.1964801400899887, 0.17660143971443176, 0.23438310623168945, 0.285065233707428, 0.2181687206029892, 0.22067978978157043, 0.21400772035121918, 0.20947253704071045, 0.21237555146217346, 0.19631309807300568, 0.19686289131641388, 0.15810653567314148, 0.19653184711933136, 0.17317691445350647, 0.17491252720355988, 0.2089473456144333, 0.18741539120674133, 0.2172330915927887]}
{"song_id": "110", "audio_path": "/workspace/downloads/110/type_11/2197b39e-23e2-4a66-b07e-dd672eab214a1650256908.wav", "type": "reference", "embedding": [0.09237482398748398, 0.14327549934387207, 0.18673209846019745, 0.1818377524614334, 0.28002700209617615, 0.15072527527809143, 0.35688966512680054, 0.1282424032688141, 0.11429616063833237, 0.2250046283006668, 0.11746195703744888, 0.08698832988739014, 0.16744524240493774, 0.19267886877059937, 0.24848856031894684, 0.22563576698303223, 0.2974456548690796, 0.13457995653152466, 0.3084132671356201, 0.15651585161685944, 0.17638058960437775, 0.28182893991470337, 0.16957224905490875, 0.1722947359085083]}
{"song_id": "111", "audio_path": "/workspace/downloads/111/type_11/7f5256e8-de5f-41c5-bf76-419e05df72d81650256908.wav", "type": "reference", "embedding": [0.12701648473739624, 0.14027750492095947, 0.15660777688026428, 0.12502633035182953, 0.14248353242874146, 0.20542755722999573, 0.5798488855361938, 0.21973171830177307, 0.17014208436012268, 0.1961366981267929, 0.14795930683612823, 0.13763731718063354, 0.17900384962558746, 0.18741770088672638, 0.20532628893852234, 0.18452343344688416, 0.18948116898536682, 0.14547105133533478, 0.15455086529254913, 0.14613166451454163, 0.17468073964118958, 0.19636474549770355, 0.15917304158210754, 0.16934967041015625]}
{"song_id": "112", "audio_path": "/workspace/downloads/112/type_11/34acd523-3c01-443d-ac3d-4ad7b9e2246f1650256907.wav", "type": "reference", "embedding": [0.3072916567325592, 0.19724711775779724, 0.1906289905309677, 0.12292812764644623, 0.12594938278198242, 0.08180926740169525, 0.14846383035182953, 0.4079470634460449, 0.20003780722618103, 0.12660852074623108, 0.13523519039154053, 0.2503071427345276, 0.2655021846294403, 0.1797436624765396, 0.23180760443210602, 0.18045704066753387, 0.19119608402252197, 0.15219107270240784, 0.13312289118766785, 0.24752527475357056, 0.14813284575939178, 0.18677233159542084, 0.17415298521518707, 0.22857306897640228]}
{"song_id": "113", "audio_path": "/workspace/downloads/113/type_11/6d9438af-5d83-434b-bb20-76e28d0bbc4e1650256907.wav", "type": "reference", "embedding": [0.35431715846061707, 0.23911245167255402, 0.24465620517730713, 0.20387834310531616, 0.23059263825416565, 0.4177981913089752, 0.1759462058544159, 0.11241994053125381, 0.11005843430757523, 0.12489016354084015, 0.1394224464893341, 0.18881598114967346, 0.17597010731697083, 0.17929288744926453, 0.23165741562843323, 0.18789054453372955, 0.16568264365196228, 0.2022501826286316, 0.13118518888950348, 0.14595478773117065, 0.15694060921669006, 0.15562653541564941, 0.16738726198673248, 0.16100086271762848]}
{"song_id": "114", "audio_path": "/workspace/downloads/114/type_11/0238ecbf-b234-470e-82e4-f3b80a267d771650256906.wav", "type": "reference", "embedding": [0.12651732563972473, 0.10027110576629639, 0.12802469730377197, 0.18769653141498566, 0.20862409472465515, 0.32468947768211365, 0.2312006801366806, 0.14912617206573486, 0.1176583394408226, 0.22750897705554962, 0.2709246277809143, 0.13282042741775513, 0.20911023020744324, 0.17754167318344116, 0.17644557356834412, 0.21608977019786835, 0.20053324103355408, 0.27627646923065186, 0.22318944334983826, 0.2016165405511856, 0.16197042167186737, 0.23252806067466736, 0.2684091031551361, 0.16913236677646637]}
{"song_id": "115", "audio_path": "/workspace/downloads/115/type_11/aabad0ff-13de-4786-aa9c-40e1f957ed9f1650256906.wav", "type": "reference", "embedding": [0.179878368973732, 0.17446258664131165, 0.14604492485523224, 0.15842846035957336, 0.22699204087257385, 0.28520315885543823, 0.2951984405517578, 0.26090702414512634, 0.2814539968967438, 0.35812801122665405, 0.2827986478805542, 0.22442717850208282, 0.12899573147296906, 0.14319512248039246, 0.12575498223304749, 0.12688106298446655, 0.12607944011688232, 0.1565801352262497, 0.1662718802690506, 0.1592852622270584, 0.16216188669204712, 0.1832767277956009, 0.12944604456424713, 0.16046792268753052]}
{"song_id": "116", "audio_path": "/workspace/downloads/116/type_11/da34f6ff-39e7-4dde-8265-e1bb01b6263e1650256901.wav", "type": "reference", "embedding": [0.1917741894721985, 0.21905340254306793, 0.308736652135849, 0.21765963733196259, 0.2711302638053894, 0.20413215458393097, 0.24082490801811218, 0.1844777911901474, 0.1771651804447174, 0.1965189427137375, 0.21749338507652283, 0.21713195741176605, 0.1901472508907318, 0.17283162474632263, 0.19377116858959198, 0.1660938262939453, 0.1957506239414215, 0.16662529110908508, 0.1860773265361786, 0.17839395999908447, 0.1797271966934204, 0.18633370101451874, 0.18537984788417816, 0.1892005205154419]}
{"song_id": "117", "audio_path": "/workspace/downloads/117/type_11/1e1599e6-ebbd-4ceb-a81d-a320331ef6e31650256901.wav", "type": "reference", "embedding": [0.29365643858909607, 0.1854555308818817, 0.15902569890022278, 0.22892306745052338, 0.20183074474334717, 0.31933408975601196, 0.18667900562286377, 0.3031176030635834, 0.35480087995529175, 0.16361133754253387, 0.1261240839958191, 0.1636485904455185, 0.1925216019153595, 0.1636282354593277, 0.1861574947834015, 0.20228561758995056, 0.18324914574623108, 0.21258926391601562, 0.14629212021827698, 0.1574486792087555, 0.1288272738456726, 0.11544198542833328, 0.14687807857990265, 0.14499057829380035]}
{"song_id": "118", "audio_path": "/workspace/downloads/118/type_11/db64461e-d752-4cf3-ab1d-56ff9232823d1650256901.wav", "type": "reference", "embedding": [0.3346470296382904, 0.19774065911769867, 0.2515881061553955, 0.17529696226119995, 0.2284548133611679, 0.1685330718755722, 0.14735975861549377, 0.2452985793352127, 0.1577763557434082, 0.18442101776599884, 0.14760617911815643, 0.18888357281684875, 0.22612129151821136, 0.18362511694431305, 0.24010881781578064, 0.19139747321605682, 0.20603638887405396, 0.18615970015525818, 0.15588851273059845, 0.20686106383800507, 0.17028458416461945, 0.21661044657230377, 0.20556816458702087, 0.18581029772758484]}
{"song_id": "119", "audio_path": "/workspace/downloads/119/type_11/180dfa7d-836a-449c-990f-a3bf39c11da11650256898.wav", "type": "reference", "embedding": [0.1513695865869522, 0.13624103367328644, 0.15215615928173065, 0.1381070762872696, 0.19763214886188507, 0.1589500606060028, 0.1422889083623886, 0.25526976585388184, 0.20303551852703094, 0.3955025374889374, 0.17152492702007294, 0.16053307056427002, 0.2063709944486618, 0.20317678153514862, 0.21948938071727753, 0.1869896650314331, 0.23206034302711487, 0.19958095252513885, 0.1664430797100067, 0.2488074004650116, 0.16552920639514923, 0.24333658814430237, 0.1756102740764618, 0.21441961824893951]}
{
"backend": "faiss-as-pgvector-standin",
"note": "Uses song-level aggregation compatible with a future pgvector online path.",
"overall": {
"count": 22,
"top1": 0.909091,
"top3": 0.954545,
"top10": 0.954545,
"mrr": 0.934343,
"mean_rank": 1.8182,
"median_rank": 1.0
},
"by_query_type": {
"1": {
"count": 20,
"top1": 1.0,
"top3": 1.0,
"top10": 1.0,
"mrr": 1.0,
"mean_rank": 1.0,
"median_rank": 1.0
},
"7": {
"count": 2,
"top1": 0.0,
"top3": 0.5,
"top10": 0.5,
"mrr": 0.277778,
"mean_rank": 10.0,
"median_rank": 10.0
}
},
"examples": {
"1": [
{
"song_id": "100",
"rank": 1,
"top3": [
[
"100",
0.9099869644641876,
0.9999855160713196,
0.9999855160713196,
1
],
[
"116",
0.8674689626693726,
0.9527432918548584,
0.9527432918548584,
1
],
[
"103",
0.8665370559692382,
0.9517078399658203,
0.9517078399658203,
1
]
]
},
{
"song_id": "101",
"rank": 1,
"top3": [
[
"101",
0.9099996781349182,
0.9999996423721313,
0.9999996423721313,
1
],
[
"118",
0.8930539643764497,
0.9811710715293884,
0.9811710715293884,
1
],
[
"116",
0.8920178270339967,
0.9800198078155518,
0.9800198078155518,
1
]
]
},
{
"song_id": "102",
"rank": 1,
"top3": [
[
"102",
0.9099974250793457,
0.9999971389770508,
0.9999971389770508,
1
],
[
"113",
0.878619978427887,
0.9651333093643188,
0.9651333093643188,
1
],
[
"118",
0.8727551674842834,
0.9586168527603149,
0.9586168527603149,
1
]
]
},
{
"song_id": "103",
"rank": 1,
"top3": [
[
"103",
0.9078967189788818,
0.9976630210876465,
0.9976630210876465,
1
],
[
"116",
0.8892688846588135,
0.9769654273986816,
0.9769654273986816,
1
],
[
"109",
0.8786498045921325,
0.965166449546814,
0.965166449546814,
1
]
]
},
{
"song_id": "104",
"rank": 1,
"top3": [
[
"104",
0.9099890029430389,
0.999987781047821,
0.999987781047821,
1
],
[
"109",
0.8646899795532226,
0.9496555328369141,
0.9496555328369141,
1
],
[
"116",
0.8414634442329406,
0.9238482713699341,
0.9238482713699341,
1
]
]
}
],
"7": [
{
"song_id": "111",
"rank": 18,
"top3": [
[
"109",
0.8765411591529846,
0.9628235101699829,
0.9628235101699829,
1
],
[
"116",
0.8749382710456848,
0.9610425233840942,
0.9610425233840942,
1
],
[
"118",
0.8641276276111602,
0.9490306973457336,
0.9490306973457336,
1
]
]
},
{
"song_id": "116",
"rank": 2,
"top3": [
[
"109",
0.8701787447929383,
0.9557541608810425,
0.9557541608810425,
1
],
[
"116",
0.8674952483177185,
0.9527724981307983,
0.9527724981307983,
1
],
[
"103",
0.8659579670429229,
0.95106440782547,
0.95106440782547,
1
]
]
}
]
}
}
\ No newline at end of file
#!/usr/bin/env /usr/local/miniconda3/bin/python
from __future__ import annotations
import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path
from statistics import median
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
import faiss
import numpy as np
def load_jsonl(path: Path):
return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line.strip()]
def aggregate_song_scores(song_ids, sims, idxs):
aggregated = defaultdict(list)
for score, idx in zip(sims, idxs):
aggregated[song_ids[int(idx)]].append(float(score))
ranked = []
for song_id, vals in aggregated.items():
vals.sort(reverse=True)
max_sim = vals[0]
top3_avg = sum(vals[:3]) / min(3, len(vals))
vote = len(vals)
combined = 0.6 * max_sim + 0.3 * top3_avg + 0.1 * min(vote / 10.0, 1.0)
ranked.append((song_id, combined, max_sim, top3_avg, vote))
ranked.sort(key=lambda x: x[1], reverse=True)
return ranked
def compute_metrics(ranks, topk):
if not ranks:
return {'count': 0}
return {
'count': len(ranks),
'top1': round(sum(1 for r in ranks if r == 1) / len(ranks), 6),
'top3': round(sum(1 for r in ranks if r <= 3) / len(ranks), 6),
f'top{topk}': round(sum(1 for r in ranks if r <= topk) / len(ranks), 6),
'mrr': round(sum(1.0 / r for r in ranks) / len(ranks), 6),
'mean_rank': round(sum(ranks) / len(ranks), 4),
'median_rank': median(ranks),
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument('--reference-embeddings-jsonl', required=True)
ap.add_argument('--query-embeddings-jsonl', required=True)
ap.add_argument('--topn', type=int, default=20)
ap.add_argument('--topk', type=int, default=10)
ap.add_argument('--output', required=True)
args = ap.parse_args()
refs = load_jsonl(Path(args.reference_embeddings_jsonl))
queries = load_jsonl(Path(args.query_embeddings_jsonl))
ref_matrix = np.asarray([r['embedding'] for r in refs], dtype=np.float32)
song_ids = [r['song_id'] for r in refs]
index = faiss.IndexFlatIP(ref_matrix.shape[1])
index.add(ref_matrix)
by_type = defaultdict(list)
examples = defaultdict(list)
for q in queries:
qvec = np.asarray(q['embedding'], dtype=np.float32).reshape(1, -1)
sims, idxs = index.search(qvec, args.topn)
ranked = aggregate_song_scores(song_ids, sims[0], idxs[0])
gold = q['song_id']
rank = next((i + 1 for i, item in enumerate(ranked) if item[0] == gold), len(ranked) + 1)
qtype = str(q['query_type'])
by_type[qtype].append(rank)
if len(examples[qtype]) < 5:
examples[qtype].append({'song_id': gold, 'rank': rank, 'top3': ranked[:3]})
report = {
'backend': 'faiss-as-pgvector-standin',
'note': 'Uses song-level aggregation compatible with a future pgvector online path.',
'overall': compute_metrics([r for ranks in by_type.values() for r in ranks], args.topk),
'by_query_type': {qtype: compute_metrics(ranks, args.topk) for qtype, ranks in by_type.items()},
'examples': examples,
}
out = Path(args.output)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding='utf-8')
print(json.dumps(report, ensure_ascii=False, indent=2))
if __name__ == '__main__':
main()
#!/usr/bin/env /usr/local/miniconda3/bin/python
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from scripts.local_music20_acr import REFERENCE_TYPE, SUPPORTED_QUERY_TYPES, embed_chroma, first_file
def parse_args():
ap = argparse.ArgumentParser()
ap.add_argument('--downloads-dir', default='/workspace/downloads')
ap.add_argument('--song-limit', type=int, default=20)
ap.add_argument('--duration', type=float, default=8.0)
ap.add_argument('--sr', type=int, default=22050)
ap.add_argument('--out-dir', default='data/pgvector_eval/music20')
return ap.parse_args()
def main():
args = parse_args()
downloads_dir = Path(args.downloads_dir)
out_dir = Path(args.out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
ref_path = out_dir / 'reference_embeddings.jsonl'
qry_path = out_dir / 'query_embeddings.jsonl'
ref_f = ref_path.open('w', encoding='utf-8')
qry_f = qry_path.open('w', encoding='utf-8')
ref_count = qry_count = 0
refs_seen = set()
for song_dir in sorted(p for p in downloads_dir.iterdir() if p.is_dir()):
ref = first_file(song_dir / f'type_{REFERENCE_TYPE}')
if ref and song_dir.name not in refs_seen and len(refs_seen) < args.song_limit:
row = {
'song_id': song_dir.name,
'audio_path': str(ref),
'type': 'reference',
'embedding': embed_chroma(str(ref), args.sr, args.duration).tolist(),
}
ref_f.write(json.dumps(row, ensure_ascii=False) + '\n')
ref_count += 1
refs_seen.add(song_dir.name)
for query_type in SUPPORTED_QUERY_TYPES:
kept = 0
for song_id in sorted(refs_seen):
song_dir = downloads_dir / song_id
qry = first_file(song_dir / f'type_{query_type}')
if not qry:
continue
row = {
'song_id': song_id,
'audio_path': str(qry),
'query_type': query_type,
'embedding': embed_chroma(str(qry), args.sr, args.duration).tolist(),
}
qry_f.write(json.dumps(row, ensure_ascii=False) + '\n')
qry_count += 1
kept += 1
print(f'query_type={query_type} rows={kept}')
ref_f.close()
qry_f.close()
print(json.dumps({'reference_rows': ref_count, 'query_rows': qry_count, 'out_dir': str(out_dir.resolve())}, ensure_ascii=False, indent=2))
if __name__ == '__main__':
main()
import unittest
from scripts.evaluate_songid_pgvector_path import aggregate_song_scores, compute_metrics
class SongIdPgvectorPathTests(unittest.TestCase):
def test_aggregate_song_scores_ranks_by_combined_score(self):
song_ids = ['a', 'a', 'b', 'c']
sims = [0.9, 0.85, 0.95, 0.2]
idxs = [0, 1, 2, 3]
ranked = aggregate_song_scores(song_ids, sims, idxs)
self.assertEqual(ranked[0][0], 'b')
self.assertEqual(ranked[1][0], 'a')
def test_compute_metrics(self):
metrics = compute_metrics([1, 2, 4], 5)
self.assertEqual(metrics['count'], 3)
self.assertEqual(metrics['top1'], 0.333333)
self.assertEqual(metrics['top3'], 0.666667)
self.assertEqual(metrics['top5'], 1.0)
if __name__ == '__main__':
unittest.main()
- 新增 `acr-engine/scripts/export_workspace_music20_embeddings_jsonl.py` 与 `acr-engine/scripts/evaluate_songid_pgvector_path.py`，补齐 song_id 级 pgvector 评测脚手架。
- 新增 `acr-engine/data/pgvector_eval/music20/` 评测产物，当前 `faiss-as-pgvector-standin` 结果：整体 `top1=0.9091`、`top3=0.9545`；其中 `query_type=1` 很强（`top1=1.0`），`query_type=7` 仍明显偏弱（`top1=0.0`，`top3=0.5`）。
- 新增 `acr-engine/data/local_eval/voice_workspace20_type7_eval.json`，对当前 `workspace_music20` 语义做了 20 条 `type_7` 批量验证：`top1=0.0`、`top3=0.05`，说明业务 song_id 正确性仍明显不足。
- 新增 `acr-engine/data/local_eval/voice_workspace20_type8_eval.json` 与 `voice_workspace20_type16_eval.json`，补充 business-corpus voice correctness 基线：`type_8 top1=0.0/top3=0.0`，`type_16 top1=0.0/top3=0.0`。
- architect review 当前结论：`APPROVED (WATCH)`，允许继续沿当前架构推进，但不能把当前 business-corpus 结果视作完成。
......
......@@ -51,3 +51,5 @@ flowchart TD
- handoff / changelog / docs README 已同步
- handoff 已刷新：yes（已指向 voice service runtime 当前状态与下一步排查路径）
- business-corpus song_id baseline 已生成：yes（`data/pgvector_eval/music20/songid_eval_report.json`）
......
......@@ -46,6 +46,15 @@
3. 把哼唱评测集接入 `evaluate.py` 或独立评测脚本
4. 继续做 docs 第二轮收敛，只保留当前有效主文档
- 已新增 song_id 级 pgvector 评测脚手架：
- `acr-engine/scripts/export_workspace_music20_embeddings_jsonl.py`
- `acr-engine/scripts/evaluate_songid_pgvector_path.py`
- 当前 `acr-engine/data/pgvector_eval/music20/songid_eval_report.json` 结果：
- overall: `top1=0.9091`, `top3=0.9545`
- `query_type=1`: `top1=1.0`, `top3=1.0`
- `query_type=7`: `top1=0.0`, `top3=0.5`
- 注意：当前 20-song 导出里 `query_type=8/16` 行数仍不足（0 行），说明下一步需要专门扩 business reference / query 选择，而不是只沿用当前前 20 首 reference。
- 当前 `workspace_music20` 业务正确性初测（`acr-engine/data/local_eval/voice_workspace20_type7_eval.json`）：
- `num_queries=20`
- `top1=0.0`
......