Skip to content

沈秋雨 / import_hk_songs

Go to a project
Toggle navigation
Toggle navigation pinning
Commit fac42d3e fac42d3e63240e369ede39d69b7e6db7c38c889e by 沈秋雨
Options

docs: 补充相关文档

1 parent bde8dd5f
Inline Side-by-side
Showing 2 changed files with 1780 additions and 0 deletions
1 # hk_songs_test → crawler_dev ETL 实现计划
2
3 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5 **Goal:** 将 `new_music_library.hk_songs_test`（87K 条，deleted=0）通过 hikoon-data 关联链补充平台信息、hikoon-data-spider 补充完整字段，OSS 转移所有媒体 URL，批量写入 `crawler_dev` QQ/酷狗/网易三平台共 15 张表（songs/singers/albums/singer_songs/singer_albums）。
6
7 **Architecture:** 单进程 Python ETL，分层模块：config → connections → reader（hk_songs_test + hikoon-data 关联）→ spider（hikoon-data-spider 详细数据）→ oss/lyric 处理 → writer（PostgreSQL 批量写入）→ runner（批次编排）。每步幂等，按唯一键 ON CONFLICT DO NOTHING，可重跑。
8
9 **Tech Stack:** Python 3.10、pymysql、psycopg2-binary、oss2、requests、tqdm、python-dotenv；运行于项目根目录 `.venv/`。
10
11 ## Global Constraints
12
13 - virtualenv: 项目根目录 `.venv/`，运行前 `source .venv/bin/activate`
14 - 所有外部 URL（音频、封面、歌手头像）统一转到 `archive-dev.oss-cn-beijing.aliyuncs.com`；已在 archive-dev 的 URL 直接复用
15 - 写入顺序每平台固定：singers → albums → songs → singer_songs → singer_albums
16 - 所有 PostgreSQL 写入按唯一键 ON CONFLICT DO NOTHING（幂等）
17 - 批量大小：每批 500 条 hk_songs_test 记录
18 - 平台编号：1=QQ，2=酷狗，4=网易
19 - 新增 .env 变量：`CRAWLER_DB_HOST`、`CRAWLER_DB_PORT`、`CRAWLER_DB_USER`、`CRAWLER_DB_PASSWORD`、`CRAWLER_DB_NAME`（对应 crawler_dev PostgreSQL）
20
21 ---
22
23 ## File Structure
24
25 ```
26 etl_to_crawler/
27 __init__.py # 空文件
28 config.py # 从 .env 加载全部连接参数，暴露常量
29 connections.py # 创建并返回 MySQL / PostgreSQL / OSS 连接对象
30 lyric.py # 去除 LRC [mm:ss.xx] 时间戳
31 oss.py # 将任意 URL 转移到 archive-dev OSS，已在则跳过
32 reader.py # 读 hk_songs_test（TencentDB）+ hikoon-data 关联链
33 spider.py # 从 hikoon-data-spider 获取三平台详细数据
34 writer.py # 批量写入 PostgreSQL（5 类表 × 3 平台）
35 runner.py # 批次编排、进度日志、错误收集
36
37 tests/
38 test_lyric.py
39 test_oss.py
40 test_spider.py
41 test_writer.py
42
43 run_etl.py # CLI 入口：python run_etl.py [--platform qq|kugou|netease|all]
44 ```
45
46 ---
47
48 ### Task 1: 环境准备 & 连接验证
49
50 **Files:**
51 - Modify: `requirements.txt`
52 - Create: `etl_to_crawler/__init__.py`
53 - Create: `etl_to_crawler/config.py`
54 - Create: `etl_to_crawler/connections.py`
55 - Modify: `.env`（追加 CRAWLER_DB_* 变量）
56
57 **Interfaces:**
58 - Produces:
59 - `config.SOURCE_DB: dict` — hikoon-data MySQL 连接参数
60 - `config.HK_SONGS_DB: dict` — new_music_library MySQL 连接参数
61 - `config.CRAWLER_DB: dict` — crawler_dev PostgreSQL 连接参数
62 - `config.OSS_CONFIG: dict` — archive-dev OSS 参数
63 - `connections.get_source_conn() -> pymysql.Connection` — hikoon-data
64 - `connections.get_hk_songs_conn() -> pymysql.Connection` — new_music_library
65 - `connections.get_pg_conn() -> psycopg2.connection` — crawler_dev
66 - `connections.get_oss_bucket() -> oss2.Bucket`
67
68 - [ ] **Step 1: 安装 psycopg2-binary**
69
70 ```bash
71 source .venv/bin/activate
72 pip install psycopg2-binary
73 pip freeze | grep psycopg2
74 ```
75
76 期望输出：`psycopg2-binary==2.9.x`
77
78 - [ ] **Step 2: 追加 CRAWLER_DB 变量到 .env**
79
80 在 `.env` 末尾追加（填入实际值）：
81 ```
82 CRAWLER_DB_HOST=data-center.rwlb.rds.aliyuncs.com
83 CRAWLER_DB_PORT=5432
84 CRAWLER_DB_USER=data_test
85 CRAWLER_DB_PASSWORD=z1n_z#JHW0
86 CRAWLER_DB_NAME=crawler_dev
87 ```
88
89 - [ ] **Step 3: 创建 `etl_to_crawler/__init__.py`**
90
91 ```python
92 ```
93 （空文件）
94
95 - [ ] **Step 4: 创建 `etl_to_crawler/config.py`**
96
97 ```python
98 import os
99 from dotenv import load_dotenv
100
101 load_dotenv()
102
103 SOURCE_DB = {
104 'host': os.environ['SOURCE_DB_HOST'],
105 'port': int(os.environ['SOURCE_DB_PORT']),
106 'user': os.environ['SOURCE_DB_USER'],
107 'password': os.environ['SOURCE_DB_PASSWORD'],
108 'database': os.environ['SOURCE_DB_NAME'],
109 'charset': 'utf8mb4',
110 }
111
112 HK_SONGS_DB = {
113 'host': os.environ['TARGET_DB_HOST'],
114 'port': int(os.environ['TARGET_DB_PORT']),
115 'user': os.environ['TARGET_DB_USER'],
116 'password': os.environ['TARGET_DB_PASSWORD'],
117 'database': os.environ['TARGET_DB_NAME'],
118 'charset': 'utf8mb4',
119 }
120
121 CRAWLER_DB = {
122 'host': os.environ['CRAWLER_DB_HOST'],
123 'port': int(os.environ.get('CRAWLER_DB_PORT', '5432')),
124 'user': os.environ['CRAWLER_DB_USER'],
125 'password': os.environ['CRAWLER_DB_PASSWORD'],
126 'dbname': os.environ['CRAWLER_DB_NAME'],
127 'sslmode': 'prefer',
128 }
129
130 OSS_CONFIG = {
131 'access_key_id': os.environ['OSS_ACCESS_KEY_ID'],
132 'access_key_secret': os.environ['OSS_ACCESS_KEY_SECRET'],
133 'endpoint': os.environ['OSS_ENDPOINT'],
134 'bucket_name': os.environ['OSS_BUCKET_NAME'],
135 'base_url': f"https://{os.environ['OSS_BUCKET_NAME']}.{os.environ['OSS_ENDPOINT']}",
136 }
137
138 PLATFORM_QQ = '1'
139 PLATFORM_KUGOU = '2'
140 PLATFORM_NETEASE = '4'
141 PLATFORMS = [PLATFORM_QQ, PLATFORM_KUGOU, PLATFORM_NETEASE]
142
143 BATCH_SIZE = 500
144 ```
145
146 - [ ] **Step 5: 创建 `etl_to_crawler/connections.py`**
147
148 ```python
149 import pymysql
150 import pymysql.cursors
151 import psycopg2
152 import oss2
153 from .config import SOURCE_DB, HK_SONGS_DB, CRAWLER_DB, OSS_CONFIG
154
155
156 def get_source_conn() -> pymysql.Connection:
157 return pymysql.connect(**SOURCE_DB, cursorclass=pymysql.cursors.DictCursor)
158
159
160 def get_spider_conn() -> pymysql.Connection:
161 cfg = SOURCE_DB.copy()
162 cfg['database'] = 'hikoon-data-spider'
163 return pymysql.connect(**cfg, cursorclass=pymysql.cursors.DictCursor)
164
165
166 def get_hk_songs_conn() -> pymysql.Connection:
167 return pymysql.connect(**HK_SONGS_DB, cursorclass=pymysql.cursors.DictCursor)
168
169
170 def get_pg_conn() -> psycopg2.extensions.connection:
171 return psycopg2.connect(**CRAWLER_DB)
172
173
174 def get_oss_bucket() -> oss2.Bucket:
175 auth = oss2.Auth(OSS_CONFIG['access_key_id'], OSS_CONFIG['access_key_secret'])
176 return oss2.Bucket(auth, OSS_CONFIG['endpoint'], OSS_CONFIG['bucket_name'])
177 ```
178
179 - [ ] **Step 6: 验证所有连接**
180
181 ```bash
182 source .venv/bin/activate
183 python3 -c "
184 from etl_to_crawler.connections import (
185 get_source_conn, get_spider_conn, get_hk_songs_conn, get_pg_conn, get_oss_bucket
186 )
187 c = get_source_conn(); c.close(); print('hikoon-data OK')
188 c = get_spider_conn(); c.close(); print('hikoon-data-spider OK')
189 c = get_hk_songs_conn(); c.close(); print('new_music_library OK')
190 c = get_pg_conn(); c.close(); print('crawler_dev OK')
191 b = get_oss_bucket(); b.get_bucket_info(); print('OSS OK')
192 "
193 ```
194
195 期望输出：5 行全部 OK。
196
197 - [ ] **Step 7: 更新 requirements.txt**
198
199 ```bash
200 pip freeze > requirements.txt
201 ```
202
203 - [ ] **Step 8: Commit**
204
205 ```bash
206 git add requirements.txt .env etl_to_crawler/
207 git commit -m "feat(etl): 初始化 etl_to_crawler 模块，验证三端连接"
208 ```
209
210 ---
211
212 ### Task 2: lyric.py — 去除 LRC 时间戳
213
214 **Files:**
215 - Create: `etl_to_crawler/lyric.py`
216 - Create: `tests/test_lyric.py`
217
218 **Interfaces:**
219 - Produces: `lyric.strip_timestamps(text: str | None) -> str`
220
221 - [ ] **Step 1: 写测试**
222
223 ```python
224 # tests/test_lyric.py
225 from etl_to_crawler.lyric import strip_timestamps
226
227 def test_strips_standard_timestamps():
228 lrc = "[00:12.34]第一行歌词\n[01:23.45]第二行歌词"
229 assert strip_timestamps(lrc) == "第一行歌词\n第二行歌词"
230
231 def test_strips_millisecond_timestamps():
232 lrc = "[00:12.345]带三位毫秒"
233 assert strip_timestamps(lrc) == "带三位毫秒"
234
235 def test_strips_meta_tags():
236 lrc = "[ti:歌曲名]\n[ar:歌手]\n[00:01.00]歌词"
237 result = strip_timestamps(lrc)
238 assert "歌词" in result
239 assert "[ti:" not in result
240
241 def test_empty_and_none():
242 assert strip_timestamps(None) == ''
243 assert strip_timestamps('') == ''
244
245 def test_plain_text_unchanged():
246 assert strip_timestamps("没有时间戳的歌词") == "没有时间戳的歌词"
247 ```
248
249 - [ ] **Step 2: 运行测试确认失败**
250
251 ```bash
252 source .venv/bin/activate && pytest tests/test_lyric.py -v
253 ```
254
255 期望：FAILED（ImportError 或 NameError）
256
257 - [ ] **Step 3: 实现 `etl_to_crawler/lyric.py`**
258
259 ```python
260 import re
261
262 _TIMESTAMP_RE = re.compile(r'\[\d{2}:\d{2}\.\d{2,3}\]')
263 _META_TAG_RE = re.compile(r'\[[a-zA-Z]+:[^\]]*\]')
264
265
266 def strip_timestamps(text: str | None) -> str:
267 if not text:
268 return ''
269 text = _META_TAG_RE.sub('', text)
270 text = _TIMESTAMP_RE.sub('', text)
271 lines = [line.strip() for line in text.splitlines() if line.strip()]
272 return '\n'.join(lines)
273 ```
274
275 - [ ] **Step 4: 运行测试确认通过**
276
277 ```bash
278 pytest tests/test_lyric.py -v
279 ```
280
281 期望：5 个测试全部 PASSED
282
283 - [ ] **Step 5: Commit**
284
285 ```bash
286 git add etl_to_crawler/lyric.py tests/test_lyric.py
287 git commit -m "feat(etl): 实现 LRC 时间戳去除"
288 ```
289
290 ---
291
292 ### Task 3: oss.py — URL 转移到 archive-dev
293
294 **Files:**
295 - Create: `etl_to_crawler/oss.py`
296 - Create: `tests/test_oss.py`
297
298 **Interfaces:**
299 - Consumes: `connections.get_oss_bucket()`、`config.OSS_CONFIG`
300 - Produces: `oss.transfer_url(url: str, oss_key: str, bucket: oss2.Bucket, base_url: str) -> str`
301
302 - [ ] **Step 1: 写测试**
303
304 ```python
305 # tests/test_oss.py
306 from unittest.mock import MagicMock, patch
307 from etl_to_crawler.oss import transfer_url
308
309 ARCHIVE_URL = "https://archive-dev.oss-cn-beijing.aliyuncs.com/some/path.mp3"
310 OTHER_URL = "https://hikoon-data-platform.oss-cn-beijing.aliyuncs.com/qq-audio/abc.mp3"
311 BASE_URL = "https://archive-dev.oss-cn-beijing.aliyuncs.com"
312
313 def test_already_archive_dev_returns_as_is():
314 bucket = MagicMock()
315 result = transfer_url(ARCHIVE_URL, "any/key.mp3", bucket, BASE_URL)
316 assert result == ARCHIVE_URL
317 bucket.put_object.assert_not_called()
318
319 def test_empty_url_returns_empty():
320 bucket = MagicMock()
321 result = transfer_url('', "any/key.mp3", bucket, BASE_URL)
322 assert result == ''
323 bucket.put_object.assert_not_called()
324
325 def test_none_url_returns_empty():
326 bucket = MagicMock()
327 result = transfer_url(None, "any/key.mp3", bucket, BASE_URL)
328 assert result == ''
329
330 def test_external_url_downloads_and_uploads():
331 bucket = MagicMock()
332 fake_content = b"audio_bytes"
333 with patch('etl_to_crawler.oss.requests.get') as mock_get:
334 mock_get.return_value.content = fake_content
335 mock_get.return_value.raise_for_status = MagicMock()
336 result = transfer_url(OTHER_URL, "crawler/qq/audio/abc.mp3", bucket, BASE_URL)
337 bucket.put_object.assert_called_once_with("crawler/qq/audio/abc.mp3", fake_content)
338 assert result == f"{BASE_URL}/crawler/qq/audio/abc.mp3"
339 ```
340
341 - [ ] **Step 2: 运行测试确认失败**
342
343 ```bash
344 pytest tests/test_oss.py -v
345 ```
346
347 期望：FAILED（ImportError）
348
349 - [ ] **Step 3: 实现 `etl_to_crawler/oss.py`**
350
351 ```python
352 import requests
353 import oss2
354 from urllib.parse import urlparse
355
356 ARCHIVE_DEV_HOST = 'archive-dev.oss-cn-beijing.aliyuncs.com'
357
358
359 def transfer_url(url: str | None, oss_key: str, bucket: oss2.Bucket, base_url: str) -> str:
360 """
361 将 url 指向的文件转移到 archive-dev OSS 的 oss_key 路径。
362 若 url 为空或已在 archive-dev，直接返回原 url（不上传）。
363 返回新的 archive-dev URL。
364 """
365 if not url:
366 return ''
367 if ARCHIVE_DEV_HOST in url:
368 return url
369 resp = requests.get(url, timeout=30)
370 resp.raise_for_status()
371 bucket.put_object(oss_key, resp.content)
372 return f"{base_url.rstrip('/')}/{oss_key}"
373
374
375 def build_oss_key(platform: str, category: str, filename: str) -> str:
376 """
377 构造 archive-dev 内的存储路径。
378 platform: 'qq' | 'kugou' | 'netease'
379 category: 'audio' | 'cover' | 'singer' | 'album'
380 filename: 带扩展名的文件名
381 """
382 return f"crawler/{platform}/{category}/{filename}"
383 ```
384
385 - [ ] **Step 4: 运行测试确认通过**
386
387 ```bash
388 pytest tests/test_oss.py -v
389 ```
390
391 期望：4 个测试全部 PASSED
392
393 - [ ] **Step 5: Commit**
394
395 ```bash
396 git add etl_to_crawler/oss.py tests/test_oss.py
397 git commit -m "feat(etl): 实现 OSS URL 转移工具"
398 ```
399
400 ---
401
402 ### Task 4: reader.py — 读取源数据
403
404 **Files:**
405 - Create: `etl_to_crawler/reader.py`
406
407 **Interfaces:**
408 - Consumes: `connections.get_hk_songs_conn()`、`connections.get_source_conn()`
409 - Produces:
410 - `reader.iter_hk_songs_batches(conn, batch_size: int) -> Iterator[list[dict]]` — 每批返回 hk_songs_test 记录
411 - `reader.fetch_platform_records(source_conn, song_ids: list[int]) -> list[dict]` — 返回每条的平台信息
412
413 每条 platform record 格式：
414 ```python
415 {
416 'source_song_id': int, # hk_song_platform.id
417 'platform': str, # '1'|'2'|'4'
418 'platform_unique_key': str, # QQ:mid / 酷狗:数字ID / 网易:数字ID
419 'platform_mid': str, # QQ:数字ID / 酷狗:hash
420 'album_audio_id': str,
421 }
422 ```
423
424 - [ ] **Step 1: 实现 `etl_to_crawler/reader.py`**
425
426 ```python
427 from typing import Iterator
428 import pymysql
429 from .config import PLATFORMS
430
431
432 _HK_SONGS_QUERY = """
433 SELECT id, name, lyricist, composer, audio_url, lyrics_url,
434 cover_url, singer, issue_time, source_song_id
435 FROM hk_songs_test
436 WHERE deleted = '0'
437 AND name IS NOT NULL AND name != ''
438 AND audio_url IS NOT NULL AND audio_url != ''
439 AND singer IS NOT NULL AND singer != ''
440 ORDER BY id
441 LIMIT %s OFFSET %s
442 """
443
444 _PLATFORM_QUERY = """
445 SELECT
446 sar.song_id AS source_song_id,
447 mr.platform,
448 mr.platform_unique_key,
449 mr.platform_mid,
450 mr.album_audio_id,
451 sar.is_main_version
452 FROM hk_song_and_record sar
453 JOIN hk_music_record mr ON mr.id = sar.record_id
454 WHERE sar.song_id IN ({placeholders})
455 AND mr.platform IN ('1','2','4')
456 AND mr.platform_unique_key IS NOT NULL
457 AND mr.platform_unique_key != ''
458 AND mr.deleted = 0
459 ORDER BY sar.song_id, mr.platform, sar.is_main_version DESC
460 """
461
462
463 def iter_hk_songs_batches(conn: pymysql.Connection, batch_size: int) -> Iterator[list[dict]]:
464 offset = 0
465 with conn.cursor() as cur:
466 while True:
467 cur.execute(_HK_SONGS_QUERY, (batch_size, offset))
468 rows = cur.fetchall()
469 if not rows:
470 break
471 yield rows
472 if len(rows) < batch_size:
473 break
474 offset += batch_size
475
476
477 def fetch_platform_records(source_conn: pymysql.Connection, song_ids: list[int]) -> list[dict]:
478 if not song_ids:
479 return []
480 placeholders = ','.join(['%s'] * len(song_ids))
481 query = _PLATFORM_QUERY.format(placeholders=placeholders)
482 with source_conn.cursor() as cur:
483 cur.execute(query, song_ids)
484 rows = cur.fetchall()
485
486 # 每个 (source_song_id, platform) 只保留 is_main_version 最高的一条
487 seen = {}
488 for row in rows:
489 key = (row['source_song_id'], row['platform'])
490 if key not in seen:
491 seen[key] = row
492 return list(seen.values())
493 ```
494
495 - [ ] **Step 2: 手动验证**
496
497 ```bash
498 source .venv/bin/activate
499 python3 -c "
500 from etl_to_crawler.connections import get_hk_songs_conn, get_source_conn
501 from etl_to_crawler.reader import iter_hk_songs_batches, fetch_platform_records
502
503 hk = get_hk_songs_conn()
504 src = get_source_conn()
505 batches = iter_hk_songs_batches(hk, 10)
506 first_batch = next(batches)
507 print('hk_songs sample:', first_batch[0]['name'], first_batch[0]['source_song_id'])
508
509 song_ids = [int(r['source_song_id']) for r in first_batch]
510 recs = fetch_platform_records(src, song_ids)
511 print('platform records:', len(recs))
512 for r in recs[:3]:
513 print(' ', r['platform'], r['platform_unique_key'][:20])
514 hk.close(); src.close()
515 "
516 ```
517
518 期望：打印出歌曲名、source_song_id，及多条带平台号的记录。
519
520 - [ ] **Step 3: Commit**
521
522 ```bash
523 git add etl_to_crawler/reader.py
524 git commit -m "feat(etl): 实现 hk_songs_test 读取与 hikoon-data 关联"
525 ```
526
527 ---
528
529 ### Task 5: spider.py — 从 hikoon-data-spider 获取三平台详细数据
530
531 **Files:**
532 - Create: `etl_to_crawler/spider.py`
533 - Create: `tests/test_spider.py`
534
535 **Interfaces:**
536 - Consumes: `connections.get_spider_conn()`
537 - Produces:
538 - `spider.fetch_qq_songs(conn, mids: list[str]) -> dict[str, dict]` — keyed by mid
539 - `spider.fetch_qq_singers(conn, song_ids: list[int]) -> dict[int, list[dict]]` — keyed by song_id
540 - `spider.fetch_kugou_songs(conn, song_ids: list[int]) -> dict[int, dict]` — keyed by id
541 - `spider.fetch_kugou_singers(conn, song_ids: list[int]) -> dict[int, list[dict]]`
542 - `spider.fetch_netease_songs(conn, song_ids: list[int]) -> dict[int, dict]`
543 - `spider.fetch_netease_singers(conn, song_ids: list[int]) -> dict[int, list[dict]]`
544
545 - [ ] **Step 1: 写测试**
546
547 ```python
548 # tests/test_spider.py
549 from unittest.mock import MagicMock, patch
550 from etl_to_crawler.spider import fetch_qq_songs, fetch_qq_singers
551
552 def _make_cursor(rows):
553 cur = MagicMock()
554 cur.__enter__ = MagicMock(return_value=cur)
555 cur.__exit__ = MagicMock(return_value=False)
556 cur.fetchall.return_value = rows
557 return cur
558
559 def test_fetch_qq_songs_keys_by_mid():
560 conn = MagicMock()
561 conn.cursor.return_value = _make_cursor([
562 {'mid': 'abc123', 'id': 1, 'title': '歌曲A', 'duration': 200,
563 'lyric': None, 'composer_name': None, 'lyricist_name': None,
564 'platform_index_url': None, 'published_at': None,
565 'cover': '', 'album_id': None,
566 'album_mid': None, 'album_title': None, 'album_cover': None,
567 'album_intro': None, 'album_type': None, 'company_id': 0,
568 'company': None, 'is_owner': 0, 'album_published_at': None}
569 ])
570 result = fetch_qq_songs(conn, ['abc123'])
571 assert 'abc123' in result
572 assert result['abc123']['title'] == '歌曲A'
573
574 def test_fetch_qq_songs_empty_list():
575 conn = MagicMock()
576 result = fetch_qq_songs(conn, [])
577 assert result == {}
578 conn.cursor.assert_not_called()
579
580 def test_fetch_qq_singers_keys_by_song_id():
581 conn = MagicMock()
582 conn.cursor.return_value = _make_cursor([
583 {'song_id': 1, 'singer_id': 10, 'mid': 'sg1', 'name': '歌手A',
584 'avatar': '', 'sex': 'M', 'area': '华语', 'index': 'G',
585 'intro': None, 'home_url': None}
586 ])
587 result = fetch_qq_singers(conn, [1])
588 assert 1 in result
589 assert result[1][0]['name'] == '歌手A'
590 ```
591
592 - [ ] **Step 2: 运行测试确认失败**
593
594 ```bash
595 pytest tests/test_spider.py -v
596 ```
597
598 期望：FAILED（ImportError）
599
600 - [ ] **Step 3: 实现 `etl_to_crawler/spider.py`**
601
602 ```python
603 import pymysql
604
605
606 def _ids_query(query_template: str, ids: list, conn: pymysql.Connection) -> list[dict]:
607 if not ids:
608 return []
609 placeholders = ','.join(['%s'] * len(ids))
610 query = query_template.format(placeholders=placeholders)
611 with conn.cursor() as cur:
612 cur.execute(query, ids)
613 return cur.fetchall()
614
615
616 # ─── QQ Music ────────────────────────────────────────────────────────────────
617
618 _QQ_SONGS_SQL = """
619 SELECT s.id, s.mid, s.album_id, s.cover, s.title, s.duration,
620 s.lyric, s.composer_name, s.lyricist_name, s.platform_index_url, s.published_at,
621 a.mid AS album_mid, a.cover AS album_cover, a.title AS album_title,
622 a.intro AS album_intro, a.type AS album_type, a.company_id,
623 a.company, a.is_owner, a.published_at AS album_published_at
624 FROM media_tencent_songs s
625 LEFT JOIN media_tencent_albums a ON a.id = s.album_id
626 WHERE s.mid IN ({placeholders})
627 """
628
629 _QQ_SINGERS_SQL = """
630 SELECT shs.song_id, shs.singer_id, sg.mid, sg.name, sg.avatar,
631 sg.sex, sg.area, sg.`index`, sg.intro, sg.home_url
632 FROM media_tencent_singer_has_songs shs
633 JOIN media_tencent_singers sg ON sg.id = shs.singer_id
634 WHERE shs.song_id IN ({placeholders})
635 """
636
637
638 def fetch_qq_songs(conn: pymysql.Connection, mids: list[str]) -> dict[str, dict]:
639 rows = _ids_query(_QQ_SONGS_SQL, mids, conn)
640 return {row['mid']: row for row in rows}
641
642
643 def fetch_qq_singers(conn: pymysql.Connection, song_ids: list[int]) -> dict[int, list[dict]]:
644 rows = _ids_query(_QQ_SINGERS_SQL, song_ids, conn)
645 result: dict[int, list] = {}
646 for row in rows:
647 result.setdefault(row['song_id'], []).append(row)
648 return result
649
650
651 # ─── Kugou ───────────────────────────────────────────────────────────────────
652
653 _KUGOU_SONGS_SQL = """
654 SELECT s.id, s.hid, s.album_audio_id, s.album_id, s.cover, s.title, s.duration,
655 s.lyric, s.composer_name, s.lyricist_name, s.platform_index_url, s.published_at,
656 a.cover AS album_cover, a.title AS album_title, a.intro AS album_intro,
657 a.type AS album_type, a.company_id, a.company, a.is_owner,
658 a.published_at AS album_published_at
659 FROM media_ku_gou_songs s
660 LEFT JOIN media_ku_gou_albums a ON a.id = s.album_id
661 WHERE s.id IN ({placeholders})
662 """
663
664 _KUGOU_SINGERS_SQL = """
665 SELECT shs.song_id, shs.singer_id, sg.name, sg.avatar,
666 sg.sex, sg.area, sg.`index`, sg.intro, sg.home_url
667 FROM media_ku_gou_singer_has_songs shs
668 JOIN media_ku_gou_singers sg ON sg.id = shs.singer_id
669 WHERE shs.song_id IN ({placeholders})
670 """
671
672
673 def fetch_kugou_songs(conn: pymysql.Connection, song_ids: list[int]) -> dict[int, dict]:
674 rows = _ids_query(_KUGOU_SONGS_SQL, song_ids, conn)
675 return {row['id']: row for row in rows}
676
677
678 def fetch_kugou_singers(conn: pymysql.Connection, song_ids: list[int]) -> dict[int, list[dict]]:
679 rows = _ids_query(_KUGOU_SINGERS_SQL, song_ids, conn)
680 result: dict[int, list] = {}
681 for row in rows:
682 result.setdefault(row['song_id'], []).append(row)
683 return result
684
685
686 # ─── Netease ─────────────────────────────────────────────────────────────────
687
688 _NETEASE_SONGS_SQL = """
689 SELECT s.id, s.album_id, s.cover, s.title, s.duration,
690 s.lyric, s.composer_name, s.lyricist_name, s.platform_index_url, s.published_at,
691 a.cover AS album_cover, a.title AS album_title, a.intro AS album_intro,
692 a.type AS album_type, a.company_id, a.company, a.is_owner,
693 a.published_at AS album_published_at
694 FROM media_netease_songs s
695 LEFT JOIN media_netease_albums a ON a.id = s.album_id
696 WHERE s.id IN ({placeholders})
697 """
698
699 _NETEASE_SINGERS_SQL = """
700 SELECT shs.song_id, shs.singer_id, sg.name, sg.avatar,
701 sg.sex, sg.area, sg.`index`, sg.intro, sg.home_url
702 FROM media_netease_singer_has_songs shs
703 JOIN media_netease_singers sg ON sg.id = shs.singer_id
704 WHERE shs.song_id IN ({placeholders})
705 """
706
707
708 def fetch_netease_songs(conn: pymysql.Connection, song_ids: list[int]) -> dict[int, dict]:
709 rows = _ids_query(_NETEASE_SONGS_SQL, song_ids, conn)
710 return {row['id']: row for row in rows}
711
712
713 def fetch_netease_singers(conn: pymysql.Connection, song_ids: list[int]) -> dict[int, list[dict]]:
714 rows = _ids_query(_NETEASE_SINGERS_SQL, song_ids, conn)
715 result: dict[int, list] = {}
716 for row in rows:
717 result.setdefault(row['song_id'], []).append(row)
718 return result
719 ```
720
721 - [ ] **Step 4: 运行测试确认通过**
722
723 ```bash
724 pytest tests/test_spider.py -v
725 ```
726
727 期望：3 个测试全部 PASSED
728
729 - [ ] **Step 5: Commit**
730
731 ```bash
732 git add etl_to_crawler/spider.py tests/test_spider.py
733 git commit -m "feat(etl): 实现 hikoon-data-spider 三平台数据获取"
734 ```
735
736 ---
737
738 ### Task 6: writer.py — 写入 PostgreSQL
739
740 **Files:**
741 - Create: `etl_to_crawler/writer.py`
742 - Create: `tests/test_writer.py`
743
744 **Interfaces:**
745 - Consumes: `connections.get_pg_conn()`
746 - Produces:
747 - `writer.upsert_qq_singers(pg_cur, singers: list[dict]) -> None`
748 - `writer.upsert_qq_albums(pg_cur, albums: list[dict]) -> None`
749 - `writer.upsert_qq_songs(pg_cur, songs: list[dict]) -> None`
750 - `writer.upsert_qq_singer_songs(pg_cur, pairs: list[tuple[int,str]]) -> None`
751 - `writer.upsert_qq_singer_albums(pg_cur, pairs: list[tuple[int,int]]) -> None`
752 - （kugou / netease 同名函数，前缀不同）
753
754 singers 格式：`{'id':int, 'mid':str, 'name':str, 'avatar':str, 'sex':str, 'area':str, 'index':str, 'intro':str|None, 'home_url':str|None}`
755
756 songs 格式：`{'song_uuid':str, 'platform_song_id':int, 'mid':str, 'album_id':int|None, 'cover':str, 'title':str, 'name':str, 'duration':int, 'lyric':str|None, 'composer_name':str|None, 'lyricist_name':str|None, 'url':str, 'lyric_url':str|None, 'platform_index_url':str|None, 'published_at':date|None, 'singers_json':str}`
757
758 - [ ] **Step 1: 写测试**
759
760 ```python
761 # tests/test_writer.py
762 from unittest.mock import MagicMock, call
763 from etl_to_crawler.writer import upsert_qq_singers, upsert_qq_singer_songs
764
765 def test_upsert_qq_singers_executes_insert():
766 cur = MagicMock()
767 singers = [{
768 'id': 1, 'mid': 'abc', 'name': '歌手', 'avatar': 'https://archive-dev.oss-cn-beijing.aliyuncs.com/a.jpg',
769 'sex': 'M', 'area': '华语', 'index': 'G', 'intro': None, 'home_url': None
770 }]
771 upsert_qq_singers(cur, singers)
772 assert cur.executemany.called
773 args = cur.executemany.call_args
774 assert 'crawler_qqmusic_singers' in args[0][0]
775 assert 'ON CONFLICT' in args[0][0]
776
777 def test_upsert_qq_singers_empty_does_nothing():
778 cur = MagicMock()
779 upsert_qq_singers(cur, [])
780 cur.executemany.assert_not_called()
781
782 def test_upsert_qq_singer_songs():
783 cur = MagicMock()
784 upsert_qq_singer_songs(cur, [(1, 'uuid-abc'), (2, 'uuid-def')])
785 assert cur.executemany.called
786 sql = cur.executemany.call_args[0][0]
787 assert 'crawler_qqmusic_singer_songs' in sql
788 ```
789
790 - [ ] **Step 2: 运行测试确认失败**
791
792 ```bash
793 pytest tests/test_writer.py -v
794 ```
795
796 期望：FAILED（ImportError）
797
798 - [ ] **Step 3: 实现 `etl_to_crawler/writer.py`**
799
800 ```python
801 import uuid
802 import json
803 import psycopg2.extras
804
805
806 # ─── QQ Music ────────────────────────────────────────────────────────────────
807
808 def upsert_qq_singers(cur, singers: list[dict]) -> None:
809 if not singers:
810 return
811 sql = """
812 INSERT INTO crawler_qqmusic_singers
813 (id, mid, name, avatar, sex, area, "index", intro, home_url, created_at, updated_at)
814 VALUES (%s, %s, %s, %s, %s::singer_sex, %s::singer_area, %s::singer_index, %s, %s, NOW(), NOW())
815 ON CONFLICT (id) DO NOTHING
816 """
817 rows = [(
818 s['id'], s['mid'], s['name'], s.get('avatar', ''),
819 s.get('sex') or 'U', s.get('area') or '其他', s.get('index') or '#',
820 s.get('intro'), s.get('home_url'),
821 ) for s in singers]
822 cur.executemany(sql, rows)
823
824
825 def upsert_qq_albums(cur, albums: list[dict]) -> None:
826 if not albums:
827 return
828 sql = """
829 INSERT INTO crawler_qqmusic_albums
830 (id, mid, cover, title, intro, type, company_id, company, is_owner, published_at, created_at, updated_at)
831 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, NOW(), NOW())
832 ON CONFLICT (id) DO NOTHING
833 """
834 rows = [(
835 a['id'], a.get('mid', ''), a.get('cover', ''), a.get('title', ''),
836 a.get('intro'), a.get('type', ''), a.get('company_id', 0) or 0,
837 a.get('company', ''), a.get('is_owner', 0) or 0, a.get('published_at'),
838 ) for a in albums]
839 cur.executemany(sql, rows)
840
841
842 def upsert_qq_songs(cur, songs: list[dict]) -> None:
843 if not songs:
844 return
845 sql = """
846 INSERT INTO crawler_qqmusic_songs
847 (id, platform_song_id, mid, album_id, cover, title, name, duration,
848 lyric, composer_name, lyricist_name, url, lyric_url,
849 platform_index_url, published_at, singers, status, created_at, updated_at)
850 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb, 0, NOW(), NOW())
851 ON CONFLICT (platform_song_id) DO NOTHING
852 """
853 rows = [(
854 s['song_uuid'], s['platform_song_id'], s['mid'], s.get('album_id'),
855 s.get('cover', ''), s.get('title', ''), s.get('name', ''), s.get('duration', 0) or 0,
856 s.get('lyric'), s.get('composer_name'), s.get('lyricist_name'),
857 s.get('url', ''), s.get('lyric_url'),
858 s.get('platform_index_url'), s.get('published_at'), s.get('singers_json', '[]'),
859 ) for s in songs]
860 cur.executemany(sql, rows)
861
862
863 def upsert_qq_singer_songs(cur, pairs: list[tuple]) -> None:
864 """pairs: [(singer_id, song_uuid), ...]"""
865 if not pairs:
866 return
867 sql = """
868 INSERT INTO crawler_qqmusic_singer_songs (id, singer_id, song_id)
869 VALUES (%s, %s, %s)
870 ON CONFLICT (singer_id, song_id) DO NOTHING
871 """
872 rows = [(str(uuid.uuid4()), singer_id, song_uuid) for singer_id, song_uuid in pairs]
873 cur.executemany(sql, rows)
874
875
876 def upsert_qq_singer_albums(cur, pairs: list[tuple]) -> None:
877 """pairs: [(singer_id, album_id), ...]"""
878 if not pairs:
879 return
880 sql = """
881 INSERT INTO crawler_qqmusic_singer_albums (id, singer_id, album_id)
882 VALUES (%s, %s, %s)
883 ON CONFLICT (singer_id, album_id) DO NOTHING
884 """
885 rows = [(str(uuid.uuid4()), singer_id, album_id) for singer_id, album_id in pairs]
886 cur.executemany(sql, rows)
887
888
889 # ─── Kugou ───────────────────────────────────────────────────────────────────
890
891 def upsert_kugou_singers(cur, singers: list[dict]) -> None:
892 if not singers:
893 return
894 sql = """
895 INSERT INTO crawler_kugou_singers
896 (id, name, avatar, sex, area, "index", intro, home_url, created_at, updated_at)
897 VALUES (%s, %s, %s, %s::kugou_singer_sex, %s::kugou_singer_area, %s::kugou_singer_index, %s, %s, NOW(), NOW())
898 ON CONFLICT (id) DO NOTHING
899 """
900 rows = [(
901 s['id'], s['name'], s.get('avatar', ''),
902 s.get('sex') or 'U', s.get('area') or '其他', s.get('index') or '#',
903 s.get('intro'), s.get('home_url', ''),
904 ) for s in singers]
905 cur.executemany(sql, rows)
906
907
908 def upsert_kugou_albums(cur, albums: list[dict]) -> None:
909 if not albums:
910 return
911 sql = """
912 INSERT INTO crawler_kugou_albums
913 (id, cover, title, intro, type, company_id, company, is_owner, published_at, created_at, updated_at)
914 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, NOW(), NOW())
915 ON CONFLICT (id) DO NOTHING
916 """
917 rows = [(
918 a['id'], a.get('cover', ''), a.get('title', ''), a.get('intro'),
919 a.get('type', ''), a.get('company_id', 0) or 0,
920 a.get('company', ''), a.get('is_owner', 0) or 0, a.get('published_at'),
921 ) for a in albums]
922 cur.executemany(sql, rows)
923
924
925 def upsert_kugou_songs(cur, songs: list[dict]) -> None:
926 if not songs:
927 return
928 sql = """
929 INSERT INTO crawler_kugou_songs
930 (id, platform_song_id, hash, album_audio_id, album_id, cover, title, name, duration,
931 lyric, composer_name, lyricist_name, url, lyric_url,
932 platform_index_url, published_at, singers, status, created_at, updated_at)
933 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb, 0, NOW(), NOW())
934 ON CONFLICT (platform_song_id) DO NOTHING
935 """
936 rows = [(
937 s['song_uuid'], s['platform_song_id'], s.get('hash', ''),
938 s.get('album_audio_id', 0) or 0, s.get('album_id'),
939 s.get('cover', ''), s.get('title', ''), s.get('name', ''), s.get('duration', 0) or 0,
940 s.get('lyric'), s.get('composer_name'), s.get('lyricist_name'),
941 s.get('url', ''), s.get('lyric_url'),
942 s.get('platform_index_url'), s.get('published_at'), s.get('singers_json', '[]'),
943 ) for s in songs]
944 cur.executemany(sql, rows)
945
946
947 def upsert_kugou_singer_songs(cur, pairs: list[tuple]) -> None:
948 if not pairs:
949 return
950 sql = """
951 INSERT INTO crawler_kugou_singer_songs (id, singer_id, song_id)
952 VALUES (%s, %s, %s)
953 ON CONFLICT (singer_id, song_id) DO NOTHING
954 """
955 cur.executemany(sql, [(str(uuid.uuid4()), s, sg) for s, sg in pairs])
956
957
958 def upsert_kugou_singer_albums(cur, pairs: list[tuple]) -> None:
959 if not pairs:
960 return
961 sql = """
962 INSERT INTO crawler_kugou_singer_albums (id, singer_id, album_id)
963 VALUES (%s, %s, %s)
964 ON CONFLICT (singer_id, album_id) DO NOTHING
965 """
966 cur.executemany(sql, [(str(uuid.uuid4()), s, a) for s, a in pairs])
967
968
969 # ─── Netease ─────────────────────────────────────────────────────────────────
970
971 def upsert_netease_singers(cur, singers: list[dict]) -> None:
972 if not singers:
973 return
974 sql = """
975 INSERT INTO crawler_netease_singers
976 (id, name, avatar, sex, area, "index", intro, home_url, created_at, updated_at)
977 VALUES (%s, %s, %s, %s::netease_singer_sex, %s::netease_singer_area, %s::netease_singer_index, %s, %s, NOW(), NOW())
978 ON CONFLICT (id) DO NOTHING
979 """
980 rows = [(
981 s['id'], s['name'], s.get('avatar', ''),
982 s.get('sex') or 'U', s.get('area') or '其他', s.get('index') or '#',
983 s.get('intro'), s.get('home_url'),
984 ) for s in singers]
985 cur.executemany(sql, rows)
986
987
988 def upsert_netease_albums(cur, albums: list[dict]) -> None:
989 if not albums:
990 return
991 sql = """
992 INSERT INTO crawler_netease_albums
993 (id, cover, title, intro, type, company_id, company, is_owner, published_at, created_at, updated_at)
994 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, NOW(), NOW())
995 ON CONFLICT (id) DO NOTHING
996 """
997 rows = [(
998 a['id'], a.get('cover', ''), a.get('title', ''), a.get('intro'),
999 a.get('type', ''), a.get('company_id', 0) or 0,
1000 a.get('company', ''), a.get('is_owner', 0) or 0, a.get('published_at'),
1001 ) for a in albums]
1002 cur.executemany(sql, rows)
1003
1004
1005 def upsert_netease_songs(cur, songs: list[dict]) -> None:
1006 if not songs:
1007 return
1008 sql = """
1009 INSERT INTO crawler_netease_songs
1010 (id, platform_song_id, album_id, cover, title, name, duration,
1011 lyric, composer_name, lyricist_name, url, lyric_url,
1012 platform_index_url, published_at, singers, status, created_at, updated_at)
1013 VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb, 0, NOW(), NOW())
1014 ON CONFLICT (platform_song_id) DO NOTHING
1015 """
1016 rows = [(
1017 s['song_uuid'], s['platform_song_id'], s.get('album_id'),
1018 s.get('cover', ''), s.get('title', ''), s.get('name', ''), s.get('duration', 0) or 0,
1019 s.get('lyric'), s.get('composer_name'), s.get('lyricist_name'),
1020 s.get('url', ''), s.get('lyric_url'),
1021 s.get('platform_index_url'), s.get('published_at'), s.get('singers_json', '[]'),
1022 ) for s in songs]
1023 cur.executemany(sql, rows)
1024
1025
1026 def upsert_netease_singer_songs(cur, pairs: list[tuple]) -> None:
1027 if not pairs:
1028 return
1029 sql = """
1030 INSERT INTO crawler_netease_singer_songs (id, singer_id, song_id)
1031 VALUES (%s, %s, %s)
1032 ON CONFLICT (singer_id, song_id) DO NOTHING
1033 """
1034 cur.executemany(sql, [(str(uuid.uuid4()), s, sg) for s, sg in pairs])
1035
1036
1037 def upsert_netease_singer_albums(cur, pairs: list[tuple]) -> None:
1038 if not pairs:
1039 return
1040 sql = """
1041 INSERT INTO crawler_netease_singer_albums (id, singer_id, album_id)
1042 VALUES (%s, %s, %s)
1043 ON CONFLICT (singer_id, album_id) DO NOTHING
1044 """
1045 cur.executemany(sql, [(str(uuid.uuid4()), s, a) for s, a in pairs])
1046 ```
1047
1048 - [ ] **Step 4: 运行测试确认通过**
1049
1050 ```bash
1051 pytest tests/test_writer.py -v
1052 ```
1053
1054 期望：3 个测试全部 PASSED
1055
1056 - [ ] **Step 5: Commit**
1057
1058 ```bash
1059 git add etl_to_crawler/writer.py tests/test_writer.py
1060 git commit -m "feat(etl): 实现三平台 PostgreSQL 写入函数"
1061 ```
1062
1063 ---
1064
1065 ### Task 7: runner.py + run_etl.py — 批次编排与主入口
1066
1067 **Files:**
1068 - Create: `etl_to_crawler/runner.py`
1069 - Create: `run_etl.py`
1070
1071 **Interfaces:**
1072 - Consumes: reader、spider、writer、oss、lyric、connections 全部模块
1073 - Produces: `runner.run(platforms: list[str]) -> None`
1074
1075 - [ ] **Step 1: 实现 `etl_to_crawler/runner.py`**
1076
1077 ```python
1078 import uuid
1079 import json
1080 import logging
1081 from tqdm import tqdm
1082
1083 from .config import PLATFORM_QQ, PLATFORM_KUGOU, PLATFORM_NETEASE, BATCH_SIZE, OSS_CONFIG
1084 from .connections import get_hk_songs_conn, get_source_conn, get_spider_conn, get_pg_conn, get_oss_bucket
1085 from .reader import iter_hk_songs_batches, fetch_platform_records
1086 from .spider import (
1087 fetch_qq_songs, fetch_qq_singers,
1088 fetch_kugou_songs, fetch_kugou_singers,
1089 fetch_netease_songs, fetch_netease_singers,
1090 )
1091 from .writer import (
1092 upsert_qq_singers, upsert_qq_albums, upsert_qq_songs,
1093 upsert_qq_singer_songs, upsert_qq_singer_albums,
1094 upsert_kugou_singers, upsert_kugou_albums, upsert_kugou_songs,
1095 upsert_kugou_singer_songs, upsert_kugou_singer_albums,
1096 upsert_netease_singers, upsert_netease_albums, upsert_netease_songs,
1097 upsert_netease_singer_songs, upsert_netease_singer_albums,
1098 )
1099 from .oss import transfer_url, build_oss_key
1100 from .lyric import strip_timestamps
1101
1102 logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
1103 log = logging.getLogger(__name__)
1104
1105
1106 def _safe_transfer(url, oss_key, bucket, base_url):
1107 try:
1108 return transfer_url(url, oss_key, bucket, base_url)
1109 except Exception as e:
1110 log.warning("OSS transfer failed for %s: %s", url, e)
1111 return url # 失败时保留原 URL，不阻断流程
1112
1113
1114 def _process_qq(hk_row: dict, pr: dict, spider_conn, pg_cur, bucket, base_url):
1115 mid = pr['platform_unique_key']
1116 songs_map = fetch_qq_songs(spider_conn, [mid])
1117 if mid not in songs_map:
1118 return
1119 sp = songs_map[mid]
1120 song_id_int = sp['id']
1121
1122 singers_map = fetch_qq_singers(spider_conn, [song_id_int])
1123 singer_list = singers_map.get(song_id_int, [])
1124
1125 # OSS 转移
1126 audio_url = _safe_transfer(
1127 hk_row['audio_url'],
1128 build_oss_key('qq', 'audio', mid + '.mp3'),
1129 bucket, base_url
1130 )
1131 cover_url = _safe_transfer(
1132 hk_row.get('cover_url') or sp.get('cover', ''),
1133 build_oss_key('qq', 'cover', str(song_id_int) + '.jpg'),
1134 bucket, base_url
1135 )
1136 album_cover = ''
1137 if sp.get('album_id') and sp.get('album_cover'):
1138 album_cover = _safe_transfer(
1139 sp['album_cover'],
1140 build_oss_key('qq', 'album', str(sp['album_id']) + '.jpg'),
1141 bucket, base_url
1142 )
1143
1144 # 歌手头像转移 + 写入 singers
1145 singer_rows = []
1146 for sg in singer_list:
1147 avatar = _safe_transfer(
1148 sg.get('avatar', ''),
1149 build_oss_key('qq', 'singer', sg['mid'] + '.jpg'),
1150 bucket, base_url
1151 )
1152 singer_rows.append({**sg, 'avatar': avatar})
1153 upsert_qq_singers(pg_cur, singer_rows)
1154
1155 # singers JSONB
1156 singers_json = json.dumps([{
1157 'name': sg['name'],
1158 'singer_id': sg['singer_id'],
1159 'platform_singer_id': sg['mid'],
1160 } for sg in singer_list], ensure_ascii=False)
1161
1162 # 写入 album
1163 album_id = None
1164 if sp.get('album_id'):
1165 album_id = sp['album_id']
1166 upsert_qq_albums(pg_cur, [{
1167 'id': sp['album_id'], 'mid': sp.get('album_mid', ''),
1168 'cover': album_cover, 'title': sp.get('album_title', ''),
1169 'intro': sp.get('album_intro'), 'type': sp.get('album_type', ''),
1170 'company_id': sp.get('company_id', 0), 'company': sp.get('company', ''),
1171 'is_owner': sp.get('is_owner', 0), 'published_at': sp.get('album_published_at'),
1172 }])
1173
1174 # 写入 song
1175 song_uuid = str(uuid.uuid4())
1176 upsert_qq_songs(pg_cur, [{
1177 'song_uuid': song_uuid,
1178 'platform_song_id': int(pr['platform_mid']) if pr.get('platform_mid') else song_id_int,
1179 'mid': mid,
1180 'album_id': album_id,
1181 'cover': cover_url,
1182 'title': sp.get('title', hk_row['name']),
1183 'name': hk_row['name'],
1184 'duration': sp.get('duration', 0) or 0,
1185 'lyric': strip_timestamps(sp.get('lyric')),
1186 'composer_name': sp.get('composer_name') or hk_row.get('composer'),
1187 'lyricist_name': sp.get('lyricist_name') or hk_row.get('lyricist'),
1188 'url': audio_url,
1189 'lyric_url': hk_row.get('lyrics_url'),
1190 'platform_index_url': sp.get('platform_index_url'),
1191 'published_at': sp.get('published_at') or hk_row.get('issue_time'),
1192 'singers_json': singers_json,
1193 }])
1194
1195 # singer_songs / singer_albums
1196 upsert_qq_singer_songs(pg_cur, [(sg['singer_id'], song_uuid) for sg in singer_list])
1197 if album_id:
1198 upsert_qq_singer_albums(pg_cur, [(sg['singer_id'], album_id) for sg in singer_list])
1199
1200
1201 def _process_kugou(hk_row: dict, pr: dict, spider_conn, pg_cur, bucket, base_url):
1202 song_id = int(pr['platform_unique_key'])
1203 songs_map = fetch_kugou_songs(spider_conn, [song_id])
1204 if song_id not in songs_map:
1205 return
1206 sp = songs_map[song_id]
1207
1208 singers_map = fetch_kugou_singers(spider_conn, [song_id])
1209 singer_list = singers_map.get(song_id, [])
1210
1211 audio_url = _safe_transfer(
1212 hk_row['audio_url'],
1213 build_oss_key('kugou', 'audio', str(song_id) + '.mp3'),
1214 bucket, base_url
1215 )
1216 cover_url = _safe_transfer(
1217 hk_row.get('cover_url') or sp.get('cover', ''),
1218 build_oss_key('kugou', 'cover', str(song_id) + '.jpg'),
1219 bucket, base_url
1220 )
1221 album_cover = ''
1222 if sp.get('album_id') and sp.get('album_cover'):
1223 album_cover = _safe_transfer(
1224 sp['album_cover'],
1225 build_oss_key('kugou', 'album', str(sp['album_id']) + '.jpg'),
1226 bucket, base_url
1227 )
1228
1229 singer_rows = []
1230 for sg in singer_list:
1231 avatar = _safe_transfer(
1232 sg.get('avatar', ''),
1233 build_oss_key('kugou', 'singer', str(sg['singer_id']) + '.jpg'),
1234 bucket, base_url
1235 )
1236 singer_rows.append({**sg, 'id': sg['singer_id'], 'avatar': avatar})
1237 upsert_kugou_singers(pg_cur, singer_rows)
1238
1239 singers_json = json.dumps([{
1240 'name': sg['name'],
1241 'singer_id': sg['singer_id'],
1242 'platform_singer_id': str(sg['singer_id']),
1243 } for sg in singer_list], ensure_ascii=False)
1244
1245 album_id = None
1246 if sp.get('album_id'):
1247 album_id = sp['album_id']
1248 upsert_kugou_albums(pg_cur, [{
1249 'id': sp['album_id'], 'cover': album_cover,
1250 'title': sp.get('album_title', ''), 'intro': sp.get('album_intro'),
1251 'type': sp.get('album_type', ''), 'company_id': sp.get('company_id', 0),
1252 'company': sp.get('company', ''), 'is_owner': sp.get('is_owner', 0),
1253 'published_at': sp.get('album_published_at'),
1254 }])
1255
1256 song_uuid = str(uuid.uuid4())
1257 upsert_kugou_songs(pg_cur, [{
1258 'song_uuid': song_uuid,
1259 'platform_song_id': song_id,
1260 'hash': sp.get('hid', pr.get('platform_mid', '')),
1261 'album_audio_id': sp.get('album_audio_id') or pr.get('album_audio_id') or 0,
1262 'album_id': album_id,
1263 'cover': cover_url,
1264 'title': sp.get('title', hk_row['name']),
1265 'name': hk_row['name'],
1266 'duration': sp.get('duration', 0) or 0,
1267 'lyric': strip_timestamps(sp.get('lyric')),
1268 'composer_name': sp.get('composer_name') or hk_row.get('composer'),
1269 'lyricist_name': sp.get('lyricist_name') or hk_row.get('lyricist'),
1270 'url': audio_url,
1271 'lyric_url': hk_row.get('lyrics_url'),
1272 'platform_index_url': sp.get('platform_index_url'),
1273 'published_at': sp.get('published_at') or hk_row.get('issue_time'),
1274 'singers_json': singers_json,
1275 }])
1276
1277 upsert_kugou_singer_songs(pg_cur, [(sg['singer_id'], song_uuid) for sg in singer_list])
1278 if album_id:
1279 upsert_kugou_singer_albums(pg_cur, [(sg['singer_id'], album_id) for sg in singer_list])
1280
1281
1282 def _process_netease(hk_row: dict, pr: dict, spider_conn, pg_cur, bucket, base_url):
1283 song_id = int(pr['platform_unique_key'])
1284 songs_map = fetch_netease_songs(spider_conn, [song_id])
1285 if song_id not in songs_map:
1286 return
1287 sp = songs_map[song_id]
1288
1289 singers_map = fetch_netease_singers(spider_conn, [song_id])
1290 singer_list = singers_map.get(song_id, [])
1291
1292 audio_url = _safe_transfer(
1293 hk_row['audio_url'],
1294 build_oss_key('netease', 'audio', str(song_id) + '.mp3'),
1295 bucket, base_url
1296 )
1297 cover_url = _safe_transfer(
1298 hk_row.get('cover_url') or sp.get('cover', ''),
1299 build_oss_key('netease', 'cover', str(song_id) + '.jpg'),
1300 bucket, base_url
1301 )
1302 album_cover = ''
1303 if sp.get('album_id') and sp.get('album_cover'):
1304 album_cover = _safe_transfer(
1305 sp['album_cover'],
1306 build_oss_key('netease', 'album', str(sp['album_id']) + '.jpg'),
1307 bucket, base_url
1308 )
1309
1310 singer_rows = []
1311 for sg in singer_list:
1312 avatar = _safe_transfer(
1313 sg.get('avatar', ''),
1314 build_oss_key('netease', 'singer', str(sg['singer_id']) + '.jpg'),
1315 bucket, base_url
1316 )
1317 singer_rows.append({**sg, 'id': sg['singer_id'], 'avatar': avatar})
1318 upsert_netease_singers(pg_cur, singer_rows)
1319
1320 singers_json = json.dumps([{
1321 'name': sg['name'],
1322 'singer_id': sg['singer_id'],
1323 'platform_singer_id': str(sg['singer_id']),
1324 } for sg in singer_list], ensure_ascii=False)
1325
1326 album_id = None
1327 if sp.get('album_id'):
1328 album_id = sp['album_id']
1329 upsert_netease_albums(pg_cur, [{
1330 'id': sp['album_id'], 'cover': album_cover,
1331 'title': sp.get('album_title', ''), 'intro': sp.get('album_intro'),
1332 'type': sp.get('album_type', ''), 'company_id': sp.get('company_id', 0),
1333 'company': sp.get('company', ''), 'is_owner': sp.get('is_owner', 0),
1334 'published_at': sp.get('album_published_at'),
1335 }])
1336
1337 song_uuid = str(uuid.uuid4())
1338 upsert_netease_songs(pg_cur, [{
1339 'song_uuid': song_uuid,
1340 'platform_song_id': song_id,
1341 'album_id': album_id,
1342 'cover': cover_url,
1343 'title': sp.get('title', hk_row['name']),
1344 'name': hk_row['name'],
1345 'duration': sp.get('duration', 0) or 0,
1346 'lyric': strip_timestamps(sp.get('lyric')),
1347 'composer_name': sp.get('composer_name') or hk_row.get('composer'),
1348 'lyricist_name': sp.get('lyricist_name') or hk_row.get('lyricist'),
1349 'url': audio_url,
1350 'lyric_url': hk_row.get('lyrics_url'),
1351 'platform_index_url': sp.get('platform_index_url'),
1352 'published_at': sp.get('published_at') or hk_row.get('issue_time'),
1353 'singers_json': singers_json,
1354 }])
1355
1356 upsert_netease_singer_songs(pg_cur, [(sg['singer_id'], song_uuid) for sg in singer_list])
1357 if album_id:
1358 upsert_netease_singer_albums(pg_cur, [(sg['singer_id'], album_id) for sg in singer_list])
1359
1360
1361 _PROCESSORS = {
1362 PLATFORM_QQ: _process_qq,
1363 PLATFORM_KUGOU: _process_kugou,
1364 PLATFORM_NETEASE: _process_netease,
1365 }
1366
1367
1368 def run(platforms: list[str]) -> None:
1369 hk_conn = get_hk_songs_conn()
1370 src_conn = get_source_conn()
1371 spider_conn = get_spider_conn()
1372 pg_conn = get_pg_conn()
1373 bucket = get_oss_bucket()
1374 base_url = OSS_CONFIG['base_url']
1375
1376 total_ok = total_err = 0
1377
1378 try:
1379 for batch in tqdm(iter_hk_songs_batches(hk_conn, BATCH_SIZE), desc='batches'):
1380 song_ids = [int(r['source_song_id']) for r in batch if r.get('source_song_id')]
1381 platform_records = fetch_platform_records(src_conn, song_ids)
1382
1383 # index platform records by source_song_id
1384 pr_by_song: dict[int, list] = {}
1385 for pr in platform_records:
1386 if pr['platform'] in platforms:
1387 pr_by_song.setdefault(int(pr['source_song_id']), []).append(pr)
1388
1389 with pg_conn.cursor() as pg_cur:
1390 for hk_row in batch:
1391 src_id = int(hk_row['source_song_id']) if hk_row.get('source_song_id') else None
1392 if not src_id or src_id not in pr_by_song:
1393 continue
1394 for pr in pr_by_song[src_id]:
1395 processor = _PROCESSORS.get(pr['platform'])
1396 if not processor:
1397 continue
1398 try:
1399 processor(hk_row, pr, spider_conn, pg_cur, bucket, base_url)
1400 total_ok += 1
1401 except Exception as e:
1402 log.error("Error processing song %s platform %s: %s",
1403 hk_row.get('name'), pr['platform'], e)
1404 total_err += 1
1405 pg_conn.commit()
1406
1407 finally:
1408 hk_conn.close()
1409 src_conn.close()
1410 spider_conn.close()
1411 pg_conn.close()
1412
1413 log.info("Done. OK=%d ERR=%d", total_ok, total_err)
1414 ```
1415
1416 - [ ] **Step 2: 创建 `run_etl.py`**
1417
1418 ```python
1419 #!/usr/bin/env python3
1420 import argparse
1421 from etl_to_crawler.config import PLATFORM_QQ, PLATFORM_KUGOU, PLATFORM_NETEASE, PLATFORMS
1422 from etl_to_crawler.runner import run
1423
1424 PLATFORM_MAP = {
1425 'qq': PLATFORM_QQ,
1426 'kugou': PLATFORM_KUGOU,
1427 'netease': PLATFORM_NETEASE,
1428 }
1429
1430 if __name__ == '__main__':
1431 parser = argparse.ArgumentParser(description='ETL: hk_songs_test → crawler_dev')
1432 parser.add_argument('--platform', default='all',
1433 choices=['qq', 'kugou', 'netease', 'all'],
1434 help='要导入的平台（默认 all）')
1435 args = parser.parse_args()
1436
1437 if args.platform == 'all':
1438 platforms = PLATFORMS
1439 else:
1440 platforms = [PLATFORM_MAP[args.platform]]
1441
1442 print(f"Starting ETL for platforms: {platforms}")
1443 run(platforms)
1444 ```
1445
1446 - [ ] **Step 3: 冒烟测试——只跑前 10 条**
1447
1448 临时修改 config.py 中 `BATCH_SIZE = 10`，再运行：
1449
1450 ```bash
1451 source .venv/bin/activate
1452 python run_etl.py --platform qq
1453 ```
1454
1455 检查：
1456 ```bash
1457 psql "postgresql://data_test:z1n_z%23JHW0@data-center.rwlb.rds.aliyuncs.com/crawler_dev?sslmode=prefer" \
1458 -c "SELECT count(*) FROM crawler_qqmusic_songs;" \
1459 -c "SELECT count(*) FROM crawler_qqmusic_singers;" \
1460 -c "SELECT count(*) FROM crawler_qqmusic_albums;"
1461 ```
1462
1463 期望：三张表均有 > 0 的行数，且无报错。
1464
1465 恢复 `BATCH_SIZE = 500`。
1466
1467 - [ ] **Step 4: 全量运行三平台**
1468
1469 ```bash
1470 source .venv/bin/activate
1471 python run_etl.py --platform all 2>&1 | tee etl_run.log
1472 ```
1473
1474 完成后检查：
1475 ```bash
1476 psql "postgresql://data_test:z1n_z%23JHW0@data-center.rwlb.rds.aliyuncs.com/crawler_dev?sslmode=prefer" -c "
1477 SELECT 'qqmusic_songs' AS tbl, count(*) FROM crawler_qqmusic_songs
1478 UNION ALL SELECT 'qqmusic_singers', count(*) FROM crawler_qqmusic_singers
1479 UNION ALL SELECT 'qqmusic_albums', count(*) FROM crawler_qqmusic_albums
1480 UNION ALL SELECT 'kugou_songs', count(*) FROM crawler_kugou_songs
1481 UNION ALL SELECT 'kugou_singers', count(*) FROM crawler_kugou_singers
1482 UNION ALL SELECT 'kugou_albums', count(*) FROM crawler_kugou_albums
1483 UNION ALL SELECT 'netease_songs', count(*) FROM crawler_netease_songs
1484 UNION ALL SELECT 'netease_singers', count(*) FROM crawler_netease_singers
1485 UNION ALL SELECT 'netease_albums', count(*) FROM crawler_netease_albums;
1486 "
1487 ```
1488
1489 期望：9 张主表均有数据。
1490
1491 - [ ] **Step 5: Commit**
1492
1493 ```bash
1494 git add etl_to_crawler/runner.py run_etl.py
1495 git commit -m "feat(etl): 实现批次编排与 CLI 入口，完成全量导入"
1496 ```
1 # 数据流文档：hk_songs_test → crawler_dev（三平台）
2
3 ---
4
5 ## 一、数据库说明
6
7 | 数据库 | 连接 | 用途 |
8 |---|---|---|
9 | `new_music_library` | TencentDB (port 60209) | 包含 `hk_songs_test`（源数据） |
10 | `hikoon-data` | AliCloud RDS (port 3307) | 包含平台关联表 `hk_song_platform`、`hk_song_and_record`、`hk_music_record` 等 |
11 | `hikoon-data-spider` | 同上 AliCloud RDS (port 3307) | 包含各平台爬虫原始表 `media_tencent_songs` 等 |
12 | `crawler_dev` | AliCloud RDS PostgreSQL | 目标写入库 |
13
14 ---
15
16 ## 二、关联链路
17
18 ```
19 new_music_library.hk_songs_test.source_song_id
20 = hikoon-data.hk_song_platform.id
21 ↓ (hk_song_and_record.song_id = hk_song_platform.id)
22 hikoon-data.hk_song_and_record
23 ↓ (record_id = hk_music_record.id)
24 hikoon-data.hk_music_record（一首歌对应多条，每个平台/版本各一条）
25 ├── platform (1=QQ, 2=Kugou, 4=Netease)
26 ├── platform_unique_key
27 │ ├─ QQ → media_tencent_songs.mid（字符串 mid）
28 │ ├─ Kugou → media_ku_gou_songs.id（数字 ID）
29 │ └─ Netease → media_netease_songs.id（数字 ID）
30 └── platform_mid
31 ├─ QQ → media_tencent_songs.id（数字 ID）
32 └─ Kugou → media_ku_gou_songs.hid（hash 字符串）
33 ```
34
35 > **重要**：`hk_songs_test.source_table_name = 'hk_song_platform'`，`source_song_id` 即为 `hk_song_platform.id`，是从 hk_songs_test 回溯到 hikoon-data 的唯一关联键。
36 >
37 > **一对多**：一条 hk_songs_test 记录可能在 QQ、酷狗、网易都有对应的 `hk_music_record`，每个平台分别写入各自的 `crawler_xx_songs` 表。
38
39 ---
40
41 ## 三、QQ音乐（platform = 1）
42
43 **Spider 核心关联**：`hk_music_record.platform_unique_key` = `media_tencent_songs.mid`
44
45 ### crawler_dev.crawler_qqmusic_songs
46
47 | 目标字段 | 来源库 | 来源表.字段 | 备注 |
48 |---|---|---|---|
49 | `id` | 生成 | — | UUID |
50 | `platform_song_id` | hikoon-data | `hk_music_record.platform_mid` | QQ 数字歌曲 ID，等同于 `media_tencent_songs.id` |
51 | `mid` | hikoon-data | `hk_music_record.platform_unique_key` | QQ mid 字符串，等同于 `media_tencent_songs.mid` |
52 | `title` / `name` | new_music_library | `hk_songs_test.name` | |
53 | `cover` | new_music_library → OSS | `hk_songs_test.cover_url` | 转换至 archive-dev OSS |
54 | `duration` | hikoon-data-spider | `media_tencent_songs.duration` | hk_songs_test 无此字段 |
55 | `lyric` | hikoon-data-spider | `media_tencent_songs.lyric` | 去掉 `[mm:ss.xx]` 时间戳后存储 |
56 | `composer_name` | hikoon-data-spider | `media_tencent_songs.composer_name` | 备选 `hk_songs_test.composer` |
57 | `lyricist_name` | hikoon-data-spider | `media_tencent_songs.lyricist_name` | 备选 `hk_songs_test.lyricist` |
58 | `url` | new_music_library → OSS | `hk_songs_test.audio_url` | 统一转换至 archive-dev OSS |
59 | `lyric_url` | new_music_library | `hk_songs_test.lyrics_url` | 已是 archive-dev OSS .txt，直接使用 |
60 | `published_at` | hikoon-data-spider | `media_tencent_songs.published_at` | 备选 `hk_songs_test.issue_time` |
61 | `platform_index_url` | hikoon-data-spider | `media_tencent_songs.platform_index_url` | |
62 | `album_id` | 关联后写入 | 见下方 albums 说明 | FK → `crawler_qqmusic_albums.id` |
63 | `singers` | 关联后组装 | 见下方 singers 说明 | JSONB 格式 |
64 | `album` | hikoon-data-spider | `media_tencent_albums.*` 组成 JSON | 冗余字段 |
65
66 ### crawler_dev.crawler_qqmusic_singers
67
68 **关联路径**：`media_tencent_songs.id`（by mid）→ `media_tencent_singer_has_songs.song_id` → `media_tencent_singers`（ON singer_id）
69
70 | 目标字段 | 来源表.字段 |
71 |---|---|
72 | `id` | `media_tencent_singers.id` |
73 | `mid` | `media_tencent_singers.mid` |
74 | `name` | `media_tencent_singers.name` |
75 | `avatar` | `media_tencent_singers.avatar` → OSS |
76 | `sex` | `media_tencent_singers.sex` |
77 | `area` | `media_tencent_singers.area` |
78 | `index` | `media_tencent_singers.index` |
79 | `intro` | `media_tencent_singers.intro` |
80 | `home_url` | `media_tencent_singers.home_url` |
81
82 **`crawler_qqmusic_songs.singers` JSONB 格式**：
83 ```json
84 [{"name": "歌手名", "singer_id": <media_tencent_singers.id>, "platform_singer_id": "<media_tencent_singers.mid>"}]
85 ```
86
87 ### crawler_dev.crawler_qqmusic_albums
88
89 **关联路径**：`media_tencent_songs.album_id` → `media_tencent_albums.id`
90
91 | 目标字段 | 来源表.字段 |
92 |---|---|
93 | `id` | `media_tencent_albums.id` |
94 | `mid` | `media_tencent_albums.mid` |
95 | `cover` | `media_tencent_albums.cover` → OSS |
96 | `title` | `media_tencent_albums.title` |
97 | `intro` | `media_tencent_albums.intro` |
98 | `type` | `media_tencent_albums.type` |
99 | `company_id` | `media_tencent_albums.company_id` |
100 | `company` | `media_tencent_albums.company` |
101 | `is_owner` | `media_tencent_albums.is_owner` |
102 | `published_at` | `media_tencent_albums.published_at` |
103
104 ---
105
106 ## 四、酷狗（platform = 2）
107
108 **Spider 核心关联**：`hk_music_record.platform_unique_key` = `media_ku_gou_songs.id`
109
110 ### crawler_dev.crawler_kugou_songs
111
112 | 目标字段 | 来源库 | 来源表.字段 | 备注 |
113 |---|---|---|---|
114 | `id` | 生成 | — | UUID |
115 | `platform_song_id` | hikoon-data | `hk_music_record.platform_unique_key` | 酷狗数字歌曲 ID，等同于 `media_ku_gou_songs.id` |
116 | `hash` | hikoon-data | `hk_music_record.platform_mid` | 酷狗 hash 字符串，等同于 `media_ku_gou_songs.hid` |
117 | `album_audio_id` | hikoon-data | `hk_music_record.album_audio_id` | |
118 | `title` / `name` | new_music_library | `hk_songs_test.name` | |
119 | `cover` | new_music_library → OSS | `hk_songs_test.cover_url` | 转换至 archive-dev OSS |
120 | `duration` | hikoon-data-spider | `media_ku_gou_songs.duration` | |
121 | `lyric` | hikoon-data-spider | `media_ku_gou_songs.lyric` | 去时间戳后存储 |
122 | `composer_name` | hikoon-data-spider | `media_ku_gou_songs.composer_name` | |
123 | `lyricist_name` | hikoon-data-spider | `media_ku_gou_songs.lyricist_name` | |
124 | `url` | new_music_library → OSS | `hk_songs_test.audio_url` | 统一转换至 archive-dev OSS |
125 | `lyric_url` | new_music_library | `hk_songs_test.lyrics_url` | 已是 archive-dev OSS .txt，直接使用 |
126 | `published_at` | hikoon-data-spider | `media_ku_gou_songs.published_at` | |
127 | `platform_index_url` | hikoon-data-spider | `media_ku_gou_songs.platform_index_url` | |
128 | `album_id` | 关联后写入 | 见 albums | FK → `crawler_kugou_albums.id` |
129 | `singers` | 关联后组装 | 见 singers | JSONB |
130
131 ### crawler_dev.crawler_kugou_singers
132
133 **关联路径**：`media_ku_gou_songs.id` → `media_ku_gou_singer_has_songs.song_id` → `media_ku_gou_singers`（ON singer_id）
134
135 | 目标字段 | 来源表.字段 |
136 |---|---|
137 | `id` | `media_ku_gou_singers.id` |
138 | `name` | `media_ku_gou_singers.name` |
139 | `avatar` | `media_ku_gou_singers.avatar` → OSS |
140 | `sex` | `media_ku_gou_singers.sex` |
141 | `area` | `media_ku_gou_singers.area` |
142 | `index` | `media_ku_gou_singers.index` |
143 | `intro` | `media_ku_gou_singers.intro` |
144 | `home_url` | `media_ku_gou_singers.home_url` |
145
146 **`crawler_kugou_songs.singers` JSONB 格式**：
147 ```json
148 [{"name": "歌手名", "singer_id": <media_ku_gou_singers.id>, "platform_singer_id": "<id字符串>"}]
149 ```
150
151 ### crawler_dev.crawler_kugou_albums
152
153 **关联路径**：`media_ku_gou_songs.album_id` → `media_ku_gou_albums.id`
154
155 | 目标字段 | 来源表.字段 |
156 |---|---|
157 | `id` | `media_ku_gou_albums.id` |
158 | `cover` | `media_ku_gou_albums.cover` → OSS |
159 | `title` | `media_ku_gou_albums.title` |
160 | `intro` | `media_ku_gou_albums.intro` |
161 | `type` | `media_ku_gou_albums.type` |
162 | `company_id` | `media_ku_gou_albums.company_id` |
163 | `company` | `media_ku_gou_albums.company` |
164 | `is_owner` | `media_ku_gou_albums.is_owner` |
165 | `published_at` | `media_ku_gou_albums.published_at` |
166
167 ---
168
169 ## 五、网易云（platform = 4）
170
171 **Spider 核心关联**：`hk_music_record.platform_unique_key` = `media_netease_songs.id`
172
173 ### crawler_dev.crawler_netease_songs
174
175 | 目标字段 | 来源库 | 来源表.字段 | 备注 |
176 |---|---|---|---|
177 | `id` | 生成 | — | UUID |
178 | `platform_song_id` | hikoon-data | `hk_music_record.platform_unique_key` | 网易云数字歌曲 ID，等同于 `media_netease_songs.id` |
179 | `title` / `name` | new_music_library | `hk_songs_test.name` | |
180 | `cover` | new_music_library → OSS | `hk_songs_test.cover_url` | 转换至 archive-dev OSS |
181 | `duration` | hikoon-data-spider | `media_netease_songs.duration` | |
182 | `lyric` | hikoon-data-spider | `media_netease_songs.lyric` | 去时间戳后存储 |
183 | `composer_name` | hikoon-data-spider | `media_netease_songs.composer_name` | |
184 | `lyricist_name` | hikoon-data-spider | `media_netease_songs.lyricist_name` | |
185 | `url` | new_music_library → OSS | `hk_songs_test.audio_url` | 统一转换至 archive-dev OSS |
186 | `lyric_url` | new_music_library | `hk_songs_test.lyrics_url` | 已是 archive-dev OSS .txt，直接使用 |
187 | `published_at` | hikoon-data-spider | `media_netease_songs.published_at` | |
188 | `platform_index_url` | hikoon-data-spider | `media_netease_songs.platform_index_url` | |
189 | `album_id` | 关联后写入 | 见 albums | FK → `crawler_netease_albums.id` |
190 | `singers` | 关联后组装 | 见 singers | JSONB |
191
192 ### crawler_dev.crawler_netease_singers
193
194 **关联路径**：`media_netease_songs.id` → `media_netease_singer_has_songs.song_id` → `media_netease_singers`（ON singer_id）
195
196 | 目标字段 | 来源表.字段 |
197 |---|---|
198 | `id` | `media_netease_singers.id` |
199 | `name` | `media_netease_singers.name` |
200 | `avatar` | `media_netease_singers.avatar` → OSS |
201 | `sex` | `media_netease_singers.sex` |
202 | `area` | `media_netease_singers.area` |
203 | `index` | `media_netease_singers.index` |
204 | `intro` | `media_netease_singers.intro` |
205 | `home_url` | `media_netease_singers.home_url` |
206
207 **`crawler_netease_songs.singers` JSONB 格式**：
208 ```json
209 [{"name": "歌手名", "singer_id": <media_netease_singers.id>, "platform_singer_id": "<id字符串>"}]
210 ```
211
212 ### crawler_dev.crawler_netease_albums
213
214 **关联路径**：`media_netease_songs.album_id` → `media_netease_albums.id`
215
216 | 目标字段 | 来源表.字段 |
217 |---|---|
218 | `id` | `media_netease_albums.id` |
219 | `cover` | `media_netease_albums.cover` → OSS |
220 | `title` | `media_netease_albums.title` |
221 | `intro` | `media_netease_albums.intro` |
222 | `type` | `media_netease_albums.type` |
223 | `company_id` | `media_netease_albums.company_id` |
224 | `company` | `media_netease_albums.company` |
225 | `is_owner` | `media_netease_albums.is_owner` |
226 | `published_at` | `media_netease_albums.published_at` |
227
228 ---
229
230 ## 六、入库顺序与过滤条件
231
232 ### 过滤条件
233
234 | 条件 | 实际字段 | 处理 |
235 |---|---|---|
236 | 平台记录存在 | `hk_music_record.platform_unique_key` 不为空 | 关联后若无对应平台记录则跳过 |
237 | 标题不为空 | `hk_songs_test.name` | 空则跳过 |
238 | 音频URL不为空 | `hk_songs_test.audio_url` | 空则跳过；有则统一转换至 archive-dev OSS |
239 | 歌手不为空 | `hk_songs_test.singer` | 文本字段，空则跳过；实际 singers 数据从 spider 关联表获取 |
240 | 歌词 | `hk_songs_test.lyrics_url` | 已是 archive-dev OSS .txt，直接写入 `lyric_url`；spider 的 `lyric` 字段去时间戳后写入 `lyric` 列 |
241
242 ### 写入顺序（每首歌，每个平台）
243
244 1. **singers** — 按 `id` 做 `ON CONFLICT DO NOTHING`
245 2. **albums** — 按 `id` 做 `ON CONFLICT DO NOTHING`
246 3. **songs** — 填入 `album_id`（FK）和 `singers`（JSONB），按 `platform_song_id` 做唯一约束
247 4. **singer_songs** — songs 写入后填充，按 `(singer_id, song_id)` 做 `ON CONFLICT DO NOTHING`
248 5. **singer_albums** — albums 写入后填充，按 `(singer_id, album_id)` 做 `ON CONFLICT DO NOTHING`
249
250 ---
251
252 ## 七、歌手关联表
253
254 ### crawler_xx_singer_songs
255
256 | 目标字段 | 类型 | 来源 |
257 |---|---|---|
258 | `id` | uuid | 生成 |
259 | `singer_id` | int/bigint | `crawler_xx_singers.id`（已写入的歌手主键） |
260 | `song_id` | uuid | `crawler_xx_songs.id`（已写入的歌曲主键） |
261
262 **数据来源**：`media_xx_singer_has_songs`（singer_id, song_id），spider 表的 singer_id / song_id 与 PG 各表主键直接对应。
263
264 | PG 目标表 | Spider 来源表 | 唯一约束 |
265 |---|---|---|
266 | `crawler_qqmusic_singer_songs` | `media_tencent_singer_has_songs` | `(singer_id, song_id)` |
267 | `crawler_kugou_singer_songs` | `media_ku_gou_singer_has_songs` | `(singer_id, song_id)` |
268 | `crawler_netease_singer_songs` | `media_netease_singer_has_songs` | `(singer_id, song_id)` |
269
270 ### crawler_xx_singer_albums
271
272 | 目标字段 | 类型 | 来源 |
273 |---|---|---|
274 | `id` | uuid | 生成 |
275 | `singer_id` | int/bigint | `crawler_xx_singers.id`（已写入的歌手主键） |
276 | `album_id` | bigint | `crawler_xx_albums.id`（已写入的专辑主键） |
277
278 **数据来源**：`media_xx_singer_has_albums`（singer_id, album_id）。
279
280 | PG 目标表 | Spider 来源表 | 唯一约束 |
281 |---|---|---|
282 | `crawler_qqmusic_singer_albums` | `media_tencent_singer_has_albums` | `(singer_id, album_id)` |
283 | `crawler_kugou_singer_albums` | `media_ku_gou_singer_has_albums` | `(singer_id, album_id)` |
284 | `crawler_netease_singer_albums` | `media_netease_singer_has_albums` | `(singer_id, album_id)` |