osiman/reader
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

keyword cleanup: v2 engine, alias rule layer, LLM semantic suggestions

Browse Source 
- build_review_bundle.py: 新增 _compute_percentile/_compute_growth,
  候选池从固定阈值改为百分位排名 + 增速因子 (v2 policy)
- term_cleanup_policy.json: 升级 v2 schema
- generate_term_cleanup_suggestions.py: 新增 _prepare_alias_suggestions,
  规则层输出 alias (大小写/单复数/分词变体)
- generate_term_cleanup_semantic_suggestions.py: 新增 LLM 语义建议脚本
  (DeepSeek API, 产出 semantic alias/stopword/promote)
- SKILL.md: 更新为 5 Phase 工作流程
- 首轮清洗 apply: interest 54, aliases 17组, stopwords 17个
- docs/design/keyword-cleanup-flow-overview.md: 流程文档
- plans/: 引擎设计方案
This commit is contained in:
root
2026-05-14 17:17:49 +08:00
parent 4399c9ca90
commit 590d050218
13 changed files with 1592 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/filter_context.personal.json
+20
View File
@@ -27,13 +27,21 @@
"Agent Skills",
"AgentScope",
"AI Agent",
"AI Coding Agent",
"AliSQL",
"Anthropic",
"Claude",
"Claude Code",
"CLAUDE.md",
"Context Engineering",
"Cursor",
"DeepSeek",
"FastAPI",
"Gin",
"Go",
"gRPC",
"Harness Engineering",
"Hermes Agent",
"Java",
"Kafka",
"Kubernetes",
@@ -49,13 +57,25 @@
"RAG",
"ReActAgent",
"Redis",
"Skill",
"SKILL.md",
"Skills",
"Spring",
"SubAgent",
"TypeScript",
"Vibe Coding",
"Workflow",
"上下文压缩",
"上下文工程",
"上下文管理",
"云原生",
"代码审查",
"可观测性",
"向量数据库",
"多Agent协作",
"大模型",
"微服务",
"渐进式披露",
"知识库"
]
}
configs/term_aliases.json
+15 -1
View File
@@ -1,5 +1,19 @@
{
"AI助手": "AI Agent",
"图文RAG": "RAG",
"Prompt架构": "Prompt Engineering"
"Prompt架构": "Prompt Engineering",
"Agent Skill": "Agent Skills",
"Binlog": "binlog",
"Coding Agent": "AI Coding Agent",
"Subagent": "SubAgent",
"Subagents": "SubAgent",
"vibe coding": "Vibe Coding",
"Agent架构": "AI Agent",
"Agent专业化": "AI Agent",
"Agent Teams": "多Agent协作",
"Agentic Engineering": "AI Agent",
"CLI工具": "CLI",
"AI编程": "AI Coding Agent",
"记忆管理": "上下文管理",
"会话管理": "上下文管理"
}
configs/term_change_log.json
+437
View File
@@ -99,6 +99,443 @@
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-04-08.json",
"suggestion_date": "2026-04-08",
"based_on_days": 7
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Anthropic",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=13, days_seen=10, recent_count=13.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Harness Engineering",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=12, days_seen=11, recent_count=12.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Skill",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=11, days_seen=9, recent_count=11.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "上下文工程",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=6, days_seen=6, recent_count=6.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "多Agent协作",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=6, days_seen=6, recent_count=6.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Claude",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=6, days_seen=5, recent_count=6.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "上下文管理",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=6, days_seen=5, recent_count=6.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "渐进式披露",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=5, days_seen=5, recent_count=5.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Skills",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=5, days_seen=4, recent_count=5.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "SKILL.md",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=4, days_seen=4, recent_count=4.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "CLAUDE.md",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=4, days_seen=4, recent_count=4.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "上下文压缩",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=4, days_seen=4, recent_count=4.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "AI Coding Agent",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=3, days_seen=3, recent_count=3.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Hermes Agent",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=3, days_seen=3, recent_count=3.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Vibe Coding",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=5, days_seen=4, recent_count=5.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Context Engineering",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=3, days_seen=3, recent_count=3.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "Cursor",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=3, days_seen=3, recent_count=3.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T08:20:35.134979Z",
"action": "add_interest_keyword",
"term": "大模型",
"reason": "Meets the configured interest-keyword review threshold and is not yet covered by interest keywords or stopwords. Evidence: total_count=4, days_seen=3, recent_count=4.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_interest_keyword",
"term": "TypeScript",
"reason": "Core language for AI agent development (e.g., Claude Code, Cursor) and backend engineering, complements existing Python/Java/Go keywords.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_interest_keyword",
"term": "代码审查",
"reason": "Chinese term for 'code review', a key practice in backend engineering and AI agent development workflows.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "Channels",
"reason": "Too generic; could refer to communication channels, YouTube channels, or software channels, not specific to user's focus areas.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "Memory",
"reason": "Extremely broad term; could refer to computer memory, human memory, or memory in various contexts, not discriminative enough.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "Prompt",
"reason": "Already covered by 'Prompt Engineering' as a more specific term; 'Prompt' alone is too broad and matches many unrelated articles.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "AGI",
"reason": "Too broad and speculative; not directly actionable for the user's practical engineering focus areas.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "AI日报",
"reason": "Generic news term; not a technical concept or tool, would add noise to the keyword index.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "AIHOT",
"reason": "Unclear meaning, likely a brand or aggregator, not a specific technical term.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "All In Code",
"reason": "Too vague; could refer to a podcast, a philosophy, or a project, not a specific technical concept.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "auto-twitter-campaign",
"reason": "Too specific to a single project/tool, not a general interest keyword for the user's focus areas.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "ChangeSet",
"reason": "Generic term used in version control and databases; too broad to be a useful filter.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "Lumina",
"reason": "Unclear reference; could be a product, framework, or brand, not clearly aligned with user's focus.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "OpenViking",
"reason": "Unclear reference; not a known tool or concept in the user's stated focus areas.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "Seedance 2.0",
"reason": "Unclear reference; likely a product or version, not a general technical term.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_stopword",
"term": "质量门禁",
"reason": "Chinese term for 'quality gate', too generic in software engineering; not specific to user's focus areas.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Agent Skill",
"reason": "Singular variant",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "Agent Skills",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Binlog",
"reason": "Case variant (auto-ranked)",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "binlog",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Coding Agent",
"reason": "Abbreviated form of 'AI Coding Agent', referring to the same concept.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "AI Coding Agent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Subagent",
"reason": "Case variant",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "SubAgent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Subagents",
"reason": "Plural variant",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "SubAgent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "vibe coding",
"reason": "Case variant",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "Vibe Coding",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Agent架构",
"reason": "Chinese translation of 'Agent architecture', a core concept in AI Agent engineering.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "AI Agent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Agent专业化",
"reason": "Chinese term for 'Agent specialization', directly related to Agent engineering.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "AI Agent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Agent Teams",
"reason": "English equivalent of 'Multi-Agent collaboration', same concept.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "多Agent协作",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "Agentic Engineering",
"reason": "Broader term for engineering with AI agents, closely related to Agent engineering focus.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "AI Agent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "CLI工具",
"reason": "Chinese translation of 'CLI tool', same concept.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "CLI",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "AI编程",
"reason": "Chinese term for 'AI programming', closely related to AI Coding Agent.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "AI Coding Agent",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "记忆管理",
"reason": "Chinese term for 'memory management', closely related to context management in LLM applications.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "上下文管理",
"suggestion_date": "2026-05-14",
"based_on_days": 365
},
{
"applied_at": "2026-05-14T09:12:50.749877Z",
"action": "add_alias",
"term": "会话管理",
"reason": "Chinese term for 'session management', related to context management in LLM applications.",
"suggestions_path": "outputs/term_index/review/term-cleanup-suggestions-2026-05-14.json",
"value": "上下文管理",
"suggestion_date": "2026-05-14",
"based_on_days": 365
}
]
}
configs/term_cleanup_policy.json
+11 -10
View File
@@ -1,14 +1,12 @@
{
"schema_version": "v1",
"schema_version": "v2",
"interest_keyword_review": {
"min_total_count": 3,
"min_days_seen": 2
"percentile_max": 0.05,
"growth_promotion": 0.5
},
"watch_term_review": {
"min_total_count": 1,
"min_days_seen": 1,
"max_total_count": 2,
"max_days_seen": 2
"percentile_min": 0.05,
"percentile_max": 0.20
},
"alias_review": {
"min_total_count": 2,
@@ -19,7 +17,10 @@
"max_days_seen": 2
},
"notes": [
"当前阶段采用保守阈值，避免在低样本条件下直接扩充 interest_keywords。",
"watch_terms 先用于观察，后续再决定是否升格为 interest_keywords 或进入 alias/stopword 配置。"
"v2: interest/watch 使用百分位排名 + 增速因子替代固定阈值",
"percentile 越小表示排名越高（top 5% = percentile 0.05）",
"growth = recent_count / total_count，衡量近期活跃度",
"watch_term_review 的 percentile_min 可理解为兴趣边界下限，低于此值的词归入 interest 候选",
"growth_promotion（默认 0.5）用于识别近期集中爆发词，即使排位不高也主动推荐确认"
]
}
}
configs/term_stopwords.json
+13
View File
@@ -1,6 +1,19 @@
[
"AGI",
"AIHOT",
"AI日报",
"All In Code",
"auto-twitter-campaign",
"ChangeSet",
"Channels",
"Lumina",
"Memory",
"OpenViking",
"Prompt",
"Seedance 2.0",
"奋斗文化",
"小银",
"质量门禁",
"银行客户经理",
"飞盘物理"
]
configs/term_watchlist.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"schema_version": "v1",
"updated_at": "2026-04-08T02:34:14.194320Z",
"updated_at": "2026-05-14T09:12:50.749877Z",
"terms": [
{
"term": "A2A",