102 lines
2.9 KiB
Markdown
102 lines
2.9 KiB
Markdown
---
|
|
name: llm-summary-review
|
|
description: Validate and refine LLM-generated summary JSON for extracted articles in this repository. Use when the user wants to review, validate, repair, or iterate on `outputs/result.json` or similar summary outputs produced from extracted article JSON.
|
|
---
|
|
|
|
# LLM Summary Review
|
|
|
|
Use this skill when working on the repository's summary loop after article extraction is done.
|
|
|
|
## What This Skill Does
|
|
|
|
- Verifies that an LLM summary result matches the expected JSON contract
|
|
- Reuses the repository validator instead of re-checking fields manually
|
|
- Repairs invalid outputs by telling the LLM exactly what to fix
|
|
- Keeps the workflow aligned with the extraction JSON produced by this project
|
|
|
|
## Inputs
|
|
|
|
Typical files:
|
|
|
|
- Extracted article JSON: `outputs/*.extracted.json`
|
|
- LLM summary result JSON: `outputs/result.json`
|
|
- Prompt template: `outputs/llm-summary-prompt.txt`
|
|
|
|
## Workflow
|
|
|
|
1. Validate the current summary result with the repository validator:
|
|
|
|
```bash
|
|
python -m summary_mcp.validate_llm_result outputs/result.json --extracted outputs/read-flow-2026.extracted.json
|
|
```
|
|
|
|
2. If validation passes:
|
|
- Report that the result is structurally valid
|
|
- Briefly note any warnings
|
|
- Do not rewrite the result unless the user asks
|
|
|
|
3. If validation fails:
|
|
- Read the validator errors carefully
|
|
- Ask the LLM to regenerate or repair only the failing parts
|
|
- Re-run the validator until it passes or a retry limit is hit
|
|
|
|
## Repair Prompt Pattern
|
|
|
|
When asking an LLM to repair a bad result, provide:
|
|
|
|
- The original extracted article JSON
|
|
- The current invalid summary JSON
|
|
- The validator error list
|
|
- A strict instruction to preserve valid fields and fix only the failing ones
|
|
|
|
Use this repair template:
|
|
|
|
```text
|
|
请修复下面这份不符合要求的摘要 JSON。
|
|
|
|
要求：
|
|
- 只输出合法 JSON
|
|
- 保留已经正确的字段
|
|
- 只修复 validator 报出的错误
|
|
- 不要补充解释
|
|
|
|
validator errors:
|
|
{{errors}}
|
|
|
|
原始提取结果:
|
|
{{extracted_json}}
|
|
|
|
当前摘要结果:
|
|
{{result_json}}
|
|
```
|
|
|
|
## Validation Rules
|
|
|
|
The validator currently enforces:
|
|
|
|
- Required fields exist
|
|
- Field types are correct
|
|
- `category` is one of: `资讯` `方法论` `工具实践` `观点评论`
|
|
- `summary` length is within bounds
|
|
- `highlights`, `keywords`, and `topics` counts are within bounds
|
|
- `keywords` and `topics` do not overlap
|
|
- `title` and `url` match the extracted article when an extracted JSON file is provided
|
|
|
|
## Repository Implementation
|
|
|
|
Relevant code:
|
|
|
|
- Validator model: `src/summary_mcp/models/llm_result.py`
|
|
- Validator logic: `src/summary_mcp/validators/llm_result.py`
|
|
- CLI entry: `src/summary_mcp/validate_llm_result.py`
|
|
|
|
Prefer using the existing validator rather than recreating checks in free-form reasoning.
|
|
|
|
## When To Stop
|
|
|
|
Stop when one of these is true:
|
|
|
|
- The validator returns `valid: true`
|
|
- The user asks to inspect the remaining failures manually
|
|
- Repeated retries fail and the user should decide how to proceed
|