osiman/reader
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add async resume jobs and doc navigation

Browse Source
This commit is contained in:
root
2026-04-14 15:55:02 +08:00
parent 6705613aa4
commit b8727f1885
26 changed files with 2791 additions and 665 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/openclaw/openclaw-handoff.md
+172 -263
View File
@@ -1,49 +1,54 @@
# OpenClaw Handoff
## Role
This file is the integration overview for OpenClaw maintainers.
Use it for:
- reader capability boundary
- production MCP entrypoints
- environment requirements
- integration rules and limitations
Do not use it as the step-by-step runbook.
For formal orchestration, read `docs/openclaw/openclaw-orchestration-flow.md`.
For field contracts, read:
- `docs/openclaw/openclaw-candidate-input-field-spec.md`
- `docs/openclaw/openclaw-delivery-payload-spec.md`
Historical plans and incident documents live under `docs/openclaw/archive/`.
## Purpose
This repository provides a FreshRSS-first reading pipeline for OpenClaw:
reader is the upstream FreshRSS processing service for OpenClaw:
`FreshRSS unread items -> RSS content extraction -> LLM summary -> rule engine -> OpenClaw delivery payload`
OpenClaw should treat this repository as an MCP-backed upstream content processor.
This repository is responsible only for upstream reading-pipeline work:
reader is responsible for:
- FreshRSS pull
- content extraction
- LLM summary generation/validation
- LLM summary generation and validation
- rule-based filtering
- OpenClaw delivery payload generation
- selected-article summary capability based on existing extracted text
- run-state persistence, status lookup, result lookup, and minimal resume for the FreshRSS workflow
- run-state persistence and run/result lookup
- async resume control for the FreshRSS workflow
- async selected-article summary generation from existing extracted files
This repository should **not** take over downstream orchestration responsibilities that belong to OpenClaw / skills, such as:
reader is not responsible for:
- Hugo publishing
- chat reporting
- user confirmation handling
- IMA upload orchestration
## Production Entrypoint
## Production Surface
reader 当前正式工作流服务启动入口是 MCP tool:
Current MCP tool count: 21.
- `start_freshrss_pipeline_job`
OpenClaw 应先拿到 `job_id`，轮询 job 状态，再在成功后读取 `run_id` 作为正式后续句柄。
`run_freshrss_openclaw_pipeline` 仍保留，但定位是同步 debug / fallback 路径，而不是正式生产启动入口。
OpenClaw should treat the returned `run_id` from `get_freshrss_pipeline_job_result` as the only stable handle for follow-up reads. Do not hand-build `outputs/freshrss/rerun/...` paths in OpenClaw.
Job state is written under `outputs/freshrss/pipeline_jobs/<job_id>/` and will minimally contain `run-state.json`, `input.json`, `result.json` on success, and `job-report.json`.
## Supported MCP Tools
Current MCP tools: 17 total, including the FreshRSS workflow set plus async job tools for both the main pipeline and article-summary flow.
Workflow service tools:
Main daily workflow:
- `start_freshrss_pipeline_job`
- `get_freshrss_pipeline_job_status`
@@ -53,41 +58,139 @@ Workflow service tools:
- `list_run_artifacts`
- `get_delivery_payload`
- `get_run_report`
Resume workflow:
- `inspect_resume_plan`
- `start_resume_job`
- `get_resume_job_status`
- `get_resume_job_result`
- `resume_run`
Article-summary tools:
Selected-article summary workflow:
- `start_article_summary_job`
- `get_article_summary_job_status`
- `get_article_summary_job_result`
- `generate_article_summaries`
Single-step / debug tools:
Debug / single-step tools:
- `run_freshrss_openclaw_pipeline`（同步模式，仅适合 debug / fallback）
- `run_freshrss_openclaw_pipeline`
- `extract_url_content`
- `extract_item_content`
- `filter_summary_result`
- `generate_article_summaries`（同步模式，仅适合轻量调试）
## Recommended Selected-Article Flow
Production rules:
For OpenClaw selected-article follow-up, prefer the async job path:
- main production start path is `start_freshrss_pipeline_job`
- production resume path is `inspect_resume_plan -> start_resume_job -> get_resume_job_status -> get_resume_job_result`
- `run_freshrss_openclaw_pipeline` is sync debug / fallback only
- `resume_run` is sync debug / fallback only
- `generate_article_summaries` is sync debug / fallback only
1. Call `start_article_summary_job` with a real extracted file path plus a non-empty `selected_ids` list.
2. Poll `get_article_summary_job_status` until `status` becomes `success` or `failed`.
3. On success, call `get_article_summary_job_result` and continue downstream processing from `written_paths`.
4. Use `generate_article_summaries` only as a synchronous debug fallback, not as the default production path.
## Production Contract
Job state is written under `outputs/freshrss/article_summary_jobs/<job_id>/` and will minimally contain:
OpenClaw should treat the returned `run_id` from `get_freshrss_pipeline_job_result` as the only stable handle for follow-up reads.
OpenClaw should not hand-build these paths:
- `outputs/freshrss/rerun/<run_dir>/run-state.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/openclaw-delivery-payload.json`
- `outputs/freshrss/rerun/<run_dir>/run-report.json`
If filesystem access is needed for debugging, only consume paths returned by MCP:
- `output_dir`
- `artifact.path`
- `delivery_output`
- `report_output`
Top-level `status` is the only status field callers should branch on.
`status_source` and `state_conflict` are explanatory fields for reconciled status.
## Minimal Production Sequence
Daily workflow:
1. Call `start_freshrss_pipeline_job`
2. Poll `get_freshrss_pipeline_job_status`
3. On success, read `get_freshrss_pipeline_job_result`
4. Persist the returned `run_id`
5. Use `get_run_status`, `get_delivery_payload`, and `get_run_report` for follow-up reads
Resume workflow:
1. Call `inspect_resume_plan(run_id)`
2. Only if `can_resume=true` and `recommended_action=resume`, call `start_resume_job`
3. Poll `get_resume_job_status`
4. Read `get_resume_job_result`
Selected-article summary workflow:
1. Call `start_article_summary_job` with a real extracted file path and non-empty `selected_ids`
2. Poll `get_article_summary_job_status`
3. Read `get_article_summary_job_result`
## Capability Boundary
Formal workflow boundary:
- only workflow `freshrss_daily_digest`
- every current FreshRSS run writes `run-state.json`
- `get_run_status` / `list_runs` / `list_run_artifacts` can still infer basic state for older runs without `run-state.json`
- resume requires a valid `run-state.json`; inferred historical runs are not resumable
Resume boundary:
- resume in place on the original `run_id`
- supported resume points:
- `generate_summaries`
- `apply_filters`
- `build_delivery_payload`
- `write_run_report`
- unsupported resume points:
- `fetch_feed`
- `extract_articles`
- production resume prefers:
- `summary/summary-batch.json`
- `candidates/candidate-batch.json`
- if required artifacts are missing, recovery should return non-resumable instead of silently falling back
Selected-article summary boundary:
- uses existing extracted files as input
- should not re-fetch original URLs
## Output Expectations
Main daily pipeline core artifacts:
- `outputs/freshrss/rerun/<run_dir>/run-state.json`
- `outputs/freshrss/rerun/<run_dir>/raw/freshrss.raw.json`
- `outputs/freshrss/rerun/<run_dir>/summary/summary-batch.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/candidate-batch.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/openclaw-delivery-payload.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/digest-brief.json`
- `outputs/freshrss/rerun/<run_dir>/run-report.json`
- `outputs/freshrss/rerun/<run_dir>/extracted/item-XX.extracted.json`
Async job state directories:
- main pipeline job: `outputs/freshrss/pipeline_jobs/<job_id>/`
- resume job: `outputs/freshrss/resume_jobs/<job_id>/`
- article-summary job: `outputs/freshrss/article_summary_jobs/<job_id>/`
Each job directory minimally contains:
- `run-state.json`
- `input.json`
- `result.json` on success
- `job-report.json`
## Required Environment Variables
## Environment And Startup
The MCP server process must have these variables available:
Required environment variables:
- `FRESHRSS_API_BASE_URL`
- `FRESHRSS_USERNAME`
@@ -96,54 +199,13 @@ The MCP server process must have these variables available:
- `LLM_API_KEY`
- `LLM_MODEL`
Example:
```powershell
set FRESHRSS_API_BASE_URL=http://127.0.0.1:8081/api/greader.php
set FRESHRSS_USERNAME=osiman
set FRESHRSS_API_PASSWORD=your-api-password
set LLM_API_URL=https://api.deepseek.com
set LLM_API_KEY=your-llm-api-key
set LLM_MODEL=deepseek-chat
```
## Server Startup
Install dependencies:
Startup:
```bash
pip install -e .
```
Start the MCP server:
```bash
summary-mcp
```
## Recommended MCP Workflow
Recommended production path:
1. Call `start_freshrss_pipeline_job` and persist the returned `job_id`
2. Poll `get_freshrss_pipeline_job_status(job_id)` until `status` becomes `success` or `failed`
3. On success, call `get_freshrss_pipeline_job_result(job_id)` and persist the returned `run_id`
4. Use `get_run_status(run_id)` as the authoritative run-state read for status, stage, artifacts, and recovery
5. Use `list_runs(...)` when OpenClaw needs recent-run discovery or high-level inspection
6. Use `list_run_artifacts(run_id)` when OpenClaw needs to inspect what this run actually produced
7. Use `get_delivery_payload(run_id)` and `get_run_report(run_id)` as the formal result-reading APIs
8. Use `resume_run(run_id)` only when the run falls inside the minimal supported resume scope
OpenClaw should not directly derive or hardcode:
- `outputs/freshrss/rerun/<run_dir>/run-state.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/openclaw-delivery-payload.json`
- `outputs/freshrss/rerun/<run_dir>/run-report.json`
If filesystem access is needed for debugging, consume only paths returned by MCP such as `output_dir`, `artifact.path`, `delivery_output`, or `report_output`.
## Recommended MCP Call
Recommended production start call:
```json
@@ -157,204 +219,51 @@ Recommended production start call:
}
```
Recommended semantics:
## Data And Content Policy
- Use `mark_read=true` for normal production runs.
- Use `mark_read=false` only for debug, test, or validation runs.
- Keep `debug_artifacts=false` for routine production runs.
- Set `debug_artifacts=true` only when troubleshooting a bad batch.
- Treat the returned `job_id` as the startup handle, and the later `run_id` from `get_freshrss_pipeline_job_result` as the stable identifier for all follow-up run reads.
- If no real `openclaw-delivery-payload.json` was produced, OpenClaw should stop instead of generating a digest from placeholders or examples.
- Use `run_freshrss_openclaw_pipeline` only when a synchronous debug / fallback path is explicitly needed.
## Formal Capability Boundary
reader 当前正式 MCP workflow service 的边界如下：
- formal workflow: only `freshrss_daily_digest`
- run truth: every FreshRSS run writes `run-state.json`
- main production start path: `start_freshrss_pipeline_job` / `get_freshrss_pipeline_job_status` / `get_freshrss_pipeline_job_result`
- state query tools: `get_run_status`, `list_runs`, `list_run_artifacts`
- result read tools: `get_delivery_payload`, `get_run_report`
- `digest-brief.json` is generated and registered as an artifact, but there is no standalone `get_digest_brief` tool yet
- `run_freshrss_openclaw_pipeline` is still supported, but only as a synchronous debug / fallback path
- the FreshRSS daily workflow now has a minimal background job model backed by a detached runner process, not a full queue / worker system
- article summary now has a minimal asynchronous job model with `start_article_summary_job` / `get_article_summary_job_status` / `get_article_summary_job_result`
- `generate_article_summaries` is still supported, but it is a synchronous debug path and outside the formal `resume_run` scope
Historical compatibility note:
- `get_run_status` / `list_runs` / `list_run_artifacts` can still infer basic state for older run directories without `run-state.json`
- `resume_run` does **not** support those inferred historical runs; it requires a valid `run-state.json`
## What The Async Job Returns
Primary return fields from `get_freshrss_pipeline_job_result`:
- `job_id`
- `run_id`
- `output_dir`
- `raw_output`
- `delivery_output`
- `report_output`
- `digest_brief_output`
- `pulled_count`
- `delivered_count`
- `marked_read_count`
- `status_counts`
- `keyword_index`
Optional:
- `items`
- returned only when `include_item_reports=true`
`run_freshrss_openclaw_pipeline` still returns the same synchronous payload for debug / fallback use.
Follow-up structured reads should use MCP tools rather than re-reading these files directly.
## Minimal Output Files
By default the pipeline writes these core artifacts:
- `outputs/freshrss/rerun/<run_dir>/run-state.json`
- `outputs/freshrss/rerun/<run_dir>/raw/freshrss.raw.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/openclaw-delivery-payload.json`
- `outputs/freshrss/rerun/<run_dir>/candidates/digest-brief.json`
- `outputs/freshrss/rerun/<run_dir>/run-report.json`
- `outputs/freshrss/rerun/<run_dir>/extracted/item-XX.extracted.json` (one per item)
It also updates local runtime keyword data:
- `data/term_index/daily/YYYY-MM-DD.json`
- `data/term_index/term_stats.json`
Per-item extracted files live under `extracted/` and are always written.
If `debug_artifacts=true`, the pipeline additionally writes normalized items, summaries, filter decisions, candidate records, and candidate inputs.
The main pipeline does not emit a batch-level `freshrss.extracted.json` file by default.
## `resume_run` Minimal Scope
`resume_run` currently supports only the minimum resume contract:
- only runs with a valid `run-state.json`
- only workflow `freshrss_daily_digest`
- resume in place on the original `run_id`
- supported resume points: `generate_summaries`, `apply_filters`, `build_delivery_payload`, `write_run_report`
- unsupported resume points: `fetch_feed`, `extract_articles`
- if required artifacts are missing, the tool returns a non-resumable response instead of silently falling back to an earlier stage
Artifact expectations by resume point:
- `generate_summaries`: requires `raw/freshrss.raw.json` and `extracted/`
- `apply_filters`: requires the above plus per-item summary outputs
- `build_delivery_payload`: requires per-item candidate inputs consistent with filter-stage output
- `write_run_report`: requires `candidates/openclaw-delivery-payload.json`; if `mark_read=true`, raw input must still be present
## Payload Specs
OpenClaw payload field specs live here:
- `docs/openclaw/openclaw-candidate-input-field-spec.md`
- `docs/openclaw/openclaw-delivery-payload-spec.md`
## Read-State Semantics
The pipeline reads from FreshRSS unread items by default.
If `mark_read=true`:
- items are marked as read only after the final `openclaw-delivery-payload.json` has been written successfully
- only successfully delivered items are marked as read
- failed or skipped items remain unread
## FreshRSS Content Policy
For FreshRSS items, the pipeline is RSS-first and does not re-crawl webpages.
Behavior:
FreshRSS processing is RSS-first:
- use `item.raw_content` first
- if missing, use `item.raw_summary`
- if neither contains usable content, skip the item
- do not fetch the original webpage again for FreshRSS items
This is intentional.
Read-state policy:
## Keyword Cleanup Governance
- items are marked read only after successful delivery payload write
- only successfully delivered items are marked read
This repository also includes a lightweight keyword-governance flow for downstream review.
Downstream boundary:
Current pieces:
- runtime keyword stats
- `data/term_index/daily/YYYY-MM-DD.json`
- `data/term_index/term_stats.json`
- governance config
- `configs/term_cleanup_policy.json`
- `configs/term_watchlist.json`
- `configs/term_change_log.json`
- review bundle builder
- `skills/keyword-cleanup-review/scripts/build_review_bundle.py`
- accepted-suggestion writer
- `scripts/apply_term_suggestions.py`
Current status:
- OpenClaw can read the keyword review bundle as maintenance input
- accepted suggestions still require explicit human confirmation
- the repository can write accepted watch / alias / stopword / interest-keyword changes after confirmation
- this governance flow is not yet wired into a periodic scheduler inside the repository
Boundary:
- keyword cleanup is a maintenance flow, not the production RSS ingestion path
- the repository does not auto-apply cleanup suggestions without confirmation
- current keyword stats are built from the delivered candidate payload, not yet from a final `DailyDigest`
## Known Limitations
- Some sources put only partial content in RSS; those items may be skipped if RSS content is insufficient.
- WeChat articles often block direct crawling, but this pipeline now avoids that path for FreshRSS items and uses RSS-provided content when available.
- Rule behavior is still conservative in some cases; many items may land in `review` depending on current rules.
- Paywall heuristics may produce false positives for some Chinese text patterns.
- Keyword cleanup governance is usable now, but periodic scheduling and before/after evaluation are not implemented yet.
## Files OpenClaw Should Read First
Recommended reading order for a new maintainer:
1. `README.md`
2. `docs/openclaw/openclaw-handoff.md`
3. `docs/openclaw/openclaw-candidate-input-field-spec.md`
4. `docs/openclaw/openclaw-delivery-payload-spec.md`
5. `docs/design/daily-keyword-index-design.md`
6. `skills/keyword-cleanup-review/SKILL.md`
7. `docs/current/context-reset-brief.md`
## Downstream Boundary Rules
For the daily-digest workflow:
- the digest should go to Hugo and chat reporting, not directly into IMA
- the full daily digest should **not** be uploaded to IMA
- the daily digest goes to Hugo and chat reporting
- the full daily digest should not be uploaded to IMA
- only explicitly user-selected article summaries should be uploaded to IMA
- selected-article summaries should be generated from existing extracted text, not by re-fetching original URLs
## Current Recommendation
## Related Maintenance Flow
For integration handoff, the repository is usable now.
Keyword cleanup exists as a separate maintenance flow, not the main RSS ingestion path.
The minimum you need to give OpenClaw is:
- the repository code
- the MCP server startup command
- the required environment variables in the target environment
- the instruction to call `run_freshrss_openclaw_pipeline`
- the rule that follow-up state/result reads must go through MCP tools first, not handwritten filesystem paths
If OpenClaw will also participate in keyword-governance review, additionally point it to:
Relevant files:
- `docs/design/daily-keyword-index-design.md`
- `skills/keyword-cleanup-review/SKILL.md`
- `scripts/apply_term_suggestions.py`
## Known Limitations
- some sources expose only partial RSS content; those items may be skipped
- rule behavior is still conservative; many items may land in `review`
- paywall heuristics may still produce false positives on some Chinese text
- keyword cleanup governance is usable but not yet wired to periodic scheduling
## Read First
Recommended reading order for a new maintainer:
1. `README.md`
2. `docs/openclaw/README.md`
3. `docs/openclaw/openclaw-handoff.md`
4. `docs/openclaw/openclaw-orchestration-flow.md`
5. `docs/openclaw/openclaw-candidate-input-field-spec.md`
6. `docs/openclaw/openclaw-delivery-payload-spec.md`
7. `docs/current/context-reset-brief.md`