Feature: Promise Tracker & Daily Webcrawl (The Talk Ledger)

File naming: feat-promise-tracker.md

---

1. Overview

---

2. Functional Requirements

2.1 User Stories / Use Cases

2.2 Acceptance Criteria

• AC-01: The Talk Ledger feed loads showing quote/summary, source, date, and Walk-o-Meter score per card.
• AC-02: Search and filter (region, politician, year, status) return the correct subset of promises.
• AC-03: For long sources (PDFs, long articles), a 3-bullet AI summary is shown: What, When, Budget.
• AC-04: Daily cron runs successfully; new/updated items appear in the feed after processing.
• AC-05: Crawled items are tagged: New Promise, Progress Update, or Fulfillment.
• AC-06: Authenticated users can follow a promise from the card; the button switches to Following, and a second tap confirms unfollow.
• AC-07: Authenticated users can flag a promise; at ≥10 flags an admin alert is triggered.
• AC-08: Authenticated users can comment (max 500 chars); comments with ≥5 flags are auto-hidden.
• AC-09: User-submitted promises enter a pending moderation queue and are shown with a "Under review" banner. Admin approval/rejection UI and submitter notification are deferred to backlog.
• AC-10: Watchdog Commentary is generated at crawl time, stored in DB, and displayed on the card.
• AC-11: Source URLs with 404 status show: "Original source no longer available."

2.3 Business Rules

• BR-01: A promise must have at minimum: quote or summary, source URL, and date. Walk-o-Meter score defaults to 0 until community input exists.
• BR-02: Crawler must not duplicate existing promises. Dedup by source URL + quote fingerprint.
• BR-03: AI summaries are generated only for sources above a length threshold (e.g. PDFs, articles > 500 words). Short quotes are used as-is.
• BR-04: Each user may cast one Like per promise. One follow per promise. One flag per promise (per reason type).
• BR-05: Watchdog Commentary uses a fixed system prompt; output is stored and may be reviewed by an admin before publishing. Max 2 sentences. No personal attacks.
• BR-06: Source URL validation runs weekly. source_status is updated to 404 or paywalled accordingly.
• BR-07: User-submitted promises are attributed "Submitted by community · Verified by WIWOKDETOK team" after approval.
• BR-08: Flag threshold for admin review: ≥10 unique users. Admin can dismiss, edit, or remove.

2.4 Feature Dependencies

---

3. Non-Functional Requirements

3.1 Performance

• Latency: The Talk Ledger FCP < 2s (p95); search/filter response < 1.5s.
• Throughput: Feed and search support concurrent reads; crawl runs once per day.
• Data volume: Pagination at 20 items per page; support 10k+ promises without degradation.

3.2 Availability & Reliability

• Uptime: Aligns with Vercel/Supabase SLA. Feed is read-heavy; no hard dependency on crawl completion for serving.
• Error handling: Failed crawl does not break the feed. Stale data shown with "last updated" indicator. Crawl retries with exponential backoff.

3.3 Security & Privacy

• Auth: Feed and search are public (no login required for read). Write actions (Flag, Comment, Follow, Submit promise) require authentication.
• Data: Source URLs and quotes are public by design. Crawler must respect robots.txt and rate limits. No PII in promise content.
• Compliance: Ethical scraping and attribution.

3.4 Accessibility & UX

• A11y: WCAG 2.1 AA; min tap target 48×48dp.
• Localization: ID primary; designed for future EN.
• Offline / low data: Data-Saver mode reduces image quality; no heavy animations; system font fallback.

3.5 Scalability & Limits

• Crawl: Rate-limit external fetches; use Vercel Cron + queue (e.g. Inngest/QStash) for crawl jobs.
• Storage: Promises and summaries in Supabase; raw crawl cache optional with defined retention.
• Comments: Paginated at 20 per load; flat (non-threaded) for MVP.

---

4. Technical Requirements

4.1 Architecture Context

• Layer: Frontend (Next.js App Router), Server Actions, cron routes, AI (summarization + Watchdog commentary).
• Entry points: /promise-tracker (Talk Ledger UI), / (Command Center linking into the feed), src/app/actions/promises.ts, /api/cron/crawl, /api/cron/validate-urls.

4.2 Feature-Specific Packages & Libraries

4.3 Data Model & APIs

Entities / tables used:

• regions: id, parent_id, level (0–4), name, code
• promises: id, region_id, quote, source_url, source_domain, source_status (active|404|paywalled), crawled_at, source_type, date, category (new_promise|progress_update|fulfillment), walk_o_meter_score, summary_what, summary_when, summary_budget, watchdog_commentary, submitted_by (nullable), created_at, updated_at
• promise_reactions: id, promise_id, user_id, type (like|follow|flag), flag_reason (nullable), created_at
• promise_comments: id, promise_id, user_id, text (max 500), like_count, flag_count, hidden (bool), created_at
• promise_submissions: id, promise_id (nullable, set on approval), submitted_by, quote, source_url, politician_name, date, status (pending|approved|rejected), admin_reason (nullable), created_at
• crawl_runs: id, started_at, finished_at, status, items_processed, error_log

Key APIs / Server Actions / Routes:

• getPromiseFeed({ search, category, region, year, status, page }) — paginated feed server action
• getPromiseFeedFacets() — fetch available region/year filter values for the UI
• reactToPromise(promiseId, type) — like / follow / flag
• commentOnPromise(promiseId, text) — insert comment; enforce 500-char limit
• submitNewPromise(data) — enter moderation queue
• GET /api/cron/crawl — cron trigger; fetch, classify, summarize, generate Watchdog commentary, write to DB
• GET /api/cron/validate-urls — update source_status

External APIs / services:

• News/social sources: whitelist of RSS feeds and scraped URLs (TBD)
• Google Gemini API: summarization and Watchdog commentary generation

4.4 Configuration & Environment

• Env vars: NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, GOOGLE_GEMINI_API_KEY, CRAWL_SOURCES, CRAWL_CRON_SECRET
• Feature flags: FEATURE_PROMISE_SEARCH — advanced search; FEATURE_USER_SUBMIT_PROMISE — user submissions

---

5. Sequence Diagram (Feature & Data Flow)

5.1 User opens The Talk Ledger and searches promises

5.2 Daily crawl and AI summarization

5.3 User follows and unfollows a promise

---

6. Edge Cases & UI States

6.1 Empty States

6.2 Loading States

6.3 Error States

6.4 Permission & Auth States

6.5 Moderation & Submission States

---

7. Open Questions / Decisions

• Q1: Exact list of crawl sources (whitelist of domains/RSS) and rate limits.
• Q2: Deduplication strategy: same quote from multiple URLs — one promise or multiple?
• Q3: Whether to store raw PDFs in Supabase Storage for audit and re-summarization.
• Q4: Watchdog Commentary moderation: auto-publish or admin-review queue before going live?

---

8. Changelog