Problem Being Solved

This page explains the core problem that dokobot-intel-toolkit solves: the daily, manual grind of monitoring thirteen AI and technology platforms for content worth repurposing into revenue-generating posts. Understanding the problem helps you decide which parts of the pipeline matter most for your workflow — whether that is authenticated scraping, incremental comparison, report generation, or the standalone WorkBuddy Skills.

The Manual Monitoring Problem

If you produce content around AI or technology, you already know the routine: open X, scroll Reddit, check Product Hunt, browse V2EX, glance at Bilibili, and repeat — every day, across more platforms than any single person can efficiently cover. By the time you have visited all thirteen sources, your first tab is already stale, and you have no systematic record of what changed since yesterday. The signal you were looking for gets buried in the noise.

The real cost is not the time spent browsing. It is the absence of a structured, comparable record: without a saved baseline from yesterday, you cannot tell whether a topic is newly exploding or just persistently popular. Without that distinction, you cannot prioritize.

Login Walls and Data Access

Many of the highest-signal platforms — X (Twitter), Reddit, Xiaohongshu — require an authenticated session to return useful content. Anonymous scraping either fails outright or returns a degraded feed. Storing your credentials in a script is a security liability and often breaks on bot-detection challenges anyway.

dokobot-intel-toolkit solves this through the dokobot Edge bridge: a Microsoft Edge browser extension paired with the dokobot CLI that relays scraping requests through your existing, already-logged-in browser session. Your credentials never leave your machine, and the scraper inherits exactly the same access you have when you browse manually.

Before any authenticated scrape can run, dokobot_ready.py checks whether the bridge is active and prints either a DEVICE_ID:<uuid> (ready to proceed) or NO_DEVICE (you need to open Edge and activate the extension first). This single gate prevents silent failures deeper in the pipeline.

From Raw HTML to Ranked Signals

Raw scrape output is noisy: inconsistent encodings, duplicated entries, irrelevant metadata, and platform-specific formatting artefacts. The Python parse and clean scripts handle this normalization — fixing encoding issues such as UTF-16-LE output, removing noise, and extracting a consistent set of structured fields (title, popularity score, URL) that downstream steps can reason about uniformly across platforms.

Once the data is clean, the incremental comparison scripts diff today's results against yesterday's saved baseline. Topics that are new since your last run are annotated with 🆕. This cross-day comparison is the step that turns a raw list into an actionable list: instead of reviewing everything, you focus only on what changed.

Monetization as a First-Class Goal

Most monitoring tools stop at "here is what is trending." dokobot-intel-toolkit goes one step further by framing every signal through a specific monetization lens: is this topic hot enough, and positioned correctly, to repurpose into a revenue-generating post on Xiaohongshu (Little Red Book)?

The summary and report generation scripts aggregate ranked signals from all platforms into a single daily intelligence report that surfaces the best opportunities — not just the most-discussed topics. The accompanying xhs-commercial-growth WorkBuddy Skill encodes a structured framework (three currencies: attention, trust, and purchase intent; two systems: virality and monetization; and the concept of commercial distance) that you can apply to evaluate any candidate topic before investing production effort.

Reusable Skills Beyond the Core Pipeline

The pipeline produces the intelligence. The WorkBuddy Skills help you act on it:

• The bili-subtitle Skill extracts a full transcript from any Bilibili video given its BV number — using the official subtitle track when available, or falling back to local faster-whisper speech recognition — so you can quickly assess or repurpose video content without watching it in full.
• The deck-studio Skill turns a topic brief into a complete slide deck outline with visual and content deliverables, compressing the production step that typically follows a content decision.
• The xhs-commercial-growth Skill, described above, gives you a repeatable framework for evaluating monetization potential before committing to a topic.

Each Skill is self-contained and can be enabled independently inside WorkBuddy by placing it in ~/.workbuddy/skills/ — they do not require the rest of the scraping pipeline to be running.

Check whether the Edge bridge is ready before scraping

Always run this first. If the bridge is not active, all authenticated scraping steps will fail silently or return degraded results.

python scripts/dokobot_ready.py

Expected output when ready:

DEVICE_ID:3a7f2c1d-84bb-4e09-a12e-5d6790bc1234

Expected output when the bridge is not active:

NO_DEVICE

If you see NO_DEVICE, open Microsoft Edge and activate the dokobot bridge extension, then re-run the check before proceeding.

Run an authenticated scrape of X (Twitter)

Once the bridge is confirmed ready, run the platform-specific PowerShell scraping script. The example below scrapes X across multiple search threads, serializing requests to avoid rate-limit interruptions.

powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1

The script saves raw output files to disk. No output is expected on stdout for normal operation; errors are suppressed with 2>$null to prevent rate-limit noise from interrupting the run.

Parse a platform feed and compare against yesterday's baseline

After scraping, parse the raw output into structured fields and then diff it against the previous day's saved baseline to surface newly trending topics.

# Extract title, popularity score, and URL from the raw feed
python scripts/parse_linuxdo.py

# Compare today's cleaned data against yesterday's baseline
python scripts/compare_linuxdo.py

Expected output from compare_linuxdo.py (illustrative):

[🆕] Rust-based AI inference engine — score: 482 — https://...
[🆕] Local LLM fine-tuning on consumer GPU — score: 317 — https://...
    Claude context window update — score: 891 — https://...

Topics marked 🆕 are newly present since yesterday's baseline. Unmarked topics were already tracked in the previous run.

• dokobot Edge bridge — How the browser extension and dokobot CLI combine to enable authenticated scraping without storing credentials.
• Incremental comparison — The cross-day diff mechanism that annotates newly trending topics with 🆕 and reduces review volume.
• Monetization signal — What qualifies a trending topic as a strong candidate for Xiaohongshu repurposing, and how the pipeline ranks signals.
• WorkBuddy Skill — How self-contained Skill packages are stored, enabled, and used independently of the core scraping pipeline.
• bili-subtitle Skill — Transcript extraction from Bilibili videos using subtitle tracks or faster-whisper fallback.
• deck-studio Skill — Automated slide deck outline generation from a topic brief.
• xhs-commercial-growth Skill — The commercial-growth framework (attention, trust, purchase intent; virality and monetization systems; commercial distance) for evaluating topic monetizability.