Architecture

This page describes the internal architecture of dokobot-intel-toolkit: how its components are arranged, how data flows from raw platform scrapes through to a structured intelligence report, and why the system is designed the way it is. Understanding this architecture helps you reason about where failures occur, how to extend the pipeline, and how the optional WorkBuddy Skills fit alongside the core scraping workflow.

dokobot_ready.py — CLI readiness probe The mandatory first step in any authenticated scraping run. You invoke this Python script to verify that the dokobot Edge bridge is active. It queries the dokobot CLI for a connected device and prints DEVICE_ID:<uuid> on success or NO_DEVICE if the bridge is unavailable. It also attempts to auto-launch Microsoft Edge and retries up to three times before giving up. Nothing downstream should run if this check fails.

PowerShell Scraping Scripts — CLI data collectors A set of PowerShell scripts (_catch_web.ps1, _catch_web2.ps1, _catch_x2.ps1, _catch_extra.ps1, scrape_0716.ps1) that perform the actual platform fetches. Each script is rate-limit-aware and runs serially to avoid triggering platform throttling. They delegate all authenticated HTTP traffic through the dokobot CLI so that your existing browser login sessions are reused. Raw output is written to local files for the next stage.

dokobot CLI — Authenticated browser automation runtime The bridge between your scraping scripts and login-walled platforms. It pairs with the dokobot Edge browser extension to relay scrape requests through Microsoft Edge, reusing your active logged-in sessions for platforms such as X (Twitter) and Reddit. You never store passwords; the CLI uses the live browser session instead.

Python Parse & Clean Scripts — CLI data normalization layer After raw files are written, these Python scripts (parse_linuxdo.py, clean_0715.py, clean_0716.py, clean2_0715.py, enc_0715.py, urls_0715.py) repair encoding issues (including UTF-16-LE output from dokobot), remove noise, and extract structured fields — title, popularity score, and URL — from each platform's raw format. The output of this stage is consistent, clean records ready for comparison.

Incremental Comparison Scripts — CLI baseline diff engine These scripts (compare_linuxdo.py, compare_linuxdo_07xx.py, compare_0714.py) diff today's cleaned data against yesterday's saved baseline. Topics that are new since the last run are annotated with 🆕. This stage is what surfaces newly trending signals rather than re-surfacing stale ones.

Summary & Report Generation Scripts — CLI report assembler summary_0715.py and gen_linuxdo_0716.py aggregate the annotated signals from all platforms into a single structured daily intelligence report. The report is organized around monetization opportunities — specifically, topics with the highest potential to be repurposed for Xiaohongshu revenue.

bili-subtitle Skill — WorkBuddy Skill library An independently activatable WorkBuddy Skill stored in ~/.workbuddy/skills/. Given a Bilibili video's BV number, it extracts the full transcript using the official subtitle track when available, or falls back to local faster-whisper speech recognition. It operates entirely outside the scraping pipeline and can be enabled on its own.

deck-studio Skill — WorkBuddy Skill library A PPT production agent WorkBuddy Skill (v3.1.0). You provide a topic or brief; the skill recommends a slide style, builds a complete outline, and delivers a content draft or per-slide visual descriptions. It is designed to accelerate turning an intelligence report signal into publishable content.

xhs-commercial-growth Skill — WorkBuddy Skill library A WorkBuddy Skill encoding a structured Xiaohongshu monetization framework: three currencies (attention, trust, purchase intent), two systems (virality and monetization), and the concept of commercial distance. You use it to evaluate how monetizable a given trending topic actually is before investing in content production.

The following trace walks through a full daily intelligence run from bridge check to final report.

Step 1 — Bridge readiness check You run dokobot_ready.py. The script calls the dokobot CLI to list connected devices. If the Edge bridge extension is active, it prints DEVICE_ID:<uuid> and exits with code 0. If not, it attempts to launch Microsoft Edge automatically and retries three times at ten-second intervals. A NO_DEVICE output means you must open Edge and activate the bridge extension before proceeding — all authenticated scraping depends on this step passing.

Step 2 — Authenticated platform scraping With the bridge confirmed ready, you execute the PowerShell scraping scripts. For example, _catch_x2.ps1 runs multiple X (Twitter) search queries serially to avoid rate limiting; _catch_extra.ps1 collects data from Xiaohongshu, Bilibili, and YouTube. Each script sends its requests through the dokobot CLI, which relays them via your active Edge browser session to bypass login walls. Raw output files (.txt, .json) are written to the local filesystem. These files are intentionally excluded from version control due to their size and commercial sensitivity.

Step 3 — Parsing and cleaning Python parse and clean scripts process the raw output files. enc_0715.py corrects encoding issues — particularly UTF-16-LE artifacts produced by dokobot output. parse_linuxdo.py extracts structured fields (title, popularity score, URL) from platform-specific JSON. clean_0716.py and related scripts remove noise and normalize formatting across all platform outputs. The result is a set of clean, consistently structured records.

Step 4 — Incremental comparison against baseline The incremental comparison scripts load today's cleaned records and diff them against yesterday's saved baseline. Any topic absent from yesterday's data is annotated with 🆕. compare_linuxdo.py handles linux.do topics; compare_0714.py performs the cross-platform signal comparison. The output is a focused list of what is newly trending today — not a full re-dump of everything seen.

Step 5 — Report generation summary_0715.py and gen_linuxdo_0716.py aggregate the annotated signals across all platforms into a single daily intelligence report. The report is structured around monetization signals — trending topics ranked by their potential to be repurposed into revenue-generating content on Xiaohongshu.

Optional — WorkBuddy Skills At any point independent of the scraping pipeline, you can enable the bili-subtitle, deck-studio, or xhs-commercial-growth Skills by placing them in ~/.workbuddy/skills/ and activating them inside WorkBuddy. For example, after identifying a trending Bilibili video in the report, you pass its BV number to the bili-subtitle Skill to extract a transcript, then feed that into deck-studio to produce a slide deck, and use xhs-commercial-growth to evaluate the monetization angle before publishing.

Browser-session relay over stored credentials The system delegates all authenticated HTTP traffic through the dokobot Edge bridge rather than storing platform credentials. This means it reuses your live logged-in browser sessions to access X (Twitter), Reddit, and other login-walled platforms. The trade-off is a hard dependency on Microsoft Edge being open and the bridge extension being active, but it eliminates credential storage and management entirely, which significantly reduces security surface area.

PowerShell for scraping, Python for processing Scraping scripts are written in PowerShell because they rely on Windows-native path conventions and must call the dokobot CLI via node using Windows backslash paths — using Git Bash for these calls causes path misresolution (e.g., /e/... being interpreted as c:\e\...). Python handles parsing, cleaning, comparison, and report generation because it is better suited to text processing, encoding correction, and structured data manipulation with the standard library.

Serial scraping with rate-limit awareness Platform scrapes run serially rather than in parallel. This is a deliberate choice to avoid triggering rate limits or anomaly detection on platforms such as X. The cost is total runtime; the benefit is reliability and continued access to login-walled content.

Incremental comparison rather than full daily snapshots The pipeline saves a baseline from each day's run and diffs against it the next day, rather than generating a full snapshot every time. This surfaces newly trending topics (🆕) rather than re-surfacing everything that is trending, which would require you to manually identify what is new. The baseline approach keeps the signal-to-noise ratio high.

WorkBuddy Skills as independent, opt-in packages The three Skills (bili-subtitle, deck-studio, xhs-commercial-growth) are shipped as standalone packages in skills/ with disable: true by default, rather than being integrated into the scraping pipeline. This allows you to adopt them independently of the intelligence pipeline and keeps the pipeline deterministic — it does not depend on WorkBuddy being installed.

Hard dependency on Windows and Microsoft Edge The PowerShell scraping scripts and the dokobot Edge bridge extension require Windows and Microsoft Edge respectively. There is no supported path to run the authenticated scraping stage on macOS or Linux. If you need cross-platform support, you would need to replace the scraping layer entirely.

Edge bridge must be manually activated before each run If dokobot_ready.py outputs NO_DEVICE, the entire authenticated scraping stage is blocked until you open Edge and activate the bridge extension. There is no fully headless or server-side execution mode for the authenticated scraping layer. Automation requiring unattended overnight runs would need a persistent Edge session.

Raw data files are ephemeral and unversioned Raw scrape output (.txt, .json, .err) and daily reports are excluded from version control. They are archived locally only. This means there is no built-in history or audit trail across runs beyond the baseline file retained for the next day's incremental comparison. If you need long-term trend analysis, you must implement your own archiving strategy.

Date-stamped script variants Many scripts carry date suffixes (e.g., _0715, _0716) reflecting day-of-iteration snapshots rather than a single canonical versioned script per function. This means you should prefer the most recent variant when reusing logic, but there is no automated deprecation or consolidation of older variants.

faster-whisper is optional but required for full bili-subtitle coverage The bili-subtitle Skill falls back to local faster-whisper speech recognition only when the optional faster-whisper library is installed. Without it, videos that lack an official Bilibili subtitle track will not produce a transcript. If broad Bilibili coverage is important to your workflow, installing faster-whisper is effectively mandatory.

No built-in deduplication across platforms The pipeline cleans and compares data per platform, but cross-platform deduplication (e.g., the same story appearing on both X and Product Hunt) is handled only by compare_0714.py at the comparison stage rather than at ingestion. If a signal appears on many platforms simultaneously, it may appear multiple times in the report under different platform sections.