Getting Started

This page walks you through making your first request with the dokobot-intel-toolkit pipeline — from verifying that your browser bridge is active to running a scrape, cleaning the output, and generating a daily intelligence report. Getting this initial run working confirms that every layer of the stack (Edge bridge, authenticated scraping, parse scripts, and comparison scripts) is wired together correctly, so you can build on it with confidence.

Before you begin, make sure the following are installed and available on your machine:

• Python ≥ 3.11 — required for all parse, clean, comparison, and report scripts (standard library only for most scripts; faster-whisper is optional and only needed for the bili-subtitle Skill's ASR fallback)
• Node.js ≥ 22 — required runtime for the dokobot CLI
• dokobot CLI — the authenticated browser automation agent; must be installed and on your PATH
• Microsoft Edge with the dokobot bridge extension installed and enabled — this is what lets scraping scripts reuse your existing login sessions on platforms like X and Reddit
• Windows OS with PowerShell — the scraping scripts are PowerShell-based and must be run from PowerShell (not Git Bash; see Troubleshooting for path pitfalls)
• WorkBuddy application (optional) — only needed if you want to use the bili-subtitle, deck-studio, or xhs-commercial-growth Skills
• faster-whisper (optional) — only needed as an ASR fallback inside the bili-subtitle Skill when a Bilibili video has no subtitle track

The minimal path to a working first run:

1. Open Microsoft Edge and activate the dokobot bridge extension.
2. Confirm the bridge is ready:
   Copy

   python scripts/dokobot_ready.py
   

   You should see DEVICE_ID:<uuid>. If you see NO_DEVICE, return to step 1.
3. Run an authenticated scrape (X/Twitter example):
   Copy

   powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1
   

4. Parse and clean the raw output:
   Copy

   python scripts/parse_linuxdo.py
   

5. Compare today's data against yesterday's baseline:
   Copy

   python scripts/compare_linuxdo.py
   

   Newly trending topics appear marked with 🆕.
6. (Optional) Generate a structured intelligence report by running your summary script after the comparison step.

Follow these steps in order. Each step has a clear success condition — do not proceed to the next step until you see it.

Step 1 — Activate the Edge bridge

Open Microsoft Edge and click the dokobot bridge extension icon to activate it. The extension must be running before any authenticated scraping can proceed. This is how the toolkit reuses your existing login sessions without storing your passwords.

Step 2 — Check bridge readiness

From a PowerShell terminal (not Git Bash), run:

python scripts/dokobot_ready.py

dokobot_ready.py contacts the Edge bridge and prints either DEVICE_ID:<uuid> (bridge is active and ready) or NO_DEVICE (bridge is not reachable). All authenticated scraping depends on this check passing. If you see NO_DEVICE, confirm that Edge is open and the bridge extension is enabled, then re-run this command.

✅ Success: Output contains DEVICE_ID: followed by a UUID.

Step 3 — Run an authenticated scrape

Choose the platform script that matches what you want to scrape. Scripts are PowerShell-based and must be invoked from PowerShell:

Example — scrape X:

powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1

The dokobot CLI relays each request through the Edge bridge using your logged-in session. Scripts run serially to stay within platform rate limits.

✅ Success: Raw output files (.txt / .json) are written to your local output directory. Errors are suppressed to $null by _catch_x2.ps1 so rate-limit interruptions do not halt the run.

Step 4 — Parse and clean the raw data

Raw scrape output often has encoding issues and noisy formatting. Run the appropriate parsing and cleaning scripts to extract structured fields (title, popularity score, URL) and fix encoding:

# Parse linux.do JSON and extract structured topics
python scripts/parse_linuxdo.py

# Fix encoding issues (e.g. UTF-16-LE from dokobot output)
python scripts/enc_0715.py

# Clean and deduplicate text
python scripts/clean_0716.py

Run the scripts that correspond to your scrape sources. The cleaned output is what feeds the comparison and report steps.

✅ Success: Clean, structured data files are present and free of encoding artifacts.

Step 5 — Compare against yesterday's baseline

The incremental comparison step diffs today's cleaned data against the previous day's saved baseline to surface what is newly trending:

# Compare linux.do topics against yesterday's baseline
python scripts/compare_linuxdo.py

# Cross-platform signal comparison
python scripts/compare_0714.py

Newly trending topics are annotated with 🆕. If this is your very first run, there is no baseline yet — the comparison script will treat all topics as new.

✅ Success: Output lists topics with 🆕 markers for anything not present in yesterday's data.

Step 6 — Generate the daily intelligence report

Aggregate all signals from the day's platforms into a single structured report that highlights the strongest monetization opportunities:

# Summarise all platforms
python scripts/summary_0715.py

# Generate a structured linux.do topic listing
python scripts/gen_linuxdo_0716.py

✅ Success: A structured intelligence report is written locally, surfacing the day's top monetizable signals ready for repurposing (e.g. to Xiaohongshu content).

Example 1 — Bridge readiness check

Verify that the dokobot Edge bridge is active before starting any scrape.

python scripts/dokobot_ready.py

Expected output when the bridge is active:

DEVICE_ID:a1b2c3d4-e5f6-7890-abcd-ef1234567890

Expected output when the bridge is not active:

NO_DEVICE

If you see NO_DEVICE, open Microsoft Edge, activate the bridge extension, and re-run the command.

Example 2 — Authenticated X (Twitter) scrape

Scrape X using multiple search threads with serial execution and rate-limit suppression.

powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1

Expected outcome: raw output files written to your local data directory. No visible errors — 2>$null suppresses rate-limit stderr noise.

Example 3 — Parse linux.do data and compare against baseline

A two-step sequence: extract structured topics from a raw JSON dump, then diff against yesterday.

python scripts/parse_linuxdo.py
python scripts/compare_linuxdo.py

Expected output from compare_linuxdo.py (excerpt):

🆕 [新增] 用 AI 自动生成小红书爆款标题  (热度: 412)
🆕 [新增] Claude 3.5 在代码补全上的真实表现  (热度: 389)
   [已有] V2EX 每周问一次：大家都在用什么 RSS 阅读器  (热度: 301)

Topics marked 🆕 were not present in yesterday's baseline and are your primary signals for the day.

Example 4 — Full pipeline in sequence

The complete ordered pipeline from readiness check to report generation.

# 1. Confirm bridge is ready
python scripts/dokobot_ready.py

# 2. Scrape platforms
powershell -ExecutionPolicy Bypass -File scripts/_catch_web.ps1
powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1
powershell -ExecutionPolicy Bypass -File scripts/_catch_extra.ps1

# 3. Parse and clean
python scripts/parse_linuxdo.py
python scripts/enc_0715.py
python scripts/clean_0716.py

# 4. Compare against baseline
python scripts/compare_linuxdo.py
python scripts/compare_0714.py

# 5. Generate report
python scripts/summary_0715.py

Expected outcome: a structured local intelligence report aggregating all platform signals, with 🆕-annotated topics ready for content repurposing.

Issue: dokobot_ready.py outputs NO_DEVICE

• Symptom: Running python scripts/dokobot_ready.py prints NO_DEVICE instead of DEVICE_ID:<uuid>.
• Likely cause: The dokobot bridge extension in Microsoft Edge is not active, or Edge is not open.
• Fix: Open Microsoft Edge, navigate to the extensions panel, and confirm the dokobot bridge extension is enabled and running. Re-run python scripts/dokobot_ready.py. Do not proceed to any scraping step until you see DEVICE_ID:<uuid>.

Issue: PowerShell scrape script is blocked by execution policy

• Symptom: Running a .ps1 script produces an error like File cannot be loaded because running scripts is disabled on this system.
• Likely cause: PowerShell's default execution policy blocks unsigned scripts.
• Fix: Invoke the script with the bypass flag:
  Copy

  powershell -ExecutionPolicy Bypass -File scripts/_catch_x2.ps1
  

Issue: Scrape script hangs or produces garbled/empty output

• Symptom: A scrape script runs but writes empty files or output contains mojibake (garbled characters).
• Likely cause (empty output): The bridge session was not ready, or the platform rate-limited the request mid-run.
• Likely cause (garbled text): dokobot outputs UTF-16-LE encoded text, which needs the encoding-fix script.
• Fix for empty output: Re-run python scripts/dokobot_ready.py to confirm the bridge is still active, then retry the scrape script.
• Fix for encoding issues: Run python scripts/enc_0715.py on the raw output file before any other parse or clean step.

Issue: Paths are misresolved when running scripts from Git Bash

• Symptom: Commands that reference paths like /e/... fail or resolve to unexpected locations such as c:\e\....
• Likely cause: Git Bash translates POSIX-style drive paths incorrectly when invoking Node.js or dokobot.
• Fix: Use PowerShell for all script execution, and write paths using Windows backslash notation (e.g. E:\path\to\scripts). Do not run dokobot or Node.js-dependent commands from Git Bash.

Issue: No 🆕 topics appear in comparison output on first run

• Symptom: Running compare_linuxdo.py shows all topics without any 🆕 marker, or the script errors with a missing baseline file.
• Likely cause: This is your first run, so no yesterday's baseline exists yet.
• Fix: This is expected behaviour on day one. The output from today's run becomes tomorrow's baseline. Run the full pipeline again the next day — incremental comparison will work once a baseline snapshot is saved.

Issue: faster-whisper process exits silently with no transcript output

• Symptom: The bili-subtitle Skill's whisper fallback starts but produces no transcript.txt, or the process exits immediately.
• Likely cause: Running the transcription script from PowerShell causes faster-whisper's model-loading warnings (written to stderr) to be interpreted as errors, killing the process before the model finishes loading.
• Fix: Always run the transcribe.py script from bash using nohup and redirect both streams:
  Copy

  nohup "$VENV/Scripts/python.exe" transcribe.py > transcribe.log 2>&1 &
  

  Then poll for completion with tail transcribe.log and wait for the ^DONE marker before reading transcript.txt.