Configuration

This page explains how to configure the dokobot-intel-toolkit so every stage of the pipeline — bridge readiness checks, authenticated scraping, data cleaning, incremental comparison, and report generation — runs with the correct paths, device identity, and feature flags. Because the toolkit depends on hardcoded binary paths, a device ID tied to your Edge bridge session, and per-script output directories, getting configuration right is a prerequisite for any successful scrape. Work through the sections below before running any script for the first time.

Before configuring the toolkit, confirm you have the following in place:

• Python ≥ 3.11 — required by dokobot_ready.py and all parse, clean, comparison, and report scripts
• Node.js ≥ 22 — the dokobot CLI is a Node.js binary; the scripts call it directly via the node executable
• dokobot CLI — installed at a known path on your machine (see Installation for the exact path shape used by the scripts)
• Microsoft Edge with the dokobot bridge extension — must be open and the extension must be active before any authenticated scrape; dokobot_ready.py verifies this
• Windows OS with PowerShell — all scraping scripts are .ps1 files and require a Windows PowerShell environment
• WorkBuddy application (optional) — needed only if you intend to activate and use the bili-subtitle, deck-studio, or xhs-commercial-growth Skills
• faster-whisper (optional) — needed only if you want the bili-subtitle Skill to fall back to local speech recognition when a Bilibili video has no subtitle track

Step 1 — Install the dokobot CLI

The PowerShell scraping scripts expect the dokobot CLI to live inside your npm global modules directory. Install it via npm:

npm install -g @dokobot/cli

After installation, confirm the binary exists at the path the scripts reference:

E:\OpenClaw\npm-global\node_modules\@dokobot\cli\dist\cli\bin\dokobot.js

If your npm global prefix differs, update the $doko / $dokobot variable in each script to match your actual path (see Configuration).

Step 2 — Install Node.js 22

The scripts call Node.js via an explicit binary path rather than relying on PATH. Place (or symlink) your Node.js 22 executable so it is reachable at:

C:\Users\<you>\.workbuddy\binaries\node\versions\22.22.2\node.exe

Replace <you> with your Windows username. If WorkBuddy manages this binary for you automatically, verify the path exists after opening WorkBuddy at least once.

Step 3 — Install Python dependencies

From the repository root, install the Python requirements:

pip install -r requirements.txt

Step 4 — Install faster-whisper (optional)

Only required if you want speech-recognition fallback in the bili-subtitle Skill:

pip install faster-whisper

Step 5 — Enable the Edge bridge extension

Open Microsoft Edge, navigate to edge://extensions/, locate the dokobot bridge extension, and toggle it on. Keep Edge open whenever you run scraping scripts. Run the readiness check to confirm:

python dokobot_ready.py

Expected output when the bridge is active:

DEVICE_ID:20618bf1-a086-4c7c-85db-28b07bcc0e74

If you see NO_DEVICE instead, the extension is not active — return to Edge and re-enable it before proceeding.

Step 6 — Install WorkBuddy Skills (optional)

Skills are stored in ~/.workbuddy/skills/. Copy or clone each skill package into that directory, then activate them inside the WorkBuddy application UI. The three available Skills are bili-subtitle, deck-studio, and xhs-commercial-growth.

All configuration in the dokobot-intel-toolkit is script-level: there is no single central config file. Each PowerShell scraping script declares its own variables at the top. You configure the toolkit by editing those variables before running the scripts.

Core path variables

Every scraping script shares three path variables. Set them to match your local environment:

Why explicit paths? The scripts invoke Node.js and the dokobot CLI directly rather than via PATH to avoid version collisions when multiple Node.js versions are installed on the same machine.

Output directory ($cwd)

Only the main daily scrape script (scrape_0716.ps1) uses a $cwd variable to define where raw output .txt files are written:

Change this to a directory you control. Use a date-stamped folder name to keep each day's raw data isolated and ready for the incremental comparison step.

The _catch_*.ps1 utility scripts stream output directly to the console (Out-String) rather than writing files, so they do not use $cwd.

Timeout

Each dokobot read call accepts a --timeout flag (in seconds). The scraping scripts use two values:

Increase the timeout if you are on a slow connection or scraping platforms that take longer to render (Xiaohongshu and Bilibili are common candidates).

Inter-request delay

The main scrape script pauses between requests with:

Start-Sleep -Seconds 12

This 12-second gap is the rate-limiting strategy. Lower it only if you have confirmed the target platform tolerates faster polling; raising it reduces the chance of temporary bans.

Error handling flag

The _catch_x2.ps1 and _catch_web2.ps1 scripts suppress non-terminating errors:

$ErrorActionPreference = 'SilentlyContinue'

This prevents noisy stderr output from the dokobot CLI from cluttering console output during retries. Remove or change this to 'Stop' if you need full error visibility during debugging.

WorkBuddy Skills configuration

Skills have no environment-variable configuration exposed at the toolkit level. They are configured inside the WorkBuddy application after activation. The bili-subtitle Skill will automatically use faster-whisper for speech recognition if the library is installed and the video has no subtitle track — no additional flag is needed.

Check bridge readiness before every run

Always start by verifying the Edge bridge is active. This tells you whether authenticated scraping will work and gives you the device ID the scraping scripts need:

python dokobot_ready.py

If the output is DEVICE_ID:<uuid>, proceed. If it prints NO_DEVICE, open Microsoft Edge and activate the dokobot bridge extension before continuing.

Run the main daily scrape

Once the bridge is confirmed active, run the main scraping script. It iterates through all configured platforms, waits 12 seconds between each request, and writes one .txt file per platform into $cwd:

powershell -ExecutionPolicy Bypass -File scripts\scrape_0716.ps1

The script scrapes X/Twitter (five queries), Reddit (four subreddits), Xiaohongshu, Bilibili, and YouTube. Each completed scrape prints DONE <filename> to the console. When all scrapes finish, it prints ALL DONE.

Re-scrape specific platform groups

If a platform times out or you want to refresh a subset of results, use the targeted utility scripts instead of re-running the full batch:

These scripts stream output to the console rather than writing files, so redirect output yourself if you need to persist the results:

powershell -ExecutionPolicy Bypass -File scripts\_catch_extra.ps1 | Out-File -Encoding utf8 extra_retry.txt

Parse and clean raw output

After scraping, run the Python parse and clean scripts against the .txt files in your $cwd directory. These fix encoding issues, remove noise, and extract structured fields (title, popularity score, URL):

python parse_clean.py --input "C:\Users\<you>\WorkBuddy\automation-2026-07-08-11-54-48"

Run incremental comparison

With today's cleaned data and yesterday's baseline in place, run the comparison script to annotate newly trending topics with 🆕:

python compare_incremental.py --today cleaned_today/ --baseline cleaned_yesterday/

Generate the daily intelligence report

Run the report generation script to aggregate all platform signals into a single structured output that surfaces the strongest monetization signals:

python generate_report.py --input compared_output/ --out daily_report.md

Use a WorkBuddy Skill

Activate a Skill inside the WorkBuddy application, then interact with it through the WorkBuddy UI. For example, to extract a Bilibili transcript, activate the bili-subtitle Skill and provide a BV number. To generate a slide deck outline, activate deck-studio and supply a topic brief. To evaluate a trending topic's monetization potential, activate xhs-commercial-growth.

Example 1 — Verify bridge readiness

Run the readiness check and confirm you see a valid device ID.

python dokobot_ready.py

Expected output (bridge active):

DEVICE_ID:20618bf1-a086-4c7c-85db-28b07bcc0e74

Output when bridge is not active:

NO_DEVICE

Example 2 — Full daily scrape run

Run the main scraping script. Each platform prints a completion line as it finishes.

powershell -ExecutionPolicy Bypass -File scripts\scrape_0716.ps1

Expected console output:

DONE x_vibe_2026-07-16
DONE x_indie_2026-07-16
DONE x_claude_2026-07-16
DONE x_agent_2026-07-16
DONE x_xhs_2026-07-16
DONE reddit_claudeai_2026-07-16
DONE reddit_localllama_2026-07-16
DONE reddit_sideproject_2026-07-16
DONE reddit_aiagents_2026-07-16
DONE xhs_2026-07-16
DONE bili_2026-07-16
DONE yt_2026-07-16
ALL DONE

Raw .txt files are written to the $cwd directory defined in the script.

Example 3 — Retry Xiaohongshu, Bilibili, and YouTube with a longer timeout

Use _catch_extra.ps1 when those three platforms time out during the main batch. The script uses a 150-second timeout and streams results to the console.

powershell -ExecutionPolicy Bypass -File scripts\_catch_extra.ps1

Expected console output (abbreviated):

===== EXTRA 1 : https://www.xiaohongshu.com/search_result?keyword=AI%E5%89%AF%E4%B8%9A =====
<page content>
===== EXTRA 2 : https://search.bilibili.com/all?keyword=AI =====
<page content>
===== EXTRA 3 : https://www.youtube.com/results?search_query=AI+agent =====
<page content>

Example 4 — Retry V2EX, Product Hunt, and IndieHackers

powershell -ExecutionPolicy Bypass -File scripts\_catch_web.ps1

Expected console output (abbreviated):

===== WEB 1 : https://www.v2ex.com/?tab=tech =====
<page content>
===== WEB 2 : https://www.producthunt.com/ =====
<page content>
===== WEB 3 : https://www.indiehackers.com/products =====
<page content>

Example 5 — Scrape specific X/Twitter queries

Use _catch_x2.ps1 to re-fetch three targeted X search queries without re-running the full batch.

powershell -ExecutionPolicy Bypass -File scripts\_catch_x2.ps1

Expected console output (abbreviated):

===== X LINE 1 : https://x.com/search?q=indie%20hacker%20AI&src=typed_query =====
<page content>
===== X LINE 2 : https://x.com/search?q=Claude%20Code%20GPT-5.6&src=typed_query =====
<page content>
===== X LINE 3 : https://x.com/search?q=AI%20agent%20tool&src=typed_query =====
<page content>

Example 6 — Customise the device ID for a different machine

If you are running on a second machine with a different device ID (as printed by dokobot_ready.py), open the relevant script in a text editor and replace the $dev value:

# Before
$dev = '20618bf1-a086-4c7c-85db-28b07bcc0e74'

# After — replace with your own device ID
$dev = '<your-device-id-from-dokobot_ready.py>'

Apply the same change to every script (scrape_0716.ps1, _catch_web.ps1, _catch_web2.ps1, _catch_x2.ps1, _catch_extra.ps1) so they all use the same active bridge session.

dokobot_ready.py prints NO_DEVICE

Symptom: Running python dokobot_ready.py outputs NO_DEVICE instead of DEVICE_ID:<uuid>.

Likely cause: The dokobot bridge extension in Microsoft Edge is not active, or Edge is not open.

Fix: Open Microsoft Edge, go to edge://extensions/, find the dokobot bridge extension, and toggle it on. Keep Edge open. Re-run python dokobot_ready.py — you should now see DEVICE_ID:<uuid>.

Script exits immediately with node: command not found or a path error

Symptom: Running a scraping script fails instantly with an error about the node binary not being found.

Likely cause: The $node path in the script does not match the actual location of node.exe on your machine.

Fix: Run where.exe node in PowerShell to find your Node.js binary, then update $node in the script to that absolute path. Ensure the version is 22 or higher (node --version).

Script fails with Cannot find path ... dokobot.js

Symptom: The scraping script throws a path error pointing at the dokobot CLI entry point.

Likely cause: The $doko / $dokobot variable points to the wrong location. This happens when the npm global prefix differs from E:\OpenClaw\npm-global.

Fix: Run npm root -g to find your global node_modules directory, then update $doko / $dokobot in every script to <npm-root>\@dokobot\cli\dist\cli\bin\dokobot.js.

A platform's .txt file is empty or contains only an error message

Symptom: After running the main scrape, one or more output files in $cwd are empty or contain a timeout/auth error rather than page content.

Likely cause (timeout): The platform took longer than 90 seconds to respond.

Fix: Re-run the affected platform using the corresponding _catch_*.ps1 utility script, which uses a 150-second timeout. For Xiaohongshu, Bilibili, or YouTube, use _catch_extra.ps1. For V2EX, Product Hunt, or IndieHackers, use _catch_web.ps1 or _catch_web2.ps1. For X/Twitter queries, use _catch_x2.ps1.

Likely cause (auth failure): You are not logged in to that platform in Edge.

Fix: Open Edge, navigate to the platform manually, log in, then re-run the scrape script.

PowerShell refuses to run the script (execution policy error)

Symptom: PowerShell prints a message such as File ... cannot be loaded because running scripts is disabled on this system.

Likely cause: The PowerShell execution policy on your machine blocks unsigned scripts.

Fix: Run the script with an explicit bypass:

powershell -ExecutionPolicy Bypass -File scripts\scrape_0716.ps1

Alternatively, set the policy for the current user permanently (if your organisation permits it):

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

bili-subtitle Skill produces no transcript output

Symptom: The bili-subtitle Skill returns nothing or reports that no subtitles were found.

Likely cause: The Bilibili video has no official subtitle track, and faster-whisper is not installed, so the speech-recognition fallback cannot run.

Fix: Install faster-whisper:

pip install faster-whisper

Then retry the Skill in WorkBuddy with the same BV number.

Output files from different days are mixed in the same directory

Symptom: The incremental comparison step produces confusing results because today's and yesterday's files share the same $cwd folder.

Likely cause: The $cwd variable was not updated to a new date-stamped directory before running today's scrape.

Fix: Before each daily run, update $cwd in scrape_0716.ps1 to a new directory path that includes today's date (for example, automation-2026-07-17-09-00-00). Keep yesterday's directory intact so the comparison scripts can use it as the baseline.