Architecture

This page explains how TIDE's processes fit together, how data moves from upstream sources to your browser, and why the system is structured the way it is. Understanding the architecture helps you reason about what happens when a data source is slow or unavailable, how to extend the dashboard with new metrics, and where to look when something goes wrong. The design centres on a single shared DuckDB file that decouples ingestion from serving, so a failing scrape can never take the dashboard offline.

tide-ingest CLI (cli) The command-line tool responsible for pulling data from every upstream source and writing rows into DuckDB. You run it directly with make ingest-all, make ingest-watchlist, and make backfill-composite, or it is invoked automatically by the Scheduler. Because the CLI and the API server share only the database file and never hold a live connection to each other, a slow or crashing ingest job cannot affect dashboard availability.

Scheduler (tide-scheduler, make scheduler) An optional background service that wraps the ingest CLI in two APScheduler cron jobs running in the America/New_York timezone:

• daily_ingest — Mon–Fri at 17:00, after US market close. Refreshes every metric whose cadence is "daily", refreshes the watchlist, and recomputes the composite history.
• release_ingest — Fri at 18:00. Refreshes every non-daily metric (weekly, monthly, quarterly) to catch new FRED, FINRA, Treasury, and CFTC releases within a week.

The Scheduler persists per-job telemetry (last run, last success, last error, next run) to the scheduler_status DuckDB table so the Sources page can surface it without needing an IPC channel to the scheduler process.

FastAPI backend (:8765) A Python FastAPI application that serves a JSON API over HTTP. It reads from DuckDB at request time — it never holds a persistent connection to any upstream data source. Key routes include:

• GET /api/dashboard — composite z-score, all four tiers, all metric readings, tally
• GET /api/watchlist — eight healthcare tickers with relative returns and sparkline data
• GET /api/sources — data source status surfaced from scheduler_status
• GET /api/scheduler — APScheduler job state
• GET /api/metrics/{id}/detail — per-metric detail
• GET /api/tier/{tier}/history — 252-day tier history

CORS is enabled for http://localhost:5173 and http://127.0.0.1:5173 so client-side navigations in development work alongside the SSR path.

DuckDB database (data/tide.duckdb) The single shared artefact that all processes communicate through. It holds three tables:

• observations — every time-series data point keyed by (metric_id, ts). Also stores watchlist prices (wl_<ticker>) and the pre-computed composite history (_composite_z) as ordinary rows.
• metrics — the MetricDefinition registry: names, tiers, sources, cadences, z-score windows, and sort orders for the 19 registered indicators.
• scheduler_status — one row per APScheduler job, updated by the Scheduler so the Sources page has something to display without polling the scheduler process directly.

The database file path is resolved by pydantic-settings in config.py relative to the project root; there is no external database server.

SvelteKit frontend (:5173) A server-side-rendered web application. On every page request, +page.server.ts issues parallel fetches to /api/dashboard and /api/watchlist and passes the full payload as props to the page component. Because rendering happens on the server, the browser receives a complete HTML page on first load. Key UI components include CompositeHero (the composite z-score and 252-day SVG history chart), TierGrid and MetricCard (per-indicator readings with as-of dates and z-scores), Watchlist with Sparkline (healthcare tickers vs XLV), and Tally (the bull/neutral/bear vote count).

TIDE Dashboard (browser, http://localhost:5173) The main page you open to read the composite Capital Pressure score, all four tier indicators, and the watchlist. This is what the SvelteKit frontend renders.

Sources page (/sources) A page within the dashboard that shows the status of every data source — when it last ran, when it will run next, and any errors that occurred. Data is served from the scheduler_status DuckDB table via /api/sources.

The following traces the full path from a market data publication to the number you read on the dashboard, using M2 money supply as a concrete example.

1. Ingestion — tide-ingest writes to DuckDB

You run make ingest-all (or the Scheduler fires daily_ingest at Mon–Fri 17:00 NY time). The tide-ingest CLI loads each metric module, finds the ones with an ingest_fn, and calls them in sequence. For M2, the FRED client requests the M2SL series using your FRED_API_KEY environment variable and receives a list of (date, value) tuples. Those rows are upserted into the observations table with PRIMARY KEY (metric_id, ts), so re-running ingest is idempotent. Stale or missing days from slow-publishing sources (M2 is monthly) are left as gaps rather than interpolated — the last known value is used by the composite.

2. Composite history — backfill-composite

After ingest completes, make backfill-composite (or the Scheduler's post-ingest step) walks the past 252 business days. For each day it reads the latest available observation for every metric, computes each metric's rolling z-score over its configured window (3 years by default, 1 year for HY spread), applies directional_z so that inverted-convention metrics contribute with the correct sign, averages within each of the four tiers, then averages the four tier averages into a single composite z-score. The resulting (date, composite_z) rows are written into observations under the series id _composite_z.

3. API — FastAPI reads and shapes JSON

When a browser navigates to the dashboard, the SvelteKit page server calls GET /api/dashboard. The FastAPI handler queries DuckDB: it reads observations and metrics, computes the current composite from live readings, assembles tier groupings, and returns a single JSON payload. The API process reads the database file; it has no connection to any upstream data source and no connection to the ingest CLI. A wedged scrape cannot affect this step.

4. SSR — SvelteKit renders HTML

+page.server.ts issues parallel fetches to /api/dashboard and /api/watchlist on every page request. It passes the full payloads as props to +page.svelte. SvelteKit renders the complete page server-side and sends HTML to the browser. The composite hero, tier grids, metric cards with their as-of dates, and the watchlist sparklines are all present in the initial response.

5. Sources page — scheduler telemetry

When you navigate to /sources, +page.server.ts fetches /api/sources, which reads the scheduler_status DuckDB table. The Scheduler writes one row per job (daily_ingest, release_ingest) after each run, recording last_run_at, last_success_at, last_error, and next_run_at. No live IPC channel exists between the Scheduler process and the API; the database row is the communication medium.

Shared database, no shared process

The most consequential design decision is that the ingest CLI, the API server, and the Scheduler communicate exclusively through the DuckDB file. No message queue, no shared memory, no RPC. This means a crashing or slow scrape job — AAII's bot detection cycling between 200 and 403, CFTC's TLS quirks, or a hung Yahoo Finance request — cannot take the dashboard down. The API keeps serving the last known values from DuckDB while ingest retries in the background.

Stale data shown, not hidden

Because different sources publish on very different schedules (M2 monthly, CFTC weekly, VIX daily), hiding indicators until they are fresh would leave large gaps in the dashboard. TIDE instead shows the last known value for every metric and labels each card with its as-of date. The composite uses last-known values for lagging series. This gives you a complete picture with explicit staleness rather than a partial picture with implicit omissions.

Equal-weight tier averaging for the composite

The composite z-score is the equal-weight average of the four tier averages, not the equal-weight average of all 19 indicators. This ensures that a tier with more indicators (Tier III has four; Tier II has two) does not dominate the composite. Each of the four aspects of capital pressure — Macro Liquidity, Capital Flows, Market Microstructure, Sentiment & Positioning — contributes equally regardless of how many metrics it currently has.

directional_z for inverted-convention metrics

Several indicators are bullish when their raw value is low: a falling VIX, a tightening HY spread, and a negative COT net position all signal easier conditions. Rather than inverting raw values at ingestion, each MetricDefinition carries a direction_kind flag and the system computes a directional_z at read time. The composite and tally use directional_z exclusively, so you can add a new inverted metric without touching the composite logic.

Direct HTTP to Yahoo Finance, no yfinance dependency

The yfinance library wraps Yahoo Finance's private API and has historically suffered from auth-cookie and crumb breakage every few weeks, requiring library updates to stay functional. TIDE instead hits query1.finance.yahoo.com/v8/finance/chart/ directly over HTTP. This surface has been stable in practice, and removing the dependency eliminates the maintenance burden of tracking upstream library changes.

Free-tier data sources only

All 19 indicators (including the four stubbed ones) use publicly available, free-tier data. This is a deliberate constraint, not a limitation to work around. It means the dashboard can be self-hosted with no ongoing data costs and no vendor lock-in. Each stubbed metric's unblock path is documented in its tier's __init__.py so you know exactly what would need to change if a free source became available.

3-year rolling z-scores, configurable per-metric

A 3-year window is long enough to capture a full credit cycle in most indicators while remaining sensitive to recent regime changes. The window is configurable per-metric via MetricDefinition.zscore_years for cases where the source data history is shorter. HY spread (BAMLH0A0HYM2) uses a 1-year window because FRED truncated that series to a 3-year history in April 2026; using a shorter z-score window than the available history would produce misleading scores.

DuckDB, not a server database

DuckDB runs in-process, stores to a single file, and requires no server to manage. For a dashboard updated once per day by a single ingest process, the concurrency constraints of a file-based database are not a practical problem. The choice avoids the operational overhead of running and maintaining a separate database server for a personal or small-team deployment.

DuckDB write concurrency

DuckDB allows one writer at a time. If you run make ingest-all while the API server holds the file open for a long-running query, the ingest process will wait or error. In practice this is rare because API queries are fast point reads, but it is something to be aware of if you build long-running analytical queries against the same file. The mitigation is to keep API queries short, which the current implementation does.

No real-time updates

Because the frontend fetches data via SSR on each page load and the Scheduler runs at most twice per weekday, the dashboard does not reflect intraday price moves. This is intentional — TIDE is designed as a daily macro signal, not a streaming ticker. If you need intraday updates you would need to add a client-side polling mechanism and a more frequent ingest cadence.

Four stubbed metrics

ici_etf_flows, buyback_yield, put_call, and uvol_dvol are registered in the MetricDefinition registry but have no ingest_fn. Their cards appear in the dashboard without live readings. The unblock paths are documented in each tier's __init__.py. Until those paths are implemented, the composite is computed from the 15 live metrics only.

FRED API key required

Six of the 15 live metrics come from FRED and require a FRED_API_KEY environment variable. This is a free key available at fred.stlouisfed.org, but it is a manual setup step that is not needed for the Yahoo Finance, CFTC, AAII, SqueezeMetrics, or Treasury TIC sources. Running make ingest-all without a FRED key will produce errors for those six metrics while the remaining nine ingest successfully.

AAII bot detection

The AAII sentiment source serves a binary .xls file behind bot-detection logic that cycles between HTTP 200 and HTTP 403 responses with the same headers. The ingest client retries four times with backoff and validates the OLE2 magic bytes on success. This makes AAII ingestion non-deterministic; a run that fails can succeed minutes later. There is no alternative free source for AAII Bull/Bear data.

TIC SLT data caps at 2023-01

The Treasury TIC SLT slt1d_globl.csv file that TIDE downloads caps at January 2023. Newer data is published across per-press-release URLs rather than a single cumulative file. Until the ingest client is updated to follow those URLs, Foreign Equity Holdings will be stale beyond that date.

No horizontal scaling

A single DuckDB file and a single ingest process mean TIDE is not designed for horizontal scaling or multi-instance deployments sharing state. For a personal or small-team observatory this is not a constraint. If you need to run multiple API replicas or parallel ingest workers you would need to move to a server database and redesign the ingest coordination layer.