---
title: Architecture
product: tide
doc_type: architecture
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/wabam/docs/tide/tide-capital-pressure-observatory/architecture
---

# Architecture

_System components, data flow, and design decisions_

## Overview

This page explains how TIDE's processes fit together, how data moves from upstream sources to your browser, and why the system is structured the way it is. Understanding the architecture helps you reason about what happens when a data source is slow or unavailable, how to extend the dashboard with new metrics, and where to look when something goes wrong. The design centres on a single shared DuckDB file that decouples ingestion from serving, so a failing scrape can never take the dashboard offline.

## Architecture diagram

```mermaid
flowchart TD
    subgraph External["External Data Sources (free-tier only)"]
        FRED["FRED API\n(M2, WALCL, HY spread,\nNFCI, TOTLL, Margin Debt)"]
        YF["Yahoo Finance Chart API\n(MOVE, VIX, VIX3M, SPY,\nTLT, HYG, watchlist tickers)"]
        CFTC["CFTC TFF Archives\n(COT S&P spec net)"]
        AAII["AAII XLS\n(Bull/Bear spread)"]
        SM["SqueezeMetrics CSV\n(Dealer Gamma / DIX)"]
        TIC["Treasury TIC SLT\n(Foreign Equity Holdings)"]
    end

    subgraph Ingest["Ingestion Layer"]
        CLI["tide-ingest CLI"]
        SCH["Scheduler\n(APScheduler cron jobs)"]
    end

    subgraph Store["Storage"]
        DB[("DuckDB\ndata/tide.duckdb")]
    end

    subgraph Backend["Backend — FastAPI :8765"]
        API["/api/dashboard\n/api/watchlist\n/api/sources\n/api/scheduler\n/api/metrics\n/api/tier"]
    end

    subgraph Frontend["Frontend — SvelteKit SSR :5173"]
        PS["+page.server.ts\n(parallel fetch on each SSR request)"]
        UI["Browser UI\n(CompositeHero, TierGrid,\nWatchlist, MetricCard)"]  
    end

    FRED & YF & CFTC & AAII & SM & TIC --> CLI
    SCH -- "triggers" --> CLI
    CLI -- "writes observations &\nscheduler_status rows" --> DB
    DB -- "read-only queries" --> API
    API -- "JSON payload" --> PS
    PS -- "props" --> UI
    UI -- "browser navigates to /sources" --> API
```

## Components

**tide-ingest CLI** (`cli`)
The command-line tool responsible for pulling data from every upstream source and writing rows into DuckDB. You run it directly with `make ingest-all`, `make ingest-watchlist`, and `make backfill-composite`, or it is invoked automatically by the Scheduler. Because the CLI and the API server share only the database file and never hold a live connection to each other, a slow or crashing ingest job cannot affect dashboard availability.

---

**Scheduler** (`tide-scheduler`, `make scheduler`)
An optional background service that wraps the ingest CLI in two APScheduler cron jobs running in the `America/New_York` timezone:
- `daily_ingest` — Mon–Fri at 17:00, after US market close. Refreshes every metric whose `cadence` is `"daily"`, refreshes the watchlist, and recomputes the composite history.
- `release_ingest` — Fri at 18:00. Refreshes every non-daily metric (weekly, monthly, quarterly) to catch new FRED, FINRA, Treasury, and CFTC releases within a week.

The Scheduler persists per-job telemetry (last run, last success, last error, next run) to the `scheduler_status` DuckDB table so the Sources page can surface it without needing an IPC channel to the scheduler process.

---

**FastAPI backend** (`:8765`)
A Python FastAPI application that serves a JSON API over HTTP. It reads from DuckDB at request time — it never holds a persistent connection to any upstream data source. Key routes include:
- `GET /api/dashboard` — composite z-score, all four tiers, all metric readings, tally
- `GET /api/watchlist` — eight healthcare tickers with relative returns and sparkline data
- `GET /api/sources` — data source status surfaced from `scheduler_status`
- `GET /api/scheduler` — APScheduler job state
- `GET /api/metrics/{id}/detail` — per-metric detail
- `GET /api/tier/{tier}/history` — 252-day tier history

CORS is enabled for `http://localhost:5173` and `http://127.0.0.1:5173` so client-side navigations in development work alongside the SSR path.

---

**DuckDB database** (`data/tide.duckdb`)
The single shared artefact that all processes communicate through. It holds three tables:
- `observations` — every time-series data point keyed by `(metric_id, ts)`. Also stores watchlist prices (`wl_<ticker>`) and the pre-computed composite history (`_composite_z`) as ordinary rows.
- `metrics` — the MetricDefinition registry: names, tiers, sources, cadences, z-score windows, and sort orders for the 19 registered indicators.
- `scheduler_status` — one row per APScheduler job, updated by the Scheduler so the Sources page has something to display without polling the scheduler process directly.

The database file path is resolved by `pydantic-settings` in `config.py` relative to the project root; there is no external database server.

---

**SvelteKit frontend** (`:5173`)
A server-side-rendered web application. On every page request, `+page.server.ts` issues parallel fetches to `/api/dashboard` and `/api/watchlist` and passes the full payload as props to the page component. Because rendering happens on the server, the browser receives a complete HTML page on first load. Key UI components include `CompositeHero` (the composite z-score and 252-day SVG history chart), `TierGrid` and `MetricCard` (per-indicator readings with as-of dates and z-scores), `Watchlist` with `Sparkline` (healthcare tickers vs XLV), and `Tally` (the bull/neutral/bear vote count).

---

**TIDE Dashboard** (browser, `http://localhost:5173`)
The main page you open to read the composite Capital Pressure score, all four tier indicators, and the watchlist. This is what the SvelteKit frontend renders.

---

**Sources page** (`/sources`)
A page within the dashboard that shows the status of every data source — when it last ran, when it will run next, and any errors that occurred. Data is served from the `scheduler_status` DuckDB table via `/api/sources`.

## Data flow

The following traces the full path from a market data publication to the number you read on the dashboard, using M2 money supply as a concrete example.

**1. Ingestion — tide-ingest writes to DuckDB**

You run `make ingest-all` (or the Scheduler fires `daily_ingest` at Mon–Fri 17:00 NY time). The `tide-ingest` CLI loads each metric module, finds the ones with an `ingest_fn`, and calls them in sequence. For M2, the FRED client requests the `M2SL` series using your `FRED_API_KEY` environment variable and receives a list of `(date, value)` tuples. Those rows are upserted into the `observations` table with `PRIMARY KEY (metric_id, ts)`, so re-running ingest is idempotent. Stale or missing days from slow-publishing sources (M2 is monthly) are left as gaps rather than interpolated — the last known value is used by the composite.

**2. Composite history — backfill-composite**

After ingest completes, `make backfill-composite` (or the Scheduler's post-ingest step) walks the past 252 business days. For each day it reads the latest available observation for every metric, computes each metric's rolling z-score over its configured window (3 years by default, 1 year for HY spread), applies `directional_z` so that inverted-convention metrics contribute with the correct sign, averages within each of the four tiers, then averages the four tier averages into a single composite z-score. The resulting `(date, composite_z)` rows are written into `observations` under the series id `_composite_z`.

**3. API — FastAPI reads and shapes JSON**

When a browser navigates to the dashboard, the SvelteKit page server calls `GET /api/dashboard`. The FastAPI handler queries DuckDB: it reads `observations` and `metrics`, computes the current composite from live readings, assembles tier groupings, and returns a single JSON payload. The API process reads the database file; it has no connection to any upstream data source and no connection to the ingest CLI. A wedged scrape cannot affect this step.

**4. SSR — SvelteKit renders HTML**

`+page.server.ts` issues parallel fetches to `/api/dashboard` and `/api/watchlist` on every page request. It passes the full payloads as props to `+page.svelte`. SvelteKit renders the complete page server-side and sends HTML to the browser. The composite hero, tier grids, metric cards with their as-of dates, and the watchlist sparklines are all present in the initial response.

**5. Sources page — scheduler telemetry**

When you navigate to `/sources`, `+page.server.ts` fetches `/api/sources`, which reads the `scheduler_status` DuckDB table. The Scheduler writes one row per job (`daily_ingest`, `release_ingest`) after each run, recording `last_run_at`, `last_success_at`, `last_error`, and `next_run_at`. No live IPC channel exists between the Scheduler process and the API; the database row is the communication medium.

## Design decisions

**Shared database, no shared process**

The most consequential design decision is that the ingest CLI, the API server, and the Scheduler communicate exclusively through the DuckDB file. No message queue, no shared memory, no RPC. This means a crashing or slow scrape job — AAII's bot detection cycling between 200 and 403, CFTC's TLS quirks, or a hung Yahoo Finance request — cannot take the dashboard down. The API keeps serving the last known values from DuckDB while ingest retries in the background.

**Stale data shown, not hidden**

Because different sources publish on very different schedules (M2 monthly, CFTC weekly, VIX daily), hiding indicators until they are fresh would leave large gaps in the dashboard. TIDE instead shows the last known value for every metric and labels each card with its as-of date. The composite uses last-known values for lagging series. This gives you a complete picture with explicit staleness rather than a partial picture with implicit omissions.

**Equal-weight tier averaging for the composite**

The composite z-score is the equal-weight average of the four tier averages, not the equal-weight average of all 19 indicators. This ensures that a tier with more indicators (Tier III has four; Tier II has two) does not dominate the composite. Each of the four aspects of capital pressure — Macro Liquidity, Capital Flows, Market Microstructure, Sentiment & Positioning — contributes equally regardless of how many metrics it currently has.

**directional_z for inverted-convention metrics**

Several indicators are bullish when their raw value is low: a falling VIX, a tightening HY spread, and a negative COT net position all signal easier conditions. Rather than inverting raw values at ingestion, each MetricDefinition carries a `direction_kind` flag and the system computes a `directional_z` at read time. The composite and tally use `directional_z` exclusively, so you can add a new inverted metric without touching the composite logic.

**Direct HTTP to Yahoo Finance, no yfinance dependency**

The `yfinance` library wraps Yahoo Finance's private API and has historically suffered from auth-cookie and crumb breakage every few weeks, requiring library updates to stay functional. TIDE instead hits `query1.finance.yahoo.com/v8/finance/chart/` directly over HTTP. This surface has been stable in practice, and removing the dependency eliminates the maintenance burden of tracking upstream library changes.

**Free-tier data sources only**

All 19 indicators (including the four stubbed ones) use publicly available, free-tier data. This is a deliberate constraint, not a limitation to work around. It means the dashboard can be self-hosted with no ongoing data costs and no vendor lock-in. Each stubbed metric's unblock path is documented in its tier's `__init__.py` so you know exactly what would need to change if a free source became available.

**3-year rolling z-scores, configurable per-metric**

A 3-year window is long enough to capture a full credit cycle in most indicators while remaining sensitive to recent regime changes. The window is configurable per-metric via `MetricDefinition.zscore_years` for cases where the source data history is shorter. HY spread (`BAMLH0A0HYM2`) uses a 1-year window because FRED truncated that series to a 3-year history in April 2026; using a shorter z-score window than the available history would produce misleading scores.

**DuckDB, not a server database**

DuckDB runs in-process, stores to a single file, and requires no server to manage. For a dashboard updated once per day by a single ingest process, the concurrency constraints of a file-based database are not a practical problem. The choice avoids the operational overhead of running and maintaining a separate database server for a personal or small-team deployment.

## Trade-offs

**DuckDB write concurrency**

DuckDB allows one writer at a time. If you run `make ingest-all` while the API server holds the file open for a long-running query, the ingest process will wait or error. In practice this is rare because API queries are fast point reads, but it is something to be aware of if you build long-running analytical queries against the same file. The mitigation is to keep API queries short, which the current implementation does.

**No real-time updates**

Because the frontend fetches data via SSR on each page load and the Scheduler runs at most twice per weekday, the dashboard does not reflect intraday price moves. This is intentional — TIDE is designed as a daily macro signal, not a streaming ticker. If you need intraday updates you would need to add a client-side polling mechanism and a more frequent ingest cadence.

**Four stubbed metrics**

`ici_etf_flows`, `buyback_yield`, `put_call`, and `uvol_dvol` are registered in the MetricDefinition registry but have no `ingest_fn`. Their cards appear in the dashboard without live readings. The unblock paths are documented in each tier's `__init__.py`. Until those paths are implemented, the composite is computed from the 15 live metrics only.

**FRED API key required**

Six of the 15 live metrics come from FRED and require a `FRED_API_KEY` environment variable. This is a free key available at `fred.stlouisfed.org`, but it is a manual setup step that is not needed for the Yahoo Finance, CFTC, AAII, SqueezeMetrics, or Treasury TIC sources. Running `make ingest-all` without a FRED key will produce errors for those six metrics while the remaining nine ingest successfully.

**AAII bot detection**

The AAII sentiment source serves a binary `.xls` file behind bot-detection logic that cycles between HTTP 200 and HTTP 403 responses with the same headers. The ingest client retries four times with backoff and validates the OLE2 magic bytes on success. This makes AAII ingestion non-deterministic; a run that fails can succeed minutes later. There is no alternative free source for AAII Bull/Bear data.

**TIC SLT data caps at 2023-01**

The Treasury TIC SLT `slt1d_globl.csv` file that TIDE downloads caps at January 2023. Newer data is published across per-press-release URLs rather than a single cumulative file. Until the ingest client is updated to follow those URLs, Foreign Equity Holdings will be stale beyond that date.

**No horizontal scaling**

A single DuckDB file and a single ingest process mean TIDE is not designed for horizontal scaling or multi-instance deployments sharing state. For a personal or small-team observatory this is not a constraint. If you need to run multiple API replicas or parallel ingest workers you would need to move to a server database and redesign the ingest coordination layer.