Refresh Cycle

This page explains how TIDE keeps its 19 indicators current — both when you want to trigger a refresh yourself and when you want the system to handle it automatically. Because different data sources publish on different schedules (daily, weekly, monthly, or quarterly), TIDE separates the refresh cycle into three idempotent commands you can run by hand and two scheduled cron jobs that fire automatically after market close. Understanding this cycle tells you exactly how fresh each reading is and what to do when something has not updated as expected.

Before working with the refresh cycle, make sure you have the following in place:

• Python ≥ 3.10 installed and available in your virtual environment
• Node.js and npm (required if you are also running the frontend)
• make available in your shell
• A FRED API key exported as the FRED_API_KEY environment variable — obtain one free at fred.stlouisfed.org
• An active internet connection — all upstream sources (FRED, Yahoo Finance query endpoint, CFTC, FINRA, Treasury) are remote
• TIDE dependencies installed — run make install if you have not done so already
• The database initialised — run make init-db once before the first ingest

The refresh tooling ships with TIDE and requires no separate installation step beyond the standard project setup.

1. Install all dependencies (backend Python package and frontend npm packages):

make install

2. Initialise the DuckDB database (creates schema and seeds the metrics registry). Run this once — re-running it is safe:

make init-db

3. Verify the metrics registry by listing every registered metric:

cd backend && $(VENV)/bin/python -m tide.ingest.cli list

You should see 19 entries. Metrics marked ✓ have a live ingest_fn; metrics marked — are stubbed (registered but not yet fetching live data).

4. Run a first full ingest to populate the database before opening the dashboard:

make ingest-all
make ingest-watchlist
make backfill-composite

Your dashboard now has data. To keep it current automatically, proceed to the Scheduler section below.

TIDE's refresh behaviour is controlled by a small set of environment variables and per-metric settings baked into the metrics registry.

Environment variables

Set variables in your shell before running any make target:

export FRED_API_KEY=your_key_here
export VENV=/path/to/your/venv
make ingest-all

Per-metric cadence

Each metric in the registry carries a cadence field (daily, weekly, monthly, or quarterly). The scheduler uses this to decide which job pulls which metrics:

• daily_ingest pulls every metric with cadence="daily", plus all watchlist tickers.
• release_ingest pulls every metric with cadence in {weekly, monthly, quarterly} — all non-daily cadences are polled on Friday so new FRED / FINRA / Treasury / CFTC releases are captured within a week.

You cannot change cadences from the command line; they are set in each metric's MetricDefinition inside the source code.

Z-score window

Z-scores default to a 3-year rolling window (zscore_years=3). The exception is the FRED series BAMLH0A0HYM2 (HY spread), which uses zscore_years=1 because that series was truncated to a 3-year window in April 2026. These values are set per-metric in MetricDefinition and are not runtime-configurable.

Composite history length

The make backfill-composite target defaults to 252 business days (approximately one trading year). You can override this when calling the CLI directly:

cd backend && $(VENV)/bin/python -m tide.ingest.cli backfill-composite --days 504

The Makefile target always uses the 252-day default.

Manual refresh

Run these three commands in order. Each is idempotent — re-running any of them is safe and will not corrupt existing data.

# 1. Pull all live metrics from upstream sources into DuckDB
make ingest-all

# 2. Pull daily closes for every watchlist ticker and the XLV benchmark
make ingest-watchlist

# 3. Recompute the 252-business-day composite history
make backfill-composite

After all three commands complete, reload the TIDE dashboard in your browser. The composite reading and all metric cards will reflect the latest data.

Why three separate commands? Separating ingest, watchlist, and composite backfill means a failure in one step does not silently corrupt the others. If ingest-all fails partway through, your existing observations remain intact and you can re-run only the failing step.

Automated refresh with the Scheduler

To have TIDE refresh itself automatically after market close, start the Scheduler as a long-running background process:

make scheduler

The Scheduler runs two APScheduler cron jobs in the America/New_York timezone:

The Scheduler is a separate process from the FastAPI server — a stalled or failing ingest job cannot take the dashboard down.

Scheduler state (last run time, last success, last error, next run time) is persisted to the scheduler_status DuckDB table and is visible on the Sources page at /sources.

Running the Scheduler once (smoke test)

To verify that every job body runs without error before committing to a long-running process, use:

make scheduler-once

This runs each job function exactly once, sequentially, then exits. It does not start the background cron loop.

Example 1 — Full manual refresh from scratch

A complete manual update sequence starting from an already-initialised database.

export FRED_API_KEY=abcdef1234567890
make ingest-all
make ingest-watchlist
make backfill-composite

Expected output (abbreviated):

  [pull] m2 ← FRED
  [ok]   m2: 780 obs (2022-01-03 → 2025-06-13)
  [pull] hy_spread ← FRED
  [ok]   hy_spread: 365 obs (2024-06-14 → 2025-06-13)
  ...
  [pull] wl_XLV ← Yahoo · XLV
  [ok]   wl_XLV: 615 obs (2023-01-03 → 2025-06-13)
  ...
  [ok] composite history: 252 business-day points written

Example 2 — Starting the Scheduler as a background service

export FRED_API_KEY=abcdef1234567890
make scheduler

Expected output on startup:

  [scheduler] started
  [scheduler] daily_ingest         Mon–Fri 17:00 America/New_York
  [scheduler] release_ingest       Fri 18:00 America/New_York

The process then runs silently until 17:00 ET on a weekday (or 18:00 ET on Friday), at which point you will see log lines like:

2025-06-16 17:00:02 INFO  tide.scheduler: [scheduler] daily_ingest done — metrics 11 ok / 0 fail, watchlist 9 ok / 0 fail, composite 252 pts

Send SIGINT (Ctrl-C) or SIGTERM to shut the Scheduler down gracefully.

Example 3 — Smoke-testing the Scheduler before going live

export FRED_API_KEY=abcdef1234567890
make scheduler-once

Expected output:

  [scheduler] --fire-once: running each job body sequentially
  [scheduler] --fire-once: done

Check the Sources page at /sources immediately after — both jobs should show a last_success_at timestamp.

Example 4 — Checking Scheduler status on the Sources page

With the dashboard running (make dev-api and make dev-web in separate terminals), open:

http://localhost:5173/sources

The Sources page reads from the scheduler_status DuckDB table and displays each job's last run time, last success, next scheduled run, and any error message from the most recent failure.

Example 5 — Overriding the virtual environment path

If your virtual environment is not at the default path, set VENV before any make call:

export VENV=/home/you/.venvs/tide
export FRED_API_KEY=abcdef1234567890
make ingest-all

FRED metrics fail with an authentication error

Symptom: ingest-all prints [fail] m2: 403 or API key missing for one or more FRED metrics.

Cause: The FRED_API_KEY environment variable is not set or is set to an invalid value.

Fix: Export a valid key before running any ingest command:

export FRED_API_KEY=your_key_here
make ingest-all

Obtain a free key at fred.stlouisfed.org. The key must be in scope for every shell session that runs ingest or the Scheduler.

make ingest-all completes but the dashboard still shows stale readings

Symptom: Metric cards on the dashboard show old as-of dates even after a successful ingest-all.

Cause: You ran ingest-all and ingest-watchlist but skipped make backfill-composite. The composite history and per-tier z-scores are not recomputed automatically after ingest.

Fix: Always run all three steps in order:

make ingest-all
make ingest-watchlist
make backfill-composite

Some metrics are always — (stubbed) and never update

Symptom: The metrics ici_etf_flows, buyback_yield, put_call, and uvol_dvol never show live readings.

Cause: These four metrics are intentionally stubbed — they are registered in the MetricDefinition registry but have no ingest_fn. Their free data sources are not yet integrated.

Fix: This is expected behaviour. The per-tier __init__.py docstrings document the unblock paths for each stubbed metric. Until they are unblocked, the dashboard shows the last known value (or no value) with a clear as-of date so you always know the reading is not current.

The Scheduler process starts but the Sources page shows no last_run_at

Symptom: make scheduler starts without error, but /sources shows null values for both jobs.

Cause: The Scheduler has not yet fired — it is waiting for the next scheduled time (17:00 ET Monday–Friday for daily_ingest, 18:00 ET Friday for release_ingest).

Fix: Use make scheduler-once to run both job bodies immediately and verify everything works:

make scheduler-once

Then recheck /sources. If last_success_at is still null after scheduler-once, check the terminal output for [fail] lines and resolve any upstream errors before restarting the long-running Scheduler.

make scheduler exits immediately instead of staying alive

Symptom: Running make scheduler prints the startup lines and then returns to the shell prompt within a second or two.

Cause: The process received a signal (e.g. your shell sent SIGHUP when a terminal closed), or you accidentally ran make scheduler-once.

Fix: Run make scheduler in a persistent session (a tmux or screen pane, or configure it as a systemd user service). Do not confuse make scheduler (long-running) with make scheduler-once (exits immediately after one run).

Watchlist tickers show no sparklines or price data

Symptom: The watchlist section of the dashboard is empty or shows dashes for all tickers.

Cause: make ingest-watchlist has not been run, or the Yahoo Finance HTTP endpoint (query1.finance.yahoo.com/v8/finance/chart/) was unreachable during the last run.

Fix: Run make ingest-watchlist and check the output for [fail] lines. Note that TIDE fetches Yahoo Finance data by hitting the chart endpoint directly over HTTP — it does not use the yfinance library. Transient network errors are safe to retry by re-running the command.

make backfill-composite reports 0 points written

Symptom: The command completes with composite history: 0 business-day points written.

Cause: The observations table is empty — no metrics have been ingested yet.

Fix: Run make ingest-all first, then make ingest-watchlist, and finally make backfill-composite. The composite computation requires at least some live metric observations to be present in DuckDB.