Data Sources

This page documents every upstream data provider that TIDE uses, the cadence at which each publishes new data, and the known limitations you will encounter in production. Understanding your data sources matters because different providers publish on different schedules — some daily, some weekly, some monthly — and TIDE shows stale readings rather than hiding them, so you need to know what "stale" means for each metric. All sources are free-tier only; no paid data vendors are used or planned.

Before reading this page you should already have TIDE running. If you have not completed initial setup, do that first. For context on this page specifically:

• FRED API key — required for the six FRED-sourced metrics. Get one free at fred.stlouisfed.org and export it as FRED_API_KEY in your environment.
• Python ≥ 3.10 — the ingest layer is pure Python; older versions are not supported.
• xlrd package — required to parse the AAII binary .xls file. It is included in the backend dependencies, but if you see an import error, install it manually: pip install xlrd.
• Internet access — all sources are fetched live over HTTPS (with one exception noted below under CFTC).
• make — the primary interface for running ingestion commands.

Data source support is built into the backend and requires no separate installation beyond the standard TIDE setup. The steps below confirm that each source is reachable and correctly configured.

1. Set your FRED API key.

   Copy

   export FRED_API_KEY=your_key_here
   

   Add this to your shell profile or .env file so it persists across sessions. Without it, TIDE raises FredKeyMissing when it tries to fetch any FRED series and those metrics will not update.

2. Run a full ingest to verify all sources.

   Copy

   make ingest
   

   This calls tide-ingest for every live metric in sequence. Watch the output for per-source success or error messages.

3. Run the watchlist ingest separately.

   Copy

   make ingest-watchlist
   

   Watchlist tickers are fetched from the Yahoo Finance chart API independently of the macro indicators.

4. Check the Sources page to confirm ingest results.

   Open http://localhost:<PORT>/sources in your browser. Every live metric should show a recent Latest obs date and a live status badge. Stubbed metrics show a stub badge and will never have observation data until their ingest functions are implemented.

5. (Optional) Start the scheduler for automatic daily refresh.

   Copy

   make scheduler
   

   The scheduler runs two APScheduler cron jobs in the America/New_York timezone: daily_ingest Monday–Friday at 17:00, and release_ingest on Fridays at 18:00. Scheduler state is persisted to the scheduler_status DuckDB table and is visible on the Sources page.

   To run a single one-off batch without starting the persistent scheduler:

   Copy

   make scheduler-once
   

Source behavior is controlled by a combination of environment variables and per-metric settings in the MetricDefinition registry.

Environment variables

Per-metric z-score window (zscore_years)

Z-scores are computed over a 3-year rolling window by default. Individual metrics can override this via MetricDefinition.zscore_years. The one currently configured override is the HY spread series (BAMLH0A0HYM2): FRED truncated this series to a 3-year window in April 2026, so TIDE sets zscore_years=1y for that metric to match the available history. If you extend TIDE with new metrics, set a shorter zscore_years for any series with limited history.

Stubbed metrics

Four metrics are registered in the MetricDefinition registry but have no ingest_fn and therefore never fetch live data: ici_etf_flows, buyback_yield, put_call, and uvol_dvol. They appear on the Sources page with a stub status badge. The per-tier __init__.py docstrings document what free data source would unblock each one. Do not expect these metrics to show readings until you implement their ingest functions.

Watchlist (watchlist.toml)

The watchlist is configured in watchlist.toml at the project root. It controls which tickers appear in the dashboard watchlist panel and which ETF is used as the benchmark for relative return calculations.

Live fields (current price, 30-day return, relative return, sparkline) are refreshed automatically by make ingest-watchlist. Manual fields (target_price, fwd_pe, status) must be edited in watchlist.toml directly and do not have an ingest path.

Checking source health

Open the Sources page at /sources in the TIDE Dashboard. It shows a table of every metric with its tier, upstream provider, publication cadence, latest observation date, last ingest time, observation count, and live/stub status. Staleness is color-coded against each source's expected cadence: a daily metric flagged stale after 5 days, a weekly metric after 14 days, a monthly metric after 60 days, and a quarterly metric after 200 days.

The top of the Sources page also shows the Scheduler panel. If the scheduler is running, each job shows its last run time, last successful run, next scheduled run, and any error from the most recent attempt.

Refreshing data manually

Run a full refresh of all macro indicators:

make ingest

Run only the watchlist refresh:

make ingest-watchlist

You can run these at any time. The dashboard reflects the new data as soon as tide-ingest writes to the DuckDB file — no restart of the dashboard process is needed, because the backend reads the database on each request.

Interpreting as-of dates

Every metric card on the main dashboard shows an as of date. Because sources publish on different schedules, the as-of dates across cards will differ. For example, FRED's M2 money supply is monthly and lags by several weeks, while VIX from Yahoo Finance updates daily. TIDE always shows the last known reading for every metric, even when that reading is days or weeks old. The Sources page is the authoritative place to see exactly how fresh each reading is.

Setting up automatic refresh

Start the scheduler to avoid running make ingest manually each day:

make scheduler

The scheduler runs in the background and triggers ingestion automatically after market close each weekday. You can confirm it is active by visiting /sources and checking that the Scheduler panel shows a next run time for each job.

Example 1 — Run a full ingest and watch the output

make ingest

Expected output (abbreviated):

[tide-ingest] m2_yoy          OK  2024-12-01  obs=420
[tide-ingest] fed_balance      OK  2025-01-15  obs=1012
[tide-ingest] hy_spread        OK  2025-01-16  obs=780
[tide-ingest] nfci             OK  2025-01-10  obs=2860
[tide-ingest] loan_growth      OK  2024-12-01  obs=960
[tide-ingest] margin_debt      OK  2024-11-01  obs=360
[tide-ingest] move_index       OK  2025-01-16  obs=6800
[tide-ingest] vix_spot         OK  2025-01-16  obs=8700
[tide-ingest] vix3m            OK  2025-01-16  obs=3100
[tide-ingest] spy_return       OK  2025-01-16  obs=8800
[tide-ingest] tlt_return       OK  2025-01-16  obs=5800
[tide-ingest] hyg_return       OK  2025-01-16  obs=4200
[tide-ingest] dix              OK  2025-01-15  obs=3400
[tide-ingest] gex              OK  2025-01-15  obs=3400
[tide-ingest] aaii_bull_bear   OK  2025-01-16  obs=1960
[tide-ingest] cot_sp_spec_net  OK  2025-01-14  obs=780
[tide-ingest] foreign_eq_hold  OK  2023-01-01  obs=264
[tide-ingest] ici_etf_flows    STUB  (no ingest_fn)
[tide-ingest] put_call         STUB  (no ingest_fn)

Stubbed metrics are logged but not treated as errors.

Example 2 — Refresh only the watchlist

make ingest-watchlist

Expected output:

[tide-ingest] watchlist MRK    OK  close=92.14  rel_30d=+1.8pp vs XLV
[tide-ingest] watchlist UNH    OK  close=318.50 rel_30d=-2.1pp vs XLV
[tide-ingest] watchlist TMO    OK  close=504.20 rel_30d=+0.4pp vs XLV
...
[tide-ingest] watchlist MDT    OK  close=87.33  rel_30d=-0.9pp vs XLV

The benchmark (XLV) is fetched first; each ticker's relative return is computed against it.

Example 3 — Start the scheduler and confirm it is running

make scheduler &

Then open /sources in your browser. The Scheduler panel should show:

daily_ingest    Mon–Fri 17:00 ET    last run: —    next run: today 17:00
release_ingest  Fri 18:00 ET        last run: —    next run: Fri 18:00

After the first scheduled run completes, the last run and last success columns populate.

Example 4 — Verify FRED connectivity before a full ingest

export FRED_API_KEY=your_key_here
tide-ingest --metric m2_yoy

If the key is missing or invalid, you will see:

FredKeyMissing: FRED_API_KEY not set. Get one at https://fred.stlouisfed.org/docs/api/api_key.html

If the key is valid, the command exits cleanly and updates only the m2_yoy metric.

FredKeyMissing on ingest

Symptom: Running make ingest prints FredKeyMissing: FRED_API_KEY not set and all FRED metrics fail.

Cause: The FRED_API_KEY environment variable is not exported in the shell where you ran make.

Fix: Export the variable before running make:

export FRED_API_KEY=your_key_here
make ingest

To persist it, add the export to your shell profile or to a .env file that your environment loads automatically.

AAII fetch returns a 403 or the xls check fails

Symptom: The AAII ingest logs AAII fetch failed after 4 attempts or AAII returned non-xls content (likely 403 page).

Cause: AAII's server intermittently blocks automated clients with a 403 response. The ingest layer retries up to four times with exponential backoff and rotates browser-shaped headers, but some request windows still fail.

Fix: Wait a few minutes and re-run make ingest. AAII publishes weekly (each Thursday), so a single failed daily attempt does not cause data loss — the previous week's value remains in DuckDB and is shown as a stale reading on the dashboard until the next successful fetch.

CFTC fetch fails with a TLS certificate error

Symptom: The COT ingest prints a TLS verification error referencing cftc.gov.

Cause: cftc.gov serves a certificate chain that fails validation on many CA bundles. This is a known issue with the CFTC's infrastructure.

Fix: No action required. TLS verification is intentionally disabled for cftc.gov requests only — this is set in the source code (verify=False applies only to cftc.gov; all other upstream calls use standard TLS). If you are seeing this error despite that setting, confirm you are running the current version of the backend.

Treasury TIC foreign_eq_hold metric is stuck at 2023-01

Symptom: The Sources page shows foreign_eq_hold with a latest observation of 2023-01-01 and the reading never advances, even after repeated ingests.

Cause: The static file at https://ticdata.treasury.gov/Publish/slt1d_globl.csv is capped at January 2023. Treasury moved newer monthly releases to per-release URLs that require scraping the TIC press-release pages — a pattern TIDE does not yet implement.

Fix: This is a known limitation of the free static endpoint. The metric will continue to show the last known value (January 2023) rather than hiding the reading. If you need current TIC data, you must implement per-release URL scraping against the Treasury TIC website and update the treasury_tic.py ingest function accordingly.

HY spread z-score looks compressed compared to other metrics

Symptom: The hy_spread metric z-score is consistently smaller in magnitude than you expect relative to its historical behavior.

Cause: In April 2026, FRED truncated the BAMLH0A0HYM2 series to a 3-year window. TIDE responds by setting zscore_years=1y for this metric so the rolling window matches the available data. A 1-year window is shorter than the 3-year default and will produce different z-score magnitudes.

Fix: This is expected behavior. The z-score is computed correctly given the data FRED now provides. If FRED restores the full series in the future, you can update MetricDefinition.zscore_years for hy_spread to match.

Yahoo Finance fetch fails for a ticker or index symbol

Symptom: An ingest for VIX, MOVE, SPY, TLT, HYG, or a watchlist ticker fails with YahooFetchError.

Cause: TIDE fetches Yahoo Finance data by calling https://query1.finance.yahoo.com/v8/finance/chart/<symbol> directly over HTTP. Yahoo occasionally changes response shapes, rate-limits aggressively, or returns transient errors. Note that yfinance is intentionally not a dependency — TIDE calls the endpoint directly to avoid yfinance's recurring auth/crumb breakage.

Fix: Wait a few minutes and retry. The fetch logic retries up to three times with exponential backoff. If failures persist for a specific symbol, verify the symbol is correct in the registry or watchlist.toml. Do not attempt to switch to yfinance — it is not a supported dependency.

Sources page shows the scheduler as "not started"

Symptom: The Scheduler panel on /sources shows not started and no jobs are listed.

Cause: The scheduler process is not running. It is an optional component that must be started explicitly.

Fix: Start the scheduler:

make scheduler

Or run a single one-off batch without starting the persistent scheduler:

make scheduler-once

The scheduler panel will update to show running once the process is active and has written its state to the scheduler_status DuckDB table.