---
title: Data Sources
product: tide
doc_type: guide
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/wabam/docs/tide/tide-capital-pressure-observatory/data-sources
---

# Data Sources

_Upstream data providers, cadence, and known limitations_

## Overview

This page documents every upstream data provider that TIDE uses, the cadence at which each publishes new data, and the known limitations you will encounter in production. Understanding your data sources matters because different providers publish on different schedules — some daily, some weekly, some monthly — and TIDE shows stale readings rather than hiding them, so you need to know what "stale" means for each metric. All sources are free-tier only; no paid data vendors are used or planned.

## Prerequisites

Before reading this page you should already have TIDE running. If you have not completed initial setup, do that first. For context on this page specifically:

- **FRED API key** — required for the six FRED-sourced metrics. Get one free at [fred.stlouisfed.org](https://fred.stlouisfed.org/docs/api/api_key.html) and export it as `FRED_API_KEY` in your environment.
- **Python ≥ 3.10** — the ingest layer is pure Python; older versions are not supported.
- **`xlrd` package** — required to parse the AAII binary `.xls` file. It is included in the backend dependencies, but if you see an import error, install it manually: `pip install xlrd`.
- **Internet access** — all sources are fetched live over HTTPS (with one exception noted below under CFTC).
- **`make`** — the primary interface for running ingestion commands.

## Configuration

Source behavior is controlled by a combination of environment variables and per-metric settings in the `MetricDefinition` registry.

### Environment variables

| Variable | Default | Effect |
|---|---|---|
| `FRED_API_KEY` | *(none)* | Required. Authenticates requests to the FRED API. Without it, all FRED-sourced metrics fail with `FredKeyMissing`. |
| `VENV` | *(project default)* | Overrides the Python virtual environment path used by `make` targets. |
| `PORT` | *(project default)* | Overrides the port the TIDE Dashboard listens on. |

### Per-metric z-score window (`zscore_years`)

Z-scores are computed over a **3-year rolling window by default**. Individual metrics can override this via `MetricDefinition.zscore_years`. The one currently configured override is the HY spread series (`BAMLH0A0HYM2`): FRED truncated this series to a 3-year window in April 2026, so TIDE sets `zscore_years=1y` for that metric to match the available history. If you extend TIDE with new metrics, set a shorter `zscore_years` for any series with limited history.

### Stubbed metrics

Four metrics are registered in the `MetricDefinition` registry but have no `ingest_fn` and therefore never fetch live data: `ici_etf_flows`, `buyback_yield`, `put_call`, and `uvol_dvol`. They appear on the Sources page with a **stub** status badge. The per-tier `__init__.py` docstrings document what free data source would unblock each one. Do not expect these metrics to show readings until you implement their ingest functions.

### Watchlist (`watchlist.toml`)

The watchlist is configured in `watchlist.toml` at the project root. It controls which tickers appear in the dashboard watchlist panel and which ETF is used as the benchmark for relative return calculations.

| Field | Type | Cadence | Description |
|---|---|---|---|
| `benchmark` | string | — | The sector ETF used for relative-return comparison. Defaults to `"XLV"`. |
| `ticker` | string | — | Yahoo Finance ticker symbol for the stock. |
| `target_price` | float | Quarterly | Your personal price target. Not fetched automatically. |
| `fwd_pe` | string | Quarterly | Forward P/E as a display string (e.g. `"12.4×"`). Set manually — free-tier Yahoo Finance does not expose this field without crumb authentication. |
| `status` | string | As needed | One of `"watching"`, `"starter"`, or `"building"`. |

Live fields (`current price`, `30-day return`, `relative return`, `sparkline`) are refreshed automatically by `make ingest-watchlist`. Manual fields (`target_price`, `fwd_pe`, `status`) must be edited in `watchlist.toml` directly and do not have an ingest path.