Adding a New Metric

This page walks you through adding a new indicator to TIDE from scratch — writing the ingest function, implementing the compute function, registering the metric definition, and wiring everything into the dashboard. Every indicator in TIDE follows the same four-file pattern (one module per metric, one import in the tier's __init__.py, one make init-db call, and one tide-ingest run call), so once you have done it once you can add subsequent metrics in minutes. Understanding the pattern also lets you audit and extend any of the 15 live metrics already in the codebase.

Before you begin, make sure you have:

• Python ≥ 3.10 with the TIDE virtual environment activated (the Makefile uses VENV to locate it).
• A configured .env file at the project root containing at least FRED_API_KEY=<your_key> if your new metric pulls from FRED. Get a free key at fred.stlouisfed.org.
• DuckDB initialised — run make init-db at least once before you begin so the MetricDefinition registry table exists.
• make available on your PATH.
• Familiarity with Python dataclasses and a basic understanding of what a z-score represents (see the Key Concepts page).
• Decided which of the four tiers your metric belongs to:
  • Tier 1 — Macro Liquidity
  • Tier 2 — Capital Flows
  • Tier 3 — Market Microstructure
  • Tier 4 — Sentiment & Positioning

Follow these steps in order. Replace <tier_number> with 1, 2, 3, or 4, and <metric_id> with a short, lowercase, underscore-separated identifier (e.g. corp_bond_issuance).

Step 1 — Create the metric module

Create a new file at:

backend/tide/metrics/tier<tier_number>/<metric_id>.py

For example, for a Tier 2 metric with ID corp_bond_issuance:

touch backend/tide/metrics/tier2/corp_bond_issuance.py

Step 2 — Implement _ingest()

Add an _ingest() function that fetches raw data from an upstream free-tier source and returns a list of (date, float) tuples sorted in ascending date order. TIDE has ready-made source helpers for FRED, Yahoo Finance (via direct HTTP, not yfinance), CFTC, AAII, and Treasury TIC.

from __future__ import annotations
from datetime import date

def _ingest() -> list[tuple[date, float]]:
    from tide.ingest.sources.fred import fetch_series
    return fetch_series("<FRED_SERIES_ID>", start="2000-01-01")

For Yahoo Finance data, use the yahoo source helper which hits query1.finance.yahoo.com/v8/finance/chart/ directly — do not add yfinance as a dependency.

Step 3 — Implement _compute()

Add a _compute() function that reads stored observations from DuckDB and returns a populated Reading dataclass (or None if not enough data is available yet).

Choose level_series if your raw value is already the indicator, or yoy_series if you want the year-over-year percentage change:

from typing import Optional
import duckdb
from tide.compute.zscore import latest_with_z, level_series   # or yoy_series
from tide.metrics.base import Reading, Vote

def _compute(conn: duckdb.DuckDBPyConnection) -> Optional[Reading]:
    points = level_series(conn, "<metric_id>", window=252)
    latest = latest_with_z(points)
    if latest is None:
        return None

    prior_idx = max(0, len(points) - 66)  # ~13 weeks of trading days
    delta = latest.indicator - points[prior_idx].indicator
    sparkline = [p.indicator for p in points[-60:]]

    z = float(latest.z)
    # Choose direction_kind to suit the metric (see Configuration section).
    # For an inverted metric (low value = bull), flip the sign:
    # directional = -z
    directional = z

    if directional > 0.3:
        direction, reason = "bull", "<explain why this reading is bullish>"
    elif directional < -0.5:
        direction, reason = "bear", "<explain why this reading is bearish>"
    else:
        direction, reason = "neutral", "<explain near-mean interpretation>"

    z_class = "pos" if directional > 0.3 else ("neg" if directional < -0.3 else "neu")

    return Reading(
        metric_id="<metric_id>",
        name="<Display Name>",
        tier=<tier_number>,
        source="<Source Label>",
        unit="<unit string>",
        value=f"{latest.indicator:+.2f}",
        delta=f"{delta:+.2f} 13W",
        delta_class="pos" if delta >= 0 else "neg",
        z=z,
        z_label=f"{z:+.1f}\u03c3",
        z_class=z_class,
        directional_z=directional,
        as_of=latest.ts,
        sparkline=sparkline,
        vote=Vote(direction=direction, reason=reason),
    )

Step 4 — Register the MetricDefinition

At the bottom of the same file, call register() with a fully populated MetricDefinition. The four fields indicator_kind, indicator_lag, indicator_window, and direction_kind are required for the composite history backfill to work correctly:

from tide.metrics import register
from tide.metrics.base import MetricDefinition

register(MetricDefinition(
    id="<metric_id>",
    name="<Display Name>",
    tier=<tier_number>,
    source="<Source Label>",
    source_kind="fred",          # fred | yahoo | cftc | aaii | treasury_tic | squeezemetrics | derived
    source_series="<SERIES_ID>",
    unit="<unit string>",
    cadence="daily",             # daily | weekly | monthly | quarterly
    zscore_years=3,              # default; override per-metric if needed
    sort_order=99,
    indicator_kind="level",      # REQUIRED — 'level' or 'yoy'
    indicator_lag=0,             # REQUIRED — only used when indicator_kind == 'yoy'
    indicator_window=252,        # REQUIRED — rolling window in observations
    direction_kind="natural",    # REQUIRED — 'natural' | 'inverted' | 'contrarian_long'
    description=(
        "One to three plain-language sentences: what it measures, why it matters, "
        "and how to read the sign."
    ),
    ingest_fn=_ingest,
    compute_fn=_compute,
))

Step 5 — Add the import to the tier's __init__.py

Open backend/tide/metrics/tier<tier_number>/__init__.py and add your module to the import list. The __init__.py docstring also documents the unblock paths for any stubbed metrics in that tier — read it before editing.

from . import <metric_id>  # noqa: F401  — triggers register() call

Step 6 — Seed the registry

Run make init-db to insert your new MetricDefinition into the DuckDB registry table:

make init-db

Step 7 — Pull data

Use the tide-ingest CLI to fetch observations for your new metric:

tide-ingest run --metric <metric_id>

Step 8 — Backfill the composite history

After ingest succeeds, recalculate the 252-business-day composite history so your new indicator is included in the chart:

make backfill-composite

Your metric now appears in the dashboard the next time you load or refresh it.

All configuration for a metric lives in its MetricDefinition dataclass. The fields below have the most impact on how the dashboard calculates and displays your indicator.

indicator_kind

This field is used by composite_history to recompute directional z-scores for every historical date without re-running _compute() for each one.

indicator_lag

Only consulted when indicator_kind == "yoy". Set it to the number of observations that represent one year:

• Weekly series → 52
• Monthly series → 12
• Quarterly series → 4
• Daily series → 252

indicator_window

The rolling window (in observations) used to compute the z-score. The default zscore_years=3 translates to:

• Daily → 756 observations
• Weekly → 156 observations
• Monthly → 36 observations
• Quarterly → 12 observations

You can override zscore_years per metric when historical data is limited. For example, hy_spread uses zscore_years=1 and indicator_window=252 because FRED truncated BAMLH0A0HYM2 to a rolling 3-year window in April 2026.

direction_kind

Controls how the raw z-score is converted to a directional_z (bull-positive) for the composite:

The composite averages directional_z across all four tier averages, not the raw z. Choosing the wrong direction_kind will flip your metric's contribution to the composite score.

include_in_composite

Defaults to True. Set to False for "loaded spring" or fragility gauges that convey magnitude of risk rather than a directional bull/bear signal. The metric will still appear as a tier card and count in the vote tally — it just will not move the composite z-score.

zscore_years

Default 3. Overriding this changes both the z-score window and, by extension, how far back _compute() requests data. Match this to the realistic depth of the upstream series.

cadence

A display string ("daily", "weekly", "monthly", "quarterly") shown on the Sources page. It does not drive scheduling logic — the Scheduler always runs tide-ingest for all metrics regardless of cadence.

Environment variables

Two environment variables affect all metrics globally (set in .env or your shell):

The Makefile also honours VENV (path to the virtual environment) and PORT (API server port).

Once your metric module is wired in, you interact with it through two tools: the tide-ingest CLI and make targets.

Pulling data for a single metric

During development, refresh only the metric you are working on to keep iteration fast:

tide-ingest run --metric <metric_id>

This calls your _ingest() function, writes the returned (date, float) pairs to DuckDB, and then calls _compute() to verify the reading resolves correctly. If _compute() returns None, the CLI prints a warning — you likely need more historical data or a wider indicator_window.

Pulling data for all metrics

Once you are satisfied with the new metric, run a full ingest to keep all indicators in sync:

tide-ingest run

Rebuilding the composite history

The 252-business-day composite history chart is pre-computed and stored in DuckDB. Run the backfill after any ingest that adds or changes a metric:

make backfill-composite

Skipping this step means your new metric will appear in the live composite reading but will not yet be reflected in the historical chart.

Checking the result in the dashboard

Open the TIDE Dashboard in your browser (default http://127.0.0.1:8000 — see PORT in your Makefile). Your new metric card should appear in the correct tier with its current value, z-score label, vote chip, and sparkline. The as-of date on the card reflects the most recent observation returned by _ingest().

If the metric card is missing, check that the import was added to the tier's __init__.py and that make init-db was re-run after the change.

Verifying source status

Navigate to the Sources page at /sources in the dashboard. It shows when each data source last ran, when it will next run, and any errors. A newly added metric appears here automatically once its first ingest completes.

Automating daily refresh

The Scheduler runs tide-ingest automatically after market close each weekday. Start it with:

make scheduler

The Scheduler fires two APScheduler cron jobs in the America/New_York timezone: daily_ingest Monday–Friday at 17:00, and release_ingest on Fridays at 18:00 (for data that publishes after the close). Your new metric is included in both jobs without any additional configuration. Scheduler state persists to the scheduler_status DuckDB table and is visible on the Sources page.

Example 1 — A natural-direction FRED level metric (daily)

This example adds a hypothetical "10-Year Real Yield" metric using FRED series DFII10.

backend/tide/metrics/tier1/real_yield_10y.py

"""10-Year Real Yield — FRED DFII10 (daily).

Indicator: raw level (inflation-adjusted yield in %). INVERTED: falling real
yields = looser financial conditions = bull.
"""
from __future__ import annotations

from datetime import date
from typing import Optional

import duckdb

from tide.compute.zscore import latest_with_z, level_series
from tide.metrics import register
from tide.metrics.base import MetricDefinition, Reading, Vote

def _ingest() -> list[tuple[date, float]]:
    from tide.ingest.sources.fred import fetch_series
    return fetch_series("DFII10", start="2003-01-01")

def _compute(conn: duckdb.DuckDBPyConnection) -> Optional[Reading]:
    points = level_series(conn, "real_yield_10y", window=756)
    latest = latest_with_z(points)
    if latest is None:
        return None

    prior_idx = max(0, len(points) - 66)  # ~13 weeks
    delta = latest.indicator - points[prior_idx].indicator
    sparkline = [p.indicator for p in points[-60:]]

    z = float(latest.z)
    directional = -z  # INVERTED — falling real yield = bull

    if directional > 0.3:
        direction, reason = "bull", "Real yields falling — easier financial conditions"
    elif directional < -0.5:
        direction, reason = "bear", "Real yields rising — tightening pressure on risk assets"
    else:
        direction, reason = "neutral", "Real yields near long-run mean"

    z_class = "pos" if directional > 0.3 else ("neg" if directional < -0.3 else "neu")

    return Reading(
        metric_id="real_yield_10y",
        name="10Y Real Yield",
        tier=1,
        source="FRED \u00b7 DFII10",
        unit="% · real",
        value=f"{latest.indicator:+.2f}%",
        delta=f"{delta:+.2f}pp 13W",
        delta_class="pos" if delta <= 0 else "neg",
        z=z,
        z_label=f"{z:+.1f}\u03c3",
        z_class=z_class,
        directional_z=directional,
        as_of=latest.ts,
        sparkline=sparkline,
        vote=Vote(direction=direction, reason=reason),
    )

register(MetricDefinition(
    id="real_yield_10y",
    name="10Y Real Yield",
    tier=1,
    source="FRED \u00b7 DFII10",
    source_kind="fred",
    source_series="DFII10",
    unit="% · real",
    cadence="daily",
    zscore_years=3,
    sort_order=60,
    indicator_kind="level",
    indicator_lag=0,
    indicator_window=756,
    direction_kind="inverted",
    description=(
        "The yield on 10-year Treasury Inflation-Protected Securities (TIPS) — what "
        "you earn above inflation by lending to the US government for a decade. Rising "
        "real yields make stocks and bonds less attractive relative to Treasuries; "
        "falling real yields encourage risk-taking. Negative z-score on this card is "
        "bullish because falling yields ease financial conditions."
    ),
    ingest_fn=_ingest,
    compute_fn=_compute,
))

Add to backend/tide/metrics/tier1/__init__.py:

from . import real_yield_10y  # noqa: F401

Seed, ingest, and backfill:

make init-db
tide-ingest run --metric real_yield_10y
make backfill-composite

Expected terminal output (abridged):

[init-db] Registered 20 metrics in MetricDefinition registry.
[ingest] real_yield_10y: fetched 5480 observations from FRED · DFII10
[ingest] real_yield_10y: compute OK — as_of=2025-07-14, z=+0.6σ, vote=neutral
[backfill] Recalculating composite history for 252 business days... done.

Example 2 — A YoY metric with a quarterly FRED series

This mirrors the pattern used by margin_debt. For a quarterly series, set indicator_lag=4 and indicator_window=12.

Key excerpt (registration block only):

register(MetricDefinition(
    id="nonfinancial_credit",
    name="Nonfinancial Sector Credit",
    tier=2,
    source="FRED \u00b7 QUSPAMUSDA",
    source_kind="fred",
    source_series="QUSPAMUSDA",
    unit="YoY · quarterly",
    cadence="quarterly",
    zscore_years=3,
    sort_order=30,
    indicator_kind="yoy",
    indicator_lag=4,       # 4 quarters = 1 year
    indicator_window=12,   # 12 quarters = 3 years
    direction_kind="natural",
    description=(
        "Total credit outstanding to the US nonfinancial sector from the BIS. Growing "
        "faster than trend means credit is expanding and supporting economic activity. "
        "Shown as year-over-year change; quarterly data with a ~1-quarter lag."
    ),
    ingest_fn=_ingest,
    compute_fn=_compute,
))

Ingest and verify:

tide-ingest run --metric nonfinancial_credit

Expected output:

[ingest] nonfinancial_credit: fetched 136 observations from FRED · QUSPAMUSDA
[ingest] nonfinancial_credit: compute OK — as_of=2025-01-01, z=-0.3σ, vote=neutral

Because this is a quarterly series, the as_of date will be several months in the past — this is expected stale-data behaviour. TIDE shows the last known value with its as-of date rather than hiding it.

Example 3 — Verifying the metric appears on the Sources page

After running make scheduler (or waiting for the next automatic run), open the dashboard and navigate to /sources:

http://127.0.0.1:8000/sources

You should see a row for your new metric showing:

• Last run: the timestamp of your manual tide-ingest run --metric <id> call
• Next run: the next weekday at 17:00 America/New_York
• Status: ok

If Status shows an error, see the Troubleshooting section below.

Use the following format for each issue: Symptom → Likely cause → Fix.

Symptom: FredKeyMissing: FRED_API_KEY not set when running tide-ingest run --metric <id>.

Likely cause: The .env file at the project root is missing the FRED_API_KEY entry, or it was not loaded into the current shell session.

Fix: Add the key to .env:

FRED_API_KEY=your_key_here

Then re-run the ingest command. You do not need to restart the dashboard; tide-ingest reads the environment fresh on each run.

Symptom: tide-ingest run --metric <id> prints compute OK but the metric card does not appear in the dashboard.

Likely cause: The import was not added to the tier's __init__.py, so register() was never called and the MetricDefinition was not written to the DuckDB registry.

Fix: Verify that from . import <metric_id> appears in backend/tide/metrics/tier<N>/__init__.py, then re-run make init-db and reload the dashboard.

Symptom: _compute() returns None and the card shows no reading.

Likely cause: Not enough stored observations to satisfy indicator_window. For example, a 3-year window (window=756) on a series that has only 200 stored rows cannot produce a z-score.

Fix: Either lower indicator_window and zscore_years in the MetricDefinition to match the available history, or fetch data from an earlier start date in _ingest(). Re-run tide-ingest run --metric <id> after making the change.

Symptom: The composite score does not change after adding the new metric.

Likely cause: make backfill-composite was not run after the ingest completed, so the pre-computed composite history table is still based on the old set of metrics.

Fix:

make backfill-composite

Refresh the dashboard. The composite history chart and the live composite z-score should now reflect the new indicator.

Symptom: The metric card is coloured green (bullish) when you expect it to be red, or vice versa.

Likely cause: direction_kind is set to "natural" for a metric where a high value is actually bearish (e.g. a spread or volatility index), or "inverted" when a high value is actually bullish.

Fix: Update direction_kind in the MetricDefinition to match the metric's economic convention:

• High value = bull → "natural"
• High value = bear → "inverted"
• Contrarian (extreme in either direction = opposite signal) → "contrarian_long"

Also ensure directional_z in _compute() is consistent: for "inverted" metrics, assign directional = -z; for "natural" metrics, assign directional = z. Re-run make init-db, re-ingest, and backfill after the fix.

Symptom: make init-db reports the same number of registered metrics as before — the new metric is not listed.

Likely cause: A syntax error or import-time exception in the new module prevented Python from executing the register() call. Errors during import are often silently swallowed unless you inspect the process output carefully.

Fix: Run the module directly to surface the error:

python -c "import tide.metrics.tier<N>.<metric_id>"

Fix the reported error, then re-run make init-db.

Symptom: The Sources page at /sources shows an error for the new metric after the Scheduler runs.

Likely cause: The upstream data source returned an unexpected response (HTTP error, changed schema, empty series) that your _ingest() function did not handle.

Fix: Run the ingest manually to see the full traceback:

tide-ingest run --metric <metric_id>

Fix the fetch logic in _ingest(), re-run the command, and confirm the Sources page clears the error on the next scheduler cycle. Note that TLS verification is disabled only for cftc.gov requests — for all other sources, standard TLS applies and certificate errors are real errors that must be resolved.