Deployment Guide

This guide explains how to run TIDE — Capital Pressure Observatory as a persistent service so your dashboard stays current without manual intervention. It covers adapting the development servers (make dev-api and make dev-web) for production use, configuring the Scheduler to refresh data automatically after market close each weekday, and keeping the Scheduler alive across reboots using a systemd unit. If you have already completed the quickstart, start at the Running as a persistent service section.

Before following this guide you should have already completed the TIDE quickstart — the database must be initialised, at least one full ingest must have run, and the composite history must have been backfilled. You also need:

• Python ≥ 3.10 (the backend pyproject.toml requires ≥ 3.11 in practice)
• Node.js and npm (for the SvelteKit frontend build)
• make (the Makefile is the primary developer interface)
• A FRED API key — set as FRED_API_KEY in your .env file (free at fred.stlouisfed.org)
• A virtualenv at the path recorded in VENV (default: ~/sandbox/envs/tideenv)
• systemd if you want the Scheduler to survive reboots (Linux only; adapt to your process manager on other platforms)
• An always-on internet connection — TIDE pulls from FRED, Yahoo Finance, CFTC, AAII, SqueezeMetrics, and Treasury TIC on every ingest cycle

TIDE has two runtime processes you must keep running for a live dashboard: the FastAPI backend (serves /api/*) and the SvelteKit frontend (serves the browser UI). A third optional process — the Scheduler — drives automated data refreshes.

1. Build the frontend for production

The make dev-web target runs Vite's development server, which is not suitable for production. Build a Node-adapter bundle instead:

cd frontend
npm run build

This produces a frontend/build/ directory that can be served by Node.js directly:

node frontend/build/index.js

The SvelteKit Node adapter reads PORT and HOST from the environment, so you can set them in your process manager or systemd unit.

2. Start the backend API

For production, run uvicorn without --reload and bind to the port you want to expose:

make dev-api PORT=8765

Or call uvicorn directly to drop the --reload flag:

/home/murali/sandbox/envs/tideenv/bin/uvicorn tide.api.app:app --port 8765 --host 0.0.0.0

The Makefile honors both VENV and PORT environment variables. If your virtualenv lives elsewhere, prefix every make call: make dev-api VENV=/opt/tide/venv PORT=9000.

3. Start the Scheduler (automated data refresh)

The Scheduler is an optional but recommended background process. Start it with:

make scheduler

This command blocks in the foreground and runs two APScheduler cron jobs (America/New_York timezone):

• daily_ingest — Mon–Fri at 17:00. Pulls every metric with cadence="daily", refreshes the watchlist, and recomputes the composite history.
• release_ingest — Fri at 18:00. Pulls every non-daily metric (weekly, monthly, quarterly) and recomputes the composite history. This catches new FRED, FINRA, Treasury, and CFTC releases within a week.

Scheduler state (next run, last success, last error) persists to the scheduler_status DuckDB table and is visible on the Sources page at /sources.

To test job logic without waiting for the cron trigger, run:

make scheduler-once

This fires both job bodies once and exits — useful for smoke-testing after a configuration change.

4. Keep the Scheduler alive across reboots with systemd

Drop the following unit file at /etc/systemd/system/tide-scheduler.service, adjusting User, WorkingDirectory, and ExecStart to match your environment:

# /etc/systemd/system/tide-scheduler.service
[Unit]
Description=TIDE ingestion scheduler
After=network.target

[Service]
Type=simple
User=murali
WorkingDirectory=/home/murali/sandbox/tide
ExecStart=/home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli scheduler
Restart=on-failure
RestartSec=30s
Environment="PYTHONUNBUFFERED=1"

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now tide-scheduler

Tail the logs at any time:

journalctl -u tide-scheduler -f

No Docker or Kubernetes configuration is currently provided. If you want to containerise TIDE, adapt the systemd units above into your container entrypoint. The only shared state between the backend API and the ingest process is the DuckDB file, so a wedged ingest never takes the API down — each process can be managed independently.

5. Wrap the API and frontend in systemd (optional)

Create analogous unit files for the API and frontend, replacing ExecStart with the uvicorn and node commands from steps 1–2. The pattern is identical to the Scheduler unit above. Pass FRED_API_KEY and any other secrets via EnvironmentFile= pointing to your .env file:

[Service]
...
EnvironmentFile=/home/murali/sandbox/tide/.env

TIDE's runtime behaviour is controlled by a single .env file at the project root, read by pydantic-settings in backend/tide/config.py. The Makefile exposes two override points at invocation time.

Environment variables (.env)

The frontend reads the same .env file (Vite is configured with envDir: '..' to read from the project root rather than frontend/).

DuckDB file path

The database path is resolved by pydantic-settings in config.py relative to the project root. There is no external database server — DuckDB is an embedded file database. Do not move the database file without updating the path in config.py or the corresponding environment variable, and ensure the process user has read/write access to the file.

Makefile variables

The Makefile accepts two variables that can be set in the environment or on the command line:

Example overrides:

make dev-api VENV=./backend/.venv PORT=9000
make install VENV=/opt/tide/venv

Z-score windows

Z-scores are 3-year rolling by default. Each metric can override this via MetricDefinition.zscore_years. One metric currently overrides the default: the HY Credit Spread (BAMLH0A0HYM2) uses a 1-year window because FRED truncated the series to a 3-year window in April 2026. This is set in the metric definition and does not require any .env change.

Scheduler timezone

The Scheduler always runs in America/New_York timezone. This is hardcoded in the APScheduler configuration and is not currently user-configurable. If your server clock is UTC (the common Linux default), the Scheduler handles the conversion automatically — you do not need to change your system timezone.

Once all processes are running, your day-to-day interaction with TIDE falls into three patterns: reading the dashboard, checking data freshness, and managing refreshes.

Opening the dashboard

Navigate to http://localhost:5173 (or your server's LAN IP on port 5173 for the development frontend, or the port you configured for the production Node build). The TIDE Dashboard shows:

• The composite Capital Pressure reading and its 252-business-day history chart at the top
• A vote tally (Bullish / Neutral / Bearish) in the header, summarising how individual indicators are reading
• Four tier grids (Macro Liquidity, Capital Flows, Market Microstructure, Sentiment & Positioning), each with per-indicator z-scores and as-of dates
• The watchlist — eight healthcare tickers with 30-day relative returns versus XLV and sparklines

Every metric card shows its as-of date so you can tell immediately whether a reading reflects today's data or a lagged publication (for example, monthly or quarterly FRED series). TIDE shows stale readings rather than hiding them.

Checking data source status

Go to /sources (the Sources page) to see each data source's last-run time, next scheduled run, and any errors. This is especially useful after the Scheduler runs to confirm that all sources updated correctly.

Manual data refresh

If you are not running the Scheduler, or if you want to force an immediate update outside the schedule, run the three refresh commands in sequence:

make ingest-all          # re-pulls all metrics that have a live ingest function
make ingest-watchlist    # re-pulls the 8 healthcare tickers + XLV
make backfill-composite  # recomputes the 252-day composite history from current data

All three commands are idempotent — running them more than once is safe.

Refreshing a single metric

To pull just one metric without a full ingest cycle, use the tide-ingest CLI directly:

/home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli run --metric m2

Or use the Makefile shortcut for M2 specifically:

make ingest-m2

After pulling a single metric, re-run make backfill-composite so the composite history reflects the updated value.

Automated refresh via the Scheduler

With tide-scheduler.service enabled, data refreshes happen automatically:

• Weekdays at 17:00 ET — daily metrics, watchlist, and composite history
• Fridays at 18:00 ET — weekly/monthly/quarterly metrics and composite history

You do not need to run any manual commands on days when the Scheduler fires successfully. Monitor it via /sources or journalctl -u tide-scheduler -f.

Example 1 — Full first-run production setup

This sequence takes you from a cloned repo to a running persistent service.

# 1. Copy and populate the environment file
cp .env.example .env
# Edit .env and set FRED_API_KEY=your_key_here

# 2. Install backend and frontend dependencies
make install

# 3. Initialise the database and seed the metrics registry
make init-db

# 4. Pull all live metrics and the watchlist (~30 s)
make ingest-all
make ingest-watchlist

# 5. Compute the 252-day composite history
make backfill-composite

# 6. Build the frontend production bundle
cd frontend && npm run build && cd ..

# 7. Start the API (background or separate terminal)
/home/murali/sandbox/envs/tideenv/bin/uvicorn tide.api.app:app --port 8765 --host 0.0.0.0 &

# 8. Start the frontend production server
PORT=5173 node frontend/build/index.js &

# 9. Enable the Scheduler as a systemd service
sudo systemctl daemon-reload
sudo systemctl enable --now tide-scheduler

Expected result: the dashboard is accessible at http://your-server-ip:5173, all 15 live metrics show today's readings, and the Scheduler will keep them current automatically.

Example 2 — Manual refresh outside the schedule

Run this when you want to force an immediate update, for example after a FRED data release:

make ingest-all
make ingest-watchlist
make backfill-composite

Expected output (abridged):

cd backend && /home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli run --all
[INFO] m2: fetched 36 observations
[INFO] walcl: fetched 36 observations
[INFO] hy_spread: fetched 252 observations
...
[INFO] ingest complete: 15 metrics updated, 4 stubbed (skipped)
cd backend && /home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli watchlist
[INFO] watchlist: fetched 9 tickers (8 + XLV)
cd backend && /home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli backfill-composite
[INFO] backfill-composite: wrote 252 composite observations

Example 3 — Using a custom virtualenv and port

If your virtualenv is not at the default path, or you need the API on a different port:

make dev-api VENV=/opt/tide/venv PORT=9000

The uvicorn process starts on port 9000 using the Python binary from /opt/tide/venv/bin/python.

Example 4 — Smoke-testing the Scheduler without waiting for cron

make scheduler-once

Expected output (abridged):

[INFO] Firing daily_ingest immediately (scheduler-once mode)
[INFO] daily_ingest: ingest-all complete
[INFO] daily_ingest: ingest-watchlist complete
[INFO] daily_ingest: backfill-composite complete
[INFO] Firing release_ingest immediately (scheduler-once mode)
[INFO] release_ingest: non-daily ingest complete
[INFO] release_ingest: backfill-composite complete
[INFO] scheduler-once: done, exiting

After this exits, check /sources to confirm last-run timestamps updated.

Example 5 — Tailing Scheduler logs

journalctl -u tide-scheduler -f

Expected output during a scheduled run:

May 12 17:00:01 hostname python[12345]: [INFO] daily_ingest triggered
May 12 17:00:31 hostname python[12345]: [INFO] daily_ingest: complete in 30.4 s
May 12 17:00:31 hostname python[12345]: [INFO] scheduler_status updated

The Scheduler service fails to start

Symptom: sudo systemctl status tide-scheduler shows failed or activating repeatedly.

Likely cause: The ExecStart path does not match your actual virtualenv location, or WorkingDirectory points to a directory the User cannot read.

Fix: Verify the paths:

ls /home/murali/sandbox/envs/tideenv/bin/python
ls /home/murali/sandbox/tide/backend

Update the unit file to match, then:

sudo systemctl daemon-reload
sudo systemctl restart tide-scheduler

If the error persists, check journalctl -u tide-scheduler -n 50 for the Python traceback.

FRED metrics fail with an authentication error

Symptom: make ingest-all prints an error like FRED_API_KEY not set or Bad Request: 400 for M2, WALCL, or HY spread.

Likely cause: FRED_API_KEY is missing from .env, or the .env file is not being read because the working directory is wrong.

Fix: Confirm the key is present:

grep FRED_API_KEY .env

If the key is missing, add it. If it is present but ingest still fails, make sure you are running make from the project root (where .env lives), not from inside backend/.

Dashboard shows stale readings after the Scheduler ran

Symptom: The as-of dates on metric cards have not advanced even though journalctl -u tide-scheduler shows a successful run.

Likely cause: The data source itself publishes on a lag (e.g., FINRA margin debt is monthly; Treasury TIC is monthly). TIDE shows stale readings rather than hiding them — this is expected behaviour.

Fix: Check the Sources page at /sources. The last-run time tells you when TIDE last attempted to pull the data; the as-of date on the metric card tells you when the source last published. A gap between the two means the upstream data has not been released yet, not that the Scheduler failed.

make backfill-composite produces fewer than 252 data points

Symptom: The composite history chart shows a shorter history than expected.

Likely cause: The initial make ingest-all was not run, or one or more metrics have no observations in the database yet, limiting how far back the composite can be computed.

Fix: Run the full sequence in order:

make ingest-all
make ingest-watchlist
make backfill-composite

If the problem persists, re-run make init-db to ensure the schema and metrics registry are correctly seeded (this is idempotent), then repeat the ingest.

The AAII Bull/Bear metric fails intermittently

Symptom: ingest-all succeeds most of the time but occasionally logs a 403 error for the AAII source.

Likely cause: AAII's server uses bot detection that cycles between 200 and 403 responses, sometimes minutes apart. The ingest client retries four times with backoff and validates the OLE2 magic bytes of the downloaded file.

Fix: This is a known upstream behaviour. Re-running make ingest-all a few minutes later usually succeeds. If the metric remains stale, check /sources for the last-error detail and retry manually:

/home/murali/sandbox/envs/tideenv/bin/python -m tide.ingest.cli run --metric aaii

Four metrics always show no live data

Symptom: The metric cards for ici_etf_flows, buyback_yield, put_call, and uvol_dvol never update, even after a successful ingest.

Likely cause: These four metrics are stubbed — they are registered in the MetricDefinition registry with name and description but have no ingest_fn. This is intentional: their free data sources are either gated or require disproportionate engineering effort. The per-tier __init__.py docstrings document what would be needed to unblock each one.

Fix: No fix is required; this is expected behaviour. The dashboard displays their last-known value (or a blank reading) alongside the as-of date. If you want to implement one of these metrics, follow the instructions in the Adding a new metric section of the README and consult the relevant tier's __init__.py.

The frontend cannot reach the API

Symptom: The dashboard loads but shows no data, and the browser console shows failed requests to /api/dashboard.

Likely cause: The FastAPI backend is not running, or it is bound to a different port than the frontend expects.

Fix: Confirm the API is up:

curl http://localhost:8765/api/dashboard | head -c 200

If the API is on a non-default port, ensure the frontend's server-side fetch is pointing to the correct address. The +page.server.ts route fetches from the API; if you changed PORT, update the fetch URL accordingly. For the development server, PORT defaults to 8765 as set in the Makefile.