Deployment Guide

This guide walks you through deploying TIDE to a production Linux server as a persistent web service. You will install the Python backend and SvelteKit frontend, initialise the DuckDB database, run the first full ingestion, start the FastAPI backend and SvelteKit frontend as long-running processes, and optionally place the scheduler under systemd so data stays fresh across reboots. TIDE runs entirely on the local filesystem — there are no containers, no orchestration platform, and no external database to manage — so the deployment surface is intentionally small.

Before you begin, make sure the following are in place on your target server:

• Python ≥ 3.10 (3.11+ recommended; pyproject.toml requires >=3.11)
• Node.js and npm ≥ 18 (for the SvelteKit frontend)
• GNU make (drives the full lifecycle — install, init, ingest, start)
• A FRED API key — free from fred.stlouisfed.org/docs/api/api_key.html; required to pull M2, the Fed balance sheet, HY spread, NFCI, and bank credit series
• Internet access from the server to FRED, Yahoo Finance, AAII, CFTC, SqueezeMetrics, and the US Treasury TIC endpoint
• xlrd Python package (optional) — required only for the AAII Bull/Bear sentiment metric; it is listed as a standard dependency in pyproject.toml so it installs automatically
• systemd (optional) — needed only if you want the scheduler to survive reboots; not required for manual or one-shot deployments

A Python virtual environment is strongly recommended. The Makefile defaults to ~/sandbox/envs/tideenv; you can override this with the VENV variable at any make invocation.

Step 1 — Clone the repository

git clone <your-repo-url> tide
cd tide

Step 2 — Create a Python virtual environment

The Makefile resolves the interpreter from the VENV path. Create the environment before running any make target:

python3 -m venv ~/sandbox/envs/tideenv

If you prefer a different path, export VENV so every subsequent make call picks it up:

export VENV=/srv/tide/.venv
python3 -m venv $VENV

Step 3 — Configure environment variables

cp .env.example .env

Open .env in your editor and set your FRED API key (see Configuration for all available keys):

FRED_API_KEY=your_key_here

Step 4 — Install dependencies

make install installs the tide Python package into your virtual environment in editable mode and runs npm install in the frontend directory:

make install
# or, to use a custom venv path:
make install VENV=/srv/tide/.venv

To install only the backend or frontend separately:

make install-backend
make install-frontend

Step 5 — Initialise the database

This creates the DuckDB schema and seeds the metrics registry with all 19 indicator definitions:

make init-db

You must run this before any ingestion. It is safe to re-run — existing data is not overwritten.

Step 6 — Pull data and build history

Run the three ingestion commands in order. The first two pull upstream data; the third recomputes the 252-business-day composite history from what was stored:

make ingest-all          # fetches all 15 live metrics (~30 s)
make ingest-watchlist    # fetches 8 healthcare tickers + XLV benchmark
make backfill-composite  # (re)computes the composite history series

ingest-all is idempotent — it re-pulls each source's full series and upserts into DuckDB, so re-running it does not create duplicates.

Step 7 — Start the API and web servers

Open two terminal sessions (or use a process manager such as screen, tmux, or systemd service units — see Keeping services alive below):

Terminal 1 — FastAPI backend (default port 8765):

make dev-api

Terminal 2 — SvelteKit frontend (default port 5173):

make dev-web

The frontend binds 0.0.0.0 and is reachable at http://<server-ip>:5173 from your browser. Both servers reload on source changes, making this setup suitable for a live production deployment on a dedicated server.

To run the API on a different port:

make dev-api PORT=9000

Keeping services alive

For a production deployment where the servers must survive terminal disconnects and server reboots, manage them with systemd service units analogous to the scheduler unit shown in Usage. Create a unit for dev-api (calling uvicorn directly) and one for dev-web (calling npm run dev in the frontend directory), then enable both with sudo systemctl enable --now.

TIDE is configured through a .env file in the project root. The pydantic-settings-based config.py reads these values at startup and resolves paths relative to the project root.

DuckDB database path is resolved by config.py against the project root. There is no environment variable to change it; the database lives at a fixed path inside the backend directory. This is intentional — the API and the ingest CLI share the same file and must agree on its location.

Per-metric z-score window is stored in the metrics registry (MetricDefinition.zscore_years) and defaults to 3 years. The HY credit spread metric is an exception: its window is set to 1 year to match FRED's April 2026 license-enforced data truncation. You configure this by editing the relevant MetricDefinition in the tier module, then re-running make init-db and make backfill-composite.

Scheduler timezone is fixed to America/New_York. The daily ingest job fires Mon–Fri at 17:00 and the release ingest job fires Fri at 18:00. These are not currently configurable via environment variables.

Once the servers are running, the FastAPI backend exposes the following endpoints for the frontend and any external API clients:

All endpoints return JSON. The SvelteKit frontend calls /dashboard and /watchlist on every page load via its page server, so the dashboard is always server-side rendered with fresh data.

Daily refresh

To keep data current without the scheduler, run these three commands once per day after US market close:

make ingest-all
make ingest-watchlist
make backfill-composite

Automated refresh with the scheduler

Start the scheduler as a blocking foreground process:

make scheduler

The scheduler runs two jobs on a cron schedule (America/New_York):

• daily_ingest — Mon–Fri at 17:00: pulls all cadence="daily" metrics, refreshes the watchlist, and runs backfill-composite.
• release_ingest — Fri at 18:00: pulls all weekly/monthly/quarterly metrics (FRED, FINRA, Treasury, CFTC), then runs backfill-composite.

To test the job bodies once and exit without waiting for the cron schedule:

make scheduler-once

Job status persists to the scheduler_status DuckDB table and is visible on the /sources endpoint and the Sources page of the dashboard.

Running the scheduler under systemd

To keep the scheduler alive across reboots, create a systemd unit. Adjust User, WorkingDirectory, and ExecStart to match your server's paths:

# /etc/systemd/system/tide-scheduler.service
[Unit]
Description=TIDE ingestion scheduler
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/tide
ExecStart=/path/to/envs/tideenv/bin/python -m tide.ingest.cli scheduler
Restart=on-failure
RestartSec=30s
Environment="PYTHONUNBUFFERED=1"

[Install]
WantedBy=multi-user.target

Enable and start the unit:

sudo systemctl enable --now tide-scheduler

Tail the scheduler logs:

journalctl -u tide-scheduler -f

Ingesting a single metric

If you want to refresh only one indicator without running the full suite:

# Replace m2 with any metric_id from the registry
cd backend && python -m tide.ingest.cli run --metric m2

List all registered metrics:

cd backend && python -m tide.ingest.cli list

After any targeted ingestion, run make backfill-composite so the history chart reflects the updated data.

Check the API is up

After starting make dev-api, confirm the backend is responding:

curl -s http://localhost:8765/health | python3 -m json.tool

Expected output (exact fields depend on implementation):

{
  "status": "ok"
}

Fetch the composite dashboard payload

curl -s http://localhost:8765/dashboard | python3 -m json.tool | head -60

The response includes the composite z-score, the tally of Bullish/Neutral/Bearish votes, the four tier averages, and a card for each of the 19 indicators with its current reading, directional vote, and as_of date.

Fetch watchlist data

curl -s http://localhost:8765/watchlist | python3 -m json.tool

Returns the 8 healthcare tickers benchmarked against XLV, with 30-day relative returns and sparkline data points.

Fetch the 252-day composite history

Use the tier history endpoint to retrieve historical z-scores for a specific tier (valid values: 1, 2, 3, 4):

curl -s http://localhost:8765/tier/1/history | python3 -m json.tool

Fetch detail and history for a single metric

# Detail card for the M2 metric
curl -s http://localhost:8765/metrics/m2/detail | python3 -m json.tool

# Observation history for the M2 metric
curl -s http://localhost:8765/metrics/m2/history | python3 -m json.tool

Full cold-start on a new server (end-to-end)

# 1. Install everything
make install VENV=/srv/tide/.venv

# 2. Initialise the database
make init-db VENV=/srv/tide/.venv

# 3. Pull all live metrics
make ingest-all VENV=/srv/tide/.venv
make ingest-watchlist VENV=/srv/tide/.venv

# 4. Build the 252-day history
make backfill-composite VENV=/srv/tide/.venv

# 5. Start the API (background, or a second terminal)
make dev-api VENV=/srv/tide/.venv PORT=8765 &

# 6. Start the frontend
make dev-web &

Open http://<server-ip>:5173 in your browser. The composite reading, history chart, and watchlist should all be populated.

Restrict the composite backfill to a shorter window

The backfill-composite command accepts a --days flag if you want to rebuild only a portion of the history (useful when testing a new metric):

cd backend && python -m tide.ingest.cli backfill-composite --days 30

Check scheduler status

While the scheduler is running, query its status via the API:

curl -s http://localhost:8765/scheduler | python3 -m json.tool

The response shows each job's next scheduled run time and the result of the last execution. Per-job history is also visible at /sources.

FRED_API_KEY not set — ingestion fails for FRED metrics

Symptom: make ingest-all exits with an authentication error or raises fredapi.exceptions.FREDRequestError for M2, WALCL, HY spread, NFCI, or TOTLL.

Cause: The .env file is missing or FRED_API_KEY is empty.

Fix: Open .env and ensure the key is set:

FRED_API_KEY=your_key_here

The key is free from fred.stlouisfed.org/docs/api/api_key.html. After saving .env, re-run make ingest-all.

make init-db fails with a module-not-found error

Symptom: No module named 'tide' or similar when running make init-db.

Cause: The backend package has not been installed into the virtual environment, or the VENV path does not match where the environment actually lives.

Fix: Run make install first. If you are using a custom venv path, pass it explicitly:

make install VENV=/your/venv/path
make init-db VENV=/your/venv/path

Dashboard shows stale data or missing metrics

Symptom: The composite reads from several days ago, or individual metric cards show an as of date that is old.

Cause: The daily ingestion and backfill have not been run recently. TIDE shows stale data rather than hiding it, so the dashboard remains available even when data is lagging.

Fix: Run the three-command refresh sequence:

make ingest-all
make ingest-watchlist
make backfill-composite

If you want this to happen automatically, start the scheduler (make scheduler) or install the systemd unit.

AAII ingestion returns 403 intermittently

Symptom: make ingest-all or a targeted AAII run logs 403 responses and fails to fetch the sentiment file.

Cause: AAII's server uses bot detection that cycles — the same request headers can return 200 or 403 minutes apart. The ingest client retries up to 4 times with backoff and validates the OLE2 magic bytes of the downloaded file.

Fix: Wait a few minutes and re-run. If the failure persists, the AAII endpoint may be temporarily unavailable. The composite continues to use the last known AAII value; the metric card's as of date will reveal the staleness.

tide-scheduler.service fails to start under systemd

Symptom: sudo systemctl start tide-scheduler fails; journalctl -u tide-scheduler -f shows a path error or permission denial.

Cause: The ExecStart path in the unit file does not match your virtual environment or working directory, or the User directive names a user that does not own those directories.

Fix: Verify all paths in the unit file:

# Confirm the interpreter path
ls /path/to/envs/tideenv/bin/python

# Confirm the working directory
ls /path/to/tide/backend

After editing the unit file, reload the daemon:

sudo systemctl daemon-reload
sudo systemctl restart tide-scheduler

Tail logs to confirm it starts cleanly:

journalctl -u tide-scheduler -f

History chart is empty after a fresh install

Symptom: The 252-day composite history chart in the hero panel renders with no data or a flat line.

Cause: make backfill-composite was not run after ingestion, so the composite history table is empty.

Fix:

make backfill-composite

This recomputes the full history from stored indicator observations. You must run it after every ingest-all or ingest-watchlist call for the chart to reflect current data.

FastAPI starts but the frontend shows no data

Symptom: The SvelteKit frontend loads but all panels are empty or display an error banner.

Cause: The frontend cannot reach the FastAPI backend. This is usually a port mismatch — the API is running on a non-default port but the frontend is configured to call port 8765.

Fix: Ensure both servers agree on the port. If you started the API with make dev-api PORT=9000, update the frontend's API base URL accordingly. The default port for the API is 8765; do not change it unless you also update the frontend configuration.