Dev

The abaca.dev package provides two subsystems used during bring-up and data-path validation of a Trilio Share Protection deployment: the datapath probe (abaca.dev.datapath) and the fidelity fixture (abaca.dev.fidelity_fixture), plus the findings writer (abaca.dev.findings_writer) that serialises probe results to disk.

Use the datapath probe when you need to verify that a real OpenStack environment can support the full Abacá backup/restore cycle end-to-end: share access, NFS mounting, Manila snapshot-and-clone, S3 reachability, Kopia snapshot creation, Kopia restore, and metadata fidelity. The probe runs from a host that can SSH into a worker IP and reach the OpenStack API; it is not part of the production control-plane and is never invoked by the conductor or the worker agent.

ProbeArgs (dataclass)

Constructed by the caller and passed to Probe.__init__. All fields are positional unless a default is shown.

Probe.__init__

fidelity_fixture.seed_script(base_dir)

fidelity_fixture.manifest_script(base_dir)

fidelity_fixture.verify(original, restored)

findings_writer.timestamp()

No parameters.

findings_writer.write(findings_dir, ts, report)

findings_writer.render_markdown(report)

Probe.run() -> int

Returns 0 when all seven probe steps complete successfully (overall_status: "completed"). Returns 1 if any step raises an exception (overall_status: "aborted"). In both cases teardown runs unconditionally in the finally block and findings files are written to findings_dir before the method returns.

fidelity_fixture.seed_script(base_dir) -> str

Returns a multi-line bash script string. When executed on the worker VM (with passwordless sudo available, which is the default for the Ubuntu 24.04 cloud image), the script creates the fixture tree under base_dir and prints the manifest JSON to stdout. The caller is responsible for running the script via the shell driver and capturing stdout.

fidelity_fixture.manifest_script(base_dir) -> str

Returns a bash script string that, when executed on the worker VM, walks an existing tree at base_dir and prints its manifest JSON to stdout. Use this to read the restored tree after a Kopia restore without modifying it.

fidelity_fixture.verify(original, restored) -> dict

Returns a structured findings dict with the following top-level keys:

{
  "matched_paths":        [str, ...],
  "missing_from_restore": [str, ...],
  "extra_in_restore":     [str, ...],
  "diffs": {
    "<rel>": {
      "mode":  {"orig": int, "restored": int, "verdict": "fail" | "ok"},
      ...
    }
  },
  "categories": {
    "uid_gid":                 "preserved" | "lost" | "n/a",
    "mode":                    "preserved" | "partial" | "lost",
    "mtime":                   "preserved" | "coarsened_1s" | "lost",
    "sparse":                  "preserved" | "materialized" | "n/a",
    "symlink":                 "preserved" | "lost",
    "hardlink_as_copy":        "confirmed" | "unexpectedly_linked",
    "xattrs":                  "preserved" | "not_preserved" | "not_present",
    "mtime_source_granularity": float
  }
}

Mtime is compared with a tolerance of 1.0 second (MTIME_TOLERANCE_SECONDS). Sparseness is compared with a ratio tolerance of 4.0× (SPARSE_BLOCKS_TOLERANCE_RATIO). Hardlink pairs are detected from the original manifest's inode numbers and are expected to restore as independent copies (nlink == 1 on each); this is recorded but does not cause a "fail" verdict. POSIX ACLs and extended attributes are recorded under xattrs but are out of scope for phase 1 and do not produce a fail verdict.

findings_writer.timestamp() -> str

Returns a UTC timestamp string in the format YYYYMMDDTHHMMSSz (e.g. "20240115T123456Z"), safe for use in filenames.

findings_writer.write(findings_dir, ts, report) -> tuple[str, str]

Returns a two-element tuple (json_path, md_path) with the absolute paths of the files written:

• <findings_dir>/<ts>-datapath.json — full probe report as pretty-printed JSON (schema_version field injected).
• <findings_dir>/<ts>-datapath.md — human-readable Markdown rendering of the same report.

findings_writer.render_markdown(report) -> str

Returns a Markdown string. Sections rendered (in order): probe inputs, source share capability matrix, Kopia environment, mount options, access-rule propagation latency, root-squash probes, S3 reachability, fidelity verdicts and per-path diffs, per-step timings, surprises, and operator recommendations. Sections whose source key is absent or empty are omitted silently.

Probe.run

The run() method catches all exceptions internally via a bare except Exception handler. Individual step failures abort the probe at that step, append a surprise entry, and set overall_status: "aborted" — they do not propagate to the caller. run() itself therefore does not raise; it always returns an int.

However, the steps raise RuntimeError internally under the following conditions, which run() catches:

Surprises (unexpected but non-fatal conditions) are appended to self.report["surprises"] rather than raised. Examples include stale Manila access rules found and cleaned up before the probe, setfattr/setfacl being unsupported on the share, and an aborted-probe exception message.

fidelity_fixture.verify

Does not raise. All comparisons are defensive; missing fields on individual entries are handled gracefully and recorded in diffs rather than causing an exception.

findings_writer.write

May raise OSError if findings_dir cannot be created or if the output files cannot be opened for writing (e.g. permissions, disk full). The caller (Probe.run) does not suppress these.

Run the datapath probe against a live Manila share

The probe is invoked programmatically. Wire up a shell driver that runs commands on your worker VM over SSH, and a manila client that wraps the Manila API.

from abaca.dev.datapath import Probe, ProbeArgs

args = ProbeArgs(
    share_id="e3b2a1c0-1234-5678-abcd-000000000001",
    worker_ip="10.0.0.42",
    findings_dir="/tmp/abaca-probe-findings",
    repo_backend="s3",
    s3_endpoint="https://s3.wasabisys.com",
    s3_bucket="my-abaca-probe-bucket",
    keep_worker=False,
    keep_clone=False,
    keep_snapshot=False,
    keep_source_rule=False,
    clone_name=None,
    s3_locked_bucket="my-abaca-probe-locked",
    s3_region="us-east-1",
    allow_insecure_tls=False,
)

probe = Probe(
    args,
    shell=my_ssh_shell,       # your SSH shell driver
    manila=my_manila_client,  # your Manila API wrapper
    s3_credentials={
        "AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
        "AWS_SECRET_ACCESS_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    },
)

exit_code = probe.run()
# 0 = all steps completed; 1 = probe aborted at a step
print("findings:", probe.report["findings_paths"])

Expected output (logged at INFO level):

[probe] findings: /tmp/abaca-probe-findings/20240115T123456Z-datapath.json / /tmp/abaca-probe-findings/20240115T123456Z-datapath.md

Use the filesystem backend for a MinIO cluster with a self-signed certificate

Pass allow_insecure_tls=True only when your S3 endpoint uses a self-signed certificate that Kopia cannot verify. Do not use this flag against Wasabi, AWS S3, or Ceph RGW.

args = ProbeArgs(
    share_id="e3b2a1c0-1234-5678-abcd-000000000002",
    worker_ip="10.0.0.43",
    findings_dir="/tmp/abaca-probe-findings",
    repo_backend="s3",
    s3_endpoint="http://minio.example.internal:9000",
    s3_bucket="probe-bucket",
    keep_worker=True,   # keep the VM for further inspection
    keep_clone=False,
    keep_snapshot=False,
    keep_source_rule=False,
    clone_name="abaca-probe-clone-manual",
    allow_insecure_tls=True,  # self-signed MinIO only
)

Read a fidelity manifest from a restored tree

After a Kopia restore completes, use manifest_script to re-read the tree and then verify to compare it against the original manifest captured during seeding.

from abaca.dev.fidelity_fixture import manifest_script, verify
import json

# Run the manifest script on the worker VM against the restore directory.
restore_base = "/var/tmp/abaca-probe-restore/fixture"
script = manifest_script(restore_base)
result = my_ssh_shell.run(f"bash -s <<'EOF'\n{script}\nEOF")
restored_manifest = json.loads(result.stdout)

# Load the original manifest saved by the probe.
with open("/tmp/abaca-probe-findings/20240115T123456Z-source-manifest.json") as fh:
    original_manifest = json.load(fh)

findings = verify(original_manifest, restored_manifest)

print("Fidelity categories:", findings["categories"])
print("Missing from restore:", findings["missing_from_restore"])
print("Extra in restore:", findings["extra_in_restore"])

Expected output (example — results vary by NFS version and Manila driver):

Fidelity categories: {
  'uid_gid': 'preserved',
  'mode': 'preserved',
  'mtime': 'coarsened_1s',
  'sparse': 'preserved',
  'symlink': 'preserved',
  'hardlink_as_copy': 'confirmed',
  'xattrs': 'not_preserved',
  'mtime_source_granularity': 1.0
}
Missing from restore: []
Extra in restore: []

Write findings to disk manually

If you invoke verify or build a report dict outside the Probe class, use findings_writer directly.

from abaca.dev.findings_writer import timestamp, write

ts = timestamp()  # e.g. "20240115T123456Z"
json_path, md_path = write("/tmp/my-findings", ts, my_report_dict)
print(f"JSON: {json_path}")
print(f"Markdown: {md_path}")

Output:

JSON: /tmp/my-findings/20240115T123456Z-datapath.json
Markdown: /tmp/my-findings/20240115T123456Z-datapath.md

Probe step order is intentional and must not be reordered. Each step registers teardown hooks as it creates external resources (Manila access rules, mounts, snapshots, clones). Teardown runs in reverse registration order regardless of whether the probe completed or aborted, and additionally sweeps every access rule on both shares whose access_to matches worker_ip. This means teardown is safe even when the caller does not track handles cleanly.

Idempotency on re-run. The seed step sweeps stale Manila access rules for worker_ip before granting new ones, and sudo rm -rf "$BASE" before recreating the fixture tree. A prior aborted probe run therefore does not block a fresh run with the same arguments. Stale rules are recorded as surprises in the report.

The fixture requires passwordless sudo on the worker VM. The seed script uses sudo chown, sudo touch, and sudo bash -c to create files owned by arbitrary UIDs and a root-owned 0600 file (used to probe root-squash behaviour). This is the default for the Ubuntu 24.04 cloud image (ubuntu user). The probe is not designed for RHEL-based worker images, and --distro ubi9 is not implemented in the worker image builder.

NFS version auto-selection. The probe tries NFSv4.1 first (vers=4.1,rw) and falls back to NFSv3 (vers=3,rw). The chosen options are recorded in report["mount"]. NFSv3 coarsens mtimes to 1-second granularity; this is reflected in the fidelity category mtime: "coarsened_1s" and in mtime_source_granularity. The mtime tolerance is 1.0 second (MTIME_TOLERANCE_SECONDS) to accommodate NFSv3.

POSIX ACLs and xattrs are out of scope for phase 1. The fixture attempts setfattr and setfacl but treats failure as non-fatal (a note is written to stderr, not a surprise). If xattrs or ACLs are not preserved through the Kopia snapshot/restore cycle, the fidelity verdict records "not_preserved" rather than "fail".

Hardlinks restore as independent copies. Kopia does not reconstruct hardlink relationships on restore. Each member of a hardlink pair restores as a file with nlink == 1 and identical content. verify detects this using the original inode pairing and records hardlink_as_copy: "confirmed". This is expected behaviour and does not constitute a failure.

S3 TCP reachability probe. The s3_reachability step checks TCP connectivity to the S3 endpoint on port 443 (and 9000 for plain HTTP MinIO) with a 5-second timeout (S3_TCP_TIMEOUT). A failure here blocks the kopia_snapshot_restore step because S3 connectivity is mandatory.

allow_insecure_tls disables TLS certificate verification in Kopia. Use this flag only against self-signed MinIO endpoints on private clusters. It passes --disable-tls-verification to Kopia repository create and reachability probe commands. Real S3 providers (Wasabi, AWS, Ceph RGW) must not receive this flag; doing so silently suppresses certificate errors.

The Kopia repository password is never passed via argv. The probe password is supplied to Kopia exclusively through the environment variable ABACA_PROBE_KOPIA_PASSWORD. This matches the production pattern where Kopia credentials flow through Barbican-managed secrets and are injected into the environment, never appearing in process command lines.

Findings files have a stable schema. schema_version is currently 1. New keys may be added in future versions but existing keys will not be removed or renamed. Downstream tooling that parses findings JSON should use schema_version to guard against breaking changes.

The probe is not invoked by the production control plane. abaca.dev.datapath is part of the abaca-dev developer/operator CLI used during bring-up and testing. It is not a code path exercised by abaca-conductor, abaca-worker-agent, or any production API handler. It requires direct network access to a worker VM IP and to the Manila and S3 APIs from the host where it runs.

Kopia repository directories on the worker. The probe creates its repository at /var/tmp/abaca-probe-repo and its restore target at /var/tmp/abaca-probe-restore. These paths are constants (KOPIA_REPO_DIR, KOPIA_RESTORE_DIR) and are not configurable via ProbeArgs. They are on the worker VM's local filesystem, not on the NFS share.