Project build plan

Project decisions are labeled as decisions—not universal truths. The architecture should remain understandable, recoverable, and useful even when the home cluster is unavailable.

1. Goal and product sequence

The project turns music-release credits into evidence-backed paths between artists. It begins with a private seed derived from a personal record collection, expands through adjacent catalog credits, and then scales toward a broader catalog graph.

Static MVP: daily or curated challenges, precomputed evidence paths, and published findings.
Committed phase two: bounded arbitrary two-artist search through a live API.
Later modes: manual relay, hidden contributor, role constraints, and collection-inspired challenges.

2. Logical roles

Role	Responsibility in this build	Important distinction
Infrastructure control host	Runs Ansible and applies repeatable host configuration.	An Ansible control node is not the same concept as a Swarm manager, even when one machine performs both roles.
Swarm manager and application-state host	Maintains Swarm state; pins PostgreSQL, Redis, canonical snapshots, API, and publishing services.	The first version intentionally uses one manager and is therefore not control-plane high availability.
Four Raspberry Pi 3B workers	Run bounded, independent worker jobs against versioned read-only graph snapshots.	All four should use a 64-bit lightweight OS so production images can target `linux/arm64`.
Optional workstation-class build node	Accelerates full ingest, compaction, image development, and expensive analysis.	It may join for heavy work but is not required for public uptime.

3. Selected architecture—with explicit tradeoffs

Selected

Ansible + Docker Swarm

Ansible configures hosts; Swarm places and reconciles containerized services across nodes.

Tradeoff: simpler than a Kubernetes control plane, but less ecosystem depth and only one initial manager.

Selected

Redis + RQ for operational jobs

Independent, retryable tasks such as scoring, validation, challenge generation, and path batches use RQ.

Tradeoff: deliberately simpler than Celery or a coordinated Dask graph.

Optional study

Dask Distributed

Dask is introduced later for a workload whose tasks have real dependencies or benefit from distributed collections and futures.

Selected

Parquet + DuckDB + PostgreSQL

Parquet is the durable analytical layer, DuckDB transforms and validates it, and PostgreSQL stores mutable application state.

Leading design

Memory-mapped adjacency arrays

Compact integer arrays are the current production direction, not a final commitment. They must beat simpler libraries on memory, speed, evidence lookup, and operational complexity.

Selected

Static-first public delivery

The core public experience remains available independently of the home API. Live search degrades gracefully when unavailable.

4. Data and graph flow

Catalog source snapshots

Streaming Python parse

Normalized Parquet

DuckDB transforms

Versioned graph snapshots

The graph starts as a bipartite artist–release graph so every path can retain release-level evidence. An artist-only projection may be derived for selected analyses, but it should not replace the evidence-bearing source graph.

5. Stateful services and snapshot placement

Swarm schedules containers; it does not automatically turn local disks into distributed storage. PostgreSQL, Redis, and canonical graph files are therefore constrained to the state host and backed up there. Workers receive immutable, checksummed graph snapshots and must reject jobs that request a different snapshot version.

Recovery posture: with one manager, worker services may keep running after manager loss, but cluster management requires recovery. The mitigation is documented state backup, reproducible Ansible configuration, versioned data artifacts, and a public application that does not require the cluster.

6. Public and private data boundary

Public	Private
Logical roles, tool choices, algorithms, aggregate findings, derived catalog paths, and the fact that four Raspberry Pi 3B workers participate.	Collection membership, exact inventory, hostnames, addressing, switch/router models, port maps, mount paths, credentials, deployment commands, backups, and incident notes.

Discogs releases its monthly catalog data dumps—releases, artist names, and credits—under CC0, while collection membership and other user-associated data are restricted. Separately, API access is governed by the Discogs API Terms of Use. The collection is therefore a private seed; public outputs should use derived catalog facts and should follow applicable notices and linking requirements.

7. Alternatives considered

Decision	Selected	Why not the main alternatives?
Multi-node services	Docker Swarm	Compose is single-host; K3s adds valuable Kubernetes learning but more control-plane concepts and overhead than this first build needs.
Background work	RQ	Celery is more feature-rich but heavier; Dask is better when tasks form a dependency graph rather than a queue of mostly independent jobs.
Analytical storage	Parquet + DuckDB	Putting all normalized credits in PostgreSQL would work, but makes immutable analytical snapshots and broad scans less portable.
Graph execution	Benchmark-gated compact arrays	NetworkX is ideal for fixtures but object-heavy at scale; a graph database adds a server and query model before measurements show it is needed.
Public availability	Static first	A live-only application would make home connectivity, maintenance, and power part of the public reliability contract.

8. Implementation phases

Phase	Exit condition
Baseline	64-bit Pi OS, repeatable Ansible bootstrap, wired connectivity, storage, and recovery notes.
Swarm skeleton	One multi-architecture service runs successfully on every worker, and manager-state backup is tested.
Collection slice	Private collection seed plus one-hop catalog expansion produces a manually verified evidence path.
Durable contracts	Parquet schema, role taxonomy, stable IDs, manifests, provenance, and PostgreSQL search are versioned.
Medium graph	RQ batches, snapshot distribution, challenge generation, and performance measurements work repeatedly.
Graph benchmark gate	NetworkX fixtures validate results; compact arrays are compared with at least one optimized graph library before final selection.
Full scale	Full-catalog ingest, compact publication, rollback, and reproducible rebuild complete within acceptable resource limits.
Static release	Daily challenge and findings remain useful with the home API disabled.
Live search	Arbitrary paths are bounded, cached, evidence-backed, observable, and resilient to service unavailability.

Underlying concepts and tools

These explanations describe the technologies themselves. Each section then shows how the idea applies to this build.

1. Configuration management: Ansible

Ansible is an agentless automation system that applies a declared configuration from a control node to managed nodes listed in an inventory. Playbooks describe tasks and desired outcomes. A well-designed task is idempotent: running it again should leave an already-correct system unchanged.

Applied here: install Docker, create users and directories, configure services, and verify the four Pi workers from one infrastructure control host.

Official Ansible getting-started guide

2. Containers, images, and multi-platform builds

A container is an isolated process created from an image. An image packages the filesystem and metadata required to run an application. Different CPU architectures need compatible image variants; a multi-platform image manifest lets one tag resolve to an AMD64 or ARM64 image as appropriate.

Applied here: GitHub Actions and Docker Buildx publish both linux/amd64 and linux/arm64 variants. The Pi 3B nodes therefore need a 64-bit OS.

Exercise: inspect a public image with docker buildx imagetools inspect IMAGE and identify its supported platforms.

Docker multi-platform builds · GitHub Container Registry

3. Orchestration: Docker Swarm

Docker Swarm is Docker Engine’s built-in orchestration mode. A swarm contains manager and worker nodes. You declare a service as desired state; managers schedule tasks on nodes and reconcile the actual state toward that declaration. Managers maintain cluster state through Raft consensus.

Applied here: one manager schedules API and worker services. A single manager is intentionally simple but not highly available; running tasks can continue after manager failure, while cluster management requires recovery.

Exercise:

docker swarm init
docker service create --name lab-hello --replicas 2 nginx:alpine
docker service ps lab-hello

Observe the difference between the service declaration and the individual tasks placed on nodes.

Swarm key concepts · Manager and worker nodes

4. Queues and distributed task graphs: RQ versus Dask

RQ is a Python background-job queue backed by Redis or Valkey. Producers enqueue callable work; workers reserve jobs, execute them, and record status or failure. Dask instead represents computations as task graphs and can manage dependencies, futures, distributed collections, data movement, and coordinated execution.

Question	RQ	Dask
Primary abstraction	Queued job	Task graph, future, array, dataframe, or bag
Best fit	Independent or loosely coupled background work	Dependent computations and parallel analytical workflows
Project role	Default operational queue	Optional advanced experiment after a suitable workload appears

Exercise:

from redis import Redis
from rq import Queue

queue = Queue(connection=Redis())
job = queue.enqueue(sum, [1, 2, 3])
print(job.id)

Run an rq worker process and observe the job lifecycle.

RQ documentation · Dask futures · Dask task-graph specification

5. Analytical files and databases

Parquet is a column-oriented file format designed for efficient analytical storage and selective reads. DuckDB is an embedded analytical SQL engine that can query Parquet directly. PostgreSQL is a server database designed for durable, concurrent, transactional state.

Applied here: normalized catalog credits live in versioned Parquet; DuckDB performs broad transformations and validation; PostgreSQL stores mutable search indexes, challenges, caches, and snapshot registry records.

Exercise:

SELECT role, count(*) AS credit_count
FROM read_parquet('credits/*.parquet')
GROUP BY role
ORDER BY credit_count DESC;

Explain why this broad aggregation fits DuckDB better than a request-time API transaction.

Apache Parquet overview · DuckDB and Parquet · PostgreSQL architecture

6. Graph models and shortest paths

A graph contains nodes and edges. A bipartite graph has two node sets, with edges only between sets. Here those sets are artists and releases. In an unweighted graph, breadth-first search finds a path with the fewest edges in O(V + E) time. An artist-only projection can simplify analysis, but may obscure the release evidence that justifies each connection.

Applied here: preserve artist → release → artist paths as the evidence-bearing model. Use NetworkX on small fixtures as a readable correctness oracle.

Exercise:

import networkx as nx

G = nx.Graph()
G.add_nodes_from(['artist:a', 'artist:b'], bipartite='artist')
G.add_node('release:1', bipartite='release')
G.add_edges_from([
    ('artist:a', 'release:1'),
    ('release:1', 'artist:b'),
])
print(nx.shortest_path(G, 'artist:a', 'artist:b'))

Confirm that the path retains its release-level evidence.

NetworkX bipartite algorithms · NetworkX shortest paths

7. Compact adjacency arrays and memory mapping

An adjacency list stores the neighbors of each node. A compact CSR-like representation can store all neighbors in one integer array plus an offsets array that marks where each node’s neighbors begin and end. Memory mapping lets the operating system page portions of a file-backed array into memory as needed rather than reading the entire file at startup.

Applied here: memory-mapped arrays are the leading production design because the Pi workers have limited memory. This remains benchmark-gated: evidence lookup, build complexity, snapshot portability, and speed matter as much as raw traversal throughput.

NumPy memory mapping

8. APIs, static-first delivery, and network boundaries

FastAPI builds typed HTTP APIs; Pydantic validates and serializes data against declared models. A static-first application publishes its essential HTML, JavaScript, and data artifacts ahead of time, so basic use does not depend on a live origin service. Cloudflare Tunnel can publish a bounded service over outbound connections, while Tailscale provides a private administrative network.

Applied here: curated challenges remain static. A later FastAPI endpoint handles bounded live search. Cloudflare carries public API traffic; Tailscale remains the administrative path.

FastAPI · Pydantic · Cloudflare Tunnel · Tailscale

9. Testing, provenance, and observability

Unit tests verify small transformations and graph rules. Integration tests exercise queues, databases, and services. Browser tests validate the public experience. Provenance records which source snapshot, parser version, role rules, and graph version produced a result. Observability exposes health, logs, queue depth, latency, and active snapshot identity.

Applied here: every accepted path must be reproducible from a versioned snapshot and ruleset; a pretty path without evidence and provenance is not a valid result.

pytest · Playwright · Ruff · mypy

10. Data rights and responsible publication

Technical availability is not the same as permission to republish. Discogs releases its monthly database exports—including releases, artists, and credits—under CC0, while collection, wantlist, marketplace, and user-associated information are restricted. API use also carries notice, linking, freshness, and other contractual requirements.

Applied here: keep collection membership private, avoid treating album art as a launch dependency, publish derived catalog paths thoughtfully, and review the current terms before choosing between API-derived and dump-derived production workflows.

Discogs API Terms of Use

Deck

Primary references

These links favor official project documentation and current terms. External references require a network connection even when the study lab itself is opened from the offline HTML download.

Music-Credit Graph
Study Lab

Project build plan

1. Goal and product sequence

2. Logical roles

3. Selected architecture—with explicit tradeoffs

Ansible + Docker Swarm

Redis + RQ for operational jobs

Dask Distributed

Parquet + DuckDB + PostgreSQL

Memory-mapped adjacency arrays

Static-first public delivery

4. Data and graph flow

5. Stateful services and snapshot placement

6. Public and private data boundary

7. Alternatives considered

8. Implementation phases

Underlying concepts and tools

1. Configuration management: Ansible

2. Containers, images, and multi-platform builds

3. Orchestration: Docker Swarm

4. Queues and distributed task graphs: RQ versus Dask

5. Analytical files and databases

6. Graph models and shortest paths

7. Compact adjacency arrays and memory mapping

8. APIs, static-first delivery, and network boundaries

9. Testing, provenance, and observability

10. Data rights and responsible publication

Primary references

Infrastructure

Distributed Python

Data systems

Graphs and arrays

Application delivery

Quality and data use