test: e2e client lifecycle + ADRs with sequence diagrams
- client_lifecycle_test.go: full end-to-end integration test planned → provisioning (enroll) → active → migrating → active → deprecated → failed. Validates age keypair generation, attrs, context/secrets endpoints, invalid transition blocking, compute entity provisioning with relationship edges and status tracking. Also tests enrollment rejection for invalid states and duplicate slug rejection for provisioning. - adr/0011-client-lifecycle-flows.md: workstation self-enrollment, compute entity provisioning, deprecation/destruction flows with Mermaid sequence diagrams. Full lifecycle state diagram. Transition check enforcement documentation. - adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction flow through OODA loop phases. Thin client bootstrap. Internal component interactions (scheduler, actuator, notifier). Complete 30-tool ownership matrix. - Fix: migration 012 FK reference (executions.id → executions.entity_id) - Fix: provision handler null attributes JSONB - Fix: provisioning steps use entity_id for execution FK All 3 integration tests pass, go vet clean.
This commit is contained in:
205
docs/adr/0011-client-lifecycle-flows.md
Normal file
205
docs/adr/0011-client-lifecycle-flows.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# ADR 0011 — Client lifecycle sequence diagrams
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-08
|
||||
|
||||
## Context
|
||||
|
||||
The Oikos client lifecycle spans two distinct onboarding paths — workstation
|
||||
self-enrollment and compute entity (LXC/VM) provisioning — sharing the same
|
||||
infrastructure lifecycle state machine (`planned → provisioning → active →
|
||||
migrating → deprecated → destroyed`). These flows must be documented and
|
||||
validated against the Go implementation.
|
||||
|
||||
## Decision
|
||||
|
||||
All machines in the homelab follow the same lifecycle state machine defined
|
||||
in `seeds/ontology.yaml` under the `infrastructure` lifecycle. The
|
||||
implementation lives in `internal/httpapi/impl.go` (handlers),
|
||||
`internal/actuator/actuator.go` (SSH provisioning), and
|
||||
`internal/ontology/validate.go` (transition check enforcement).
|
||||
|
||||
## Workstation self-enrollment flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant BS as bootstrap.sh
|
||||
participant API as Oikos API (:8090)
|
||||
participant DB as Postgres
|
||||
participant Inf as Infisical
|
||||
|
||||
Op->>API: POST /entities {slug:"ws:laptop", type:"workstation", state:"planned"}
|
||||
API->>DB: INSERT entities (planned)
|
||||
API-->>Op: 201 + ETag
|
||||
|
||||
Op->>BS: curl bootstrap.sh | sudo bash
|
||||
BS->>BS: detect hostname, mesh IP
|
||||
BS->>BS: fetch CLIENTS.md, AGENTS.md, OIKOS.md, tools/
|
||||
BS->>BS: install age, curl, jq
|
||||
|
||||
BS->>API: POST /clients/enroll {slug, hostname, mesh_ip}
|
||||
API->>DB: validate state = planned|provisioning
|
||||
API->>API: generateAgeKeypair()
|
||||
API->>API: store age key in Infisical (best-effort)
|
||||
API->>DB: UPDATE state→provisioning, set age_pubkey, mesh_ip, enrolled_at
|
||||
API->>DB: INSERT audit_log (client.enrolled)
|
||||
API-->>BS: {age_private_key, age_public_key, infisical_client_id, infisical_client_secret}
|
||||
|
||||
BS->>BS: write /etc/age/key.txt (0600)
|
||||
BS->>BS: write /etc/infisical/identity (0600)
|
||||
BS->>BS: install context-poller.sh (launchd/systemd, 5min)
|
||||
|
||||
loop Every 5 minutes
|
||||
BS->>API: GET /clients/ws:laptop/context?since={timestamp}
|
||||
API->>DB: SELECT context_files, context_version
|
||||
API-->>BS: {agent_files_changed, sops_config_changed, tools_changed}
|
||||
end
|
||||
|
||||
Op->>API: PATCH /entities/ws:laptop {state:"active"}
|
||||
API->>DB: validate lifecyle transition provisioning→active
|
||||
API->>DB: UPDATE state→active, version+1
|
||||
API->>DB: INSERT audit_log (client.activated)
|
||||
API-->>Op: 200 + ETag
|
||||
```
|
||||
|
||||
### Enforced transition checks
|
||||
|
||||
Before `provisioning → active`, the `TransitionChecks` map in
|
||||
`internal/ontology/validate.go` validates:
|
||||
|
||||
| Check | Workstation | LXC/VM |
|
||||
|-------|------------|--------|
|
||||
| `age-key-enrolled-if-needed` | Requires `age_pubkey` in attrs | Skipped (no age key for compute entities) |
|
||||
| `mesh-joined-if-needed` | Requires `mesh_ip` in attrs | Skipped |
|
||||
| `health-check-answering` | Queries `entity_status.health != 'down'` | Same |
|
||||
| `doc-page-complete` | Requires at least 1 `documents` edge | Same |
|
||||
|
||||
### Before `deprecated → destroyed`:
|
||||
|
||||
| Check | Description |
|
||||
|-------|-------------|
|
||||
| `no-inbound-edges` | Zero `depends-on`, `hosts`, `provides`, `mounts`, `routes-to`, `stores-on` edges |
|
||||
| `secrets-revoked-and-rekeyed` | `age_pubkey` must be removed from attributes |
|
||||
| `backups-verified` | Audit log must have a `backup-verified` entry in last 30 days |
|
||||
| `ingress-and-dns-removed` | No remaining `routes-to`/`provides`/`hosts` edges |
|
||||
|
||||
## Compute entity provisioning flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator/Hermes
|
||||
participant API as Oikos API (:8090)
|
||||
participant DB as Postgres
|
||||
participant Act as Actuator
|
||||
participant PVE as Proxmox Host
|
||||
|
||||
Op->>API: POST /entities/provision {slug:"lxc:jellyfin", host:"host:hubris", attrs:{vmid,cores,...}}
|
||||
API->>DB: validate host exists, slug not taken
|
||||
API->>DB: INSERT entities (planned)
|
||||
API->>DB: INSERT executions (provision)
|
||||
API->>DB: INSERT provisioning_steps (6 steps, all pending)
|
||||
API->>DB: INSERT relationships (host:hubris hosts lxc:jellyfin)
|
||||
API->>DB: INSERT audit_log (entity.provisioned)
|
||||
API-->>Op: 201 {entity, execution_id}
|
||||
|
||||
Note over Act,PVE: Actuator loop picks up provisioning execution
|
||||
|
||||
Act->>API: GET execution for lxc:jellyfin
|
||||
Act->>DB: resolve host:hubris → (mesh IP, ssh user)
|
||||
|
||||
par Provisioning steps
|
||||
Act->>DB: UPDATE provisioning_step[1] (validate-constraints)
|
||||
Act->>PVE: ssh: pct status {vmid}
|
||||
PVE-->>Act: "does not exist" → ok
|
||||
Act->>DB: UPDATE provisioning_step[1] (ok)
|
||||
|
||||
Act->>DB: UPDATE provisioning_step[2] (create-container)
|
||||
Act->>PVE: ssh: pct create {vmid} --cores N --memory M --rootfs ...
|
||||
PVE-->>Act: container created
|
||||
Act->>DB: UPDATE provisioning_step[2] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct exec {vmid} -- apt install -y service1 service2
|
||||
Act->>DB: UPDATE provisioning_step[4] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct set {vmid} -mp0 /mnt/library,/mnt/library
|
||||
Act->>DB: UPDATE provisioning_step[5] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct exec {vmid} -- systemctl is-system-running
|
||||
PVE-->>Act: "running" → ok
|
||||
Act->>DB: UPDATE provisioning_step[6] (ok)
|
||||
end
|
||||
|
||||
Act->>DB: UPDATE entity state→active
|
||||
Act->>DB: UPDATE execution status→completed
|
||||
```
|
||||
|
||||
## Deprecation and destruction flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant API as Oikos API
|
||||
participant DB as Postgres
|
||||
|
||||
Note over Op,DB: Active entity → Deprecated
|
||||
|
||||
Op->>API: PATCH /entities/lxc:jellyfin {state:"deprecated"}
|
||||
API->>DB: validate lifecycle transition active→deprecated
|
||||
API->>DB: UPDATE state→deprecated
|
||||
API-->>Op: 200
|
||||
|
||||
Note over Op,DB: Deprecated → Destroyed (with gate checks)
|
||||
|
||||
Op->>API: PATCH /entities/lxc:jellyfin {state:"destroyed"}
|
||||
alt Inbound edges exist
|
||||
API->>DB: TransitionChecks["no-inbound-edges"] → count > 0
|
||||
API-->>Op: 409 (inbound edges still exist)
|
||||
else All checks pass
|
||||
API->>DB: TransitionChecks → all pass
|
||||
API->>DB: UPDATE state→destroyed
|
||||
API->>DB: INSERT audit_log (client.destroyed)
|
||||
API-->>Op: 200
|
||||
end
|
||||
```
|
||||
|
||||
## Full lifecycle state diagram
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> planned : POST /entities
|
||||
|
||||
planned --> provisioning : POST /clients/enroll (workstation)<br/>POST /entities/provision (compute)
|
||||
planned --> destroyed : cancelled via PATCH
|
||||
|
||||
provisioning --> active : PATCH state="active"<br/>checks: age-key, mesh, health
|
||||
provisioning --> failed : error during provisioning
|
||||
|
||||
active --> migrating : PATCH state="migrating"
|
||||
active --> deprecated : PATCH state="deprecated"
|
||||
active --> failed : PATCH state="failed"
|
||||
|
||||
migrating --> active : PATCH state="active"
|
||||
migrating --> failed : PATCH state="failed"
|
||||
|
||||
failed --> active : PATCH state="active"<br/>check: recovery-verified
|
||||
|
||||
deprecated --> active : PATCH state="active" (un-deprecate)
|
||||
deprecated --> destroyed : PATCH state="destroyed"<br/>checks: no-inbound-edges,<br/>secrets-revoked, backups-verified,<br/>ingress-dns-removed
|
||||
|
||||
destroyed --> [*]
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Workstations** self-enroll via `bootstrap.sh` → `POST /clients/enroll`.
|
||||
The age keypair is generated server-side and delivered once.
|
||||
- **Compute entities** are provisioned by the actuator over SSH. The operator
|
||||
declares intent via `POST /entities/provision`; the actuator executes
|
||||
step-by-step with DB-tracked progress.
|
||||
- **Transition gates** are enforced by named checks in
|
||||
`internal/ontology/validate.go`. The `deprecated → destroyed` gate blocks
|
||||
until all inbound edges are severed — preventing orphan references.
|
||||
- **Thin clients** poll `GET /clients/{slug}/context` for agent file deltas
|
||||
instead of `git pull`. The control plane host (mac-mini) keeps the full
|
||||
repo clone.
|
||||
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# ADR 0012 — Hermes/Oikos interaction architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-08
|
||||
|
||||
## Context
|
||||
|
||||
Hermes (the AI agent) is the primary operator interface for the hubris
|
||||
homelab. It communicates with Oikos via the MCP protocol. The MCP tools
|
||||
map to the OODA loop phases (Observe, Orient, Decide, Act). A thin client
|
||||
model distributes agent context via API deltas instead of git clones.
|
||||
|
||||
## Decision
|
||||
|
||||
Hermes interacts with Oikos through three surface layers: MCP tools
|
||||
(agent-facing), REST API endpoints (operator-facing and agent-facing), and
|
||||
the SSH actuator (internal). All read paths go through the Postgres DB as
|
||||
the single source of truth. All writes go through the API with audit
|
||||
logging and policy classification.
|
||||
|
||||
## Hermes → Oikos interaction flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as Hermes (AI Agent)
|
||||
participant MCP as MCP Server (:8090/mcp)
|
||||
participant API as REST API (:8090/api/v1)
|
||||
participant DB as Postgres (TimescaleDB)
|
||||
participant Act as Actuator
|
||||
participant PVE as Proxmox Hosts
|
||||
participant Matrix as Matrix Notifier
|
||||
|
||||
Note over H,Matrix: ── OODA: Observe ──
|
||||
|
||||
H->>MCP: get_entity("service:caddy")
|
||||
MCP->>DB: SELECT * FROM entities WHERE slug=$1
|
||||
DB-->>MCP: {slug, type, state, health, attrs}
|
||||
MCP-->>H: entity record
|
||||
|
||||
H->>MCP: get_state_snapshot()
|
||||
MCP->>DB: SELECT e.slug, st.health, st.disk_usage_pct FROM entities e LEFT JOIN entity_status st
|
||||
DB-->>MCP: [{slug, health, disk_pct, drift_count}, ...]
|
||||
MCP-->>H: fleet health snapshot
|
||||
|
||||
H->>MCP: search_knowledge("jellyfin hardware acceleration")
|
||||
MCP->>DB: SELECT ... WHERE search @@ to_tsquery('jellyfin & hardware & acceleration')
|
||||
DB-->>MCP: [documents, runbooks]
|
||||
MCP-->>H: ranked FTS results
|
||||
|
||||
H->>MCP: get_blast_radius("service:caddy")
|
||||
MCP->>DB: SELECT blast_radius($1, 3) -- recursive CTE
|
||||
DB-->>MCP: [{entity, depth}, ...]
|
||||
MCP-->>H: what breaks if caddy goes down
|
||||
|
||||
Note over H,Matrix: ── OODA: Orient ──
|
||||
|
||||
H->>MCP: explain("lxc:jellyfin")
|
||||
MCP->>DB: SELECT e.*, st.health, st.last_check FROM entities e LEFT JOIN entity_status st
|
||||
MCP->>DB: SELECT r.type, se.slug, te.slug FROM relationships r WHERE ...
|
||||
DB-->>MCP: compact context card
|
||||
MCP-->>H: {type, state, health, relations, version, updated_at}
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "restart")
|
||||
MCP->>DB: SELECT risk_class, approval FROM classification_for($1, $2)
|
||||
DB-->>MCP: {risk_class: "reversible_low", approval: "auto-act"}
|
||||
MCP-->>H: safe to auto-act
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "deploy")
|
||||
MCP->>DB: ...
|
||||
DB-->>MCP: {risk_class: "config_mutation", approval: "operator-approval"}
|
||||
MCP-->>H: needs operator approval
|
||||
|
||||
Note over H,Matrix: ── OODA: Decide ──
|
||||
|
||||
alt reversible_low (auto-act)
|
||||
H->>MCP: request_execution("lxc:caddy", "restart")
|
||||
MCP->>API: POST /executions {action:"restart", target:"lxc:caddy"}
|
||||
API->>DB: INSERT executions (auto_approved)
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: ssh systemctl restart caddy
|
||||
Act->>DB: UPDATE execution status→completed
|
||||
MCP-->>H: execution {status: completed}
|
||||
else config_mutation (escalate)
|
||||
H->>MCP: request_execution("lxc:jellyfin", "deploy")
|
||||
MCP->>API: POST /executions
|
||||
API->>DB: INSERT executions (proposed, needs approval)
|
||||
API->>Matrix: send approval request via notifier
|
||||
Matrix->>Operator: "Approve deploy lxc:jellyfin? ✅/❌"
|
||||
Operator->>Matrix: ✅
|
||||
Matrix->>API: POST /approvals/{id}/approve
|
||||
API->>DB: UPDATE execution status→approved
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: run deploy procedure
|
||||
MCP-->>H: execution {status: completed}
|
||||
end
|
||||
|
||||
Note over H,Matrix: ── OODA: Act (mutation gated) ──
|
||||
|
||||
H->>MCP: tail_log("caddy", lines=200)
|
||||
MCP->>PVE: ssh journalctl -u caddy -n 200
|
||||
PVE-->>MCP: log lines
|
||||
MCP-->>H: caddy logs
|
||||
|
||||
H->>MCP: get_service_status("caddy")
|
||||
MCP->>PVE: ssh systemctl show caddy
|
||||
PVE-->>MCP: {ActiveState, SubState, ...}
|
||||
MCP-->>H: service status
|
||||
|
||||
H->>MCP: list_lxcs()
|
||||
MCP->>DB: SELECT * FROM entity_status WHERE type='lxc'
|
||||
DB-->>MCP: [lxc:caddy, lxc:jellyfin, ...]
|
||||
MCP-->>H: all LXCs with state
|
||||
|
||||
H->>MCP: get_lxc_state("lxc:caddy")
|
||||
MCP->>PVE: ssh pct status {vmid} --verbose
|
||||
PVE-->>MCP: RAM, CPU, disk, uptime
|
||||
MCP-->>H: LXC resource state
|
||||
```
|
||||
|
||||
## Thin client bootstrap flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant New as New Client (bare machine)
|
||||
participant Gitea as Gitea (raw URL)
|
||||
participant API as Oikos API
|
||||
participant DB as Postgres
|
||||
|
||||
New->>Gitea: curl bootstrap.sh
|
||||
Gitea-->>New: bootstrap.sh
|
||||
|
||||
New->>Gitea: fetch CLIENTS.md, AGENTS.md, OIKOS.md, tools/
|
||||
Gitea-->>New: agent orientation files
|
||||
|
||||
New->>API: POST /clients/enroll {slug, hostname, mesh_ip}
|
||||
API->>DB: validate state, mesh IP
|
||||
API->>API: generate age keypair
|
||||
API->>DB: store pubkey, transition→provisioning
|
||||
API-->>New: {age_private_key, age_public_key, infisical_identity}
|
||||
|
||||
New->>New: write /etc/age/key.txt, /etc/infisical/identity
|
||||
New->>New: install context-poller (launchd/systemd, every 5min)
|
||||
|
||||
loop Every 5 minutes
|
||||
New->>API: GET /clients/ws:{hostname}/context?since={timestamp}
|
||||
API->>DB: SELECT files changed since {timestamp}
|
||||
API-->>New: {agent_files_changed, sops_config_changed, tools_changed}
|
||||
New->>Gitea: fetch only changed files
|
||||
end
|
||||
```
|
||||
|
||||
## Internal Oikos component interactions
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Sched as Scheduler
|
||||
participant API as API Server
|
||||
participant Notif as Notifier
|
||||
participant Act as Actuator
|
||||
participant DB as Postgres
|
||||
|
||||
Note over Sched,DB: The OODA loop (internal)
|
||||
|
||||
Sched->>DB: probe endpoints (HTTP, TCP, disk, cert-expiry)
|
||||
DB-->>Sched: results
|
||||
Sched->>DB: INSERT signals (dedup, flap suppression)
|
||||
Sched->>DB: UPDATE entity_status (health, disk, drift_count)
|
||||
|
||||
Act->>DB: poll executions with status=auto_approved
|
||||
DB-->>Act: pending executions
|
||||
Act->>Act: circuit breaker check
|
||||
Act->>SSH: execute procedure
|
||||
Act->>DB: UPDATE execution status+result
|
||||
|
||||
Notif->>DB: poll pending approvals
|
||||
DB-->>Notif: [approval requests]
|
||||
Notif->>Matrix: send approval messages
|
||||
|
||||
API->>DB: INSERT audit_log (every mutation)
|
||||
API->>DB: NOTIFY oikos_events (SSE streaming)
|
||||
```
|
||||
|
||||
## Tool ownership matrix
|
||||
|
||||
| Tool | Interface | Package | DB query | SSH |
|
||||
|------|-----------|---------|----------|-----|
|
||||
| `get_entity` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `list_entities` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_relations` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_blast_radius` | MCP + REST | both | CTE function | — |
|
||||
| `search_knowledge` | MCP | `internal/mcp/server.go` | FTS query | — |
|
||||
| `get_health_summary` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `whoami` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `explain` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `preflight` | MCP | `internal/mcp/server.go` | CASE expression | — |
|
||||
| `get_change_history` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_state_snapshot` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `list_my_secrets` | MCP | `internal/mcp/server.go` | attributes query | — |
|
||||
| `tail_log` | MCP | `internal/mcp/server.go` | — | journalctl |
|
||||
| `get_service_status` | MCP | `internal/mcp/server.go` | — | systemctl show |
|
||||
| `list_lxcs` | MCP | `internal/mcp/server.go` | entity_status query | — |
|
||||
| `get_lxc_state` | MCP | `internal/mcp/server.go` | relationship query | pct status |
|
||||
| `ping_service` | MCP | `internal/mcp/server.go` | — | probe |
|
||||
| `request_execution` | MCP | `internal/mcp/server.go` | executions INSERT | SSH via actuator |
|
||||
| `get_agent_activity` | MCP | `internal/mcp/server.go` | agent_activity query | — |
|
||||
| `get_signal_history` | MCP | `internal/mcp/server.go` | signals query | — |
|
||||
| `get_patterns` | MCP | `internal/mcp/server.go` | patterns query | — |
|
||||
| `get_skills` | MCP | `internal/mcp/server.go` | skills query | — |
|
||||
| `get_audit_trail` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_trend` | MCP | `internal/mcp/server.go` | metrics query | — |
|
||||
| `get_event_timeline` | MCP | `internal/mcp/server.go` | events query | — |
|
||||
| `query_metrics` | MCP | `internal/mcp/server.go` | metric_samples query | — |
|
||||
| `enroll` | REST | `internal/httpapi/impl.go` | entities+audit+events | — |
|
||||
| `context` | REST | `internal/httpapi/impl.go` | context_files query | — |
|
||||
| `secrets` | REST | `internal/httpapi/impl.go` | secrets.Manager.List | — |
|
||||
| `provision` | REST | `internal/httpapi/impl.go` | entities+steps+relations | SSH via actuator |
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Hermes is the primary operator interface.** All operator actions flow
|
||||
through Hermes → MCP → Oikos. The old `bin/homelab` CLI is dead.
|
||||
- **MCP tools are read-only by design.** Mutations go through
|
||||
`request_execution`, which is policy-gated and requires operator
|
||||
approval for `config_mutation` and `destructive` actions.
|
||||
- **The actuator holds the SSH key.** Hermes has no direct SSH access.
|
||||
The security boundary is Hermes → MCP → API → execution queue →
|
||||
actuator → SSH.
|
||||
- **Thin clients poll for context deltas.** No git clones on workstations.
|
||||
The 5-minute poll replaces `git pull` with HTTP queries to
|
||||
`GET /clients/{slug}/context`.
|
||||
- **The DB is the single source of truth.** All state transitions,
|
||||
audit entries, and event emissions go through Postgres. The scheduler,
|
||||
actuator, notifier, and API all read/write the same tables.
|
||||
Reference in New Issue
Block a user