fix: blast radius walks dependency direction, and reachability survives no ICMP
Two things the entity window redesign surfaced but deliberately left alone. **blast_radius answered the wrong question.** It walked source→target for every relationship type, but which end of an edge is the dependent differs per type: "machine hosts container" means the target breaks, while "service depends-on service" and "ingress routes-to service" mean the SOURCE breaks. Walking everything forwards was right for hosts/provides and backwards for everything else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping, so the result contained tasks and executions that cannot break. Direction is now declared per relationship type in seeds/ontology.yaml (blast_direction: forward | backward | none), the same shape as the entity types' monitoring: declaration, and defaults to none so an undeclared edge contributes nothing rather than a confidently wrong answer. It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so nothing recorded that all 21 public hostnames are terminated by caddy. A `served-by` edge type now says so. pool:ludo-lvm 2 -> 23 (every container storing on it, then their services) lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes) service:authentik 7 (what authenticates via it) **Every ping check was reporting down.** Not a host:strong false positive: all seven, including ws:mac-mini — the Docker host itself. The scheduler runs in Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed and every LAN ping fails. Under health aggregation each broken probe dragged its entity to down. The question the check exists to answer is "is it reachable", and ICMP is only one way to ask it. checkPing now falls back to a TCP connect before concluding anything, which restores an honest verdict for the four hosts that are genuinely up while leaving the genuinely unreachable ones down. TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1, authentik=2 — the cycle walked the wrong way); it now asserts the corrected depths, and its exact-node-count check is relaxed because walking the right way also surfaces the seed's own real dependents, which are correct answers. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -605,6 +605,9 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Count int `json:"count"`
|
||||
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||
// that answer on something else (a Home Assistant VM has no sshd).
|
||||
Port int `json:"port"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -638,9 +641,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
|
||||
output, err := cmd.Output()
|
||||
if err != nil {
|
||||
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||
// simply unavailable from here. On this deployment the scheduler runs
|
||||
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||
// ping checks reported "down" for hosts that were demonstrably up
|
||||
// (including the Docker host itself). Under health aggregation that one
|
||||
// broken probe was enough to drag each entity to down.
|
||||
//
|
||||
// The question this check exists to answer is "is it reachable", and
|
||||
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||
// concluding anything.
|
||||
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||
return checkResult{
|
||||
health: "healthy",
|
||||
metrics: map[string]float64{},
|
||||
}
|
||||
}
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ping",
|
||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
||||
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -654,6 +674,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
if timeout <= 0 {
|
||||
timeout = 5 * time.Second
|
||||
}
|
||||
d := net.Dialer{Timeout: timeout}
|
||||
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
conn.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||
|
||||
func parsePingLatency(output []byte) float64 {
|
||||
|
||||
Reference in New Issue
Block a user