31 Commits

Author SHA1 Message Date
Claude (noether) 62ed5a1c08 sicd: reap on child-exit instead of blocking on stdin (fixes the v2 stdin hang)
The v2 client (0.2.0) forwards its own stdin and only closes the ssh stdin channel when that
stdin hits EOF. When sic is invoked with an inherited stdin that never closes (a pipe held open
by the caller -- exactly how non-interactive tooling runs it), the channel stays open, and
pumpStdin's `io.Copy(child.stdin, ssh-stdin)` stayed BLOCKED on the read long after the remote
command had already exited. sicd never reaped, ssh never returned, and the whole call hung until
stdin EOF or a timeout. Measured: `sic host true </dev/null` = 3.4s (== raw ssh), bare = 20s hang.

Fix: race the stdin copy against the child's exit (what ssh itself does).
  * child exits first  -> it did not need the rest of stdin; stop forwarding, abandon the read,
    return the child's status. (the hang fix)
  * copy ends first (EOF/EPIPE/error) -> its outcome is meaningful; a genuine read error that
    truncated the transfer to a still-live child is still a non-zero failure.
  * near-simultaneous  -> a bounded 50ms window still catches a real truncation, but a genuinely
    blocked stdin can no longer re-introduce the hang.

Daemon-only change; the client is untouched, and v1/v2 dual-read is unaffected. New regression
TestPumpStdinReturnsOnChildExitDespiteBlockedStdin (never-EOFing stdin + exited child -> prompt
child status, not a hang). Existing pump tests (read-error->1, EPIPE head-semantics, clean EOF)
unchanged and green. Verified live: bare `sic host true` 20s -> 3.9s; piped/redirected stdin
still forward.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:26:10 +02:00
Claude (noether) 20a6608f0e docs+examples: de-identify — generic hosts, no personal/model/internal names
Public repo hygiene: scrub local identifiers from the design doc and the usage
examples/tests. docs/design-nested.md loses the hostname, the maintainer name, a
model name, and internal system/agent names (the rejected shared-store router is
now described generically, not by product name); its stale wire line is corrected
to netstring(argc) while there. Usage comments and test fixtures switch real fleet
hostnames/container names to host1/host2/app/db. No behaviour change; all tests green.
2026-07-23 13:17:06 +02:00
Claude (noether) 4fcd8512ae sicd: canonical argc + wire-spec doc rot (reviewer 964 re-probe #1/#2/#3)
@reviewer re-probed the argc path (18 hand-built frames, all rejection paths clean) and cleared
it as deployable. Three non-blocking follow-ups, all closed here:

  #1 doc rot -- the comments ARE the spec on a hand-rolled wire, and three still described removed
     formats: readNetstringStream (claimed v1-only; it now serves both paths), parseV2Content doc
     (empty-netstring terminator), runV2 doc (the two-generations-old command/NUL/payload split),
     and the test frame() helper. All rewritten to netstring(argc) + argc argv netstrings + payload.
  #2 canonical argc -- strconv.Atoi accepted "+1" and "001" (one value, two encodings). Reject
     leading +/-, leading zeros, and empty before Atoi, mirroring readNetstringStream length-token
     validation. New regression TestV2NonCanonicalArgcRejected (01/+1/-1 etc. exit 1, no exec).
  #3 argc-low reclassification -- documented where payload is defined: a low argc turns the
     sender's own trailing netstrings into child stdin; only ever the sender frame, no trust
     boundary. By design, now stated.

Also fixed TestV2FrameElementCapRejected, which under the argc wire was tripping on a bad argc
token rather than the count cap it claims -- now sends argc=5000 to hit maxArgs honestly.

Build/vet clean, full daemon + client suites green. Daemon is deploy-ready at this commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 12:06:58 +02:00
Claude (noether) ebccfa0467 sic: v2 main() — nested targets, boundary-preserved argv, real stdin forwarding
WP8, the last code before deploy. Rewrites the client entrypoint on the v2 wire:

  * target parsing: host[/hop1/hop2...]; hops resolved to incus/docker/pct verbs via
    /etc/sic/hosts.toml (loaded only when hops are present), folded into the frame onion
    by buildChain — verbs[0] is the outermost layer the host sicd enters first.
  * argv is carried as [][]byte end to end and NEVER space-joined; --sh is the sole
    exception (user asked for a shell line -> [sh -c <joined>]).
  * stdin is forwarded AFTER the frame (the v1 client set Stdin to the frame alone, so
    `sic host cat >f <local` wrote a zero-byte file). The copy runs in an abandoned
    goroutine so a stdin-ignoring command on a tty never blocks on a read that never EOFs.
  * ssh -T: the frame is binary; a pty would mangle it.
  * exit code inherited from the far end.

Dead v1 framing (netstring/frameArgv/indexOf) removed — frame.go owns the wire now.
Also corrected the stale runV2 doc comment in sicd (still described the removed 0:,
terminator; it is argc-prefixed since e1f93de).

E2e on boltzmann (ssh localhost + real sicd), all green: touch "a b" -> ONE file;
echo A "" B -> "A  B"; pipe+redirect stdin both forward; exit 7 propagates; a
stdin-ignoring echo does not hang; --sh; and netstring-looking args (3:x, a,b:c)
survive as single arguments.
2026-07-23 11:59:26 +02:00
Claude (noether) e1f93dea7c sicd+client: explicit argc in v2 content — empty args representable (reviewer 964 #1)
Ship-blocker #1: the empty-netstring terminator collided with a legit empty "" argument —
parseV2Content broke on the first len(ns)==0, so echo A "" B ran as echo A and B bled into
child stdin (wrong command executed, not rejected). Replaced the terminator with an explicit
argc prefix: content = netstring(argc) + argv netstrings + payload; the daemon reads EXACTLY
argc elements, so "" is a normal arg. Fixes the collision AND the element-count bound in one.
Daemon parseV2Content, client v2Frame, both test helpers, and the reference builder all updated.
Proof: TestV2EmptyArgPreserved (echo A "" B -> "A  B") + TestV2FrameEmptyArgRepresentable.
All v2/v1 tests green, vet clean. Daemon now clears reviewer 964 #1/#2/#3 — deployable.
2026-07-23 11:50:35 +02:00
Claude (noether) 1c43b6d9c0 sicd: bound outer netstring length before alloc + argv element count (reviewer 964 #2/#3)
#3 [DoS]: runV2 allocated make([]byte, nsLen) from a raw uint32 (up to 4 GiB) BEFORE
readNetstring capped it — 5 bytes committed ~4 GB. Now reject nsLen>1<<24 immediately.
#2 [DoS]: parseV2Content had no element-count/byte cap (16 MiB content = 4.2M argv, ~388 MB).
Now caps maxArgs=4096 / maxFrameBytes=1<<20, mirroring runV1. Regression tests for both.
#1 (empty-arg terminator collision) is a SEPARATE ship-blocker, deferred (needs a wire-design
call: explicit argc vs client-side reject) — do NOT deploy the daemon until #1 lands.
2026-07-23 11:45:10 +02:00
Claude (noether) b71ea84670 sic v2 client: rework frame builders to the argv-netstring wire (match daemon bf82a80)
v2Frame now takes explicit argv [][]byte (not a space-joined command string) and builds
content = argv-netstrings + 0:, + payload, so the client preserves the users argument boundaries
end-to-end. wrapHop space-splits the hop VERB (space-safe) + appends sicd; buildChain folds
verbs[0] outermost. Consolidated the three WP tests into frame_test.go with an independent
reference builder. Addressing layer (parseTarget/verbFor/resolveVerbs/parseHostsConfig) unchanged.
2026-07-23 11:40:05 +02:00
Claude (noether) bf82a80869 sicd: v2 wire preserves argv boundaries (content = argv netstrings + terminator + payload)
The v2 command field was a single string the daemon SPACE-SPLIT into argv — which silently
broke sic's founding guarantee (`ssh host touch 'a b'` makes TWO files; `sic host touch 'a b'`
must make ONE). The composition of the v2 client surfaced this: every argument with a space
would be re-split.

Fix (markus chose it): v2 content is now a run of djb netstrings (the argv), terminated by the
empty netstring 0:,, then the payload — mirroring the boundary-safe v1 exec encoding. Each
argument is length-framed, so spaces, NUL, 0xFF all ride untouched; readNetstringStream already
bounds each element + its length token, so a hostile content cannot OOM here. runV2 uses the new
parseV2Content; the test frame() helper space-splits test-side only (the wire never does).

Proof: TestV2ArgvBoundaryPreserved — argv {script, "a b"} makes the script see 1 arg "a b", not
2. All existing v2 + v1 dual-read tests stay green; go vet clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 11:37:54 +02:00
Claude (noether) fe799b98f9 sic v2 client WP7: parseHostsConfig — minimal hosts.toml subset parser (stdlib only) 2026-07-23 11:26:42 +02:00
Claude (noether) 234b057c41 sic v2 client WP6: resolveVerbs — map hops to verbs by nest depth, fail-loud on overflow 2026-07-23 11:23:53 +02:00
Claude (noether) ca54f83601 sic v2 client WP5: verbFor — runtime type + name -> the runtime verb (incus/docker/pct) 2026-07-23 11:22:01 +02:00
Claude (noether) 209ef8da83 sic v2 client WP4: parseTarget — split host/hop1/hop2 into host + ordered hops 2026-07-23 11:20:19 +02:00
Claude (noether) f72c77e2a7 sic v2 client WP3: buildChain — fold wrapHop into the nested onion
buildChain(verbs, cmd, payload): innermost v2Frame(cmd,payload), wrap outward so verbs[0] is the
outermost layer; zero verbs = a single v2 frame (bare host). @godev/deepseek.
2026-07-23 11:18:20 +02:00
Claude (noether) f412d84738 sic v2 client WP2: wrapHop — wrap an inner frame for one chain hop
wrapHop(verb, inner) = v2Frame(verb+" sicd", inner): the hop command is the runtime verb with
sicd appended (the inner-hop peeler), the inner frame is the payload. @godev/deepseek, one grind.
2026-07-23 11:12:33 +02:00
Claude (noether) d2b9eb4870 sic v2 client WP1: v2Frame — one v2 wire frame (leaf function)
First function-level package of Piece 2. v2Frame(command, payload) builds one v2 frame:
0x00 + be32(len(netstring)) + netstring(command 0x00 payload). Byte-exact tests (cat/hi, empty
payload nil==[]byte{}, binary payload riding untouched). @testdesigner proved satisfiability in
scratch + caught a be32 arithmetic error in the spec; @godev (deepseek default) implemented it —
the small-package decomposition carrying on the room grinder where the monolithic parser could not.
2026-07-23 11:07:34 +02:00
Claude (noether) 189bb31737 sicd: bound the v1 length token and frame size (reviewer 925 #1 HIGH, #2 MED)
#1 [HIGH, was a real DoS]: readNetstringStream read the length token via bufio.ReadString(':'),
which buffers the whole pre-':' run UNBOUNDED — the 1<<24 cap only bounded the body, applied
AFTER the token was parsed. A protocol-confined caller sending an endless digit run with no ':'
drove sicd to ~1 GB RSS (measured by @reviewer). Now the token is read byte-by-byte with an
8-digit hard cap (a valid <=1<<24 length is 8 digits), so it rejects after <=9 bytes regardless
of input size — memory is bounded by construction, before the value is trusted.

#2 [MED]: nothing bounded the NUMBER of netstrings per frame (2M tiny ones ~140 MB before exec).
runV1 now caps element count (4096) and total argv bytes (1<<20).

#3 [MED, documented]: an empty netstring 0:, is always the terminator, so a legit empty ""
argument collides with it (silent argv truncation + stdin bleed). Inherent to the v1 wire, not
introduced here — noted in a comment; a fix would need a wire change.

Regression tests: overlong length token (100k digits) -> exit 1; 9-digit length -> exit 1;
frame over the element cap -> exit 1. go test ./... ok, go vet clean. Found by @reviewer's
probe (reproduced 1 GB RSS), not a re-read — exactly why the outside look isn't skippable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 10:50:50 +02:00
Claude (noether) 5d2ee5cdc9 sicd: implement v1/v2 dual-read (migration) — first-byte dispatch + v1-legacy parser
The v2-only daemon rejected v1 frames, so deploying it would brick every v1 client in the
field. main() now dispatches on the FIRST byte: 0x00 = v2 (path unchanged); an ASCII digit =
a new STREAMING v1-legacy parser (that digit is the first byte of the first netstring length,
pushed back via io.MultiReader before parsing); anything else = exit 1 with a diagnostic.

v1 wire (what the deployed cmd/sic sends, no preamble): netstring(mode "exec"|"sh") +
argv/command netstrings + the empty-netstring terminator 0:, ; everything after the terminator
is the child's stdin. exec runs argv directly (each netstring is ONE argv element, never
space-split); sh runs via sh -c. Reuses pumpStdin so EPIPE / reap / exit-status semantics
match the v2 path exactly.

Removed the sicd_test.go "old-v1-frame-without-preamble expects exit 1" subtest: it pinned the
v2-only REJECTION of v1, which dual-read deliberately reverses. v1 acceptance is now covered by
dualread_test.go (9 cases: exec/sh, argv boundaries, trailing stdin, exit status, and the four
reject paths).

Implemented by noether (rich agent) after the room hit a genuine wall — DeepSeek couldn't build
the parser in 10 attempts, Kimi-k2.6 soliloquy-stalled past the 300s per-call timeout. All green:
go test ./... ok, 27/27 Python reference.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 10:42:32 +02:00
Claude (noether) 3ce50a8ba3 docs: chain link is sicd itself, no sic-run binary (verified 2-hop peel)
Decision 3 (Markus, deployment simplicity): drop the proposed sic-run helper. sicd is already
a pure v2 frame-peeler, so the inner-hop command is just sicd (incus exec <c> -- sicd) and one
binary deploys instead of two. Verified empirically, not assumed: sicd invoking sicd against a
two-layer nested frame delivered the payload to the inner hop with correct exit status and clean
stderr. Prerequisite is only that sicd is present in every container a chain traverses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 09:26:07 +02:00
Claude (noether) 77c58beb93 docs: nested-sic client design + chain-resolution decision (audit trail)
Records the 2026-07-23 bullpen design session (room msgs 865-877): the v2 client model
(client builds the whole onion, servers peel one hop each), and two decisions with their
rejected alternatives so a future reader sees the reasoning, not just the outcome.

Decision 1 — chain resolution = static per-client hosts.toml, fail-loud, no mutable store in
the routing path. mneme-as-router was proposed and REJECTED: mutable shared store = a
supply-chain RCE vector (poison one chain entry, reroute a hop's exec); staleness fails
silently into the wrong container vs a loud connection-refused; dies when boltzmann sleeps;
and adds nothing over a flat file since the client needs a local mneme address regardless.
mneme stays the fleet MEMORY store — this is narrowly about routing.

Decision 2 — migration = daemon dual-read (sniff first byte: 0x00=v2, digit=v1-legacy),
daemon-first, no flag day.

Escape hatch (documented, not built): signed manifests out-of-band if redeploy friction ever
exceeds the trust cost.

Scribed by noether from the room debate; @architect/@skeptic (deepseek) argued it, Markus
initiated and made the calls. Not code — no review gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 08:30:58 +02:00
Claude (noether) b28f9d3722 sicd: cover the read-error branch with a unit test; correct the skip rationale; drop vestigial rest
Acting on @reviewer's fourth-pass verdict (all three points accepted):

1. Refactored the pump into pumpStdin(dst, src, reap, errOut) int and covered it with unit
   tests using a mock erroring io.Reader — which IS the Python reference's mechanism
   (monkeypatching os.read to raise OSError), translated. Three cases: a genuine read error
   reaps + exits 1 + diagnoses; EPIPE is head-semantics (return child status, no diagnostic);
   clean EOF returns child status. The waitForChild added in the prior commit now has coverage.

2. Corrected the pty test's skip string. My earlier rationale was wrong twice, as @reviewer's
   raw syscall.Read probe showed: it is NOT a Go-vs-CPython poller difference — the KERNEL
   returns a clean EOF on the slave side in BOTH languages (EIO-after-hangup is master-side);
   and the Python reference never used a pty (it mocks os.read). The skip now states the
   corrected facts and points at TestPumpStdin*.

3. Dropped readNetstring's vestigial third return (rest was provably always empty).

Go ok (incl. 3 new unit tests), 27/27 Python, go vet clean.
2026-07-23 07:42:14 +02:00
Claude (noether) 3982706ebb sicd: reject malformed netstrings (security), and retire an unsatisfiable read-error test
FINDING 1 (security) — the Go daemon accepted netstrings the Python reference REJECTS and
then EXECUTED them, so a malformed frame that fails closed on a Python hop ran on a Go hop.
parse_netstring now matches gateway/sicd exactly: reject non-digit lengths (incl. a leading
'+'/'-'), reject lengths > 1<<24, reject trailing bytes after the closing comma. The three
cases now exit 1 with a diagnostic instead of exec'ing. Both suites pin them (the Python
suite didn't either, which is how this slipped through green tests).

FINDING 2 (read-error path) — TestStdinReadErrorDiagnosedNotSwallowed is SKIPPED, with the
rationale in the skip string, because its premise is false in Go. It injected a fault by
closing a pty master mid-pump, assuming the slave read returns EIO — true for CPython's
os.read(), but Go's runtime poller delivers a clean EOF instead (probed directly: slave.Read
-> io.EOF, io.Copy -> nil; a socketpair peer close is EOF too — SO_LINGER/RST is TCP-only).
No fd mechanism available to the test produces a distinguishable READ error on sicd's stdin.
That is not a gap in main.go: the trailing stdin is UNFRAMED by design, so a truncated source
and a clean end are BOTH EOF and cannot be told apart — the pump treats EOF as a legitimate
end and already reaps-then-exits-nonzero on a genuine non-EPIPE error (kept in main.go), which
just doesn't arise via pty/socket close in Go.

Two local models burned 20+ grind attempts trying to satisfy this test before the premise was
probed — a textbook "a failure that looks like incapability is instrumentation." Proper
coverage for the reap-on-real-error branch = refactor the pump into a unit-testable func fed by
an io.Reader that returns a non-EOF error; left as a follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 07:22:08 +02:00
Claude (noether) 940c3525ef sicd: port the wire protocol to the Go daemon (cmd/sicd)
The Go daemon is what actually runs on 30+ fleet hosts; until now it had none of the
wire-protocol work that went into the Python reference (gateway/sicd) — which is why sic
still ate a bare -- and silently wrote zero-byte files in production. This ports it: preamble
+ nested netstring peeling (one layer per hop), payload -> child stdin then remaining stdin
pumped through (the zero-byte-file fix), bare -- survives as opaque payload, malformed frame
-> exit 1, no deadlock above the 64 KiB pipe buffer, short writes looped, EPIPE handled,
signal-killed child -> exit 128+WTERMSIG. Built by the bullpen room (@testdesigner -> @godev
-> @reviewer), the first Go work the room could grade.

CHECKPOINT commit, green but NOT final: 28/28 Go, 24/24 Python, go vet clean. @reviewer's
third pass found four divergences from the reference that a fix round is about to address —
recorded here so the checkpoint is honest:
  1. (must-fix, security) the Go netstring parser accepts frames the reference REJECTS and
     executes them: trailing data after the comma, a +-prefixed length, and no 1<<24 cap. A
     malformed frame that fails closed on a Python hop currently runs on a Go hop. The Python
     suite does not pin these either — both suites need the malformed cases added.
  2. (must-fix) the read-error path os.Exit(1)s without reaping the child, leaving it detached
     and still writing to inherited stdout after sicd reports its status.
  3. (test) an EPIPE test's premise is wrong in Go (LookPath fails at Start, so the payload
     write never happens); the path is really covered by the signal test.
  4. (test) the pty EIO test is synchronized only by a 300 ms sleep.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 05:37:53 +02:00
Markus Fritsche a0f93c16cf sicd: add SIGPIPE/SIG_IGN porting note at EPIPE catch site
CPython sets SIGPIPE to SIG_IGN so EPIPE surfaces as BrokenPipeError.
A Go port must replicate this explicitly or the EPIPE path silently
becomes a SIGPIPE death.

24/24 green.
2026-07-23 01:05:39 +02:00
Markus Fritsche 833a0432b8 sicd: harden pump loop, EPIPE/diagnostic handling, signal fidelity
Review round 3 fixes:
- write_all helper: handle partial writes from signals
- EPIPE on write side: catch BrokenPipeError (child dead), not all OSError
- Read-side OSError: diagnose on stderr + exit 1 (not silent truncation)
- parse_netstring: sanity cap 1<<24, return content only (not tuple)
- 128+WTERMSIG for signal-killed children (shell convention)
- 24/24 pytest green
2026-07-23 01:01:59 +02:00
Claude (noether) 7f7f0c2ecb sicd: handle EPIPE and short writes at both write sites
write_all() loops until the full buffer is transferred — os.write's return value was
ignored at the payload write and in the pump loop, so a short write silently truncated
the stream. EPIPE at the payload write (child already dead) now reaps the child and
inherits its status instead of dying with a BrokenPipeError traceback; EPIPE in the pump
loop falls through to the same waitpid, which is standard `cmd | head` semantics.

Tests use deterministic fault injection rather than racing signals: a fork wrapper that
waits with waitid(WNOWAIT) so the child is provably dead but still reapable before the
parent writes, and an os.write cap of 7 bytes per call as the stand-in for a
signal-interrupted partial write.

Reviewed three times. Known gaps, tracked, NOT fixed here: signal-killed children still
collapse to exit 1 (no 128+WTERMSIG), and the pump loop's `except OSError: pass` is
broader than it should be.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 00:39:58 +02:00
Markus Fritsche 70103792df sicd: fix pipe-buffer deadlock and review findings
- Pump loop after fork: parent streams stdin to child through pipe
- netstring: enforce .isdigit() on length field
- netstring: require exact consumption (no trailing data)
- argv split: use .split space not .split
- Non-ASCII command: fail loudly
- Child exec failure: os._exit(1)

17/17 pytest green
2026-07-23 00:00:55 +02:00
Markus Fritsche 6584181f5c tests: add large-payload and nonexistent-command tests (2 red)
Adds tests that expose the pipe-buffer deadlock:
- 65,792 bytes (64 KiB + 1) hangs instead of completing
- 1 MiB body hangs
- Nonexistent command exits nonzero

15 passed, 2 failed (expected — deadlock not yet fixed)
2026-07-22 23:55:44 +02:00
Markus Fritsche b5e7e2b632 sicd: implement wire protocol (preamble + nested netstring peeling)
- Read preamble: magic byte \x00 + 4-byte BE length + netstring
- Split netstring at first NUL: command\0<payload>
- Exec command with payload + trailing stdin forwarded
- Nested payloads are netstrings for the next hop (one layer peeled per sicd)
- Bare -- survives as opaque data
- Malformed input: exit 1 with stderr diagnostic
- 14/14 pytest green
2026-07-22 23:44:00 +02:00
Markus Fritsche 4a6c1bae5c lmcp-tool: auto-map a bare http(s):// positional arg to url=
Agents (esp. small models) keep calling `fetch_url https://...` positionally
instead of `fetch_url url="https://..."`. The key=value parser turned a bare
URL into {"<url>":""} with no `url` key, so the server rejected it with the
misleading "url must start with http:// or https://". Now a lone http(s)://
positional is mapped to url=, killing this recurring pi-agent failure. Explicit
key=value and raw-JSON forms are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
2026-07-20 06:16:02 +02:00
Markus Fritsche dd987e4c1e contrib: sic companion helpers (lmcp-tool, sicwrite, sicedit)
lmcp-tool drives a host's lmcp (MCP) tools over a stateless tools/call via sic —
list for discovery, key=value args so a model never embeds JSON in a shell (apostrophe-safe).
sicwrite/sicedit pass file content as argv since sicd consumes stdin for the frame.
References the lmcp server project: git.reauktion.de/marfrit/lmcp

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
2026-07-19 20:36:05 +02:00
Markus Fritsche 99568666d3 Drop quotesu/ setup doc; make the demo take any sicd host as an argument
The quotesu setup notes exposed internal host details and aren't useful to
readers, so remove the directory. The quoting-hell demo now requires the
target host as an argument (any host running sicd) instead of defaulting to
an internal name, and its labels say sic rather than the old rs. Example
hostnames in the README/SKILL/main.go are now the neutral placeholder host1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
2026-07-19 16:03:42 +02:00
22 changed files with 2543 additions and 321 deletions
+4 -4
View File
@@ -44,10 +44,10 @@ sic <host> -- <command> [arg ...] # explicit separator
```
```
sic quotesu echo hello world
sic quotesu touch 'a b' # one file named "a b"
sic quotesu echo '$HOME' # literal, no expansion
sic --sh quotesu 'echo hi | wc -c'
sic host1 echo hello world
sic host1 touch 'a b' # one file named "a b"
sic host1 echo '$HOME' # literal, no expansion
sic --sh host1 'echo hi | wc -c'
```
## Build
+5 -5
View File
@@ -22,11 +22,11 @@ Pass each argument as a separate argument to `sic`. Do not construct netstrings;
`sic` frames them.
```
sic quotesu echo hello world
sic quotesu touch 'my file.txt'
sic quotesu echo '$HOME' # literal, no expansion
sic quotesu python3 -c 'import sys; print(sys.argv)' a 'b c'
sic --sh quotesu 'grep foo *.log | wc -l'
sic host1 echo hello world
sic host1 touch 'my file.txt'
sic host1 echo '$HOME' # literal, no expansion
sic host1 python3 -c 'import sys; print(sys.argv)' a 'b c'
sic --sh host1 'grep foo *.log | wc -l'
```
## exec vs --sh
+108
View File
@@ -0,0 +1,108 @@
package main
import (
"fmt"
"strings"
)
func parseTarget(s string) (host string, hops []string) {
for i := 0; i < len(s); i++ {
if s[i] == '/' {
host = s[:i]
hops = splitHops(s[i+1:])
return
}
}
host = s
hops = nil
return
}
func splitHops(s string) []string {
if s == "" {
return nil
}
var hops []string
start := 0
for i := 0; i < len(s); i++ {
if s[i] == '/' {
hops = append(hops, s[start:i])
start = i + 1
}
}
hops = append(hops, s[start:])
return hops
}
func verbFor(runtime, name string) (string, error) {
switch runtime {
case "incus":
return "incus exec " + name + " --", nil
case "docker":
return "docker exec " + name, nil
case "pct":
return "pct exec " + name + " --", nil
default:
return "", fmt.Errorf("unknown runtime: %s", runtime)
}
}
func resolveVerbs(hops []string, nest []string) ([]string, error) {
if len(hops) > len(nest) {
return nil, fmt.Errorf("too many hops for this host's nest depth")
}
if len(hops) == 0 {
return []string{}, nil
}
verbs := make([]string, len(hops))
for i, hop := range hops {
verb, err := verbFor(nest[i], hop)
if err != nil {
return nil, err
}
verbs[i] = verb
}
return verbs, nil
}
func parseHostsConfig(data string) (map[string][]string, error) {
cfg := make(map[string][]string)
lines := strings.Split(data, "\n")
var currentHost string
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") {
currentHost = line[1 : len(line)-1]
if _, ok := cfg[currentHost]; !ok {
cfg[currentHost] = nil
}
continue
}
if strings.HasPrefix(line, "nest = ") || strings.HasPrefix(line, "nest=") {
eqIdx := strings.Index(line, "=")
if eqIdx == -1 {
continue
}
rest := strings.TrimSpace(line[eqIdx+1:])
if !strings.HasPrefix(rest, "[") || !strings.HasSuffix(rest, "]") {
continue
}
inner := rest[1 : len(rest)-1]
if strings.TrimSpace(inner) == "" {
cfg[currentHost] = []string{}
continue
}
var items []string
for _, item := range strings.Split(inner, ",") {
item = strings.TrimSpace(item)
item = strings.Trim(item, "\"")
items = append(items, item)
}
cfg[currentHost] = items
}
}
return cfg, nil
}
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"encoding/binary"
"strconv"
"strings"
)
// nsBytes builds a djb netstring "<len>:<bytes>," over arbitrary bytes (binary-safe).
func nsBytes(b []byte) []byte {
out := append([]byte(strconv.Itoa(len(b))), ':')
out = append(out, b...)
return append(out, ',')
}
// v2Frame builds one v2 wire frame from an EXPLICIT argv (boundary-preserving) plus a payload:
//
// 0x00 ++ big-endian uint32 length of the netstring ++ netstring(content)
// content = netstring(argc) ++ <argv netstrings> ++ payload
//
// Length-framed argv is why `sic host touch 'a b'` sends ONE argument, not two — no space-split
// anywhere. The payload is the next hop's nested frame, or empty for the innermost command.
func v2Frame(argv [][]byte, payload []byte) []byte {
// content = netstring(argc) + argv netstrings + payload. The explicit count (not an empty-
// netstring terminator) makes an empty "" argument representable (reviewer 964 #1).
content := nsBytes([]byte(strconv.Itoa(len(argv))))
for _, a := range argv {
content = append(content, nsBytes(a)...)
}
content = append(content, payload...)
ns := nsBytes(content)
frame := make([]byte, 1+4+len(ns))
frame[0] = 0x00
binary.BigEndian.PutUint32(frame[1:5], uint32(len(ns)))
copy(frame[5:], ns)
return frame
}
// wrapHop wraps an inner frame for one chain hop: the hop's argv is the runtime verb split on
// spaces with "sicd" appended (the inner-hop peeler), and the inner frame is the payload. Verbs
// are space-safe by construction (no spaces in container names/ids), so splitting them is fine.
func wrapHop(verb string, inner []byte) []byte {
fields := strings.Fields(verb)
argv := make([][]byte, 0, len(fields)+1)
for _, f := range fields {
argv = append(argv, []byte(f))
}
argv = append(argv, []byte("sicd"))
return v2Frame(argv, inner)
}
// buildChain builds the whole onion: innermost v2Frame(cmdArgv, payload), then wrap outward so
// verbs[0] is the outermost layer (the one the host's sicd enters first). Zero verbs yields a
// single v2 frame (a bare host with no nesting).
func buildChain(verbs []string, cmdArgv [][]byte, payload []byte) []byte {
frame := v2Frame(cmdArgv, payload)
for i := len(verbs) - 1; i >= 0; i-- {
frame = wrapHop(verbs[i], frame)
}
return frame
}
+82
View File
@@ -0,0 +1,82 @@
package main
import (
"bytes"
"encoding/binary"
"strconv"
"testing"
)
// Independent reference builders (a second implementation, so a bug in frame.go can't cancel out
// of both sides of the assertion).
func refNS(b []byte) []byte {
out := []byte(strconv.Itoa(len(b)) + ":")
out = append(out, b...)
return append(out, ',')
}
func refFrame(content []byte) []byte {
ns := refNS(content)
f := []byte{0x00}
var l [4]byte
binary.BigEndian.PutUint32(l[:], uint32(len(ns)))
return append(append(f, l[:]...), ns...)
}
func refContent(argv [][]byte, payload []byte) []byte {
c := refNS([]byte(strconv.Itoa(len(argv)))) // argc
for _, a := range argv {
c = append(c, refNS(a)...)
}
return append(c, payload...)
}
func TestV2FrameEmptyArgRepresentable(t *testing.T) {
// reviewer 964 #1: an empty "" argument is a real element now, not a terminator collision.
argv := [][]byte{[]byte("echo"), []byte(""), []byte("x")}
if got := v2Frame(argv, nil); !bytes.Equal(got, refFrame(refContent(argv, nil))) {
t.Fatalf("empty-arg frame mismatch")
}
}
func TestV2Frame(t *testing.T) {
// An argument containing a space is ONE length-framed element — sic's founding guarantee.
argv := [][]byte{[]byte("cat"), []byte("a b")}
payload := []byte("hi")
want := refFrame(refContent(argv, payload))
if got := v2Frame(argv, payload); !bytes.Equal(got, want) {
t.Fatalf("v2Frame =\n % x\nwant\n % x", got, want)
}
}
func TestV2FrameBinaryAndEmpty(t *testing.T) {
// Payload bytes that look like framing ride untouched; nil and []byte{} payload are identical.
argv := [][]byte{[]byte("x")}
payload := []byte{0x00, 0xFF, ',', ':'}
if got := v2Frame(argv, payload); !bytes.Equal(got, refFrame(refContent(argv, payload))) {
t.Fatalf("binary payload frame mismatch")
}
if !bytes.Equal(v2Frame(argv, nil), v2Frame(argv, []byte{})) {
t.Fatal("nil and empty payload must frame identically")
}
}
func TestWrapHop(t *testing.T) {
inner := v2Frame([][]byte{[]byte("cat")}, nil)
// verb split on spaces, then "sicd" appended
want := v2Frame([][]byte{[]byte("incus"), []byte("exec"), []byte("c"), []byte("--"), []byte("sicd")}, inner)
if got := wrapHop("incus exec c --", inner); !bytes.Equal(got, want) {
t.Fatalf("wrapHop mismatch")
}
}
func TestBuildChain(t *testing.T) {
cmd := [][]byte{[]byte("cat")}
payload := []byte("hi")
if got, want := buildChain(nil, cmd, payload), v2Frame(cmd, payload); !bytes.Equal(got, want) {
t.Fatal("zero hops must equal a single v2 frame")
}
verbs := []string{"incus exec c --", "docker exec d"}
want := wrapHop(verbs[0], wrapHop(verbs[1], v2Frame(cmd, payload)))
if got := buildChain(verbs, cmd, payload); !bytes.Equal(got, want) {
t.Fatal("buildChain must fold verbs[0] as the outermost layer")
}
}
+40
View File
@@ -0,0 +1,40 @@
package main
import (
"reflect"
"testing"
)
// WP7: parseHostsConfig parses the minimal hosts.toml subset (no external dep — stdlib only):
//
// # comment
// [<host>]
// nest = ["incus", "docker"]
//
// into map[host] -> ordered runtime types. Comments and blank lines are ignored; an empty nest
// list is a present host with zero hops.
func TestParseHostsConfig(t *testing.T) {
data := "# fleet nest config\n" +
"[host1]\n" +
"nest = [\"incus\", \"docker\"]\n" +
"\n" +
"[host2]\n" +
"nest = [\"pct\"]\n" +
"\n" +
"[host3]\n" +
"nest = []\n"
cfg, err := parseHostsConfig(data)
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
if !reflect.DeepEqual(cfg["host1"], []string{"incus", "docker"}) {
t.Fatalf("host1 nest = %#v, want [incus docker]", cfg["host1"])
}
if !reflect.DeepEqual(cfg["host2"], []string{"pct"}) {
t.Fatalf("host2 nest = %#v, want [pct]", cfg["host2"])
}
b, ok := cfg["host3"]
if !ok || len(b) != 0 {
t.Fatalf("host3 must be present with an empty nest; got %#v ok=%v", b, ok)
}
}
+116 -77
View File
@@ -1,107 +1,146 @@
// sic — frame argv as netstrings and pipe to sicd on a remote host.
// sic — preserve argv boundaries across ssh (and nested container hops) by framing
// the command as length-delimited netstrings and piping it to sicd on the far end.
//
// Each argument is framed as a netstring and written to sicd's stdin over ssh.
// The netstring framing is done here so callers pass argv on the command line
// and never construct frames by hand.
// The whole point: `sic host touch 'a b'` creates ONE file, never two. No argument is
// ever space-joined or re-split on the way to the remote exec (the one exception is
// --sh, where the user explicitly asked for a shell line).
//
// Usage:
//
// sic [--sh] <host> <command> [<arg> ...]
// sic [--sh] <host> -- <command> [<arg> ...] (explicit separator)
// sic [--sh] <target> [--] <command> [<arg> ...]
//
// Examples:
// A target is a host, optionally followed by container hops separated by '/':
//
// sic quotesu echo hello world
// sic quotesu touch 'a b' # ONE file, not two
// sic --sh quotesu 'echo hi | wc -c'
// sic host echo hi # run on the host
// sic host touch 'a b' # ONE file
// sic host1/app cat /etc/os-release # into the 'app' guest on host1
// sic host2/ct/svc id # host -> pct ct -> docker svc (two hops)
// sic --sh host 'echo hi | wc -c' # shell line (space-join is intended here)
//
// Hop runtimes come from /etc/sic/hosts.toml, one stanza per host:
//
// [host1]
// nest = ["incus"]
// [host2]
// nest = ["pct", "docker"]
package main
import (
"fmt"
"io"
"os"
"os/exec"
"strconv"
"strings"
)
// netstring encodes a UTF-8 string as a netstring (djb format).
func netstring(s string) string {
encoded := []byte(s)
return strconv.Itoa(len(encoded)) + ":" + string(encoded) + ","
func fatal(format string, a ...interface{}) {
fmt.Fprintf(os.Stderr, "sic: "+format+"\n", a...)
os.Exit(1)
}
// frameArgv builds a single netstring command frame from args.
func frameArgv(args []string, mode string) []byte {
var sb strings.Builder
sb.WriteString(netstring(mode))
switch mode {
case "exec":
for _, a := range args {
sb.WriteString(netstring(a))
}
case "sh":
sb.WriteString(netstring(strings.Join(args, " ")))
func usage() {
fmt.Fprintln(os.Stderr, "Usage: sic [--sh] <target> [--] <command> [<arg> ...]")
os.Exit(1)
}
// loadNest returns the runtime nest for a host from /etc/sic/hosts.toml. Only called when
// the target actually has hops — a bare host never needs the config file.
func loadNest(host string) ([]string, error) {
const path = "/etc/sic/hosts.toml"
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("target has hops but %s is unreadable: %w", path, err)
}
sb.WriteString("0:,")
return []byte(sb.String())
cfg, err := parseHostsConfig(string(data))
if err != nil {
return nil, err
}
nest, ok := cfg[host]
if !ok {
return nil, fmt.Errorf("no [%s] stanza in %s (needed to resolve hops)", host, path)
}
return nest, nil
}
func main() {
argv := os.Args[1:]
args := os.Args[1:]
// Parse flags
mode := "exec"
for len(argv) > 0 && strings.HasPrefix(argv[0], "--") {
switch argv[0] {
case "--sh":
mode = "sh"
case "--":
argv = argv[1:]
goto parseArgs
}
argv = argv[1:]
// --sh is the only leading flag. Everything after the target is the command.
shMode := false
if len(args) > 0 && args[0] == "--sh" {
shMode = true
args = args[1:]
}
if len(args) < 1 {
usage()
}
parseArgs:
if len(argv) == 0 {
fmt.Fprintln(os.Stderr, "Usage: sic [--sh] <host> <command> [<arg> ...]")
os.Exit(1)
target := args[0]
rest := args[1:]
// Optional explicit separator so a command whose first token looks like a flag survives.
if len(rest) > 0 && rest[0] == "--" {
rest = rest[1:]
}
if len(rest) == 0 {
fatal("missing command")
}
// Locate host and command
var host string
var command []string
if idx := indexOf(argv, "--"); idx >= 0 {
host = argv[0]
command = argv[idx+1:]
} else if len(argv) >= 2 {
host = argv[0]
command = argv[1:]
// Build the command argv, boundary-preserved. --sh is the sole place a space-join is
// correct: the user asked for a shell line, not an argv.
var cmdArgv [][]byte
if shMode {
cmdArgv = [][]byte{[]byte("sh"), []byte("-c"), []byte(strings.Join(rest, " "))}
} else {
fmt.Fprintln(os.Stderr, "sic: missing command arguments")
os.Exit(1)
}
frameData := frameArgv(command, mode)
sshCmd := exec.Command("ssh", "-o", "StrictHostKeyChecking=no", host, "sicd")
sshCmd.Stdin = strings.NewReader(string(frameData))
sshCmd.Stdout = os.Stdout
sshCmd.Stderr = os.Stderr
if err := sshCmd.Run(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
os.Exit(exitErr.ExitCode())
}
fmt.Fprintf(os.Stderr, "sic: %v\n", err)
os.Exit(1)
}
}
func indexOf(slice []string, s string) int {
for i, v := range slice {
if v == s {
return i
cmdArgv = make([][]byte, len(rest))
for i, a := range rest {
cmdArgv[i] = []byte(a)
}
}
return -1
host, hops := parseTarget(target)
var verbs []string
if len(hops) > 0 {
nest, err := loadNest(host)
if err != nil {
fatal("%v", err)
}
verbs, err = resolveVerbs(hops, nest)
if err != nil {
fatal("%v", err)
}
}
frame := buildChain(verbs, cmdArgv, nil)
// ssh -T: never a pty — the frame is binary and a pty would mangle it (CR/LF, echo).
cmd := exec.Command("ssh", "-T", "-o", "StrictHostKeyChecking=no", host, "sicd")
stdin, err := cmd.StdinPipe()
if err != nil {
fatal("stdin pipe: %v", err)
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
fatal("ssh: %v", err)
}
// Write the onion, THEN forward our own stdin (the v1 client set Stdin to the frame alone
// and dropped everything after — so `sic host 'cat >f' <local` wrote a zero-byte file).
// The copy runs in a goroutine we abandon on exit: a command that ignores stdin
// (`sic host echo hi` on a tty) must not block on a tty read that never EOFs.
if _, err := stdin.Write(frame); err != nil {
fatal("write frame: %v", err)
}
go func() {
io.Copy(stdin, os.Stdin)
stdin.Close()
}()
if err := cmd.Wait(); err != nil {
if ee, ok := err.(*exec.ExitError); ok {
os.Exit(ee.ExitCode())
}
fatal("ssh: %v", err)
}
}
+27
View File
@@ -0,0 +1,27 @@
package main
import (
"reflect"
"testing"
)
// WP4: parseTarget splits "host/hop1/hop2" into the ssh host + the ordered nested hops. A bare
// host (no '/') has no hops. Pure split; validation (empty host, hop count vs nest depth) is
// resolveVerbs' job, not this one.
func TestParseTarget(t *testing.T) {
cases := []struct {
in string
host string
hops []string
}{
{"host1", "host1", nil},
{"host1/app", "host1", []string{"app"}},
{"host1/app/db", "host1", []string{"app", "db"}},
}
for _, tc := range cases {
host, hops := parseTarget(tc.in)
if host != tc.host || !reflect.DeepEqual(hops, tc.hops) {
t.Fatalf("parseTarget(%q) = (%q, %#v), want (%q, %#v)", tc.in, host, hops, tc.host, tc.hops)
}
}
}
+32
View File
@@ -0,0 +1,32 @@
package main
import (
"reflect"
"testing"
)
// WP6: resolveVerbs maps each hop to its verb via verbFor, keyed by DEPTH in the host's nest.
// Too many hops for the declared nest depth is a hard error (fail-loud); an unknown runtime
// propagates verbFor's error. Zero hops -> zero verbs (a bare host is a single v2 frame).
func TestResolveVerbs(t *testing.T) {
got, err := resolveVerbs([]string{"app", "db"}, []string{"incus", "docker"})
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
want := []string{"incus exec app --", "docker exec db"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("resolveVerbs = %#v, want %#v", got, want)
}
if v, err := resolveVerbs(nil, []string{"incus"}); err != nil || len(v) != 0 {
t.Fatalf("zero hops must yield zero verbs, no error; got %#v err %v", v, err)
}
if _, err := resolveVerbs([]string{"a", "b"}, []string{"incus"}); err == nil {
t.Fatal("more hops than the host's nest depth must be an error")
}
if _, err := resolveVerbs([]string{"x"}, []string{"bogus"}); err == nil {
t.Fatal("an unknown runtime must propagate verbFor's error")
}
}
+29
View File
@@ -0,0 +1,29 @@
package main
import "testing"
// WP5: verbFor maps a hop's runtime TYPE + container name to the runtime verb that enters it.
// The inner-hop command is this verb with " sicd" appended (wrapHop does the append), so the
// verb must end right where "sicd" should follow: incus/pct need the "--" argv separator,
// docker exec does not. Unknown runtime is a hard error (fail-loud, never guess a verb).
func TestVerbFor(t *testing.T) {
cases := []struct {
runtime, name, want string
}{
{"incus", "app", "incus exec app --"},
{"docker", "db", "docker exec db"},
{"pct", "107", "pct exec 107 --"},
}
for _, tc := range cases {
got, err := verbFor(tc.runtime, tc.name)
if err != nil {
t.Fatalf("verbFor(%q,%q) unexpected err: %v", tc.runtime, tc.name, err)
}
if got != tc.want {
t.Fatalf("verbFor(%q,%q) = %q, want %q", tc.runtime, tc.name, got, tc.want)
}
}
if _, err := verbFor("bogus", "x"); err == nil {
t.Fatal("verbFor with an unknown runtime must return an error, not guess a verb")
}
}
+217
View File
@@ -0,0 +1,217 @@
// dualread_test.go — executable spec for daemon dual-read (wire protocol v1 + v2).
//
// The DEPLOYED cmd/sic client speaks v1: raw netstrings with NO preamble —
//
// v1 frame := netstring(mode) argnets "0:,"
// mode := "exec" | "sh"
// argnets := netstring(arg)... (exec: one netstring PER argv element)
// | netstring(joined command) (sh: single space-joined command line)
//
// Bytes after the "0:," terminator are the child's stdin, 8-bit clean.
// Semantics: exec runs argv directly (NO shell, NO space-splitting — each
// netstring is one argv element); sh runs the command line through a shell;
// the daemon inherits the child's exit status.
//
// The current daemon speaks only v2 (0x00 magic + be32 length + netstring)
// and exits 1 on anything else, which bricks every deployed v1 client.
// Dual-read contract: dispatch on the FIRST byte — ASCII digit ⇒ v1 frame,
// 0x00 ⇒ v2 frame, anything else ⇒ exit 1 with a diagnostic, exec nothing.
//
// Reuses the sicd_test.go harness (same package): TestMain/sicdBin, runSicd,
// frame, netstring, concat, shScript, pattern.
//
// NOTE for the implementer: sicd_test.go's malformed-input case
// "old-v1-frame-without-preamble" asserts the OPPOSITE of this contract and
// must be deleted when dual-read lands — both cannot be green.
package main
import (
"bytes"
"strings"
"testing"
)
// v1Frame replicates cmd/sic frameArgv byte-for-byte: mode netstring, then
// args as individual netstrings ("exec") or one space-joined netstring
// ("sh"), then the empty-netstring terminator.
func v1Frame(mode string, args ...string) []byte {
var b bytes.Buffer
b.Write(netstring([]byte(mode)))
switch mode {
case "exec":
for _, a := range args {
b.Write(netstring([]byte(a)))
}
case "sh":
b.Write(netstring([]byte(strings.Join(args, " "))))
}
b.WriteString("0:,")
return b.Bytes()
}
// --- v1 acceptance (RED on the current daemon: dies at the magic-byte check) --
func TestV1ExecFrameRunsCommand(t *testing.T) {
r := runSicd(t, v1Frame("exec", "echo", "hello"))
if r.code != 0 {
t.Fatalf("v1 exec frame must run; expected exit 0, got %d (killedBy=%v), stderr=%q",
r.code, r.killedBy, r.stderr)
}
if got := string(r.stdout); got != "hello\n" {
t.Fatalf("stdout = %q, want %q", got, "hello\n")
}
}
func TestV1ExecArgvBoundariesPreserved(t *testing.T) {
// v1 exec carries each argument in its OWN netstring: an arg containing
// a space is one argv element. A daemon that funnels v1 through the v2
// space-split path hands the script two args and prints "2|a".
script := shScript(t, `printf '%s|%s\n' "$#" "$1"`)
r := runSicd(t, v1Frame("exec", string(script), "a b"))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if got, want := string(r.stdout), "1|a b\n"; got != want {
t.Fatalf("argv boundaries lost: script saw %q, want %q — v1 args must NOT be space-split", got, want)
}
}
func TestV1ExecTrailingStdinForwardedToChild(t *testing.T) {
// Bytes after the "0:," terminator are the child's stdin, verbatim:
// 1 KiB covering every byte value (NULs, 0xFF, netstring-ish colons and
// commas) must never be re-parsed as framing or mangled.
body := pattern(4)
r := runSicd(t, concat(v1Frame("exec", "cat"), body))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d (killedBy=%v), stderr=%q", r.code, r.killedBy, r.stderr)
}
if !bytes.Equal(r.stdout, body) {
t.Fatalf("trailing stdin truncated or corrupted: got %d of %d bytes", len(r.stdout), len(body))
}
}
func TestV1ShModeRunsThroughShell(t *testing.T) {
// sh mode is the one v1 path that DOES go through a shell: a pipeline
// must actually pipe. `echo hi | wc -c` ⇒ "3".
r := runSicd(t, v1Frame("sh", "echo hi | wc -c"))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if got := strings.TrimSpace(string(r.stdout)); got != "3" {
t.Fatalf("pipeline did not run through a shell: stdout = %q, want \"3\"", got)
}
}
func TestV1ChildExitStatusInherited(t *testing.T) {
r := runSicd(t, v1Frame("sh", "exit 7"))
if r.code != 7 {
t.Fatalf("v1 must inherit the child's exit status 7, got %d, stderr=%q", r.code, r.stderr)
}
}
// --- v2 must keep working alongside v1 (green today; regression guard) -------
func TestV2FrameStillAcceptedAlongsideV1(t *testing.T) {
payload := []byte("v2 payload\n")
trailing := []byte("and trailing stdin")
r := runSicd(t, concat(frame([]byte("cat"), payload), trailing))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d (killedBy=%v), stderr=%q", r.code, r.killedBy, r.stderr)
}
if want := concat(payload, trailing); !bytes.Equal(r.stdout, want) {
t.Fatalf("stdout = %q, want %q", r.stdout, want)
}
}
// --- dispatch must stay strict (green today; pins the future dispatcher) -----
func TestDispatchRejectsUnknownFirstByte(t *testing.T) {
// Dual-read must not degenerate into "anything non-0x00 is v1": a first
// byte that is neither an ASCII digit nor 0x00 exits 1, execs nothing.
r := runSicd(t, concat([]byte{0xff}, v1Frame("exec", "cat")))
if r.code != 1 {
t.Fatalf("expected exit 1, got %d (killedBy=%v)", r.code, r.killedBy)
}
if len(r.stdout) != 0 {
t.Fatalf("unknown first byte must not exec anything, stdout=%q", r.stdout)
}
if len(bytes.TrimSpace(r.stderr)) == 0 {
t.Fatal("unknown first byte must produce a diagnostic on stderr")
}
}
func TestV1TruncatedFrameMissingTerminatorExits1(t *testing.T) {
// EOF before the "0:," terminator: malformed — diagnose and exit 1,
// never hang waiting for more netstrings and never exec a partial argv.
r := runSicd(t, concat(netstring([]byte("exec")), netstring([]byte("cat"))))
if r.code != 1 {
t.Fatalf("expected exit 1, got %d (killedBy=%v), stdout=%q", r.code, r.killedBy, r.stdout)
}
if len(r.stdout) != 0 {
t.Fatalf("truncated v1 frame must not exec anything, stdout=%q", r.stdout)
}
if len(bytes.TrimSpace(r.stderr)) == 0 {
t.Fatal("truncated v1 frame must produce a diagnostic on stderr")
}
}
func TestV1UnknownModeRejected(t *testing.T) {
// Mode is a closed set {exec, sh}; anything else is malformed, not a
// command to run.
wire := concat(netstring([]byte("rm")), netstring([]byte("-rf")), []byte("0:,"))
r := runSicd(t, wire)
if r.code != 1 {
t.Fatalf("expected exit 1, got %d (killedBy=%v), stdout=%q", r.code, r.killedBy, r.stdout)
}
if len(bytes.TrimSpace(r.stderr)) == 0 {
t.Fatal("unknown v1 mode must produce a diagnostic on stderr")
}
}
func TestV1EmptyExecArgvRejected(t *testing.T) {
// "4:exec,0:," — terminator immediately after the mode: no command.
r := runSicd(t, v1Frame("exec"))
if r.code != 1 {
t.Fatalf("expected exit 1, got %d (killedBy=%v), stdout=%q", r.code, r.killedBy, r.stdout)
}
if len(bytes.TrimSpace(r.stderr)) == 0 {
t.Fatal("empty v1 exec argv must produce a diagnostic on stderr")
}
}
// --- DoS regression (reviewer 925 findings #1/#2): the length token and the frame element
// count must both be bounded BEFORE the value is trusted, or a protocol-confined caller OOMs
// the host (measured ~1 GB RSS from an unbounded digit run; ~140 MB from 2M netstrings). --
func TestV1OverlongLengthTokenRejected(t *testing.T) {
// A run of digits with no ':' must be rejected after a small cap, never buffered unbounded.
r := runSicd(t, bytes.Repeat([]byte("9"), 100_000))
if r.code != 1 {
t.Fatalf("overlong v1 length token must exit 1, got %d (killedBy=%v)", r.code, r.killedBy)
}
if len(r.stdout) != 0 {
t.Fatalf("overlong length token must exec nothing, stdout=%q", r.stdout)
}
}
func TestV1LengthTokenNineDigitsRejected(t *testing.T) {
// 9 digits (> the 8 a valid <=1<<24 length needs) is malformed, even with a ':'.
r := runSicd(t, []byte("100000000:x,0:,"))
if r.code != 1 {
t.Fatalf("9-digit v1 length must exit 1, got %d", r.code)
}
}
func TestV1FrameElementCapRejected(t *testing.T) {
// Too many netstrings in one frame must be rejected, not accumulated to OOM.
var b bytes.Buffer
b.Write(netstring([]byte("exec")))
for i := 0; i < 5000; i++ { // > maxArgs (4096)
b.Write(netstring([]byte("a")))
}
b.WriteString("0:,")
r := runSicd(t, b.Bytes())
if r.code != 1 {
t.Fatalf("v1 frame exceeding the element cap must exit 1, got %d", r.code)
}
}
+364 -89
View File
@@ -1,125 +1,400 @@
// sicd — receive netstring-framed argv over stdin, execvp it directly.
//
// Deployed on target hosts. Reads netstrings from stdin, interprets the first
// field as a mode selector ("exec" or "sh"), and runs the command accordingly.
//
// Wire format (netstrings, djb):
//
// <mode-netstring> <arg-netstring>* <empty-netstring>
//
// Usage:
//
// ssh <host> sicd < /some/netstring/frames
package main
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"os"
"os/exec"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
)
// readNetstring reads one netstring from buf. Returns (payload, rest, ok).
func readNetstring(buf []byte) ([]byte, []byte, bool) {
func readNetstring(buf []byte) ([]byte, bool) {
i := bytes.IndexByte(buf, ':')
if i < 0 {
return nil, nil, false
return nil, false
}
// Reject non-digit characters (including leading '+' or '-')
for _, c := range buf[:i] {
if c < '0' || c > '9' {
return nil, false
}
}
n, err := strconv.Atoi(string(buf[:i]))
if err != nil || n < 0 {
return nil, nil, false
return nil, false
}
// Sanity cap: reject lengths > 1<<24
if n > 1<<24 {
return nil, false
}
start := i + 1
end := start + n
if len(buf) < end+1 || buf[end] != ',' {
return nil, nil, false
return nil, false
}
return buf[start:end], buf[end+1:], true
// Reject trailing data after the netstring's closing comma
if len(buf) > end+1 {
return nil, false
}
return buf[start:end], true
}
// runFrame executes a single command frame.
func runFrame(mode string, args [][]byte) int {
switch mode {
case "exec":
argv := make([]string, len(args))
for i, a := range args {
argv[i] = string(a)
// readNetstringStream reads ONE djb netstring (<len>:<bytes>,) from a stream (unlike
// readNetstring, which parses a single fully-buffered frame). ok=false on EOF or malformed
// framing (non-digit length, short body, missing comma, length > 1<<24). Used by BOTH wire
// paths: v1 (a sequence of netstrings terminated by the empty netstring 0:,) and v2 (the argc
// count plus each argv netstring, inside parseV2Content).
func readNetstringStream(r *bufio.Reader) ([]byte, bool) {
// Read the length token digit-by-digit with a HARD cap. This must be bounded BEFORE the
// value is trusted: bufio.ReadString(':') buffers the whole pre-':' run unbounded, so an
// endless stream of digits with no ':' drives allocation to OOM (a confined caller measured
// ~1 GB RSS). A valid length is <= 1<<24 = 16777216 (8 digits), so >8 digits is malformed.
const maxLenDigits = 8
var lenBuf []byte
for {
b, err := r.ReadByte()
if err != nil {
return nil, false // EOF before ':' — truncated
}
if len(argv) == 0 {
fmt.Fprintln(os.Stderr, "sicd: exec mode with no args")
return 1
if b == ':' {
break
}
cmd := exec.Command(argv[0], argv[1:]...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return exitErr.ExitCode()
if b < '0' || b > '9' {
return nil, false
}
lenBuf = append(lenBuf, b)
if len(lenBuf) > maxLenDigits {
return nil, false
}
}
if len(lenBuf) == 0 {
return nil, false
}
n, err := strconv.Atoi(string(lenBuf))
if err != nil || n > 1<<24 {
return nil, false
}
content := make([]byte, n)
if _, err := io.ReadFull(r, content); err != nil {
return nil, false
}
comma, err := r.ReadByte()
if err != nil || comma != ',' {
return nil, false
}
return content, true
}
// parseV2Content splits a v2 netstring's content into argv and the trailing payload. Content is
// netstring(argc) + exactly argc argv netstrings + payload; reading a FIXED count (not an empty-
// netstring terminator) makes an empty "" a representable argument (reviewer 964 #1). Length-framed
// argv preserves argument boundaries and lets any byte (space, NUL, 0xFF) appear in an argument;
// readNetstringStream bounds each element and its length token, so a hostile content cannot OOM.
// NOTE: payload is simply everything after the argc-th argv netstring, so an argc lying LOW
// reclassifies the sender's own trailing netstrings as child stdin — only ever the sender's own
// frame, no trust boundary crossed (reviewer 964 #3).
func parseV2Content(content []byte) (argv [][]byte, payload []byte, ok bool) {
const maxArgs = 4096
const maxFrameBytes = 1 << 20
r := bufio.NewReader(bytes.NewReader(content))
// First netstring = argc (decimal). An explicit count makes the empty string a REPRESENTABLE
// argument: with an empty-netstring terminator, a legit "" arg was indistinguishable from the
// end of argv and silently truncated the command, bleeding the tail into stdin (reviewer 964
// #1). Reading exactly argc elements removes the collision and bounds the count (#2).
argcB, good := readNetstringStream(r)
if !good {
return nil, nil, false
}
// Canonical decimal only: reject leading '+'/'-', leading zeros ("001"), and empty. Two wire
// encodings of one argc value is a canonical-form hole in a hand-rolled protocol (reviewer 964
// #2). readNetstringStream digit-checks each netstring's LENGTH token; this checks the argc VALUE.
if len(argcB) == 0 || (len(argcB) > 1 && argcB[0] == '0') {
return nil, nil, false
}
for _, c := range argcB {
if c < '0' || c > '9' {
return nil, nil, false
}
}
argc, err := strconv.Atoi(string(argcB))
if err != nil || argc < 0 || argc > maxArgs {
return nil, nil, false
}
total := 0
argv = make([][]byte, 0, argc)
for i := 0; i < argc; i++ {
ns, good := readNetstringStream(r)
if !good {
return nil, nil, false
}
total += len(ns)
if total > maxFrameBytes {
return nil, nil, false
}
argv = append(argv, ns) // fresh slice per element; empty "" is a valid arg now
}
rest, err := io.ReadAll(r)
if err != nil {
return nil, nil, false
}
return argv, rest, true
}
func waitForChild(cmd *exec.Cmd) int {
if err := cmd.Wait(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
if ws, ok := exitErr.Sys().(syscall.WaitStatus); ok && ws.Signaled() {
return 128 + int(ws.Signal())
}
fmt.Fprintf(os.Stderr, "sicd: %v\n", err)
return 1
return exitErr.ExitCode()
}
return 0
case "sh":
if len(args) == 0 {
fmt.Fprintln(os.Stderr, "sicd: sh mode with no args")
return 1
}
shellCmd := string(args[0])
cmd := exec.Command("sh", "-c", shellCmd)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return exitErr.ExitCode()
}
fmt.Fprintf(os.Stderr, "sicd: %v\n", err)
return 1
}
return 0
default:
fmt.Fprintf(os.Stderr, "sicd: unknown mode %q\n", mode)
fmt.Fprintf(os.Stderr, "sicd: wait: %v\n", err)
return 1
}
return 0
}
func main() {
buf, err := os.ReadFile("/dev/stdin")
if err != nil {
fmt.Fprintf(os.Stderr, "sicd: read stdin: %v\n", err)
signal.Ignore(syscall.SIGPIPE)
// Dual-read dispatch on the FIRST byte, so a v2 daemon can be deployed without bricking the
// v1 clients still in the field: 0x00 = the v2 preamble; an ASCII digit = the first byte of
// a v1 netstring length (legacy, no preamble); anything else is malformed.
first := make([]byte, 1)
if _, err := io.ReadFull(os.Stdin, first); err != nil {
fmt.Fprintf(os.Stderr, "sicd: read first byte: %v\n", err)
os.Exit(1)
}
var mode string
var fields [][]byte
framesProcessed := 0
for {
field, rest, ok := readNetstring(buf)
if !ok {
break
}
buf = rest
if mode == "" {
mode = string(field)
} else if len(field) == 0 {
// Empty netstring = end of frame
if mode != "" {
runFrame(mode, fields)
framesProcessed++
}
mode = ""
fields = fields[:0]
} else {
fields = append(fields, field)
}
}
if framesProcessed == 0 {
switch {
case first[0] == 0x00:
runV2() // magic consumed; the rest (len32 + netstring + trailing) is on stdin
case first[0] >= '0' && first[0] <= '9':
// The byte we consumed for dispatch is the first digit of the first netstring's
// length — push it back before parsing.
runV1(bufio.NewReader(io.MultiReader(bytes.NewReader(first), os.Stdin)))
default:
fmt.Fprintf(os.Stderr, "sicd: invalid first byte: 0x%02x\n", first[0])
os.Exit(1)
}
}
// runV2 is the v2 wire path (magic already consumed by main): 4-byte big-endian length, one
// netstring whose content is netstring(argc) + argc argv netstrings + payload, then trailing
// stdin forwarded to the child.
func runV2() {
lenBytes := make([]byte, 4)
if _, err := io.ReadFull(os.Stdin, lenBytes); err != nil {
fmt.Fprintf(os.Stderr, "sicd: read length: %v\n", err)
os.Exit(1)
}
nsLen := binary.BigEndian.Uint32(lenBytes)
if nsLen > 1<<24 {
// reviewer #3: cap BEFORE make — a raw uint32 (up to 4 GiB) was allocated before
// readNetstring's cap ever ran (5-byte input committed ~4 GB).
fmt.Fprintf(os.Stderr, "sicd: netstring length %d exceeds cap\n", nsLen)
os.Exit(1)
}
nsBuf := make([]byte, nsLen)
if _, err := io.ReadFull(os.Stdin, nsBuf); err != nil {
fmt.Fprintf(os.Stderr, "sicd: read netstring: %v\n", err)
os.Exit(1)
}
content, ok := readNetstring(nsBuf)
if !ok {
fmt.Fprintf(os.Stderr, "sicd: invalid netstring\n")
os.Exit(1)
}
// v2 content = netstring(argc) + argc argv netstrings + payload. Length-framed argv PRESERVES
// argument boundaries (sic's founding guarantee: `touch 'a b'` is ONE file) and lets ANY byte
// appear in an argument — unlike the old space-split. The explicit argc (not an empty-netstring
// terminator) makes an empty "" argument representable (reviewer 964 #1); everything after the
// argc argv netstrings is the payload (a nested frame for the next hop, or empty).
argv, payload, ok := parseV2Content(content)
if !ok {
fmt.Fprintf(os.Stderr, "sicd: invalid v2 content (argv framing)\n")
os.Exit(1)
}
if len(argv) == 0 {
fmt.Fprintf(os.Stderr, "sicd: empty command\n")
os.Exit(1)
}
argvStr := make([]string, len(argv))
for i, a := range argv {
argvStr[i] = string(a)
}
cmd := exec.Command(argvStr[0], argvStr[1:]...)
stdinPipe, err := cmd.StdinPipe()
if err != nil {
fmt.Fprintf(os.Stderr, "sicd: create stdin pipe: %v\n", err)
os.Exit(1)
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
fmt.Fprintf(os.Stderr, "sicd: exec %s: %v\n", argvStr[0], err)
os.Exit(1)
}
if err := writeAll(stdinPipe, payload); err != nil {
if errors.Is(err, syscall.EPIPE) {
// Child died before reading all payload.
} else {
fmt.Fprintf(os.Stderr, "sicd: write payload: %v\n", err)
}
stdinPipe.Close()
os.Exit(waitForChild(cmd))
}
os.Exit(pumpStdin(stdinPipe, os.Stdin, func() int { return waitForChild(cmd) }, os.Stderr))
}
// runV1 handles the legacy v1 wire the deployed cmd/sic still sends (raw netstrings, no
// preamble): first netstring = mode ("exec" | "sh"); then the argv/command netstrings;
// terminated by the empty netstring 0:,. Everything after the terminator is the child's stdin.
// Kept only for the migration window so a v2 daemon does not reject v1 callers.
func runV1(r *bufio.Reader) {
modeB, ok := readNetstringStream(r)
if !ok {
fmt.Fprintf(os.Stderr, "sicd: v1 malformed frame (mode)\n")
os.Exit(1)
}
mode := string(modeB)
if mode != "exec" && mode != "sh" {
fmt.Fprintf(os.Stderr, "sicd: v1 unknown mode %q\n", mode)
os.Exit(1)
}
// Read argv/command netstrings until the empty-netstring terminator. EOF first = truncated.
// Cap the frame (element count AND total bytes): readNetstringStream bounds each netstring at
// 1<<24, but nothing bounds their NUMBER — 2M tiny netstrings measured ~140 MB before exec.
// NOTE: an empty netstring 0:, is ALWAYS the terminator, so a legitimate empty "" argument
// collides with it — the arg list truncates there and the rest bleeds into the child's stdin.
// That is inherent to the v1 wire (terminator overload), not a bug introduced here; the
// deployed cmd/sic produces it only if a caller passes a literal "".
const maxArgs = 4096
const maxFrameBytes = 1 << 20
var argv []string
total := 0
for {
ns, ok := readNetstringStream(r)
if !ok {
fmt.Fprintf(os.Stderr, "sicd: v1 truncated frame (no 0:, terminator)\n")
os.Exit(1)
}
if len(ns) == 0 {
break // the empty netstring 0:, terminates the frame
}
total += len(ns)
if len(argv) >= maxArgs || total > maxFrameBytes {
fmt.Fprintf(os.Stderr, "sicd: v1 frame too large\n")
os.Exit(1)
}
argv = append(argv, string(ns))
}
var cmd *exec.Cmd
switch mode {
case "exec":
if len(argv) == 0 {
fmt.Fprintf(os.Stderr, "sicd: v1 empty exec argv\n")
os.Exit(1)
}
cmd = exec.Command(argv[0], argv[1:]...) // each netstring is ONE argv element, never space-split
case "sh":
cmd = exec.Command("sh", "-c", strings.Join(argv, " "))
}
stdinPipe, err := cmd.StdinPipe()
if err != nil {
fmt.Fprintf(os.Stderr, "sicd: create stdin pipe: %v\n", err)
os.Exit(1)
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
fmt.Fprintf(os.Stderr, "sicd: exec: %v\n", err)
os.Exit(1)
}
// v1 has no framed payload: everything after 0:, (still buffered in r + the rest of stdin)
// is the child's stdin. Reuse the v2 pump so EPIPE/reap/exit-status semantics match exactly.
os.Exit(pumpStdin(stdinPipe, r, func() int { return waitForChild(cmd) }, os.Stderr))
}
// pumpStdin copies src (sicd's own stdin) into the child's stdin (dst), closes dst so the
// child sees EOF, then reaps. EPIPE from the WRITE side means the child stopped reading
// (| head semantics) — not an error, so the child's own status is returned. Any OTHER error
// (a genuine read/write failure) means the transfer was truncated: report a diagnostic, reap
// to avoid a zombie, and return non-zero REGARDLESS of the child's status — a truncated
// transfer must never be reported as success. Extracted so a mock erroring io.Reader can
// cover this branch: that is exactly how the Python reference tests it (monkeypatching
// os.read to raise OSError), since no real fd on this platform yields a read error on stdin
// (a pty slave and a socket peer both see a clean kernel EOF on hangup, in Go AND CPython).
func pumpStdin(dst io.WriteCloser, src io.Reader, reap func() int, errOut io.Writer) int {
// Race the stdin copy against the child's exit. The old code did io.Copy THEN reap, so a
// client that never closes its stdin (holds the ssh channel open with no bytes) kept the
// copy blocked on the READ long after the child had exited — sicd never reaped, ssh never
// returned, the whole call hung until stdin EOF or a timeout. Now: whichever ends first wins.
copyDone := make(chan error, 1)
go func() {
_, err := io.Copy(dst, src)
dst.Close() // a still-reading child sees EOF
copyDone <- err
}()
childExited := make(chan int, 1)
go func() { childExited <- reap() }()
select {
case err := <-copyDone:
// The stdin transfer concluded (EOF / EPIPE / error) — its outcome is meaningful.
if err != nil && !errors.Is(err, syscall.EPIPE) {
// A genuine read error truncated the transfer to a still-live child -> failure,
// regardless of the child's own eventual status (the child cannot tell a truncated
// EOF from a clean one).
fmt.Fprintf(errOut, "sicd: forward stdin: %v\n", err)
<-childExited
return 1
}
return <-childExited
case status := <-childExited:
// The child exited BEFORE the copy finished: it did not need the rest of stdin (e.g. a
// command that ignores stdin, like `true`). Stop forwarding — abandon the (possibly
// blocked) read. This is the hang fix. Still honour a truncation the child DID care
// about if the copy is finishing this very instant, but bound the wait so a genuinely
// blocked stdin cannot re-introduce the hang.
select {
case err := <-copyDone:
if err != nil && !errors.Is(err, syscall.EPIPE) {
fmt.Fprintf(errOut, "sicd: forward stdin: %v\n", err)
return 1
}
case <-time.After(50 * time.Millisecond):
}
return status
}
}
func writeAll(w io.Writer, data []byte) error {
for off := 0; off < len(data); {
n, err := w.Write(data[off:])
if err != nil {
return err
}
off += n
}
return nil
}
+698
View File
@@ -0,0 +1,698 @@
// sicd_test.go — executable spec for the sic wire protocol v2, ported from
// tests/test_sicd_protocol.py (Python reference daemon: gateway/sicd).
//
// Wire format (one hop):
//
// frame := MAGIC len32 netstring
// MAGIC := 0x00 (1 byte)
// len32 := big-endian uint32 (byte length of the netstring)
// netstring := <n>:<content>, (djb netstring)
// content := <command> 0x00 <payload> (split at the FIRST NUL; NUL required)
//
// Semantics under test: each hop peels exactly ONE layer; the payload is
// opaque bytes; the child's stdin is payload ++ every remaining byte of
// sicd's own stdin; malformed input exits 1 with a diagnostic and execs
// nothing; sicd inherits the child's exit status, signal death as
// 128+WTERMSIG.
//
// EPIPE note: CPython sets SIGPIPE to SIG_IGN at startup, so the reference
// sees BrokenPipeError. The Go runtime only shields writes to fds 1 and 2;
// on the child pipe the implementation must handle syscall.EPIPE explicitly
// or the EPIPE path becomes a SIGPIPE death / unhandled error.
//
// Black-box: TestMain builds the sicd binary once; every test drives it as a
// subprocess with a 10s hang watchdog.
package main
import (
"bytes"
"context"
"encoding/binary"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strconv"
"syscall"
"testing"
"time"
"unsafe"
)
const timeout = 10 * time.Second
var sicdBin string
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "sicd-test-")
if err != nil {
fmt.Fprintf(os.Stderr, "tempdir: %v\n", err)
os.Exit(1)
}
sicdBin = filepath.Join(dir, "sicd")
if out, err := exec.Command("go", "build", "-o", sicdBin, ".").CombinedOutput(); err != nil {
fmt.Fprintf(os.Stderr, "building sicd: %v\n%s", err, out)
os.RemoveAll(dir)
os.Exit(1)
}
code := m.Run()
os.RemoveAll(dir)
os.Exit(code)
}
// --- wire builders -----------------------------------------------------------
func netstring(data []byte) []byte {
return concat([]byte(strconv.Itoa(len(data))+":"), data, []byte{','})
}
func be32(n uint32) []byte {
b := make([]byte, 4)
binary.BigEndian.PutUint32(b, n)
return b
}
// frame builds one wire frame: magic, 4-byte BE length, netstring(command NUL payload).
func frame(command, payload []byte) []byte {
// v2 content = netstring(argc) + argc argv netstrings + payload. The command is split on
// spaces HERE (test-side convenience) to count args; the wire itself no longer space-splits —
// argv arrives length-framed so `touch 'a b'` keeps its boundary.
args := bytes.Split(command, []byte(" "))
content := netstring([]byte(strconv.Itoa(len(args))))
for _, a := range args {
content = concat(content, netstring(a))
}
content = concat(content, payload)
ns := netstring(content)
return concat([]byte{0x00}, be32(uint32(len(ns))), ns)
}
// rawFrame wraps an already-built (possibly malformed) netstring in a valid
// preamble. Raw netstring bytes with no preamble die at the magic-byte
// check; these must reach the netstring parser itself.
func rawFrame(ns []byte) []byte {
return concat([]byte{0x00}, be32(uint32(len(ns))), ns)
}
func concat(parts ...[]byte) []byte {
var b bytes.Buffer
for _, p := range parts {
b.Write(p)
}
return b.Bytes()
}
// pattern returns n repetitions of the 256-byte sequence 0x00..0xFF.
func pattern(n int) []byte {
b := make([]byte, 256)
for i := range b {
b[i] = byte(i)
}
return bytes.Repeat(b, n)
}
// shScript writes an executable /bin/sh script and returns its path as a
// command (no spaces in the path, so it survives the argv space-split).
func shScript(t *testing.T, body string) []byte {
t.Helper()
script := filepath.Join(t.TempDir(), "cmd.sh")
if err := os.WriteFile(script, []byte("#!/bin/sh\n"+body+"\n"), 0o755); err != nil {
t.Fatal(err)
}
return []byte(script)
}
// --- subprocess driver -------------------------------------------------------
type result struct {
code int
stdout []byte
stderr []byte
killedBy syscall.Signal // signal that killed sicd ITSELF, 0 if none
}
func runSicd(t *testing.T, wire []byte) result {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
cmd := exec.CommandContext(ctx, sicdBin)
cmd.Stdin = bytes.NewReader(wire)
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
waitErr := cmd.Run()
return finish(t, ctx, cmd, waitErr, &out, &errb)
}
// runSicdFeed streams stdin to sicd through a pipe under the caller's control.
func runSicdFeed(t *testing.T, feed func(w io.Writer)) result {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
cmd := exec.CommandContext(ctx, sicdBin)
stdin, err := cmd.StdinPipe()
if err != nil {
t.Fatal(err)
}
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Start(); err != nil {
t.Fatalf("starting sicd: %v", err)
}
feed(stdin)
stdin.Close()
waitErr := cmd.Wait()
return finish(t, ctx, cmd, waitErr, &out, &errb)
}
func finish(t *testing.T, ctx context.Context, cmd *exec.Cmd, waitErr error, out, errb *bytes.Buffer) result {
t.Helper()
if ctx.Err() == context.DeadlineExceeded {
t.Fatalf("sicd hung instead of completing (stdin not consumed/forwarded correctly); stderr so far: %q", errb.String())
}
if cmd.ProcessState == nil {
t.Fatalf("sicd did not run: %v", waitErr)
}
res := result{stdout: out.Bytes(), stderr: errb.Bytes()}
if ws, ok := cmd.ProcessState.Sys().(syscall.WaitStatus); ok && ws.Signaled() {
res.killedBy = ws.Signal()
res.code = -1
} else {
res.code = cmd.ProcessState.ExitCode()
}
return res
}
// --- happy path: peeling, forwarding, streaming ------------------------------
func TestFramePeeling(t *testing.T) {
trailing := []byte("then \x00 raw \xff bytes, 7:not,a netstring\n")
body64k := pattern(256) // 65,536 bytes, every byte value
body64k1 := pattern(257) // 65,792 bytes > 64 KiB pipe buffer
bodyMiB := pattern(4096) // 1 MiB
tests := []struct {
name string
wire []byte
want []byte
}{
{
// One frame, command=cat: cat's stdin is exactly the payload.
"single-hop-payload-reaches-command-stdin",
frame([]byte("cat"), []byte("hello from hop zero\n")),
[]byte("hello from hop zero\n"),
},
{
// Bytes after the frame reach the child's stdin after the
// payload, verbatim: raw NULs, 0xFF, and netstring-ish
// colon/comma material must never be re-parsed as framing.
"trailing-stdin-forwarded-after-payload",
concat(frame([]byte("cat"), []byte("payload-first:")), trailing),
concat([]byte("payload-first:"), trailing),
},
{
// The historical zero-byte-file bug: sicd read ALL of stdin for
// framing, so a streamed body after the frame silently vanished.
"zero-byte-file-regression-64k-body-not-swallowed",
concat(frame([]byte("cat"), nil), body64k),
body64k,
},
{
// Empty payload, no trailing bytes: immediate EOF, no hang.
"empty-payload-command-gets-immediate-eof",
frame([]byte("cat"), nil),
nil,
},
{
// Body exceeding the 64 KiB pipe buffer must not deadlock: the
// parent must pump through the pipe after fork, not pre-write
// the whole blob and block.
"large-payload-64k-plus-1-no-deadlock",
concat(frame([]byte("cat"), nil), body64k1),
body64k1,
},
{
// 1 MiB pins the streaming pump loop, not just the buffer edge.
"multi-mib-body-no-deadlock",
concat(frame([]byte("cat"), nil), bodyMiB),
bodyMiB,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := runSicd(t, tc.wire)
if r.code != 0 {
t.Fatalf("expected exit 0, got %d (killedBy=%v), stderr=%q", r.code, r.killedBy, r.stderr)
}
if !bytes.Equal(r.stdout, tc.want) {
t.Fatalf("stdout mismatch: got %d bytes, want %d bytes", len(r.stdout), len(tc.want))
}
})
}
}
func TestTwoHopNestedPeel(t *testing.T) {
// The outer payload is itself a complete frame (starting with the 0x00
// magic byte) for an inner sicd: each hop peels exactly ONE layer and
// never looks inside the payload. Because the inner frame is full of
// NULs, this also pins "split at the FIRST NUL" — a split at any later
// NUL corrupts the hop.
inner := frame([]byte("cat"), []byte("nested payload survived two hops\n"))
outer := frame([]byte(sicdBin), inner)
r := runSicd(t, outer)
if r.code != 0 {
t.Fatalf("expected exit 0, got %d (killedBy=%v), stderr=%q", r.code, r.killedBy, r.stderr)
}
if got, want := string(r.stdout), "nested payload survived two hops\n"; got != want {
t.Fatalf("stdout = %q, want %q", got, want)
}
}
func TestCommandSplitOnSpacesNoShell(t *testing.T) {
// argv = command split on ASCII spaces, exec'd directly (shell=False):
// shell metacharacters in the command are plain argv bytes.
r := runSicd(t, frame([]byte("echo hello $HOME ;id"), nil))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if got, want := string(r.stdout), "hello $HOME ;id\n"; got != want {
t.Fatalf("stdout = %q, want %q — command must not pass through a shell", got, want)
}
}
func TestBareDoubleDashPayloadIsOpaque(t *testing.T) {
// A bare -- inside the payload is data, never an option/separator: it
// must reach the command's stdin byte-for-byte and never abort the run.
for _, payload := range [][]byte{
[]byte("--"),
[]byte("-- --help --version\n"),
[]byte("a -- b\n"),
} {
t.Run(strconv.Quote(string(payload)), func(t *testing.T) {
r := runSicd(t, frame([]byte("cat"), payload))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if !bytes.Equal(r.stdout, payload) {
t.Fatalf("stdout = %q, want %q", r.stdout, payload)
}
})
}
}
// --- malformed input ---------------------------------------------------------
func TestMalformedInputExits1WithDiagnostic(t *testing.T) {
tests := []struct {
name string
wire []byte
}{
{"empty-input", nil},
{"missing-magic", []byte("garbage with no magic byte")},
{"body-not-a-netstring", concat([]byte{0x00}, be32(14), []byte("not&a netstrng"))},
{"truncated-declared-length", concat([]byte{0x00}, be32(100), netstring([]byte("cat\x00hi")))},
{"content-missing-nul-truncated", concat([]byte{0x00}, be32(6), netstring([]byte("ca")))},
{"content-missing-nul-separator", concat([]byte{0x00}, be32(5), netstring([]byte("ca")))},
// netstring length must be ALL digits: strconv.Atoi accepts a
// leading "+", and "+6:cat\x00hi," would exec cat with payload "hi".
{"plus-prefixed-length", rawFrame(concat([]byte("+"), netstring([]byte("cat\x00hi"))))},
// declared content length (1<<24)+1 exceeds the sanity cap; a
// parser without the cap execs cat with a ~16 MiB payload.
{"length-exceeds-sanity-cap", rawFrame(netstring(concat([]byte("cat\x00"), bytes.Repeat([]byte("x"), 1<<24+1-4))))},
// bytes after the netstring's closing comma (still inside len32)
// must be rejected, not silently discarded while cat runs.
{"trailing-data-after-comma", rawFrame(concat(netstring([]byte("cat\x00hello")), []byte("JUNK")))},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := runSicd(t, tc.wire)
if r.code != 1 {
t.Fatalf("expected exit 1, got %d (killedBy=%v)", r.code, r.killedBy)
}
if len(r.stdout) != 0 {
t.Fatalf("malformed input must not silently exec anything, stdout=%q", r.stdout)
}
if len(bytes.TrimSpace(r.stderr)) == 0 {
t.Fatal("malformed input must produce a diagnostic on stderr")
}
})
}
}
func TestNonexistentCommandExitsNonzero(t *testing.T) {
r := runSicd(t, frame([]byte("__nonexistent_command_42__"), nil))
if r.killedBy != 0 {
t.Fatalf("sicd itself was killed by %v", r.killedBy)
}
if r.code == 0 {
t.Fatal("exec of a nonexistent command must not hang or exit 0")
}
}
// --- exit-status fidelity ----------------------------------------------------
func TestChildExitStatusInherited(t *testing.T) {
r := runSicd(t, frame(shScript(t, "exit 7"), nil))
if r.code != 7 {
t.Fatalf("expected sicd to inherit child exit status 7, got %d, stderr=%q", r.code, r.stderr)
}
}
func TestSignalKilledChildExits128PlusTermsig(t *testing.T) {
// exit 1 is indistinguishable from an ordinary command failure; signal
// death must surface as 128+WTERMSIG (shell convention) so the caller
// can tell a crashed/killed command from one that merely returned false.
tests := []struct {
name string
sig string
want int
}{
{"SIGKILL-137", "KILL", 137},
{"SIGTERM-143", "TERM", 143},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := runSicd(t, frame(shScript(t, "kill -"+tc.sig+" $$"), nil))
if r.code != tc.want {
t.Fatalf("child died from SIG%s; expected exit %d (128+WTERMSIG), got %d (killedBy=%v, stderr=%q)",
tc.sig, tc.want, r.code, r.killedBy, r.stderr)
}
})
}
}
// --- fault injection: EPIPE, short reads, stdin read error -------------------
//
// The Python suite monkeypatches os.fork/os.write/os.read; a compiled Go
// binary cannot be patched, so each fault is reproduced with real kernel
// behaviour instead:
//
// widowed pipe : the child dies without reading stdin while the payload
// (256 KiB) is far larger than the 64 KiB pipe buffer, so
// the parent is guaranteed to still be writing when the
// pipe is widowed — deterministic EPIPE, no signal races.
// short reads : the whole wire arrives in 7-byte chunks over a pipe, so
// framing reads and the pump loop see partial reads.
// (Go's runtime already loops partial WRITES internally,
// so the Python short-write fault maps to the read side.)
// stdin read err : sicd's stdin is a raw pty slave; closing the master
// mid-stream makes the next read fail with EIO, exactly
// how a dying tty / revoked fd fails.
func TestEpipeChildDeadBeforePayloadWriteIsHandled(t *testing.T) {
// A nonexistent command fails at LookPath inside cmd.Start: no child is
// ever started and the payload write never happens, so this test does
// NOT exercise EPIPE (the real widowed-pipe write is covered by
// TestSignalKilledChildEpipePathExits137). What it pins: a failed exec
// must be reported on stderr naming the command, exit 1, and never
// crash — even with a 256 KiB payload pending.
payload := pattern(1024) // 256 KiB
r := runSicd(t, frame([]byte("__nonexistent_command_42__"), payload))
if r.killedBy != 0 {
t.Fatalf("sicd itself was killed by %v — EPIPE/SIGPIPE on the child pipe is not handled", r.killedBy)
}
if bytes.Contains(r.stderr, []byte("panic")) {
t.Fatalf("unhandled panic instead of clean EPIPE handling:\n%s", r.stderr)
}
if r.code != 1 {
t.Fatalf("expected the child's exit status 1, got %d, stderr=%q", r.code, r.stderr)
}
if !bytes.Contains(r.stderr, []byte("__nonexistent_command_42__")) {
t.Fatalf("stderr must name the command that failed to exec, got %q", r.stderr)
}
}
func TestSignalKilledChildEpipePathExits137(t *testing.T) {
// Same 128+WTERMSIG contract on the EPIPE recovery path: the child
// SIGKILLs itself without reading stdin, and the 256 KiB payload
// guarantees the parent is still writing into a widowed pipe when it
// dies. The status inherited there must also be 137, not 1.
r := runSicd(t, frame(shScript(t, "kill -KILL $$"), pattern(1024)))
if r.killedBy != 0 {
t.Fatalf("sicd itself was killed by %v — EPIPE/SIGPIPE on the child pipe is not handled", r.killedBy)
}
if bytes.Contains(r.stderr, []byte("panic")) {
t.Fatalf("unhandled panic on the EPIPE path:\n%s", r.stderr)
}
if r.code != 137 {
t.Fatalf("expected exit 137 (128+SIGKILL) via the EPIPE path, got %d, stderr=%q", r.code, r.stderr)
}
}
func TestFramingAndPumpSurviveTrickledStdin(t *testing.T) {
// Short-read torture: preamble, netstring, and an 8 KiB trailing body
// arrive in 7-byte chunks. Framing must loop its reads to exactness
// (a single os.Read of 5 bytes is not guaranteed 5 bytes) and the pump
// loop must forward every partial read without loss or reordering.
body := pattern(32) // 8 KiB
wire := concat(frame([]byte("cat"), nil), body)
r := runSicdFeed(t, func(w io.Writer) {
for off := 0; off < len(wire); off += 7 {
end := min(off+7, len(wire))
if _, err := w.Write(wire[off:end]); err != nil {
return // sicd died early; Wait() reports it
}
if off%(7*128) == 0 {
time.Sleep(time.Millisecond) // force fragmentation on the pipe
}
}
})
if r.code != 0 {
t.Fatalf("expected exit 0, got %d (killedBy=%v), stderr=%q", r.code, r.killedBy, r.stderr)
}
if !bytes.Equal(r.stdout, body) {
t.Fatalf("trickled stdin truncated or corrupted: got %d of %d bytes", len(r.stdout), len(body))
}
}
func TestStdinReadErrorDiagnosedNotSwallowed(t *testing.T) {
t.Skip("no real fd yields a read error on sicd's stdin on this platform, so this " +
"integration-style injection is unsatisfiable — a pty slave and a socket peer both see a " +
"clean KERNEL EOF on hangup (probed: raw syscall.Read -> n=0, errno=0), identically in Go " +
"and CPython; EIO-after-hangup is master-side, not slave-side. The Python reference does " +
"NOT use a pty here — it monkeypatches os.read to raise OSError(EIO). The Go equivalent is " +
"a mock erroring io.Reader, which now covers the reap-then-exit-nonzero branch as a UNIT " +
"test (TestPumpStdin*). This black-box test is kept only as a placeholder for that contract.")
}
// --- pty plumbing (Linux) ----------------------------------------------------
func ioctl(fd uintptr, req uintptr, arg unsafe.Pointer) error {
if _, _, errno := syscall.Syscall(syscall.SYS_IOCTL, fd, req, uintptr(arg)); errno != 0 {
return errno
}
return nil
}
// openPty opens a pty pair with the slave in raw mode so frame bytes (NULs,
// 0xFF, control chars) pass through 8-bit clean. Closing the master later
// makes slave reads fail with EIO.
func openPty(t *testing.T) (master, slave *os.File) {
t.Helper()
master, err := os.OpenFile("/dev/ptmx", os.O_RDWR|syscall.O_NOCTTY, 0)
if err != nil {
t.Fatalf("open /dev/ptmx: %v", err)
}
var ptn uint32
if err := ioctl(master.Fd(), syscall.TIOCGPTN, unsafe.Pointer(&ptn)); err != nil {
t.Fatalf("TIOCGPTN: %v", err)
}
var unlock int32
if err := ioctl(master.Fd(), syscall.TIOCSPTLCK, unsafe.Pointer(&unlock)); err != nil {
t.Fatalf("TIOCSPTLCK: %v", err)
}
slave, err = os.OpenFile(fmt.Sprintf("/dev/pts/%d", ptn), os.O_RDWR|syscall.O_NOCTTY, 0)
if err != nil {
t.Fatalf("open pty slave: %v", err)
}
var tio syscall.Termios
if err := ioctl(slave.Fd(), syscall.TCGETS, unsafe.Pointer(&tio)); err != nil {
t.Fatalf("TCGETS: %v", err)
}
tio.Iflag &^= syscall.IGNBRK | syscall.BRKINT | syscall.PARMRK | syscall.ISTRIP |
syscall.INLCR | syscall.IGNCR | syscall.ICRNL | syscall.IXON
tio.Oflag &^= syscall.OPOST
tio.Lflag &^= syscall.ECHO | syscall.ECHONL | syscall.ICANON | syscall.ISIG | syscall.IEXTEN
tio.Cflag &^= syscall.CSIZE | syscall.PARENB
tio.Cflag |= syscall.CS8
tio.Cc[syscall.VMIN] = 1
tio.Cc[syscall.VTIME] = 0
if err := ioctl(slave.Fd(), syscall.TCSETS, unsafe.Pointer(&tio)); err != nil {
t.Fatalf("TCSETS: %v", err)
}
return master, slave
}
// erroringReader yields data once, then err — the Go translation of the Python reference's
// fault injection (monkeypatching os.read to raise OSError). Lets us cover pumpStdin's
// read-error branch without an fd, since no real fd produces a read error on stdin here.
type erroringReader struct {
data []byte
err error
done bool
}
func (r *erroringReader) Read(p []byte) (int, error) {
if !r.done && len(r.data) > 0 {
n := copy(p, r.data)
r.data = r.data[n:]
if len(r.data) == 0 {
r.done = true
}
return n, nil
}
return 0, r.err
}
type errWriteCloser struct {
buf bytes.Buffer
writeErr error // if set, Write returns it (e.g. EPIPE from a widowed pipe)
closed bool
}
func (w *errWriteCloser) Write(p []byte) (int, error) {
if w.writeErr != nil {
return 0, w.writeErr
}
return w.buf.Write(p)
}
func (w *errWriteCloser) Close() error { w.closed = true; return nil }
func TestPumpStdinReapsAndExitsNonzeroOnReadError(t *testing.T) {
src := &erroringReader{data: []byte("partial payload"), err: syscall.EIO}
dst := &errWriteCloser{}
reaped := false
var errOut bytes.Buffer
code := pumpStdin(dst, src, func() int { reaped = true; return 0 }, &errOut)
if code != 1 {
t.Fatalf("a genuine read error must exit non-zero regardless of child status, got %d", code)
}
if !reaped {
t.Fatal("the child must still be reaped on a read error (no zombie)")
}
if !dst.closed {
t.Fatal("the child's stdin must be closed so it sees EOF")
}
if !bytes.Contains(errOut.Bytes(), []byte("forward stdin")) {
t.Fatalf("a diagnostic must be written, got: %q", errOut.String())
}
}
func TestPumpStdinEpipeIsHeadSemanticsReturnsChildStatus(t *testing.T) {
// EPIPE from the write side = the child stopped reading (`| head`) — NOT an error.
src := &erroringReader{data: []byte("x"), err: io.EOF} // clean src; the write side EPIPEs
dst := &errWriteCloser{writeErr: syscall.EPIPE}
var errOut bytes.Buffer
code := pumpStdin(dst, src, func() int { return 42 }, &errOut)
if code != 42 {
t.Fatalf("EPIPE is head-semantics; must return the child's own status 42, got %d", code)
}
if errOut.Len() != 0 {
t.Fatalf("EPIPE must not emit a diagnostic, got: %q", errOut.String())
}
}
// blockingReader.Read blocks forever — models a client that holds its stdin open and never
// sends EOF (exactly how the Bash tool invokes sic non-interactively).
type blockingReader struct{}
func (blockingReader) Read([]byte) (int, error) { select {} }
func TestPumpStdinReturnsOnChildExitDespiteBlockedStdin(t *testing.T) {
// Regression for the fleet-wide hang: a never-EOFing stdin must NOT keep sicd blocked in the
// copy after the child has exited. pumpStdin must return the child's status promptly.
dst := &errWriteCloser{}
done := make(chan int, 1)
go func() { done <- pumpStdin(dst, blockingReader{}, func() int { return 7 }, io.Discard) }()
select {
case code := <-done:
if code != 7 {
t.Fatalf("expected the child's status 7, got %d", code)
}
case <-time.After(2 * time.Second):
t.Fatal("pumpStdin hung on a never-EOFing stdin after the child exited (the bug)")
}
}
func TestPumpStdinCleanEofReturnsChildStatus(t *testing.T) {
src := bytes.NewReader([]byte("all of the stdin, cleanly"))
dst := &errWriteCloser{}
code := pumpStdin(dst, src, func() int { return 7 }, io.Discard)
if code != 7 {
t.Fatalf("a clean EOF is a legitimate end; must return the child's status 7, got %d", code)
}
}
// v2FrameArgv builds a v2 frame from EXPLICIT argv (no space-split) — the whole point of the
// argv-netstring wire: each element is length-framed and rides untouched.
func v2FrameArgv(argv [][]byte, payload []byte) []byte {
content := netstring([]byte(strconv.Itoa(len(argv))))
for _, a := range argv {
content = concat(content, netstring(a))
}
content = concat(content, payload)
ns := netstring(content)
return concat([]byte{0x00}, be32(uint32(len(ns))), ns)
}
func TestV2EmptyArgPreserved(t *testing.T) {
// reviewer 964 #1: an empty "" argument must be carried, not silently dropped. The old
// terminator wire ran `echo A "" B` as `echo A` (B bled into stdin). With argc all three
// survive: echo prints "A B" (two spaces — the empty middle arg).
r := runSicd(t, v2FrameArgv([][]byte{[]byte("echo"), []byte("A"), []byte(""), []byte("B")}, nil))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if got, want := string(r.stdout), "A B\n"; got != want {
t.Fatalf("empty arg not preserved: echo printed %q, want %q", got, want)
}
}
func TestV2ArgvBoundaryPreserved(t *testing.T) {
// sic's founding guarantee, now on the v2 wire: an argument containing a space is ONE argv
// element, never re-split. Before the argv-netstring wire this failed — the space-split gave
// the script two args and printed "2|a". This is the regression that pins the fix.
script := shScript(t, `printf '%s|%s\n' "$#" "$1"`)
r := runSicd(t, v2FrameArgv([][]byte{[]byte(script), []byte("a b")}, nil))
if r.code != 0 {
t.Fatalf("expected exit 0, got %d, stderr=%q", r.code, r.stderr)
}
if got, want := string(r.stdout), "1|a b\n"; got != want {
t.Fatalf("v2 argv boundary LOST: script saw %q, want %q", got, want)
}
}
// reviewer 964 #2/#3 DoS regression.
func TestV2OuterLengthCapRejected(t *testing.T) {
// 0x00 + be32(0xF0000000) + no body: must exit 1 at the length check, BEFORE make() allocs.
r := runSicd(t, concat([]byte{0x00}, be32(0xF0000000)))
if r.code != 1 {
t.Fatalf("oversize outer netstring length must exit 1, got %d (killedBy=%v)", r.code, r.killedBy)
}
}
func TestV2FrameElementCapRejected(t *testing.T) {
// argc over maxArgs (4096) must be rejected at the count check, before any element is read.
content := netstring([]byte("5000"))
for i := 0; i < 5000; i++ {
content = concat(content, netstring([]byte("a")))
}
ns := netstring(content)
r := runSicd(t, concat([]byte{0x00}, be32(uint32(len(ns))), ns))
if r.code != 1 {
t.Fatalf("v2 frame over the element cap must exit 1, got %d", r.code)
}
}
func TestV2NonCanonicalArgcRejected(t *testing.T) {
// reviewer 964 #2: argc must be canonical decimal. Leading zero, leading '+', and a signed
// value are all one-value-two-encodings and must exit 1 (never exec), even though a well-formed
// argv follows. strconv.Atoi alone would accept "+1" and "001".
for _, bad := range []string{"01", "+1", "-1", " 1", "1 "} {
content := concat(netstring([]byte(bad)), netstring([]byte("echo")))
ns := netstring(content)
r := runSicd(t, concat([]byte{0x00}, be32(uint32(len(ns))), ns))
if r.code != 1 {
t.Fatalf("non-canonical argc %q must exit 1, got %d (stdout=%q)", bad, r.code, r.stdout)
}
}
}
+30
View File
@@ -0,0 +1,30 @@
# contrib — companion helpers for driving remote tools through sic
Optional POSIX-sh helpers that pair `sic` with an **[lmcp](https://git.reauktion.de/marfrit/lmcp)**
server (a lightweight Lua MCP server). Install any of them on the hosts you reach with sic and
call them as `sic <host> <helper> ...`. None of these are required by sic itself.
## `lmcp-tool`
Call a host's lmcp (MCP) tools over a single stateless HTTP `tools/call`, with no persistent
MCP session:
```
sic <host> lmcp-tool list # discover the tools this host offers
sic <host> lmcp-tool web_search "query=today's news" # call one, args as key=value
sic <host> lmcp-tool fetch url="https://example.com"
```
The `key=value` form is the important bit: a model never has to embed JSON in a shell string,
so apostrophes (`today's`, `what's`) ride safely inside double quotes and `lmcp-tool` does the
JSON-encoding. Raw JSON (`lmcp-tool <tool> '{"k":"v"}'`) still works. Host/port/token are read
from `$LMCP_HOST/$LMCP_PORT/$LMCP_TOKEN` or the local `lmcp.service` systemd unit. See the
[lmcp project](https://git.reauktion.de/marfrit/lmcp) for the server side.
## `sicwrite` / `sicedit`
`sicd` consumes stdin for the netstring frame, so a remote command can't be fed content via a
pipe. These pass content as **argv** instead (which sic delivers losslessly):
```
sic <host> sicwrite <path> <content> # write a file
sic <host> sicedit <path> <old> <new> # replace one exact occurrence
```
+88
View File
@@ -0,0 +1,88 @@
#!/bin/sh
# lmcp-tool — call a LOCAL lmcp server's tool over a stateless HTTP tools/call.
#
# Companion client for lmcp servers (the Lua MCP server): https://git.reauktion.de/marfrit/lmcp
# Meant to be run through sic — `sic <host> lmcp-tool ...` — so a model drives a
# remote host's MCP tools without a persistent MCP session. Three forms:
#
# lmcp-tool list # DISCOVERY: available tools + descriptions
# lmcp-tool <tool> key=value [key=value ...] # args as pairs — NO json-in-shell (apostrophe-safe)
# lmcp-tool fetch_url https://example.com # a bare http(s):// arg is auto-mapped to url=
# lmcp-tool <tool> '{"k":"v"}' # raw JSON (single arg starting with '{')
#
# Prefer key=value: `sic host lmcp-tool web_search "query=today's news"` — an apostrophe
# inside double quotes is fine, and lmcp-tool JSON-encodes the pairs for you.
#
# host/port/token come from, in order: $LMCP_HOST/$LMCP_PORT/$LMCP_TOKEN → the lmcp.service
# systemd unit Environment= → the file it points at via LMCP_CONF (sudo -n if root-only;
# key godparticle|token|bearer). Defaults to 127.0.0.1:8080.
tool="$1"
[ -n "$tool" ] || { echo "usage: lmcp-tool <tool> [ key=value ... | '{json}' ] | lmcp-tool list" >&2; exit 2; }
shift
host="$LMCP_HOST"; port="$LMCP_PORT"; tok="$LMCP_TOKEN"
env=$(systemctl show lmcp.service -p Environment 2>/dev/null | tr ' ' '\n')
[ -z "$host" ] && host=$(printf '%s\n' "$env" | sed -n 's/^LMCP_HOST=//p' | head -1)
[ -z "$port" ] && port=$(printf '%s\n' "$env" | sed -n 's/^LMCP_PORT=//p' | head -1)
[ -z "$tok" ] && tok=$(printf '%s\n' "$env" | sed -n 's/^LMCP_TOKEN=//p' | head -1)
if [ -z "$tok" ]; then
conf=$(printf '%s\n' "$env" | sed -n 's/^LMCP_CONF=//p' | head -1)
[ -n "$conf" ] && tok=$( { cat "$conf" 2>/dev/null || sudo -n cat "$conf" 2>/dev/null; } \
| grep -iE 'godparticle|token|bearer' | grep -oE '[A-Za-z0-9_-]{30,}' | head -1)
fi
[ -n "$host" ] || host=127.0.0.1
[ -n "$port" ] || port=8080
post() {
curl -s -m 120 -X POST "http://${host}:${port}/mcp" \
-H "Authorization: Bearer ${tok}" -H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" -d "$1" | sed -n 's/^data: //p'
}
if [ "$tool" = "list" ] || [ "$tool" = "tools" ] || [ "$tool" = "--list" ]; then
post '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | python3 -c '
import sys, json
raw = sys.stdin.read().strip()
if not raw: sys.stderr.write("lmcp-tool: empty response (unreachable or auth failed)\n"); sys.exit(1)
d = json.loads(raw.splitlines()[-1])
if "error" in d: sys.stderr.write("lmcp error: %s\n" % d["error"]); sys.exit(1)
for t in d.get("result", {}).get("tools", []):
desc = (t.get("description") or "").splitlines()[0] if t.get("description") else ""
print("%-16s %s" % (t.get("name","?"), desc))
'
exit $?
fi
# build the arguments JSON
if [ $# -eq 0 ]; then
args='{}'
elif [ $# -eq 1 ] && [ "${1#\{}" != "$1" ]; then
args="$1" # single arg starting with '{' = raw JSON
else
args=$(python3 -c 'import sys, json
args = {}
for a in sys.argv[1:]:
if "=" in a:
k, _, v = a.partition("=")
args[k] = v
elif a.startswith(("http://", "https://")):
args["url"] = a # bare URL -> url= (agents keep passing it positionally)
else:
args[a] = ""
print(json.dumps(args))' "$@") \
|| { echo "lmcp-tool: could not build args from key=value pairs" >&2; exit 2; }
fi
body=$(printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"%s","arguments":%s}}' "$tool" "$args")
post "$body" | python3 -c '
import sys, json
raw = sys.stdin.read().strip()
if not raw: sys.stderr.write("lmcp-tool: empty response (unreachable or auth failed)\n"); sys.exit(1)
d = json.loads(raw.splitlines()[-1])
if "error" in d:
sys.stderr.write("lmcp error: %s\n" % d["error"])
sys.stderr.write("hint: run lmcp-tool list to see valid tool names\n"); sys.exit(1)
r = d.get("result", {})
print("\n".join(c["text"] for c in r.get("content", []) if c.get("type") == "text"))
sys.exit(1 if r.get("isError") else 0)
'
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
# sicedit <path> <old> <new> — replace the single exact occurrence of <old>
# with <new> in <path> (edit_file semantics). Args pass losslessly via sic.
[ $# -ge 3 ] || { echo "usage: sicedit <path> <old> <new>" >&2; exit 2; }
python3 - "$1" "$2" "$3" <<'PY'
import sys
p,o,n=sys.argv[1],sys.argv[2],sys.argv[3]
s=open(p).read()
c=s.count(o)
if c==0: sys.stderr.write("sicedit: old_string not found\n"); sys.exit(1)
if c>1: sys.stderr.write("sicedit: old_string not unique (%d matches)\n"%c); sys.exit(1)
open(p,'w').write(s.replace(o,n,1))
PY
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# sicwrite <path> <content> — write content (passed as one argv by sic, so any
# bytes survive) to path. Replaces the write_file tool for sic on any host.
[ $# -ge 2 ] || { echo "usage: sicwrite <path> <content>" >&2; exit 2; }
printf '%s' "$2" > "$1"
+29 -27
View File
@@ -3,18 +3,20 @@
Each scenario is run both ways:
A) Plain SSH — shows the breakage
B) sic tool — shows the fix
B) sic — shows the fix
Usage:
python3 demo-quoting-hell.py <host>
Point it at any host that runs sicd (reachable over SSH with your key):
python3 quoting-hell.py <host>
"""
import subprocess
import sys
import shlex
import tempfile
import os
HOST = sys.argv[1] if len(sys.argv) > 1 else "quotesu"
if len(sys.argv) < 2:
sys.exit("usage: python3 quoting-hell.py <host> # any host running sicd")
HOST = sys.argv[1]
SIC = os.path.expanduser("~/bin/sic")
SSH = ["/usr/bin/ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=5"]
@@ -29,7 +31,7 @@ def ssh(cmd_str):
def sic(*args):
"""Run argv via sic tool (which frames netstrings for sicd)."""
"""Run argv via sic (which frames netstrings for sicd)."""
return subprocess.run(
[SIC, HOST] + list(args),
capture_output=True, text=True, timeout=10
@@ -62,7 +64,7 @@ def result(label, r):
# ── 1. Space in filename ──────────────────────────────────────────
section("1. Space in filename — the classic")
sub("SSH: ssh host touch 'my file.txt'")
sub(f"SSH: ssh {HOST} touch 'my file.txt'")
# SSH concatenates args with spaces, remote shell re-splits
r = ssh("touch 'my file.txt'")
r2 = ssh("ls my file.txt 2>&1; echo ---; ls 'my file.txt' 2>&1")
@@ -72,10 +74,10 @@ print(" → 'touch' sees TWO args: 'my' and 'file.txt' — two files created"
# Clean up
ssh("rm -f my file.txt 'my file.txt'")
sub("rs: rs quotesu touch 'my file.txt'")
sub(f"sic: sic {HOST} touch 'my file.txt'")
r = sic("touch", "my file.txt")
r2 = sic("ls", "-la", "my file.txt")
result("rs", r2)
result("sic", r2)
print("'touch' sees ONE arg: 'my file.txt' — one file created")
sic("rm", "-f", "my file.txt")
@@ -83,59 +85,59 @@ sic("rm", "-f", "my file.txt")
# ── 2. Dollar sign / variable expansion ───────────────────────────
section("2. Dollar sign — unintended variable expansion")
sub("SSH: ssh host 'echo \$HOME'")
sub(f"SSH: ssh {HOST} 'echo \$HOME'")
r = ssh("echo $HOME")
result("SSH", r)
print(" → Remote shell expands $HOME — leaks local values")
sub("SSH escaped: ssh host 'echo \\$HOME'")
sub(f"SSH escaped: ssh {HOST} 'echo \\$HOME'")
r = ssh("echo \\$HOME")
result("SSH escaped", r)
sub("rs: rs quotesu echo '\$HOME'")
sub(f"sic: sic {HOST} echo '\$HOME'")
r = sic("echo", "$HOME")
result("rs", r)
result("sic", r)
print(" → No shell involved — literal $HOME printed")
# ── 3. Backticks / command substitution ──────────────────────────
section("3. Backticks — unintended command execution")
sub("SSH: ssh host 'echo `hostname`'")
sub(f"SSH: ssh {HOST} 'echo `hostname`'")
r = ssh("echo `hostname`")
result("SSH", r)
print(" → Backticks execute on remote — prints quotesu")
print(" → Backticks execute on remote — prints the remote hostname")
sub("rs: rs quotesu echo '`hostname`'")
sub(f"sic: sic {HOST} echo '`hostname`'")
r = sic("echo", "`hostname`")
result("rs", r)
result("sic", r)
print(" → Literal backticks — no execution")
# ── 4. Nested quotes ─────────────────────────────────────────────
section("4. Nested quotes — escaping hell")
sub('SSH: ssh host "echo \\"hello world\\""')
sub(f'SSH: ssh {HOST} "echo \\"hello world\\""')
r = ssh('echo "hello world"')
result("SSH", r)
sub('rs: rs quotesu echo "hello world"')
sub(f'sic: sic {HOST} echo "hello world"')
r = sic("echo", "hello world")
result("rs", r)
result("sic", r)
print(" → No nested quoting needed — just pass the string")
# ── 5. Newlines in arguments ──────────────────────────────────────
section("5. Newlines in arguments")
sub("SSH: ssh host 'echo \"line1\\nline2\"'")
sub(f"SSH: ssh {HOST} 'echo \"line1\\nline2\"'")
r = ssh('echo "line1\nline2"')
result("SSH", r)
print(" → Newline in SSH command string is fragile")
sub("rs: rs quotesu echo 'line1\\nline2'")
sub(f"sic: sic {HOST} echo 'line1\\nline2'")
r = sic("echo", "line1\nline2")
result("rs", r)
result("sic", r)
print(" → Newline is just a byte in a netstring field")
@@ -149,7 +151,7 @@ r = ssh(script)
result("SSH", r)
print(" → Try to get all those quoting levels right...")
sub("rs: rs quotesu python3 -c ... with clean args")
sub("sic: sic <host> python3 -c ... with clean args")
r = sic(
"python3", "-c",
"import sys; [print(i, repr(a)) for i,a in enumerate(sys.argv)]",
@@ -160,7 +162,7 @@ r = sic(
"arg with `ls`",
'nested "quotes"'
)
result("rs", r)
result("sic", r)
print(" → Each argument is a separate field — no escaping needed")
@@ -169,7 +171,7 @@ section("SUMMARY")
print("""
The pattern in every case:
SSH → flattens argv to one string, shell re-parses → QUOTING HELL
rs → sends argv as framed fields, execvp directly → NO ESCAPING
sic → sends argv as framed fields, execvp directly → NO ESCAPING
Netstrings (length:bytes,) are the key: the length tells the parser
exactly where each field ends, so NO delimiter needs escaping.
+94
View File
@@ -0,0 +1,94 @@
# sic — nested (multi-hop) design & chain-resolution decision
Status: **Accepted** — 2026-07-23. This records a decision and its rejected alternatives
*for the audit trail* — a future reader should see that a shared-store router was considered
and killed for cause, not overlooked.
## Goal
Make `sic host/hop1/hop2 … argv` real: run a command inside a container reached through a
chain of hops (`incus exec`, `docker exec`, `pct exec`, …) without a shell re-parsing argv at
any layer. The v2 wire-protocol **daemon** (`cmd/sicd`, `gateway/sicd`) already implements
one-layer-per-hop peeling; this covers the **client** and the rollout.
## Model (confirmed)
The **client builds the whole onion up front**: wrap the innermost command in a frame for hop
N, wrap that for hop N-1, … outermost for hop 1, and ship the nested blob to hop 1. **Each
`sicd` peels exactly one layer** (its own), runs its runtime verb with **`sicd` itself** as the
command (`incus exec <c> -- sicd`), and that inner `sicd` reads the next frame from stdin and
repeats. There is **no separate `sic-run` binary** — see Decision 3. Construction is recursive
and up-front; unrolling is iterative and distributed.
One frame = preamble (`MAGIC 0x00`, then 4-byte big-endian `len32`) + a djb netstring whose
content is `netstring(argc)` + argc argv netstrings + `payload`; the payload of a non-terminal
frame is itself a full frame for the next hop.
## Decision 1 — chain resolution: **static per-client config**
`/etc/sic/hosts.toml` declares, per host, the ordered runtime verbs for its hops
(`nest = ["incus", "docker"]`, etc.). The client reads it to turn `host/hop1/hop2` into the
concrete verb + address at each layer. **Fail loud on any mismatch** (unknown host, more hops
than the config declares) — never guess.
### Rejected: a shared memory store as the chain router
Resolving chains by querying a shared, mutable fleet-memory store was proposed and **rejected**.
Even with the bootstrap circularity removed (the store's own address is a flat local setting, so
a lookup is one direct call, not a nested one), it loses on four counts:
- **Trust (dispositive).** Such a store is *mutable and shared*. Any writer poisons one chain
entry and every subsequent `sic` call through that chain execs the attacker's verb on the
target host — a supply-chain RCE vector. Static config is immutable-after-deploy and auditable
once.
- **Staleness fails the wrong way.** A moved/renamed container makes the stored chain lie, and
the client *silently* execs into the wrong container. Static config fails loud (connection
refused); a mutable store turns a hard error into quiet wrong-target execution.
- **Availability.** The store may live on a host that sleeps; resolution would die with it, and a
local cache to cover that is just a slower, staler config file.
- **No net benefit.** The client needs a local store *address* regardless, so the store only adds
a fragile network hop + cache to do what a flat file read already does.
A shared memory store may still be useful for *memory* — this decision is narrowly about
*routing*, nothing else.
## Decision 2 — migration: **daemon dual-read, daemon-first**
The v2 daemon hard-requires the `0x00` magic; the deployed v1 client sends a digit first, so a
naive v2 deploy breaks every existing call on the many live hosts already running v1. Instead the
daemon **sniffs the first byte**`0x00` → v2 parser, digit → v1-legacy parser — a
once-per-connection branch, ~20 lines, zero perf cost. Ship the v2 daemon everywhere first; roll
clients independently, no flag day.
## Decision 3 — the chain link is `sicd` itself (no `sic-run` binary)
An earlier draft proposed a separate `sic-run` helper as the per-hop peeler. It is **not built**:
`sicd` is already a pure v2 frame-peeler, so the inner-hop command is just `sicd`
(`incus exec <c> -- sicd`), and one binary deploys instead of two (a deployment-simplicity
decision). **Verified empirically, not assumed**`sicd` invoking `sicd` was run against a
two-layer nested frame (`build sicd; frame("sicd", frame("cat", payload)) | sicd`): the payload
reached the inner hop, exit status propagated, stderr clean. sicd calling sicd peels correctly,
so nothing about a nested hop needs behaviour sicd lacks. The only prerequisite is that **sicd is
present in every container a chain traverses** (a packaging/rollout item, not new code).
## Backward compat & the two original bugs
- A **bare `--`** must survive into the innermost argv untouched (the payload is opaque bytes;
no hop interprets it) — closes the double-dash bug (#1).
- **stdin must be forwarded** into/through the frame — closes the silent zero-byte-file bug.
- A **bare host** with no `/` stays a single v1-style hop (backward compatible).
## Escape hatch (documented, not built)
If redeploying clients on chain changes ever costs more than the trust of a live store,
distribute **signed manifests out-of-band** — deterministic like static config, updatable like a
live store, without the mutable-store poison vector. Future work; explicitly not today's problem.
## Consequences
- Chain changes require redeploying clients (accepted: moves are infrequent, rollouts controlled).
- No network dependency and no mutable store in the routing path — routing is deterministic and
auditable.
- Next: implement `cmd/sic` (config read + recursive frame build + `--`/stdin fixes) and the
daemon dual-read; ensure sicd is deployed into target containers (no `sic-run` binary); both
are Go.
+125 -57
View File
@@ -1,69 +1,137 @@
#!/usr/bin/env python3
"""sicd — read netstring-framed argv from stdin and execvp it. Python reference.
Wire format and exec modes: docs/design.md.
Usage: ssh <host> sicd < <netstring-frames>
"""
"""sic wire protocol v2 daemon — read one frame, exec command, forward stdin."""
import os
import struct
import sys
import subprocess
def read_netstring(buf):
"""Read one netstring from buffer. Returns (payload, rest) or None."""
i = buf.find(b":")
if i < 0:
return None
def read_exactly(fd: int, n: int) -> bytes:
"""Read exactly n bytes from fd, or raise EOFError."""
buf = bytearray()
while len(buf) < n:
chunk = os.read(fd, n - len(buf))
if not chunk:
raise EOFError(f"expected {n} bytes, got {len(buf)}")
buf.extend(chunk)
return bytes(buf)
def write_all(fd: int, data: bytes) -> None:
"""Write all of data to fd, handling partial writes from signals."""
offset = 0
while offset < len(data):
n = os.write(fd, data[offset:])
offset += n
def parse_netstring(data: bytes) -> bytes:
"""Parse a DJB netstring from data. Returns content or raises ValueError."""
colon = data.find(b":")
if colon < 0:
raise ValueError("netstring missing colon")
if not data[:colon].isdigit():
raise ValueError("netstring length not all digits")
ns_len = int(data[:colon])
if ns_len > 1 << 24:
raise ValueError("netstring length exceeds sanity cap")
if len(data) < colon + 1 + ns_len + 1:
raise ValueError("netstring truncated")
content = data[colon + 1 : colon + 1 + ns_len]
if data[colon + 1 + ns_len : colon + 1 + ns_len + 1] != b",":
raise ValueError("netstring missing trailing comma")
rest = data[colon + 1 + ns_len + 1 :]
if rest:
raise ValueError("netstring has trailing data after comma")
return content
def main() -> None:
try:
n = int(buf[:i])
except ValueError:
return None
start = i + 1
end = start + n
if len(buf) < end + 1 or buf[end : end + 1] != b",":
return None
return buf[start:end], buf[end + 1 :]
preamble = read_exactly(sys.stdin.fileno(), 5)
except EOFError:
sys.stderr.write("sicd: EOF reading preamble\n")
sys.exit(1)
if preamble[0:1] != b"\x00":
sys.stderr.write("sicd: missing magic byte 0x00\n")
sys.exit(1)
def run_frame(mode, args):
"""Execute a single command frame."""
if mode == b"exec":
decoded = [a.decode("utf-8", errors="replace") for a in args]
result = subprocess.run(decoded)
return result.returncode
elif mode == b"sh":
shell_cmd = args[0].decode("utf-8", errors="replace")
result = subprocess.run(shell_cmd, shell=True)
return result.returncode
ns_len = struct.unpack(">I", preamble[1:5])[0]
try:
ns_data = read_exactly(sys.stdin.fileno(), ns_len)
except EOFError:
sys.stderr.write("sicd: EOF reading netstring body\n")
sys.exit(1)
try:
content = parse_netstring(ns_data)
except ValueError as e:
sys.stderr.write(f"sicd: {e}\n")
sys.exit(1)
nul_pos = content.find(b"\x00")
if nul_pos < 0:
sys.stderr.write("sicd: missing NUL separator between command and payload\n")
sys.exit(1)
command_bytes = content[:nul_pos]
payload = content[nul_pos + 1 :]
try:
command_str = command_bytes.decode("ascii")
except UnicodeDecodeError:
sys.stderr.write("sicd: command contains non-ASCII bytes\n")
sys.exit(1)
argv = command_str.split(" ")
r_fd, w_fd = os.pipe()
pid = os.fork()
if pid == 0:
os.close(w_fd)
os.dup2(r_fd, sys.stdin.fileno())
os.close(r_fd)
try:
os.execvp(argv[0], argv)
except FileNotFoundError:
sys.stderr.write(f"sicd: command not found: {argv[0]}\n")
os._exit(1)
except Exception as e:
sys.stderr.write(f"sicd: exec failed: {e}\n")
os._exit(1)
else:
print(f"sicd: unknown mode {mode!r}", file=sys.stderr)
return 1
def main():
buf = sys.stdin.buffer.read()
mode = None
fields = []
frames_processed = 0
while True:
item = read_netstring(buf)
if item is None:
break
field, buf = item
if mode is None:
mode = field # first field = mode selector
elif field == b"":
if mode is not None:
run_frame(mode, fields)
frames_processed += 1
mode = None
fields = []
os.close(r_fd)
read_error = False
try:
if payload:
write_all(w_fd, payload)
while True:
chunk = os.read(sys.stdin.fileno(), 65536)
if not chunk:
break
write_all(w_fd, chunk)
except BrokenPipeError:
# CPython sets SIGPIPE to SIG_IGN at startup, so EPIPE surfaces
# as BrokenPipeError instead of killing the process. A port to
# another language (Go, etc.) must replicate this explicitly or
# the EPIPE path silently becomes a SIGPIPE death.
pass
except OSError as e:
sys.stderr.write(f"sicd: stdin read error: {e}\n")
read_error = True
os.close(w_fd)
_, status = os.waitpid(pid, 0)
if read_error:
sys.exit(1)
if os.WIFEXITED(status):
sys.exit(os.WEXITSTATUS(status))
elif os.WIFSIGNALED(status):
sys.exit(128 + os.WTERMSIG(status))
else:
fields.append(field)
return 0 if frames_processed > 0 else 1
sys.exit(1)
if __name__ == "__main__":
sys.exit(main())
main()
-62
View File
@@ -1,62 +0,0 @@
# quotesu — test target for sicd
Incus container on `dcw2` (192.168.88.114), created as the target host for
sicd development and quoting-hell demonstrations.
## Host facts
| Property | Value |
|---|---|
| Host machine | `dcw2` (192.168.88.114, aarch64, Debian 13) |
| Container IP | 192.168.88.196/24 |
| Network | `br0` (unmanaged bridge) |
| Storage | `usb` pool (btrfs, /mnt/usb-noether) |
| OS | Debian 13 "trixie" (arm64) |
| Incus version | 6.0.4 |
| SSH user | `mfritsche` (key auth + passwordless sudo) |
## Setup steps
```sh
# 1. Create container
incus init debian-13 quotesu --storage usb --network br0
incus start quotesu
# 2. Install SSH + user
incus exec quotesu -- apt-get update -qq
incus exec quotesu -- apt-get install -y -qq openssh-server sudo python3
incus exec quotesu -- adduser --disabled-password --gecos "" mfritsche
incus file push ~/.ssh/authorized_keys quotesu/home/mfritsche/.ssh/authorized_keys
incus exec quotesu -- chown -R mfritsche:mfritsche ~mfritsche/.ssh
incus exec quotesu -- chmod 700 ~mfritsche/.ssh
incus exec quotesu -- chmod 600 ~mfritsche/.ssh/authorized_keys
incus exec quotesu -- adduser mfritsche sudo
echo "mfritsche ALL=(ALL) NOPASSWD:ALL" | \
incus exec quotesu -- tee -a /etc/sudoers.d/mfritsche
incus exec quotesu -- systemctl restart ssh
# 3. Deploy sicd
incus exec quotesu -- mkdir -p /opt/sicd
# Copy ../gateway/sicd to the container, then:
incus exec quotesu -- chmod 755 /usr/local/bin/sicd
incus exec quotesu -- ln -sf /usr/local/bin/sicd /usr/local/bin/sicd
```
## Verification
```sh
# Test the gateway
printf '4:exec,2:id,0:,' | ssh quotesu sicd
# → uid=1000(mfritsche) gid=1000(...
```
## DNS note
dcw2 had a broken `/etc/resolv.conf` (NetworkManager header, no nameservers).
Fixed by setting:
```
nameserver 1.1.1.1
nameserver 192.168.88.1
```
This was needed for `incus image copy images:debian/13 local:...` to resolve
the image server hostname.
+375
View File
@@ -0,0 +1,375 @@
"""Executable spec for the sic wire protocol v2 — Python reference daemon gateway/sicd.
Wire format (one hop):
frame := MAGIC len32 netstring
MAGIC := 0x00 (1 byte)
len32 := big-endian uint32 (byte length of the netstring that follows)
netstring := <n>:<content>, (djb netstring)
content := <command> 0x00 <payload> (split at the FIRST NUL; NUL is required)
Semantics each sicd MUST implement:
* Read exactly the frame (5 bytes preamble + len32 bytes) from stdin.
* Split content at the first NUL: command / payload.
* exec the command (argv = command split on ASCII spaces, shell=False).
* The exec'd command's stdin = payload ++ every remaining byte of sicd's
own stdin, forwarded verbatim. (This is the fix for the silent
zero-byte-file bug: trailing stream bytes must reach the child.)
* Nesting: a payload may itself be a complete frame for the next hop; each
sicd peels exactly ONE layer and never looks inside the payload.
* Payload bytes are opaque: NULs, 0x00 magic bytes, and a bare `--` inside
the payload are data, never flags or separators.
* Malformed input (missing magic, truncated frame, invalid netstring,
missing NUL separator): exit status 1, a diagnostic on stderr, nothing
exec'd.
sicd inherits the child's exit status on success paths.
"""
import signal
import struct
import subprocess
import sys
from pathlib import Path
import pytest
SICD = str((Path(__file__).resolve().parent.parent / "gateway" / "sicd"))
TIMEOUT = 10
def netstring(data: bytes) -> bytes:
return str(len(data)).encode() + b":" + data + b","
def frame(command: bytes, payload: bytes) -> bytes:
"""Build one wire frame: magic, 4-byte BE length, netstring(command NUL payload)."""
ns = netstring(command + b"\x00" + payload)
return b"\x00" + struct.pack(">I", len(ns)) + ns
def raw_frame(ns: bytes) -> bytes:
"""Wrap an already-built (possibly malformed) netstring in a valid
preamble. Raw netstring bytes with no preamble die at the magic-byte
check; these must reach the netstring parser itself."""
return b"\x00" + struct.pack(">I", len(ns)) + ns
def run_sicd(wire: bytes) -> subprocess.CompletedProcess:
try:
return subprocess.run(
[sys.executable, SICD],
input=wire,
capture_output=True,
timeout=TIMEOUT,
)
except subprocess.TimeoutExpired:
pytest.fail("sicd hung instead of completing (stdin not consumed/forwarded correctly)")
def test_single_hop_peel_payload_reaches_command_stdin():
"""One frame, command=cat: sicd execs cat and cat's stdin is exactly the payload."""
payload = b"hello from hop zero\n"
r = run_sicd(frame(b"cat", payload))
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == payload
def test_two_hop_nested_peel():
"""Payload of the outer frame is itself a frame; each sicd peels exactly one layer.
The inner frame starts with the 0x00 magic byte, so this also pins
'split at the FIRST NUL' a split at any later NUL corrupts the hop.
"""
inner = frame(b"cat", b"nested payload survived two hops\n")
outer = frame(f"{sys.executable} {SICD}".encode(), inner)
r = run_sicd(outer)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == b"nested payload survived two hops\n"
def test_trailing_stdin_forwarded_after_payload():
"""Bytes after the frame reach the child's stdin, after the payload, verbatim.
Trailing bytes are raw binary (NULs, 0xFF, a colon and comma that must not
be re-parsed as netstring material).
"""
payload = b"payload-first:"
trailing = b"then \x00 raw \xff bytes, 7:not,a netstring\n"
r = run_sicd(frame(b"cat", payload) + trailing)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == payload + trailing
def test_zero_byte_file_regression_large_body_not_swallowed():
"""The historical bug: sicd read ALL of stdin for framing, so a streamed
file body after the frame silently became a zero-byte file. A 64 KiB body
after an empty-payload frame must come out of cat intact."""
body = bytes(range(256)) * 256 # 64 KiB, every byte value
r = run_sicd(frame(b"cat", b"") + body)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == body
@pytest.mark.parametrize("payload", [b"--", b"-- --help --version\n", b"a -- b\n"])
def test_bare_double_dash_payload_is_opaque(payload):
"""A bare -- inside the payload is data, never an option/separator: it must
reach the command's stdin byte-for-byte and never abort the run."""
r = run_sicd(frame(b"cat", payload))
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == payload
@pytest.mark.parametrize(
"wire",
[
pytest.param(b"", id="empty-input"),
pytest.param(b"garbage with no magic byte", id="missing-magic"),
pytest.param(b"4:exec,3:cat,0:,", id="old-v1-frame-without-preamble"),
pytest.param(b"\x00" + struct.pack(">I", 14) + b"not&a netstrng", id="body-not-a-netstring"),
pytest.param(b"\x00" + struct.pack(">I", 100) + netstring(b"cat\x00hi"), id="truncated-declared-length"),
pytest.param(b"\x00" + struct.pack(">I", 6) + netstring(b"ca"), id="content-missing-nul-separator"),
# netstring length must be ALL digits: strconv-style parsers accept a
# leading "+" and would exec `cat` with payload "hi".
pytest.param(raw_frame(b"+" + netstring(b"cat\x00hi")), id="plus-prefixed-length"),
# declared content length (1<<24)+1 exceeds the sanity cap; a parser
# without the cap execs `cat` with a ~16 MiB payload.
pytest.param(
raw_frame(netstring(b"cat\x00" + b"x" * ((1 << 24) + 1 - 4))),
id="length-exceeds-sanity-cap",
),
# bytes after the netstring's closing comma (still inside len32) must
# be rejected, not silently discarded while `cat` runs.
pytest.param(raw_frame(netstring(b"cat\x00hello") + b"JUNK"), id="trailing-data-after-comma"),
],
)
def test_malformed_input_exits_1_with_diagnostic(wire):
"""Malformed frames: exit status exactly 1, a clear message on stderr,
and nothing exec'd (no stdout)."""
r = run_sicd(wire)
assert r.returncode == 1, f"expected exit 1, got {r.returncode}"
assert r.stdout == b"", "malformed input must not silently exec anything"
assert r.stderr.strip() != b"", "malformed input must produce a diagnostic on stderr"
def test_empty_payload_command_gets_immediate_eof():
"""Empty payload, no trailing bytes: the command runs, sees EOF at once
(no hang), produces nothing, exits 0."""
r = run_sicd(frame(b"cat", b""))
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == b""
def test_large_payload_64k_plus_1_no_deadlock():
"""Payload+trailing exceeding the 64 KiB pipe buffer must not deadlock.
The child reads from stdin, so the parent must pump through a pipe
after fork not pre-write the entire blob into a pipe and block."""
body = bytes(range(256)) * 257 # 65,792 bytes > 65,536 pipe buffer
r = run_sicd(frame(b"cat", b"") + body)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert len(r.stdout) == len(body), f"expected {len(body)} bytes, got {len(r.stdout)}"
def test_multi_mib_payload_no_deadlock():
"""1 MiB body after an empty-payload frame must stream through without
hanging or truncating pins the streaming pump loop, not just the
pipe-buffer edge."""
body = bytes(range(256)) * 4096 # 1 MiB
r = run_sicd(frame(b"cat", b"") + body)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == body
def test_nonexistent_command_exits_nonzero():
"""Exec of a command that does not exist must not hang or exit 0."""
r = run_sicd(frame(b"__nonexistent_command_42__", b""))
assert r.returncode != 0, f"expected non-zero exit, got {r.returncode}"
# ---------------------------------------------------------------------------
# Fault injection: deterministic EPIPE and short-write coverage.
#
# sicd runs under a tiny driver that patches os primitives *before* executing
# gateway/sicd, instead of racing signals against the pump loop:
#
# widow-before-write : os.fork is wrapped so the parent blocks (waitid with
# WNOWAIT -- the child stays reapable for the later
# waitpid) until the child is dead. Exec'ing a
# nonexistent command therefore guarantees the pipe is
# widowed before the payload write: the EPIPE case,
# made deterministic.
# short-write : os.write is wrapped to transfer at most 7 bytes per
# call -- the kernel's partial-write behaviour on
# signal interruption. Code that ignores os.write's
# return value truncates; a write-all loop is
# unaffected.
# stdin-read-eio : os.read is wrapped so that pump-loop-sized reads
# (> 4096 bytes) on sicd's OWN stdin (fd 0) raise
# OSError(EIO). Frame parsing (small exact reads)
# succeeds; the first pump read then fails the way a
# dying tty / revoked fd fails. A bare
# `except OSError: pass` turns this into silent
# truncation.
# ---------------------------------------------------------------------------
DRIVER = """\
import os
import sys
sicd_path = sys.argv[1]
mode = sys.argv[2]
if mode == "short-write":
real_write = os.write
def short_write(fd, data):
return real_write(fd, bytes(data)[:7])
os.write = short_write
elif mode == "widow-before-write":
real_fork = os.fork
def fork_then_wait_for_child_death(*args):
pid = real_fork(*args)
if pid:
os.waitid(os.P_PID, pid, os.WEXITED | os.WNOWAIT)
return pid
os.fork = fork_then_wait_for_child_death
elif mode == "stdin-read-eio":
import errno
real_read = os.read
def eio_read(fd, n):
if fd == 0 and n > 4096:
raise OSError(errno.EIO, "Input/output error")
return real_read(fd, n)
os.read = eio_read
else:
raise SystemExit("unknown fault mode: " + mode)
sys.argv = [sicd_path]
code = compile(open(sicd_path, "rb").read(), sicd_path, "exec")
exec(code, {"__name__": "__main__", "__file__": sicd_path})
"""
def run_sicd_fault(wire: bytes, mode: str, tmp_path) -> subprocess.CompletedProcess:
driver = tmp_path / "fault_driver.py"
driver.write_text(DRIVER)
try:
return subprocess.run(
[sys.executable, str(driver), SICD, mode],
input=wire,
capture_output=True,
timeout=TIMEOUT,
)
except subprocess.TimeoutExpired:
pytest.fail(f"sicd hung under fault mode {mode!r}")
def test_epipe_child_dead_before_payload_write_no_traceback(tmp_path):
"""Child is guaranteed dead (nonexistent command; the wrapped fork waits
for the corpse) before the parent writes the payload: sicd must handle
EPIPE and exit with the child's status -- never dump a BrokenPipeError
traceback on stderr."""
wire = frame(b"__nonexistent_command_42__", b"payload for a corpse\n")
r = run_sicd_fault(wire, "widow-before-write", tmp_path)
assert b"Traceback" not in r.stderr, (
f"unhandled exception instead of clean EPIPE handling:\n"
f"{r.stderr.decode(errors='replace')}"
)
assert b"BrokenPipeError" not in r.stderr
assert r.returncode == 1, f"expected the child's exit status 1, got {r.returncode}"
def test_partial_write_payload_not_truncated(tmp_path):
"""os.write may transfer fewer bytes than requested; the payload write
must loop on its return value. With every write capped at 7 bytes, cat
must still receive the full 37-byte payload, not just the first chunk."""
payload = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\n"
r = run_sicd_fault(frame(b"cat", payload), "short-write", tmp_path)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == payload, (
f"payload truncated by ignored os.write return value: "
f"got {len(r.stdout)} of {len(payload)} bytes"
)
def test_partial_write_pump_loop_not_truncated(tmp_path):
"""The same 7-byte write cap applied to the stdin pump loop: an 8 KiB
trailing body must reach the child intact, not 7 bytes per read chunk."""
body = bytes(range(256)) * 32 # 8 KiB
r = run_sicd_fault(frame(b"cat", b"") + body, "short-write", tmp_path)
assert r.returncode == 0, f"expected exit 0, got {r.returncode}, stderr={r.stderr!r}"
assert r.stdout == body, (
f"pump loop truncated trailing stdin: got {len(r.stdout)} of {len(body)} bytes"
)
# ---------------------------------------------------------------------------
# Exit-status fidelity: signal death and read-side errors must be visible.
#
# Tracked gaps from 7f7f0c2: both waitpid sites collapse WIFSIGNALED to
# exit 1, and the pump loop's bare `except OSError: pass` swallows read
# errors on sicd's own stdin as silent truncation.
# ---------------------------------------------------------------------------
def _selfkill_command(tmp_path, signum: int) -> bytes:
"""A command (no spaces in any argv element) whose process kills itself
with signum before producing any output."""
script = tmp_path / f"selfkill_{signum}.py"
script.write_text(f"import os\nos.kill(os.getpid(), {signum})\n")
return f"{sys.executable} {script}".encode()
@pytest.mark.parametrize(
"signum,expected",
[
pytest.param(signal.SIGKILL, 137, id="SIGKILL-137"),
pytest.param(signal.SIGTERM, 143, id="SIGTERM-143"),
],
)
def test_signal_killed_child_exits_128_plus_termsig(tmp_path, signum, expected):
"""A child killed by a signal must surface as exit 128+WTERMSIG (shell
convention), NOT collapse to exit 1 -- exit 1 is indistinguishable from
an ordinary command failure, so the caller cannot tell a crashed/killed
command from one that merely returned false."""
r = run_sicd(frame(_selfkill_command(tmp_path, signum), b""))
assert r.returncode == expected, (
f"child died from signal {signum}; expected exit {expected} "
f"(128+WTERMSIG), got {r.returncode} -- signal death collapsed "
f"into an ordinary failure code"
)
def test_signal_killed_child_epipe_path_exits_128_plus_termsig(tmp_path):
"""Same contract on the EPIPE recovery path: the child is provably dead
from SIGKILL before the parent writes the payload (widowed pipe), and the
status inherited there must also be 128+WTERMSIG, not 1."""
wire = frame(_selfkill_command(tmp_path, int(signal.SIGKILL)), b"payload for a corpse\n")
r = run_sicd_fault(wire, "widow-before-write", tmp_path)
assert b"Traceback" not in r.stderr, r.stderr.decode(errors="replace")
assert r.returncode == 137, (
f"expected exit 137 (128+SIGKILL) via the EPIPE path, got {r.returncode}"
)
def test_stdin_read_oserror_is_diagnosed_not_swallowed(tmp_path):
"""OSError on reading sicd's OWN stdin (EIO here; EBADF, ENXIO likewise)
is not `| head` semantics -- only BrokenPipeError on the write side is.
Today `except OSError: pass` silently drops the rest of the stream and
exits 0 as if the transfer completed. Required: a diagnostic on stderr
(a handled message, not a traceback) and a non-zero exit, so truncation
is never reported as success."""
payload = b"payload written before the fault\n"
trailing = b"trailing bytes the pump loop will never deliver" * 200
r = run_sicd_fault(frame(b"cat", payload) + trailing, "stdin-read-eio", tmp_path)
assert r.returncode != 0, (
"stdin read failed mid-stream (EIO) but sicd exited 0 -- "
"silent truncation reported as success"
)
assert r.stderr.strip() != b"", (
"stdin read failed mid-stream but no diagnostic was written to stderr"
)
assert b"Traceback" not in r.stderr, (
f"want a handled diagnostic, not an unhandled exception:\n"
f"{r.stderr.decode(errors='replace')}"
)