marfrit/marfrit-packages
0
0
Fork 1
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

marfrit-publish reports false-positive failure on scp clean-shutdown #4

New Issue
Closed
opened 2026-04-29 21:19:28 +00:00 by marfrit · 2 comments
marfrit commented 2026-04-29 21:19:28 +00:00
Owner
Copy Link
Copy Source

Bug

/usr/local/bin/marfrit-publish on boltzmann reports "Connection closed" and exits non-zero immediately after the scp upload to hertz, even when the upload completed successfully. The hertz-side step (/opt/herding/bin/marfrit-publish-arch) is then never invoked, leaving the package staged in hertz:/tmp/ but not signed/repo-add'd/rsync'd.

Reproduce

On boltzmann, with a freshly-built package:

cd /home/mfritsche/marfrit-packages/arch/vulkan-panfrost
/usr/local/bin/marfrit-publish vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst

Output:

==> scp vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst -> hertz.fritz.box:/tmp/
scp: Connection closed

Exit non-zero.

Underlying

scp -v against the same target shows the file is fully transferred (Transferred: sent 2513440, received 5940 bytes ... Exit status 0) — the "Connection closed" is just sftp's normal subsystem shutdown, not an error. The script is treating it as one.

Workaround used in this case: re-run the hertz-side step manually:

ssh hertz 'sudo /opt/herding/bin/marfrit-publish-arch aarch64 /tmp/vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst'

That published cleanly on first try.

Likely fix

marfrit-publish probably runs scp ... 2>&1 and greps the stderr for "closed" or relies on a non-zero exit from a sub-process that's not the actual scp. Either:

  • Switch the scp to a check on ${PIPESTATUS[0]} of the scp specifically, or
  • Use rsync -e ssh which has cleaner exit-status semantics, or
  • After the transfer, ssh and verify the file landed before declaring failure.

Impact

Low — the workaround is one ssh away. But every marfrit-publish invocation today has hit this, so it eats one extra command per publish. Worth ~10 minutes to fix.

Context

Discovered while publishing vulkan-panfrost-1:26.0.5-1-aarch64 from the fourier campaign (2026-04-29 evening session). Same pattern hit the firefox-fourier and kwin-fourier publishes too, per the his agent's session notes from earlier the same evening.

## Bug `/usr/local/bin/marfrit-publish` on boltzmann reports "Connection closed" and exits non-zero immediately after the scp upload to hertz, even when the upload completed successfully. The hertz-side step (`/opt/herding/bin/marfrit-publish-arch`) is then never invoked, leaving the package staged in `hertz:/tmp/` but not signed/repo-add'd/rsync'd. ## Reproduce On boltzmann, with a freshly-built package: ``` cd /home/mfritsche/marfrit-packages/arch/vulkan-panfrost /usr/local/bin/marfrit-publish vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst ``` Output: ``` ==> scp vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst -> hertz.fritz.box:/tmp/ scp: Connection closed ``` Exit non-zero. ## Underlying `scp -v` against the same target shows the file is fully transferred (`Transferred: sent 2513440, received 5940 bytes ... Exit status 0`) — the "Connection closed" is just sftp's normal subsystem shutdown, not an error. The script is treating it as one. Workaround used in this case: re-run the hertz-side step manually: ``` ssh hertz 'sudo /opt/herding/bin/marfrit-publish-arch aarch64 /tmp/vulkan-panfrost-1:26.0.5-1-aarch64.pkg.tar.zst' ``` That published cleanly on first try. ## Likely fix `marfrit-publish` probably runs `scp ... 2>&1` and greps the stderr for "closed" or relies on a non-zero exit from a sub-process that's not the actual scp. Either: - Switch the scp to a check on `${PIPESTATUS[0]}` of the scp specifically, or - Use `rsync -e ssh` which has cleaner exit-status semantics, or - After the transfer, ssh and verify the file landed before declaring failure. ## Impact Low — the workaround is one ssh away. But every `marfrit-publish` invocation today has hit this, so it eats one extra command per publish. Worth ~10 minutes to fix. ## Context Discovered while publishing `vulkan-panfrost-1:26.0.5-1-aarch64` from the fourier campaign (2026-04-29 evening session). Same pattern hit the firefox-fourier and kwin-fourier publishes too, per the his agent's session notes from earlier the same evening.
marfrit commented 2026-04-29 22:21:29 +00:00
Author
Owner
Copy Link
Copy Source

Root cause found

The "Connection closed" is not just a false positive — for filenames containing a colon (epoch-versioned packages like qt6-base-1:6.11.0-2-...), scp parses the colon as a host:path separator and tries to treat qt6-base-1 as a remote host. The connection then refuses, scp aborts mid-transfer, and the file does NOT land on hertz.

Repro on boltzmann:

$ scp qt6-base-1:6.11.0-2-aarch64.pkg.tar.zst hertz.fritz.box:/tmp/
ssh: Could not resolve hostname qt6-base-1: Name or service not known
scp: Connection closed

For the earlier vulkan-panfrost-1:26.0.5-1-... case, the file did transfer despite the warning — that turned out to be rsync semantics (or the way the shell quoted that specific filename). With scp directly and a colon-bearing filename, the transfer always fails.

Fix

Either:

  1. In marfrit-publish, prefix the source argument with ./ to defeat scp host:path parsing: 
    scp "./$pkg" "$HERTZ_HOST:/tmp/"
  2. Use -- separator if scp version supports it.
  3. Switch to rsync -e ssh -- which has cleaner argument handling.

Option 1 is one line. Option 3 is cleaner long-term and would also fix the false-positive-on-clean-shutdown sub-issue.

Verified workaround

Manual scp from boltzmann with ./ prefix worked for qt6-base-1:6.11.0-2-aarch64.pkg.tar.zst and qt6-xcb-private-headers-1:6.11.0-2-aarch64.pkg.tar.zst this evening; both published cleanly via the hertz step after that.

## Root cause found The "Connection closed" is not just a false positive — for filenames containing a colon (epoch-versioned packages like `qt6-base-1:6.11.0-2-...`), `scp` parses the colon as a `host:path` separator and tries to treat `qt6-base-1` as a remote host. The connection then refuses, scp aborts mid-transfer, and the file does NOT land on hertz. Repro on boltzmann: ```sh $ scp qt6-base-1:6.11.0-2-aarch64.pkg.tar.zst hertz.fritz.box:/tmp/ ssh: Could not resolve hostname qt6-base-1: Name or service not known scp: Connection closed ``` For the earlier `vulkan-panfrost-1:26.0.5-1-...` case, the file did transfer despite the warning — that turned out to be `rsync` semantics (or the way the shell quoted that specific filename). With `scp` directly and a colon-bearing filename, the transfer always fails. ## Fix Either: 1. In `marfrit-publish`, prefix the source argument with `./` to defeat scp host:path parsing: ```bash scp "./$pkg" "$HERTZ_HOST:/tmp/" ``` 2. Use `--` separator if scp version supports it. 3. Switch to `rsync -e ssh --` which has cleaner argument handling. Option 1 is one line. Option 3 is cleaner long-term and would also fix the false-positive-on-clean-shutdown sub-issue. ## Verified workaround Manual scp from boltzmann with `./` prefix worked for `qt6-base-1:6.11.0-2-aarch64.pkg.tar.zst` and `qt6-xcb-private-headers-1:6.11.0-2-aarch64.pkg.tar.zst` this evening; both published cleanly via the hertz step after that.
marfrit commented 2026-05-18 07:10:18 +00:00
Author
Owner
Copy Link
Copy Source

Fixed in /usr/local/bin/marfrit-publish on boltzmann 2026-05-17 (option 1 from the analysis above):

# Stage to hertz /tmp; the publish helper will move it into incoming.
echo "==> scp $base -> $HERTZ_HOST:/tmp/"
case "$pkg" in /*) src="$pkg" ;; *) src="./$pkg" ;; esac
scp -q "$src" "$HERTZ_HOST:/tmp/$base"

The case prefixes relative paths with ./ (absolute paths pass through unchanged) so scp can never misparse a : inside the filename as a host:path separator. Verified live during the kwin-fourier 6.6.5 and qt6-base-fourier 6.11.1 publishes that triggered the bug originally — both pushed cleanly without intervention.

Memory entry covering the gotcha for future operators: feedback_marfrit_publish_colon.md in the noether memory store.

Closing.

Fixed in `/usr/local/bin/marfrit-publish` on boltzmann 2026-05-17 (option 1 from the analysis above): ```sh # Stage to hertz /tmp; the publish helper will move it into incoming. echo "==> scp $base -> $HERTZ_HOST:/tmp/" case "$pkg" in /*) src="$pkg" ;; *) src="./$pkg" ;; esac scp -q "$src" "$HERTZ_HOST:/tmp/$base" ``` The `case` prefixes relative paths with `./` (absolute paths pass through unchanged) so scp can never misparse a `:` inside the filename as a `host:path` separator. Verified live during the kwin-fourier 6.6.5 and qt6-base-fourier 6.11.1 publishes that triggered the bug originally — both pushed cleanly without intervention. Memory entry covering the gotcha for future operators: `feedback_marfrit_publish_colon.md` in the noether memory store. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
marfrit (Markus Fritsche)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: marfrit/marfrit-packages#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?