Skip to content

postmortem · · 19 min

An unsigned Windows program no antivirus holds

Shipping a Go program to cheap office PCs with no code-signing certificate. What broke in the older PowerShell and NSSM installer, why console windows flashed, how an endpoint antivirus froze setup by holding netsh, and why the setup window never appeared after Windows' Yes. Each fix, and the guard it left behind.

The connector is a Go program on an ordinary office PC that reads the attendance machines on the network and sends every punch to a cloud attendance platform; the architecture post has the design. This one is about getting it to install, and keep running, on the PCs it lands on: mostly Windows, mostly cheap, rarely looked after by IT, some with an endpoint antivirus that suspends programs until a person decides, some with Smart App Control on. And the Windows build has no code-signing certificate. The Mac app is signed and notarised; the Windows file is not.

So before it reads a punch, the program has three jobs: nothing may hold it, block it or hide it. This post is the failures against each, in order, each with its fix and the guard that fails without it.

Before the connector: scripts reading other programs' exit codes

The older edge agent was Python. On Windows, setup.bat started PowerShell, which built a Python virtual environment, installed packages, and registered the agent as a service through NSSM, a small service wrapper. In July and August 2026 most of those steps broke on someone's PC:

  • PowerShell 7.4 turned exit codes into errors. Under -ErrorAction Stop, its default $PSNativeCommandUseErrorActionPreference aborts on any non-zero native exit, and robocopy exits 1 after a successful copy.
  • PowerShell 5.1 turned stderr into errors. Opting out does nothing on 5.1, what stock Windows ships. There, a redirected stderr line becomes a terminating NativeCommandError even on exit 0, and nssm prints "Can't open service!" on every first install.
  • An em-dash stopped the script parsing. 5.1 reads a .ps1 with no BOM as Windows-1252. A test now requires every shipped script to be ASCII or carry a BOM; a month later it went red on six em-dashes in comments.
  • NSSM's service states. nssm remove on a running service only marks it for deletion, so a reinstall left no service. nssm start waits only briefly, so a cold Python start printed a red failure over a healthy one.
  • A reinstall froze. robocopy's default retries add up to about 347 days for one file the running agent held open.
  • Downloads at install time. Python by hand, packages from PyPI unless the zip carried a wheels folder, NSSM from its website unless bundled. The NSSM download branch crashed on an undeclared variable and closed the window with "Service install failed" and no reason.

Each was fixed, and each had one shape: a layer between "double-click" and "service running" reading another layer's output by version-specific rules.

Five layers that each read the one before, and one program that does notthe older agentsetup.batPowerShell 5.1/7.4venv + pipnssm.exeserviceem-dash, no BOM:no parseexit codes andstderr = errorsPyPI or wheels;Python by handdownload branchcrashedSTART_PENDINGread as failurethe connectorone .exeWindows asks: Yeselevated copyService Ctrl Mgrrunningnothing fetchedonce, for the servicecopies itself toan admin folderAPI calls, nota wrappersays Runningitselfred: what broke, or had to be worked around, in July and August 2026

The older chain had five places where one program read another's output under version-dependent rules. In the connector's chain there is only one program.

The connector removes the layers instead: one Go binary per architecture, nothing downloaded during install. It copies itself into a folder only administrators can write (a SYSTEM service must never run a binary from Downloads), registers with the Service Control Manager through golang.org/x/sys/windows/svc/mgr, and reports Running from its own handler, so there is no start-pending window for a wrapper to misread:

terminal
s, err := m.CreateService(name, exe, mgr.Config{StartType: mgr.StartAutomatic}, "run")
// An office whose connector died at 2am should be syncing again by 2:01.
s.SetRecoveryActions([]mgr.RecoveryAction{
	{Type: mgr.ServiceRestart, Delay: 10 * time.Second},
	{Type: mgr.ServiceRestart, Delay: 30 * time.Second},
	{Type: mgr.ServiceRestart, Delay: time.Minute},
}, uint32((24 * time.Hour).Seconds()))
s.SetRecoveryActionsOnNonCrashFailures(true) // a clean but failing exit counts too

Removing the scripts did not remove Windows. The rest is what it did next.

Console windows that flashed

What it looked like. On 2026-11-04, after a person typed the code into the setup window, black console windows flashed one after another. Then the install failed: "Could not put the program in C:\Program Files\...: Access is denied."

What was happening. Two bugs. The setup window is a GUI program, and every console program it started (netsh, arp, a network check) got a console of its own, because nothing told Windows not to make one. And Windows will not replace an .exe that is running: a window opened from that folder, or a service still stopping, keeps the file in use.

The fix. One way to start a program, and only one:

terminal
// Command is exec.CommandContext, without a console window on Windows.
func Command(ctx context.Context, name string, args ...string) *exec.Cmd {
	cmd := exec.CommandContext(ctx, name, args...)
	hide(cmd) // Windows: HideWindow + CREATE_NO_WINDOW; elsewhere a no-op
	return cmd
}

For the file in use: Windows will not overwrite a running program but lets it be renamed. If the new copy cannot be renamed into place, the running one is moved aside to .old (and put back if even that fails), and the next install deletes it. Its guard swaps rename for a fake that says "Access is denied" while the destination exists, so it runs on any OS.

The guard for the console. "Always use hidden.Command" lasts until the next person types exec.Command, so a test reads the source the way the Windows compiler would:

terminal
ctx := build.Default
ctx.GOOS, ctx.GOARCH, ctx.CgoEnabled = "windows", "amd64", false
pkg, _ := ctx.ImportDir(dir, 0) // every dir but internal/hidden: what Windows compiles
for _, f := range pkg.GoFiles { // test files are not in GoFiles
	file, _ := parser.ParseFile(token.NewFileSet(), filepath.Join(dir, f), nil, 0)
	ast.Inspect(file, func(n ast.Node) bool {
		if sel, ok := n.(*ast.SelectorExpr); ok {
			if x, ok := sel.X.(*ast.Ident); ok && x.Name == "exec" && strings.HasPrefix(sel.Sel.Name, "Command") {
				bad = append(bad, f) // → t.Fatalf("use hidden.Command: %v", bad)
			}
		}
		return true
	})
}

With GOOS=windows, go/build checks a _windows.go file even when the test runs on a Mac. Test files are left out: they do start PowerShell, to check the connector's work against Windows' own view.

The antivirus that held netsh

What it looked like. On 2026-11-05, on an office PC with an endpoint antivirus, the person typed the code, the platform accepted it, and the window sat on "Checking the code" for good.

What was happening. That antivirus suspends netsh.exe when it starts, behind a "Block / Resume process" prompt. Until someone chooses, netsh does nothing. The connector was waiting on netsh, "Checking the code" was waiting on install, and install ran six netsh firewall commands for its three rules. One held child process froze every screen above it. Note what was held: not the connector, but Windows' own network tool, started by the connector.

The fix, in five layers.

  1. Cap it. hidden.Output kills any short system command after 20 s.
  2. Name it. A capped command returns ErrTimedOut. The fixed-address step used to read any netsh failure as "this Windows is too old" and park it until a restart; a timeout is now retried on the next tick.
  3. Move it off the waited path. Install no longer touches the firewall. The service opens it at every start, in the background, and sends its first report before the address commands, which are background work too.
  4. Stop starting programs. What netsh, PowerShell, arp, route and sc.exe did is now an API call from inside the process.
  5. Guard it. A test fails if a launch of one of those comes back.
terminal
func Output(ctx context.Context, name string, args ...string) ([]byte, error) {
	ctx, cancel := context.WithTimeout(ctx, Limit) // 20 s
	defer cancel()
	cmd := Command(ctx, name, args...)
	cmd.WaitDelay = time.Second // a child holding the pipe cannot outlive the cap
	out, err := cmd.CombinedOutput()
	if err != nil && errors.Is(ctx.Err(), context.DeadlineExceeded) {
		return out, fmt.Errorf("%s %w (%s)", filepath.Base(name), ErrTimedOut, Limit)
	}
	return out, err
}
Who waits on what, before and afterbefore"Checking the code"install6 × netsh firewallantivirus: Block / Resumeeach box waits on the one to its right; the last one waits for a personafter"Checking the code"install: SCM onlyservice startsfirst report goes outfirewall via COMaddresses via IP Helperdashed: background, nobody waits;a failure is a warning, tried again

The cap and the move to the background ended the hang. Moving the work into the process removed the thing to hold.

Layer three has two guards. One reads main.go and fails if install calls anything firewall-related. The other runs the real loop with a fake address command that blocks until its context ends, as netsh was held, and fails if no report reaches the fake server within three seconds.

Layer four: asking Windows instead of its tools

Addresses, neighbours and routes go through the IP Helper API, in a package called winnet. CreateUnicastIpAddressEntry adds a fixed address in the active store only, gone after a restart as netsh's store=active was. GetIpNetTable2 replaces arp -a and GetIpForwardTable2 replaces route print. Measured on a windows-latest runner: an address added this way leaves DHCP on, with or without Windows' DHCP and static coexistence switch, so that netsh step disappeared rather than moved.

The rows are read straight out of Windows' memory, so one test pins struct sizes and offsets to netioapi.h (80 bytes for MIB_UNICASTIPADDRESS_ROW, 88 for MIB_IPNET_ROW2, 104 for MIB_IPFORWARD_ROW2), and others compare every read with Get-NetRoute, Get-NetNeighbor and Get-NetIPAddress on the same runner. The test may start PowerShell. The program may not.

Firewall rules go through the Windows Firewall COM API (INetFwPolicy2), the one WiX's firewall extension uses. Without cgo that means calling vtable slots by number from netfw.idl with syscall.SyscallN. The rules are narrow (UDP from the local subnet to this program only, relay ports on Private networks only), and a reinstall replaces them, never doubles. The Windows test adds each twice over two stale duplicates and reads the result back with Get-NetFirewallRule, field by field. To check it could fail at all, a temporary commit made the COM code write every rule as TCP, which the two UDP cases exist to catch, and was then reverted.

The old agent's service is stopped through the Service Control Manager, which waits until it has really stopped; sc.exe stop only asked. Copy and Open in the setup window use Windows' clipboard calls and ShellExecute.

Layer five: the guard

TestWindowsStartsNoSystemProgram fails on any string literal in a non-test .go file that names netsh, powershell, arp, route, sc, wmic or ipconfig with .exe. Cruder than the AST guard, and enough: a launch has to name its program. Its allowlist entry for the firewall file was deleted the day the COM rules landed. One exception is left: a PowerShell input box, used only when Edge WebView2 is missing, run from System32 with its module path pinned, because that copy runs as administrator.

The one-line install

A browser download gets a small tag beside the file, the Mark of the Web, and SmartScreen shows its blue warning for an unsigned file that carries it. A file PowerShell downloads with Invoke-WebRequest carries no such tag. So the platform shows one line to paste into PowerShell, irm …/install.ps1 | iex, and the script it fetches does this:

terminal
$want = @{ 'amd64' = '<sha256>'; '386' = '<sha256>' }[$arch]   # filled in at build time
$file = Join-Path $env:TEMP ('connector-' + $want.Substring(0, 12) + '.exe')
if (-not ((Test-Path $file) -and (Get-FileHash -Algorithm SHA256 $file).Hash -eq $want)) {
  Invoke-WebRequest -UseBasicParsing -Uri "$server/downloads/connector-windows-$arch.exe" -OutFile $file
  if ((Get-FileHash -Algorithm SHA256 $file).Hash -ne $want) {
    Remove-Item -Force $file
    throw 'The download did not match this release, so it was deleted.'
  }
}
Start-Process -FilePath $file

A CI job runs the exact line on windows-latest under Windows PowerShell 5.1, in a check-only mode that stops after the hash check, and asserts the file has no Zone.Identifier stream. Then it appends one byte to the served .exe and asserts the line fails and leaves no file behind.

Smart App Control is the one thing the line cannot get past. On Windows 11 it runs only apps from verified publishers and has no setting to allow one. So when Start-Process fails with "An Application Control policy has blocked this file", the script says so in yellow rather than red: Windows blocked it, nothing was installed, and there are two ways on. Use another PC on the same office network, or let the machine send its punches itself.

The elevated copy's short log

After Windows' Yes, setup carries on in a second, elevated copy with no console, so an error, a crash and a window that never opened all looked like nothing. Now every windowed run shows an error or crash in a dialog and writes each step to a log in the person's temp folder, reset past 64 KB:

terminal
<time> start <version> setup --server <server>
<time> asked Windows for administrator rights; that copy carries on
<time> start <version> setup --server <server>
<time> opening the window

The rest are window closed, error: …, stopped: … and done. That is all the logging there is, and it diagnosed the next bug in one look.

The window that never appeared after Yes

What it looked like. On 2026-11-06, on an office PC: the install line, Windows' Yes, then nothing. The log ended at opening the window, with no window closed. Two elevated copies were running, both with an empty MainWindowTitle, and nothing on screen.

What was happening. The non-elevated copy asks for administrator rights by starting itself again through ShellExecute with the runas verb, and it passed SW_HIDE, so the elevated copy's console would never flash. Windows hands that show state to the new process in its start-up information, and the first time the process calls ShowWindow, Windows ignores the argument and uses the start-up state instead. The first ShowWindow in the elevated copy was the WebView2 library's own. It said show. Windows said hide.

The fix. Show it a second time. The second call is honoured:

terminal
// Windows applies the way a program was started to its first ShowWindow,
// and the copy started after "Yes" is started hidden. The second is honoured.
func showInFront(hwnd uintptr) {
	procShowWindow.Call(hwnd, swShowNormal)
	procSetForegroundWindow.Call(hwnd)
}

The guard. The test binary starts itself again with SysProcAttr.HideWindow, which sets the same start-up flag and SW_HIDE that ShellExecute passes. The child opens the real window and reports through its exit code:

terminal
if os.Getenv("HIDDEN_CHILD") == "1" {
	runtime.LockOSThread()
	if w := newWindow(); w == nil {
		os.Exit(4) // no Edge WebView2
	} else if !windows.IsWindowVisible(windows.HWND(uintptr(w.Window()))) {
		os.Exit(3) // → "the window stayed hidden in a copy started hidden"
	}
	os.Exit(0)
}
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestTheWindowShowsInACopyStartedHidden$")
cmd.Env = append(os.Environ(), "HIDDEN_CHILD=1")
cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true} // as ShellExecute passes it

I revert-checked it on a windows-latest runner: with showInFront removed, the test failed. With it back, it passed.

Why the window stayed hidden, and why CI never saw itan office PCnot adminsetupShellExecute runasSW_HIDE · Yes1st ShowWindow(SHOW)start-up says hide2nd ShowWindow(NORMAL)+ SetForegroundWindowbefore the fix: two copies, no windowafter: on screen, in frontthe CI runneralready adminsetupno relaunchno start-up state1st ShowWindow(SHOW)honouredwindow found by title,screenshot kept: greenthe guard now starts a copy hidden on purpose, so CI takes the office PC's path

CI tested the code path, not the start-up state. The relaunch only happens without admin rights, and a hosted Windows runner always has them.

Why CI missed it. CI did open the window on real Windows and keep a screenshot, but a hosted runner is already administrator, so no SW_HIDE was ever passed. The new test reproduces the state the relaunch leaves behind, not the relaunch. Until the fix shipped, the workaround was to open PowerShell as administrator first, then paste the line.

What still needs a code-signing certificate, and why

Everything above works without a certificate. Some of it is a way around not having one, and a way around is not trust.

  • Smart App Control. A PC with it on cannot run the connector at all, and the script can only say so politely. Only a signature changes that.
  • The browser download. Anyone who clicks Download for Windows in a browser still gets SmartScreen's blue warning. The line is a detour.
  • The administrator prompt. Windows' Yes prompt for an unsigned program names no publisher. It is the one moment a careful person is asked whether they trust us, and it gives them nothing to go on.
  • Identity, not just integrity. The hash proves the .exe is the one this release's script expects, but both come from the same server. A signature binds the file to a publisher, wherever it came from.
  • Updates. The update path checks a manifest signed with an Ed25519 key the server never holds, and is switched off until a review's conditions are met. Windows and the antivirus cannot see that trust: to them, each new unsigned build is just another unknown file.

And one thing a certificate would not fix: the antivirus held netsh.exe, which Microsoft signs. Not starting a child process is what stops it being held. The list of what the connector still needs has the certificate near the top; the no-launch guard stays either way.

What I would tell someone shipping the same thing

  • Count the layers between double-click and running, and remove layers before fixing them. Each reads another's output by version-specific rules.
  • Nothing anyone waits on should start a program you do not control. Call the API from your own process; cap, classify and background the rest.
  • Write the rule as a test that reads the source. A rule lasts exactly as long as the test that enforces it.
  • Reproduce the start-up state, not just the code path. A runner that is already elevated never takes the path an office PC takes.
  • Give the invisible process a voice. A short log turned "nothing happened" into one line to read.
  • Say plainly what you cannot fix. A yellow sentence with two ways on beats a red error. Both are worse than the certificate.

Working through something like this? I help teams ship AI and cloud systems that hold up, and cost what they should.