Testing against hardware we do not own

The connector is a Go program on an office PC that reads biometric attendance machines and sends their punches to a cloud attendance platform. It speaks Hikvision's ISAPI, Dahua's CGI, ZKTeco's TCP protocol on port 4370 and its ADMS HTTP push, and a Sintech terminal's UDP frames; the readers post has the protocols. This one is about testing them when most of the brands have never been on my desk, and a mistake reaches an office as attendance quietly missing at the end of the month. Five things stand in for the field, and each one caught something.

devsim: machines that only exist in a test

devsim is a small program, the standard library plus the connector's own pretend ZKTeco, that behaves like an attendance machine. Each subcommand sends and answers what the connector and the platform's server expect:

• hikvision: ISAPI. Digest login, deviceInfo, the clock read and set, the event search with paging and a time window, users. --page-max 30 behaves like firmware that caps its page and says so in its capabilities.
• dahua: CGI. Digest login, magicBox.cgi, the clock, recordFinder.cgi.
• zk4370: the machine side of ZKTeco's 4370 protocol: session, comm key, serial, counters, clock, and the bulk read of people and punches in packets. Anything that would change a machine is refused.
• sintech: 32-byte UDP frames, resent every 3.3 seconds until the same bytes come back, as the real terminal does.
• zkteco-adms and hikvision-push: machines that send by themselves, over ADMS and as JSON events.

Every kind takes a fake serial (ABC1234567890 by default, counting up across a fleet), PINs, how often someone punches, days of stored history, a time zone, and a --clock-offset that makes the clock wrong and stamps the punches with it. --fleet runs up to 10,000 machines without --i-know. Without --allow-remote it refuses any target that is not this computer or a private address, so a typo cannot point a fleet at a public server.

How tests use it

The tests run the connector's real readers and relays against the simulator, in the same process: an httptest server for the HTTP machines, a listener on 127.0.0.1:0 for ZKTeco. A reader test is a full conversation:

d := newDevice("hikvision", "ABC1234567890", []string{"1", "2", "3"}, zone, 0, 3)
srv := httptest.NewServer(hikHandler(d, "admin", "pw", 0))
r := hik.New(machine.Target{Serial: d.Serial, Host: host, Port: port, Username: "admin", Password: "pw"})
r.PageSize = 4 // force several pages

got, _ := r.Punches(ctx, time.Time{}, time.Time{})
wantSamePunches(t, got, d.between(time.Time{}, time.Time{}), true)
all := d.between(time.Time{}, time.Time{})
got, _ = r.Punches(ctx, all[2].At, all[6].At) // since exclusive, until inclusive
wantSamePunches(t, got, all[3:7], true)
// then Users, SetClock and read it back, and a wrong password must be ErrAuth

A machine with --clock-offset 5m must report a clock five minutes fast and stamp its punches with it, the starting point for time being the hardest field in a punch. The relays are tested the same way, on what reaches a fake platform server.

A simulator written by the reader's author can share the reader's misunderstanding, so its output is checked against bytes I did not write. Its Sintech frames must match frames captured from a real terminal, byte for byte. Its ADMS lines must match the strings the server's own tests use, and its push events the server's fixture. The pretend ZKTeco keeps its own copy of the wire rules, refuses a request with a wrong checksum as a strict firmware would, and can be told to misbehave: a wrong comm key, a stall, a broken reply, a hang-up mid-transfer.

A real machine gets a manual, read-only test, switched on by an environment variable, that prints counts and timings but never a serial, a name or a punch. When the in-process ZKTeco reader first met a real K40, with its comm key, it read 26 people and 1,671 punches in 0.79 seconds.

One contract for every reader

Every vendor's reader implements one small interface: Info, Punches between two times, Users, and Close. The runner opens a fresh reader for every pull. So there is one contract test, in a package of its own:

// machinetest is its own package, like net/http/httptest, so that importing
// "testing" never reaches the shipped binary: only _test.go files import it.
func Contract(t *testing.T, newReader func() machine.Reader) {
	t.Run("Info does not panic", ...)
	t.Run("Punches tolerates a zero (unbounded) window", ...)
	t.Run("Punches tolerates since after until", ...) // an error is fine; a panic is not
	t.Run("Users does not panic", ...)
	t.Run("Close is safe to call once", ...)
}

Hikvision, Dahua and ZKTeco each call it with a constructor whose transport always fails (r.Dial returns "contract test: no real network"), so it never touches a network. The point is not that a machine answers. It is that nothing panics when one does not. The contract was written against a fake before the ZKTeco reader existed, so that reader arrived with its first test already waiting.

Tests that must never reach the real administrator prompt

What it looked like

On 2026-10-30, every test run on a Mac asked for an administrator password "for a script started by connector.test". Just after 11pm, one of those prompts was approved. What followed ran until just after 2am.

What was happening

A test for the setup window, TestEveryPageRequestIsAnswered, pressed every button for real, including "save extra office networks". On a Mac that step asks for the password through osascript ... with administrator privileges and starts the program again with extranets. In a test, the program is the test binary. Once the prompt was approved, connector.test extranets ran as root. A Go test binary ignores an argument it does not know and runs every test again, and as root osascript does not prompt. So it started itself again, and again: a chain of about 240 root processes over about three hours. Nothing was damaged, and the protected folder was unchanged, but a test had been running as root on a real machine.

The fix and the guard

Every step that asks for administrator rights or changes the system is now a package variable: setup, update, the proxy setting, extra networks, the machine link, remove, start, and on a Mac the password prompt and osascript themselves. TestMain replaces each one before any test runs:

const realPrompt = "a test reached the real admin prompt"

func TestMain(m *testing.M) {
	doSetup = func(string, string) (string, error) { panic(realPrompt) }
	doExtraNets = func(string) error { panic(realPrompt) }
	// ...every other admin step, and the machine-link command runner
	guardSystem() // on a Mac: runAsAdmin and osascript panic too
	os.Exit(m.Run())
}

// fakeAdmin replaces one administrator step for one test.
func fakeAdmin(t *testing.T, step *func(string) error, fake func(string) error) {
	old := *step
	*step = fake
	t.Cleanup(func() { *step = old })
}

A test that means to exercise a step swaps in a fake for itself. One that reaches a real step by accident fails loudly, with a stack showing how. The simulator's tests make the network-alias command panic, because the relays pull that package in, and the runner's ZKTeco tests stub every probe of a found address, because a search test's "found" address may be a real machine on this office's network.

The lesson is not about osascript. A prompt shown by a test teaches a person to click Allow, and a test binary with privileges it was never meant to have does whatever its arguments do not say.

Real Windows in CI

Most office PCs that run the connector are Windows, and the service, the firewall rules, file permissions and the setup window only compile there. Linux CI vets the code for Windows too, but vetting is not running. So a workflow runs on a GitHub-hosted windows-latest runner, only when the connector changes, because Windows minutes cost more: go vet, go test, a build, and the window, opened as a person would see it:

$p = Start-Process -FilePath .\connector.exe -ArgumentList 'setup' -PassThru
foreach ($i in 1..30) {
  Start-Sleep -Seconds 1
  if ($p.HasExited) { throw "The window closed by itself (exit code $($p.ExitCode))." }
  $p.Refresh()
  if ($p.MainWindowTitle -eq 'Connector') { break }
}
Start-Sleep -Seconds 4  # let WebView2 draw the page
# CopyFromScreen to a PNG, uploaded as an artifact kept for 7 days

It uses a data folder of its own, so nothing touches the runner's ProgramData. The first green run took fixes a Mac could never have shown. The punch spool holds a lock file, and Windows will not delete a test's temp folder while it is open, so every test closes its spool in t.Cleanup. Unix file modes apply only off Windows. The older agent's settings file is tested in a protected folder, as on a real PC. Later, a test that reads main.go broke on a Windows checkout's CRLF line endings.

The suite also asks Windows itself: IP Helper reads are compared with Get-NetRoute, Get-NetNeighbor and Get-NetIPAddress, firewall rules read back with Get-NetFirewallRule, Copy checked with Get-Clipboard. A test may start PowerShell; the program may not, for reasons in the antivirus post. That post also has this runner's blind spot: it is already administrator, so it never takes the path an office PC takes after Windows' Yes.

The race detector and an address goroutine

What it looked like

A go test -race run failed TestTheFirstReportNeverWaitsOnTheMachineLink with a data race: the test's cleanup writing a package-level fake while a goroutine inside Run was reading it.

What was happening

Run starts every loop under a sync.WaitGroup and returns once the context is done and the group has drained. But the goroutine that puts this PC's addresses in place and reports the machine link was started before the group existed, outside it. Run could return while it was still ticking, and the test, believing the runner had stopped, put its fakes back under it.

The fix

Both halves. Run counts the goroutine it starts, and the test waits for Run to return before its fakes go back:

var wg sync.WaitGroup // Run returns only when everything it started has stopped
wg.Add(1)
go func() {
	defer wg.Done()
	r.ensureAliases(ctx) // the OS forgets second addresses on reboot
	r.pruneAliases(ctx)
	r.tickLink(ctx)      // then report the machine link once it is in place
}()

// the test:
stopped := make(chan struct{})
go func() { defer close(stopped); r.Run(ctx) }()
t.Cleanup(func() { cancel(); <-stopped }) // before the fakes are put back

The test exists to prove that a held system command cannot hold the first report. It found a different bug on the way: the race detector reads a function's promises more literally than I do.

A flaky test and a process-wide cache

What it looked like

The in-process ZKTeco test passed alone and failed under go test -count=2, on the second run: the second read brought no punches.

What was happening

The ZKTeco reader remembers each machine's punch count by serial, for the life of the process. A routine read that finds the same count skips the transfer, because on a full machine the transfer takes minutes and almost always finds nothing new. The test used the same fake serial every run. On the second run, in the same process, the count matched, and the read skipped.

The fix, twice

The test fix is one line: a unique serial per run.

// One serial per run: internal/zkteco remembers a machine's punch count for the process.
serial := fmt.Sprintf("ABC%010d", time.Now().UnixNano()%1e10)

But the flake was describing a field bug. Moving a machine's reading position back, for a reset, a repair, or the older agent's position taken over, left the remembered count in place, so on a quiet machine a reset did nothing until someone punched. Now one rewind helper sets the position and calls ForgetCount, and both places that move a position back use it, as the older agent cleared its count marker. The guard stores one punch from 26 hours ago, reads twice (the second must bring nothing), moves the position back 48 hours, and requires the next read to bring that punch.

State that makes a test flaky is state in production too. The flake was the cheaper of the two ways to find it.

Every fix leaves one guard, and the guard is revert-checked

Every fix in this series leaves one test behind, and before I trust it I revert the fix, not the test, and watch the test fail with the message I expect. Then I put the fix back. A test that passes both ways is not a test.

A few of the revert-checks behind this series:

• The hidden window. With the second ShowWindow removed, the test that starts a copy of itself hidden failed on windows-latest; restored, it passed.
• The firewall rules. A temporary commit made the COM code write every rule as TCP, which the UDP cases exist to catch, and was then reverted.
• The serial check. Reading a machine only after it reports its own serial came with four guards, each reverted and seen failing.
• Reading Hikvision as the older agent did. Each behaviour change came with a test that fails with that change reverted.

The incident above added one rule to the discipline. A revert-check runs the code with its fix removed, which is exactly when it might reach the real system. So revert-checks get the same fakes as the tests, and nothing in a test or a revert-check runs the real osascript, netsh, ifconfig or ip.

What an adversarial review pass found

Once the connector worked, I read it again as someone trying to lose a punch. That pass found four ways, each now fixed with a guard:

• Two uploads at once. "Read all now" while the regular upload was draining a backlog sent the same batch twice and marked never-sent punches as sent. Reproduced: 200 of 1,000 lost. Now uploads take a lock with TryLock: a second caller returns at once, and the next tick sends what it skipped. The guard queues 1,000 punches, runs two uploads concurrently, five rounds, and requires all 1,000 distinct punches at the fake server and none still pending.
• Two programs on one spool. Each would count what was sent and lose or resend punches. The spool now takes an exclusive file lock (flock, or LockFileEx with fail-immediately on Windows), which the system releases even on a crash, and status opens it read-only.
• One unknown machine blocking the rest. A machine the platform no longer knew held back every machine's punches. Now it holds back only its own.
• A swapped machine. The machine's own serial was checked once an hour, so two machines that swap addresses after a power cut could, for up to an hour, record one's punches as the other's. It is now checked on every read.

func (r *Runner) FlushAll(ctx context.Context) int {
	if !r.flushMu.TryLock() { // one upload at a time: two would ack the same batch twice
		return 0
	}
	defer r.flushMu.Unlock()
	// ...
}

The same pass made the administrator window load DLLs only from System32. A later review of the old-agent handover made it fail closed, in the handover post.

What none of this proves

The README keeps a list of what only a real machine or office can prove: that a terminal accepts the first search body sent to it, that discovery answers on a real office network, how real proxies, inspection boxes and VPN clients behave. Those wait for the field, and for what the connector still needs.

What I would tell someone testing hardware they do not own

• Simulate the protocol, then check the simulator against someone else's bytes. Captures from a real device and the server's own test strings keep it from agreeing with your misunderstanding.
• Give every reader one contract. A new vendor then gets its first test for the price of a constructor.
• Make side effects a seam, and make the default a panic. A test that reaches a real prompt should fail with a stack, not ask a person.
• Run on the OS your users run. Vetting for Windows is not running on Windows, and the first green run will cost you fixes you could not see.
• Treat a flaky test as a report about shared state. It is often describing a production bug in a smaller room.
• Revert the fix and watch the guard fail. Then put it back.