What the connector still needs

This is the last post in the series on the office-PC connector. The architecture post describes what it does. This one is about what it does not do yet, and what I plan to do about it. Before writing it I asked one question about every gap: how does the rest of the industry solve this? Each answer below cites where it came from.

Start from the failures, not the features

A read-only audit of the production fleet sorted every active machine by whether its punches had arrived recently. About half had delivered in the last day. About one in twelve had stopped within the last three days. The rest, close to four in ten, had been down for weeks or had never delivered anything at all.

The surprise was in that last group. Most of it was not a protocol problem, and no amount of connector code would have fixed it:

The things a program can fix by itself, it mostly already does: a machine that moved is searched for by serial, a machine one router away is read through the router, and a reading position that went wrong is repaired. What it cannot fix is a password typed wrong on the machine, a power switch, or a setup somebody started and never finished. For those, "self-healing" means telling the right person, quickly, in words they can act on.

1. Tell a person, during working hours

This is the biggest win, and it is mostly not new code. The platform already knows each office's working hours, already distinguishes "quiet because it is night" from "quiet during a working day", and already has a re-alert ladder that gives up after two weeks. What it does not yet do for every machine type is send the notice to the office itself.

How the alerting world does this is well documented. Prometheus Alertmanager groups related alerts, inhibits an alert when a bigger one already explains it, and repeats on a schedule (Alertmanager docs). Translated to the connector, the plan is:

• One message per machine, with one fix line per failure kind: the PC is off, the machine is off, the password is wrong, the machine moved, a port is blocked, the machine was never connected.
• Inhibition. When the PC itself is offline, say "the PC is off" once, not "machine quiet" for every machine behind it.
• A dry run first. Count what would be sent across the fleet before sending anything. An alert stream that is noisy in its first week is an alert stream people learn to ignore.
• An expected-punch baseline. "Reads succeed but return almost nothing" is a failure too. Compare each machine's punches per working day with its last few weeks, the roster and the holidays, and flag a machine that reads fine but keeps nothing.

The measure is simple: the share of sites delivering every working day, which today is about half.

2. Don't lock the machine you are reading

This one I found while researching this post, and it is a real bug.

A Hikvision terminal has an "illegal login lock". After a handful of failed logins from one address, typically five to seven depending on firmware and account, it refuses that address for a while, commonly 30 minutes (an installer's write-up of the setting). The connector retries an HTTP 401 three times on a fresh connection, because a 401 with the right password does happen: the machine drops the keep-alive its digest nonce was bound to. That retry was the right fix for that problem. But the read loop runs every minute, and after a failed read nothing slows it down.

Three tries a minute is about 90 failed logins every 30 minutes, against a lock that trips at five to seven. With a wrong password, the machine is locked against the PC almost permanently. Worse, it stays locked for a while after someone types the right password, which looks exactly like "I fixed it and it still doesn't work".

The fix is small:

• Classify a 401 that survives all three fresh-connection tries as wrong_password, its own failure kind, the way a wrong ZKTeco comm key already is.
• After one such read, wait an hour before trying again: at most three tries in any 30 minutes, under every lock threshold I could find.
• Try again at once when the password is changed in the platform, because that is the moment a person is waiting to see it work.

The guard is a test that counts login attempts against a fake machine over a simulated hour with a wrong password, and fails above the threshold.

3. Sign the Windows file

The connector's Windows build is unsigned. The Windows post covers how far that can be pushed. It cannot be pushed past Smart App Control, which blocks unsigned programs and has no per-app exception (Microsoft's overview).

What I learned about the options:

• EV is no longer the shortcut. Microsoft's current guidance is that SmartScreen reputation is built per signing identity over time, and an extended-validation certificate does not skip it (SmartScreen reputation).
• The managed signing service is not open to every country. Microsoft's own cloud signing service is the cheapest route where it is available (eligibility and pricing), but it does not accept organisations from where I am.
• So: an OV certificate, renewed yearly, with the key on a hardware token held by one person, and every release timestamped. Once signed, the same publisher accumulates reputation across releases instead of starting over each time.

4. Turn self-update back on, safely

Self-update is built and switched off, waiting on three changes from a security review: install exactly the bytes that were verified, refuse an older version, and trust-check the rollback files. The research added three more, all from how the update-security literature frames the attacks:

• Health means "read a machine". Today the rollback trigger is "did not report in within five minutes". A build that reports in perfectly and reads nothing would pass. The new rule is to read at least one machine, or have none configured, within ten minutes.
• Staged rollout with a kill switch. One percent, then ten, then everyone, with a server-side switch that stops a bad version reaching the next stage.
• An expiry on the signed manifest. The Update Framework names the freeze attack: a server that keeps serving an old, validly signed manifest so clients never learn about a fix (TUF specification). A signed expiry date closes it without adopting all of TUF. Full TUF, with threshold keys and rotation, is worth it at a larger fleet size (go-tuf).

5. Notice a loop that hangs

A crashed connector is handled. The Windows service is set to restart after 10 seconds, then 30, then a minute, and systemd and launchd keep it alive elsewhere. A hung connector is not handled. If a lock is never released and the read loop never finishes, the heartbeat still goes out every minute, and from outside everything looks fine.

systemd has the pattern for this: a service pings a watchdog, and a service that stops pinging is killed and restarted (systemd.service). The portable version is for every loop to record when it last finished, put those times in the heartbeat, and exit with a non-zero code when the read or send loop has not finished within ten of its own intervals, so the service manager restarts it. On Windows that needs the recovery setting that counts non-crash exits, which Go's service manager package exposes (svc/mgr).

6. Should we build our own network?

This was the question I most wanted evidence for. The idea is attractive: put every office PC on a private overlay, WireGuard with a self-hosted control server, and reach any machine from anywhere. I compared four shapes:

The answer was clearer than I expected. The industry pattern for "a program at a site that cloud software needs to reach" is the program dials out, which is what the connector already does. An overlay would add the highest risk and the most operational work, and fix none of the failures in the audit. Each site's machines are already read by that site's PC. The overlay would only help reach a machine from another site, and nobody needs that.

So the plan is:

• Keep the outbound connector.
• Later, add an opt-in support tunnel over the connector's existing HTTPS connection, off by default, time-limited, scoped to the company's own machines on vendor ports, behind two-factor login with an audit trail. No VPN driver, no new admin prompt.
• Pilot a small box only at the few sites where "the PC was off" is the measured cause.
• Build an overlay only if the support tunnel shows real demand for raw reach across sites. Then build it in user space (tsnet on a self-hosted control server), so there is still no driver.

Smaller things the research turned up

• One discovery address may be wrong. The connector asks Dahua devices to identify themselves on one multicast group, while Dahua's own firewall guide lists another. The cheap fix is to send to both, then confirm on a real machine which one answers.
• Which clock to trust, per machine. A Hikvision whose zone setting is wrong can be right on its display (set by hand) or right in its stated instant (kept by a time server). Today the connector trusts the display for every machine; it can measure, per machine, which of the two sits closer to the platform's time and decide from that.
• Cameras look like terminals until someone types a password. An ONVIF discovery probe would label them before anyone tries.
• The machine list is not cached on disk. After a reboot with no internet, nothing is read until the internet comes back. Nothing is lost, because the punches wait on the machine. But a sealed on-disk copy would let reading start within a minute of boot.
• Everything runs privileged. Adding a second address to the network adapter needs administrator rights, so today the whole service runs as SYSTEM, including the relays that parse traffic from the office network. The fix is a tiny privileged helper that does only that one job.
• Free supply-chain checks: a vulnerability scan (govulncheck) and an SBOM (cyclonedx-gomod) on every release.

The plan, in one table

What I'd tell someone at the same point

• Audit before you build the next feature. I expected protocol bugs. I found onboarding records that never delivered and agents that were silent for weeks. The fix for the biggest group was a message, not a module.
• "Self-healing" ends at the power switch. Past that point the job is telling the right person, once, in their words.
• A retry is a policy, not a loop. The three-try retry for a stale nonce was right. Running it every minute, forever, against a machine with a lockout was not.
• Ask what the industry does before building infrastructure. It took an afternoon of reading to see that the overlay network I wanted to build would have fixed nothing on the failure list.

The same story, written up as a short paper with the audit table in it: Reading the Unreachable (PDF), DOI 10.5281/zenodo.24026722.