Handing over from the old agent without losing a punch

A replacement is only finished when the thing it replaces is switched off. For the connector that meant an older Python agent, installed as a Windows service on an office PC, reading the same machines. Switch it off too early and punches between its last read and the connector's first are lost. Never switch it off and two programs read every machine, forever.

Before writing any of the handover, I audited production, read only. What it found shaped every rule below, so here it is first, in proportions:

• Every site still ran the old agent. Not one connector was in production.
• About one live site in seven had a machine the old agent was failing to read, for hours to weeks: a wrong password, a machine switched off, a port it could not reach.
• A few old agents had silently stopped reading for weeks, some for months, while still sending heartbeats, with no error in their state.
• Most Hikvision rows had no username or password on the platform. They lived only in the old agent's settings file on the PC.
• Two machines stored as cloud connections had been silent for months while the old agent read them every minute. Nothing would ever have told the connector to read them.
• The platform never records "last contact" for a machine a PC reads. So "has it reached us by itself?" can only be asked of machines that push.

The legacy import: copy what you can prove is yours

When the connector first starts on a PC, it looks for the old agent's settings file where the old installers put it. It reads that file only if it can trust it: owned by SYSTEM or Administrators, writable by nobody else, in a folder nobody else can replace files in. Its addresses become places this PC reads attendance from and sends device passwords to, so a file anyone could have planted is refused. More on that in the antivirus post.

From each machine entry it takes the serial (or a Sintech's MAC), host, port, vendor, username and password. A ZKTeco comm key may be an integer or a string, and 0 means "not set". The list goes to the platform once per connector. The PC records that it was done, and the platform records it on the connector's own row and refuses a second import with a conflict. A network failure is not recorded, so it is retried. The platform side is where the rules live:

• Only the company's own machines. An entry is matched by serial (Sintech: by MAC) against active machines of this company, scoped to the connector's branch. It never creates a machine and never touches one another company shares with this one.
• Only private office addresses. A host is copied only if it is a private IPv4 address, so a planted file cannot repoint a machine, and its password, at the internet. Ports must be 1 to 65535.
• Credentials only where HR set none. A password or username in the old file fills an empty field and never overwrites one.
• One transaction, capped at 256 entries.

Re-pathing rows wrongly marked "sends by itself"

The dangerous rows were the ones the old agent read over the network while the platform believed the machine pushed its own punches. A leftover push flag with no path ever chosen kept those machines off the connector's read list, so their punches would stop the day the old agent stopped. The import moves such a row to "read by an office PC". A path HR chose deliberately is kept.

The audit's two machines needed one more case. They were stored as a cloud connection, not just flagged, and had been silent for months. So a cloud row counts as stale too, but only with evidence on both sides:

def stale_push?(machine, host)
  return false if machine.path_chosen_by_hr? || machine.push_only_vendor?
  return false unless machine.connect_path == "cloud"
  return true  if machine.pulled_connection_type?      # a push flag on a pulled row
  office_address?(host || machine.ip_address) &&      # an office address to read it at
    !machine.contacted_since?(2.days.ago)              # and it is not reaching us itself
end

Sintech terminals move too. The old agent's UDP listener was the terminal's receiver on this PC's port 5055, which the connector's relay takes over once the old agent stops. So a Sintech matched by MAC becomes "Sintech through this PC", for this connector, unless HR set it to send straight to the platform, it is still doing so (contact within a day), another PC already has it, or its MAC disagrees with the old file. Two rows sharing a MAC are both left alone, because which one is meant is unclear.

The retire verdict: fail closed

Every five minutes, until it is done, the connector asks whether the old agent may stop. The old agent is named by the ID in its own settings file, never by the PC's name. The connector reports a Mac's computer name while the old agent reported the host name, and names like ADMIN-PC repeat across branches, so matching by name could stop the wrong agent at the wrong office. A file with no ID means an install set up by environment variables only, and that one is turned off by hand.

The connector's half, simplified:

func Retire(ctx context.Context, c Retirer, notes Notes, covers func(LegacyDevice) bool, carry func(serial, at string)) (string, error) {
	if notes.Note(RetiredNote) != "" {
		return "", nil // done once; never fought over if someone turns it back on
	}
	path, ok := Find()
	if !ok || Trusted(path) != nil {
		return "", nil
	}
	devices, _ := Parse(path)
	uid := AgentUID(path) // its own ID, never the PC's name
	var others []LegacyDevice
	for _, d := range devices {
		if d.Unsupported != "" {
			return keep + "machine " + d.Serial + " runs on " + d.Unsupported, nil
		}
		if !covers(d) {
			others = append(others, d) // the platform must see these arrive by themselves
		}
	}
	v, err := c.LegacyRetireCheck(ctx, uid, others)
	if err != nil || !v.Ready {
		return waitReason(v), err
	}
	for serial, at := range v.Watermarks {
		carry(serial, at)
	}
	// ... stop the service, tell the platform, write RetiredNote
}

Covered means one of two things. The machine is on this connector's list and it has been reading it for 30 minutes since its first read, or the machine is assigned to this connector's relay. Why 30 minutes: a first read starts at the machine's clock, not its whole log, so the punches before it are still the old agent's to send. After half an hour the platform's other checks (a recent report, nothing pending) mean the old agent has read past the connector's start.

HR's setting alone is not proof. A machine the connector does not read has to have actually reached the platform by itself in the last two days. An earlier version trusted a list made at import time; it now asks for contact, so connectors that imported earlier retire too.

One blocker was removed on purpose. The first verdict waited while the old agent was failing to read any machine. With about one live site in seven in that state, sometimes for weeks, those sites would never have handed over, and both programs would have read the same machines indefinitely. Carrying the reading positions, below, makes a failing machine no reason to wait.

Settings the Go reader does not have keep the old agent on. A few hand-edited old installs read a ZKTeco over UDP (force_udp) or with the punch-type and verify fields read the other way round (status_field, verify_field). The connector has none of those yet, so a machine on one keeps the old agent on. The log names the setting, and the platform is not even asked.

The watermark handover

A ready verdict carries where the old agent had read each machine to: its last reported watermark per serial, in the machine's own wall clock. The connector takes each one over before it stops anything:

const carryBack = 7 * 24 * time.Hour // older is a backfill, not a handover

func (r *Runner) CarryWatermark(serial, at string) {
	t, ok := r.target(serial)
	if !ok {
		return // only machines on this PC's own list
	}
	old, err := ParseTime(at, t.Location())
	if err != nil {
		return
	}
	now := r.platformNow()
	if floor := now.Add(-carryBack); old.Before(floor) {
		old = floor // at most a week back
	}
	old = ClampWatermark(old, now)
	if cur, err := ParseTime(r.watermark(t.Serial), t.Location()); err == nil && !cur.After(old) {
		return // never forward: already reading from there, or earlier
	}
	r.rewind(t.Serial, FormatTime(old)) // sets it and forgets the ZKTeco count
}

Each bound has a reason. Never forward, because moving a position later can only skip punches. At most seven days back, because the platform keeps an older punch only as a raw record, and anything further back is what "bring in older punches" is for. Only this PC's machines, because a verdict must not move a position for a machine this connector does not read. And the carry goes through the same rewind as every other move back, which also clears the ZKTeco count-skip. The time post has the bug that happens without that.

Stopping the old Windows service through the SCM

The old agent runs as a Windows service. The first version stopped it with sc.exe. That is a separate program, which an antivirus can suspend, and sc stop only asks a service to stop. It returns before the service has stopped. The connector now talks to the Service Control Manager from its own process and waits:

func scmStop(name string) error {
	m, s, err := openService(name) // ERROR_SERVICE_DOES_NOT_EXIST → errNoService
	if err != nil {
		return err
	}
	defer m.Disconnect()
	defer s.Close()
	st, err := s.Control(svc.Stop)
	if errors.Is(err, windows.ERROR_SERVICE_NOT_ACTIVE) {
		return errNotRunning
	}
	for deadline := time.Now().Add(30 * time.Second); err == nil && st.State != svc.Stopped; {
		if time.Now().After(deadline) {
			return errors.New("it did not stop within 30 seconds")
		}
		time.Sleep(300 * time.Millisecond)
		st, err = s.Query()
	}
	return err
}

Then it sets the service's start type to disabled, so a reboot does not bring it back. "Not installed" or "not running" counts as stopped only when the platform says the old agent is not live. A live old agent with no service is running some other way, and calling it stopped would strand what it reads. Its files stay, and it can be turned back on. The connector records that it retired it, so it never fights a person who does that. Only after the stop does it tell the platform, with an exact stopped: true. The platform marks the old agent uninstalled, "replaced", and writes an audit line, so HR is not told a PC went offline.

A small reason to do this in code: on a staging PC, a deleted old agent kept calling and getting refused until it was stopped by hand, and in PowerShell sc is an alias for Set-Content, not the service tool.

Never lose a punch

The handover is only safe because the connector itself loses nothing. These are the rules underneath. The first two came from an adversarial review that set out to make it lose one.

The offline spool

Every punch is written to disk, and fsynced, before any upload is attempted. The machine's own log rolls over, so it may be the only other copy. There is one append-only JSONL file per machine plus a small file holding how many lines the platform has taken. The duplicate key is staff number, time and status, the same thing the platform hashes, so re-reading a whole log is free. A half-written last line from a power cut is dropped on load, not fatal; the machine still has it. There is no SQLite and no cgo: a person can read the file, and the build cross-compiles. An office offline for a week replays cleanly.

No punch lost to two uploads or two programs

The review found that "read all now", pressed while the regular upload was draining a backlog, sent the same batch twice and marked never-sent punches as sent. Reproduced in a test, 200 of 1,000 punches were lost. Now one upload runs at a time, and a second caller returns at once instead of queueing:

func (r *Runner) FlushAll(ctx context.Context) int {
	if !r.flushMu.TryLock() {
		return 0 // two uploads would both ack the same batch
	}
	defer r.flushMu.Unlock()
	// ...
}

The same reasoning applies across programs. Each program keeps its own count of what was sent, so two connectors on one spool would lose or resend punches. The spool folder takes an exclusive lock that the OS releases even on a crash, and status opens it read only. A machine the platform no longer knows holds back only its own punches, not every other machine's.

Park the punches the platform refuses, keep the rest flowing

A batch refused with a 4xx about the request could be one bad punch, or the whole request. The connector tells them apart with an empty upload for the same machine. If that is refused too, the problem is the machine or the request: nothing is parked, the batch stays queued, and that machine waits 30 minutes. If it is accepted, some punch is the problem, so the batch is split in halves, down to single punches. Only punches refused one by one are set aside. A batch of n with k bad punches costs about 1 + 2·k·log2(n) requests.

Set-aside punches go to a parked file beside the spool, with the platform's status, a short code and the time, never the body. They are kept 90 days, at most 10,000 per PC, and never resent on their own. A parked --retry command puts them back once, after the platform is fixed; a punch refused again is parked again, never looped. The window shows the count and the oldest date.

Back off together, spread out, drain gently

Every caller of the client shares one cool-down: heartbeat, config, commands, uploads and both relays. So one outage backs them all off together, instead of each hammering on its own timer. No network, a 5xx, 408 or 429 backs off exponentially with full jitter, 5 seconds doubling to a 5 minute cap. A Retry-After is honoured plus up to 20%, capped at an hour. A 401, or a 403 that is not "awaiting confirmation", pauses about 30 minutes. While cooling down, calls fail without touching the network, and any success clears it. Loops start at a random point in their interval and wait it ±20%, so a fleet of PCs that rebooted together does not call in together.

Backlogs drain gently: at most five batches of 200 per machine per 15-second tick, a backlog older than 20 hours goes up as a historical import, and the relays drain their stores in bounded batches. That pass found one more bug: the Sintech relay sent its frames under a key the platform did not read, so every relayed frame got a 400.

Activity lines survive a power cut

The connector's Activity lines ("Read 12 punches from Front door", "Front door's clock was 4 min slow; set it to the right time") also go to the platform with the heartbeat, so support sees what the PC saw. Unsent lines are kept in the spool's notes, not only in memory, so a restart or a power cut loses none. A line leaves the outbox only after a heartbeat carrying it is accepted. The platform drops a line it already has, matched on time and text with microsecond timestamps, in case the answer was lost on the way back. At most 500 lines wait, oldest out first.

Lessons for anyone replacing an agent in the field

• Audit production before you design the handover. Every rule above traces to a proportion in that list. I'd have guessed none of the silent ones.
• Trust the old state only if only an administrator could have written it, and copy from it only into what you can prove is yours: own machines, private addresses, empty fields, once.
• Retire on evidence, and fail closed. A fresh report, nothing pending, every machine covered or seen arriving by itself. Anything unclear leaves the old program on, which costs a duplicate read, not a lost punch.
• Name the old program by its own ID. Host names repeat.
• Carry positions, bounded on both sides. Never forward, never further back than the platform keeps.
• Stop services through the OS API, and wait for "stopped". An "asked to stop" is not a handover.
• A spool needs one writer, one uploader and one way to set a bad punch aside. Each of those was a real loss in a test before it was a rule.