Time is the hardest field in a punch

A punch has three fields: who, when, and sometimes in or out. "Who" is a staff number typed into the machine once. "In or out" is a button people often press wrongly, which the reader post covers. "When" is the hard one, because four clocks stand behind it, and in the field three of them are wrong somewhere:

• The machine's clock. Every punch carries it. Someone set it by hand, perhaps a year ahead, perhaps in the wrong time zone.
• The office PC's clock. A wrong zone, a dead battery, a laptop that has not synced for weeks.
• The platform's clock. The one that is right, and the one that refuses a signed request more than five minutes off.
• The wall clock people see. What HR checks attendance against, which is whatever the machine's screen shows.

This post is the incidents those clocks produced, each as what it looked like, what was happening, the fix and the guard. Some belong to the older Python agent and were carried into the Go connector; some are the connector's own.

Incident one: a clock a year ahead stopped a site for eighteen days

What it looked like

A site stopped delivering attendance, and nothing noticed. The agent process was up, its heartbeat was green, and it read the machine every 60 seconds. For 18 days no punch arrived; the only attendance came from a manual "bring in older punches". The same symptom had been seen at other sites before.

What was happening

The agent kept a per-machine watermark, the newest punch time it had read, and asked for punches after it. It advanced the watermark to the newest punch in each read, with no upper bound. The machine's clock had been set ahead, so it handed over punches stamped more than a year in the future. The watermark jumped there, and from then on every real punch was older than the watermark and filtered out as already read.

Payroll was safe, because the platform refuses a punch dated more than 24 hours ahead, but it could not see or undo the agent's watermark. Backfill bypasses the watermark, which is why it was the only thing that worked. There was a second way in: a first read seeded the watermark from the machine's newest record, so a clock-ahead machine was poisoned before any punch was read.

The fix: clamp on write, clamp on read, rewind instead of clear

Three rules, applied at every place a watermark is set, and carried into the connector:

const futureTolerance = 24 * time.Hour // mirrors the platform's own drift guard

// A punch dated past now + 24 h is a mis-set clock: dropped before it is
// spooled or feeds the watermark.
keep, dropped := DropFuturePunches(punches, platformNow, loc)

// A watermark can never pass now. Errs towards re-reading, never towards skipping.
func ClampWatermark(candidate, now time.Time) time.Time {
	if candidate.After(now) {
		return now
	}
	return candidate
}

The one that mattered for the sites already stuck was the third rule: the stored watermark is clamped on read, not only on write, so an affected install repairs itself the moment it upgrades:

stored, _ := ParseTime(note("watermark::"+serial), loc)
if fixed := ClampWatermark(stored, platformNow); !fixed.Equal(stored) {
	rewind(serial, fixed) // and say so: live punches were being skipped
}

The platform also got a "reset the reading position" button, and its first design was wrong. Clearing the watermark sends the next read down the first-read path, which starts from now and reads nothing, so the button would report success and change nothing. It rewinds instead: 48 hours by default, never more than 30 days, so a mistyped value cannot turn into a full-log walk. Punches older than 20 hours then go up as a historical import, because the platform discards a real-time punch more than 24 hours out. A weekend of real attendance was lost that way once, before that rule existed.

The guards

Tests drive both ways in through the command path. The heartbeat carries each machine's watermark, and the platform judges it: an hourly sweep flags a future watermark, or a site up but silent in working hours. The rule I took away: a high-water mark fed by an untrusted clock needs a ceiling, and the ceiling has to apply to the value already on disk.

Watermarks are in the machine's wall clock

Every punch time on the wire and in the spool is the machine's own wall clock, "2006-01-02 15:04:05", with no zone. That is deliberate. It is what the machine shows, what HR checks against, and the same string the ZKTeco reader, the Hikvision reader and the machines that push by themselves all produce, so the platform's duplicate check matches a punch that arrived by two paths.

So the watermark is that string too, parsed in the machine's configured zone and compared with the platform's time in that zone. A first read starts from the machine's own clock, clamped to the platform's now, and reads forward; older attendance is an explicit "bring in older punches", sent day by day.

Incident two: a factory time zone, and a clock "fixed" to the wrong time

What it looked like

This one was caught by comparing the connector's Hikvision reader with the older agent's, behaviour by behaviour, before the connector took over the production machines. The case: a machine whose time zone setting was left on the factory default, +08:00, while someone set its clock by hand to local time, +05:45. HR sees the right time on the screen. That is common.

What was happening

ISAPI writes every timestamp with an offset: "2026-10-29T11:30:00+08:00". The connector's first reader did the textbook thing and applied it, converting to an instant and then to the machine's configured zone. On this machine that moves every punch 2 hours 15 minutes early. Then the clock check compares the machine's "instant" with the platform's, finds it 2 h 15 min slow, and sets it, writing the right instant with a +05:45 offset. A machine that believes it is in +08:00 then displays 14:15 at noon. The machine's screen was the one thing that had been right.

The fix: the screen is the truth; the offset is the machine's dialect

Read every ISAPI time as the wall clock the machine shows: the first 19 characters, in the machine's configured zone, offset discarded. Remember the offset the machine reports for itself, and write that offset on everything sent back: the search window, so the machine's own filtering lines up with its own clock, and a clock set, so the screen lands on the time we meant.

// Reading: the date and time the machine shows; its offset is not applied.
func (r *Reader) toLocalWall(s string) (time.Time, bool) {
	if len(s) < 19 {
		return time.Time{}, false
	}
	t, err := time.ParseInLocation("2006-01-02T15:04:05", s[:19], r.loc)
	return t, err == nil
}

// Writing: our wall time, in the machine's own stated offset ("+08:00").
func (r *Reader) wallToISO(t time.Time) string {
	return t.Format("2006-01-02T15:04:05") + r.machineOffset()
}

The older agent read the same field differently. It converted every punch by the offset the machine stated, into the office's zone. For a machine set by hand, like this one, that conversion moved every punch by the gap between the two zones, two hours and fifteen minutes here. For a machine kept right by a time server with the wrong zone, the conversion was right and the display was wrong. Neither rule is right for both machines.

The connector trusts the display, and leans on the clock check below for the other case. A machine whose display is more than 30 seconds off the platform's time is set right, in its own stated offset, so after the first correction the display and the instant agree. What that cannot fix is a backlog read before the first correction from a machine of the second kind. Deciding per machine, from how far its display and its stated instant each sit from the platform's time, is on the list of what the connector still needs.

One more rule, learned a week earlier: a clock set keeps the machine's timeMode. The first set always sent manual, which quietly turned NTP off; now it sends back whatever mode the machine reported.

The guard

A test plays a machine on +08:00 showing noon, with a punch at 11:30. It asserts the clock reads 12:00 local, the punch reads 11:30:00, the search window is written 11:00:00+08:00, and a set to noon is written 12:00:00+08:00. It fails with the change reverted, as does each of the other parity fixes in that commit.

The PC's clock is not the platform's clock

A signed request more than five minutes from the platform's time is refused, and office PC clocks are often wrong by more than that. So the client learns the platform's time from the Date header of every answer:

// The median of the last 5 answers' Date, each carried forward by Go's
// monotonic clock since it arrived. A step of the PC's wall clock, by a
// person or by "set time automatically", does not move it.
func (c *Client) ServerNow() time.Time {
	est := make([]time.Time, len(c.samples))
	for i, s := range c.samples {
		est[i] = s.server.Add(time.Since(s.local)) // s.local keeps its monotonic reading
	}
	slices.SortFunc(est, time.Time.Compare)
	return est[len(est)/2]
}

A refusal for the timestamp drops the old samples, re-reads the time from that very answer, and retries once; a refusal for the key is not retried. The window tells the person at the PC when their clock is more than two minutes off, because other programs on it are wrong too. Nothing in the connector uses it.

The same time drives the machines: the reading position and the 24-hour future cut-off use it, so a PC hours fast cannot hold a position ahead of a machine that was put right.

Setting a machine's clock

A machine an hour slow puts everyone an hour early, so each read compares the machine's clock with the platform's time. The rules:

• Off by more than 30 seconds: set it. Half a minute is already a late mark for someone. The first version waited for two minutes.
• At most once every 6 hours per machine, so a machine whose clock will not stay set is not fought with every minute.
• Refresh first. Before setting, the connector sends one signed ping so the platform's time is fresh. If that fails, nothing is set.
• "Set the clock now" from the platform sets it at once, regardless of tolerance and cap. Under a second off is left alone.
• Punches already read are never rewritten. The report says how far off the machine was and from when to when, so HR can see which punches carry the wrong time.
• A machine set back pulls its watermark back to the time it was set to. Otherwise the punches made just after the set are older than the watermark and skipped.

Only Hikvision and Dahua clocks are written. The ZKTeco reader is read only by construction, so a ZKTeco's clock is reported, never set.

A +05:45 zone is a good test zone

Most of these offices are in a +05:45 zone. I'd now pick a zone like it to test against even if I didn't have to, because it breaks every whole-hour assumption:

• Offsets have minutes. Formatting one is %02d:%02d of hours and minutes, and any code that builds an offset from hours alone is wrong.
• Hikvision writes its zone POSIX-style, sign inverted. CST-5:45:00 means five hours forty-five ahead of UTC. The simulator builds it that way so the tests see the real shape.
• A bare Windows PC has no zone database for Go. time.LoadLocation fails there, so the machine's zone falls back to a fixed +05:45 offset. That is a marked shortcut with a known ceiling: replace it the day a customer is in another zone.
• The errors do not look like zone errors. A factory +08:00 misread is 2 h 15 min. An offset applied the wrong way round is 11 h 30 min, which is what an audit query of mine produced against a UTC column stored without a zone: one AT TIME ZONE where it needed two, (t AT TIME ZONE 'UTC') AT TIME ZONE 'the office zone'. Neither number looks like "the zone is wrong" at a glance.

Incident three: the checkout after midnight

What it looked like

HR at one company said late work was splitting across days. The audit found it was true once: people who checked in between 10:00 and 12:00 checked out at 00:57 the next morning, and that 00:57 became the next day's first check-in, which then showed either 21 hours worked or a missing checkout.

What was happening

This is the platform's day builder, not the connector: the connector delivers the wall clock exactly, and pairing is the platform's job. The builder's overnight rule pulled a morning punch back only when the previous day's last punch was at 20:00 or later, and this company's shifts all end by 22:00 with check-ins around noon. Turning on the night-shift flag would not help either: its noon-to-noon window drops a 10:00 to 12:00 check-in. And the labels cannot be trusted to settle it: across the fleet, only about one machine in six labelled check-in and check-out reliably.

The fix, chosen by a shadow run

I tried rules against sixty days of real attendance, read only, before building one. A cut at the midpoint of each rest gap moved hundreds of days and was rejected: it split unrostered nights and followed misconfigured shifts. A broad continuation rule moved thousands and built days over 16 hours. The rule that survived is narrow. Yesterday is still open and did not end on a check-out; today's first punch is not labelled check-in, comes after midnight and before the middle of the rest gap or 06:00, whichever is earlier, and is within 16 hours of yesterday's opening punch. Then it belongs to yesterday.

With the exact code, the shadow run changed about one person-day in seven thousand, each one checked by hand and better than before, with nothing duplicated and nothing lost.

It also found a second bug. A punch rebuilds only its own date, and the nightly rebuild covered yesterday only when it ran before 01:30, so a 02:00 checkout never re-closed yesterday. A day that hands a punch back now queues a rebuild of yesterday. This one is built and in testing, not yet in production.

Incident four: a rewind that did nothing

What it looked like

The production audit's line-by-line comparison caught this before anyone hit it. On a quiet ZKTeco, "reset the reading position" logged the rewind, and the next read returned nothing, and the one after, until someone punched.

What was happening

A ZKTeco cannot filter its log, so a routine read first compares the machine's punch count with the count at the last full read and skips the transfer when they match (why). A rewind moves the watermark back, but the machine's count has not changed, so the very read the rewind set up was skipped as "nothing new". The older agent had cleared its count marker on every rewind. Its original commit even listed "anything that moves the watermark backwards must invalidate the marker" as a trap. The Go port kept the cache and lost the trap.

The fix and the guard

ForgetCount(serial), and one rewind helper that sets the watermark and forgets the count together. Every site that moves a position back uses it: the reset button, the future-watermark repair from incident one, and the old agent's positions taken over at handover (that post).

// rewind moves a machine's reading position and makes sure the next read
// really reads: the count-skip must not catch a moved-back position.
func (r *Runner) rewind(serial, at string) {
	r.Buf.SetNote(noteWatermark(serial), at)
	zkteco.ForgetCount(serial)
}

The guard runs a pretend ZKTeco holding one punch from 26 hours ago, reads it twice (nothing new), rewinds 48 hours, and asserts that the next read brings exactly that punch. Writing it found a second, smaller bug: the remembered count is process-wide, so the test passed once and failed under -count=2. Each run now uses its own serial.

What I would tell someone building this

• Keep the punch in the machine's wall clock. Convert nothing you cannot verify. The screen is what people check, and the same string from every path is what makes duplicates detectable.
• Put a ceiling on every high-water mark fed by a device clock, and apply it to the stored value on read. The read clamp is the one that repairs the field.
• Never clear a position to fix it. Rewind it, and invalidate every cache that means "nothing new" in the same function.
• Take time from the server and carry it on the monotonic clock. Use the PC's clock for nothing.
• When you write a device's clock, write it in the device's own dialect, keep its other settings, and never rewrite punches already read. Report the range instead.
• Test in a zone with minutes in it. Whole-hour zones hide a whole class of bug.