Reading four brands of attendance machine

The architecture post treats "read the machine" as one box. This post opens it. Inside are four brands that share almost nothing on the wire:

Almost every rule below came from the older Python agent the connector replaced, and most of those came from a support ticket. I've kept the ticket next to each rule. The code is generalised; the numbers are the commits'.

One interface, one registry

Every machine looks the same to the rest of the connector:

// Reader is one machine, built per use and closed after it.
type Reader interface {
	Info(ctx context.Context) (Info, error)                              // alive and authenticated; no log walk
	Punches(ctx context.Context, since, until time.Time) ([]Punch, error) // (since, until]
	Users(ctx context.Context) ([]User, error)
	Close() error
}

// Optional abilities, found with a type assertion.
type Identifier interface{ Identify(ctx context.Context) (serial string, err error) }
type Clock interface {
	Clock(ctx context.Context) (time.Time, error)
	SetClock(ctx context.Context, t time.Time) error
}
type Detected interface{ Detected() (vendor, model, serial string) }

Each vendor package registers an opener from its init. Open refuses an unknown vendor, and refuses a host that is not an IP address or a plain host name: no slash, ?, #, @ or space, so an address pasted into a URL can only ever name a machine. A missing vendor is an error, not a crash, worded for the person who will read it on the machine's card in the platform.

A reader is built fresh for every pull, so anything it learns has to live outside it, or it is forgotten every sixty seconds.

Before any read: the machine says its own serial

Two machines in one office can swap addresses. The platform still has the old ones, so nothing short of asking can tell, and reading the wrong machine would file its punches, and its staff numbers, under the other. So every open asks:

if id, ok := reader.(Identifier); ok && target.Serial != "" {
	got, err := id.Identify(ctx)
	if err != nil {
		reader.Close()
		return nil, err
	}
	if got != "" && !SameSerial(got, target.Serial) {
		reader.Close()
		return nil, &WrongMachineError{Host: target.Host, Want: target.Serial, Got: got}
	}
}

SameSerial ignores case and spaces, and accepts one serial ending with the other when the shorter has eight or more characters: a Hikvision reports a long form (model, date, label serial) while a person types the label. The first version cached the answer for an hour. A review pointed out that a swap is exactly what a cache hides, so it runs on every read: one small request a minute per machine.

ZKTeco: a binary protocol on TCP 4370

ZKTeco (and eSSL, the same machines under another name) speak their own little-endian protocol:

TCP frame:  50 50 82 7D | uint32 length of the packet that follows
packet:     uint16 command | uint16 checksum | uint16 session | uint16 reply no. | data
checksum:   the RFC 1071 Internet checksum, taken with the checksum field at 0

A bulk read prepares a table (users or the log), fetches it in pieces and frees the buffer.

Read only, by construction. The package has ten request constants: connect, exit, auth, read a setting, read the counters, read the clock, read the firmware version, and the three steps of a bulk read. The send path refuses any other code, and nothing that clears, enrols, unlocks or restarts a machine exists to be sent by accident. The price is that the connector cannot set a ZKTeco's clock. I took that trade on purpose.

The comm key. A machine with a key answers CMD_CONNECT with "unauthorised" (2005) and a session number. The connector replies with CMD_AUTH (1102) and a four-byte proof: the key's bits reversed plus the session, XORed with ZKSO, halves swapped, mixed with a constant. A refused proof once came back as no error at all, so the card could only say "not reading" and support went looking for a network fault. Now it is its own error kind, and the card says the key is wrong.

Record layouts. A user is 72 bytes on current firmware and 28 on old; a punch is 40, 16 or 8. The reader picks the size that matches the machine's own counter. The 8-byte layout stores a slot number, not the staff number, so for it the user table is read and mapped across.

The count-skip cache

A ZKTeco cannot filter its log, so a read is the whole log. On the staging machine one full read took 196 seconds against a pull loop that fires every 60, and the machine serves one session at a time. It was almost never idle, and a manual "sync now" usually lost the race and died with a bare "timed out". That reads as dead hardware. It was two of our own threads queueing for one socket. The older agent's fix, kept in the connector, has two halves:

• One lock per machine, shared by the pull loop and commands. The loop skips a busy machine (the next tick is a minute away, and the watermark makes a skipped cycle free); a command waits, because a person asked.
• Skip the transfer when nothing changed. Reading the counters takes about a second. A routine read that finds the same punch count as the last full read returns nothing without fetching the log.

if until.IsZero() && !r.logRead { // a routine read, not a backfill
	c, err := r.counters(ctx)
	if err != nil {
		return nil, err // an unreadable count is never "unchanged"
	}
	if unchanged(r.serial, c.records) {
		return nil, nil
	}
}

The count lives in memory, so the first read after a restart always reads. Its known ceiling (the older agent's too): a machine at its storage cap keeps a constant count while old punches roll off. Its trap: anything that moves the reading position back must clear the count, as the time post explains.

In-process, not a sidecar

The first Go connector supervised the older Python agent, frozen into a one-file executable, as a ZKTeco sidecar. A one-file build forks a second process, so kills had to take the process group; output had to go to a file so a killed child could not hold Wait() open; CI needed a Windows job to freeze it. Two days later an 800-line Go reader replaced it, with a pretend machine for tests (how that is tested). One program now, and every download and self-update reads ZKTeco.

Hikvision: ISAPI over HTTP

Hikvision terminals speak ISAPI: JSON, sometimes XML, over HTTP. The connector reads AcsEvent, UserInfo/Search, deviceInfo and System/time. It answers an MD5 digest challenge, or Basic when the machine is set to Basic, and reads every WWW-Authenticate header, not only the first (SHA-256-only units are a known gap). Some firmwares answer XML under /ISAPI/System despite ?format=json, which left one unit's clock check blind for three days, so every reply is tried as JSON, then XML.

A 401 is not always a wrong password

In two weeks of production, more than half of Hikvision backfills failed, almost all with 401 and "check the username and password". The password was right: the live pull loop kept delivering, and a retry a minute later passed.

A Hikvision binds a digest nonce to the connection that issued it. When it drops a keep-alive, the next request replays a dead nonce. The pull loop survived that by trying again next tick; a command tries once. Now a 401 renegotiates from scratch on a new connection:

if resp.StatusCode == http.StatusUnauthorized {
	auth.forget() // drop the nonce
	if attempt == tries-1 {
		return nil, ErrAuth // three refusals, a second apart: now it is the password
	}
	client.CloseIdleConnections() // the next try handshakes on a fresh socket
	sleep(ctx, time.Second)
	continue
}

The search ladder: every parameter, not four guesses

Some firmwares refuse the AcsEvent search body with badParameters (status code 6), as JSON, as XML, or inside an HTTP 200. The older agent first shipped four fallback bodies, and one unit rejected all four. That is the case against hand-picked rungs: each round of guessing costs a visit to a client's office and teaches you one bit. So the ladder now covers every parameter an ISAPI firmware is known to be fussy about, twelve rungs ordered by what each gives up.

Three rules make it safe to ship. Rung 0 is byte for byte what a healthy terminal already accepts, so a working unit never sees a changed request. The ladder engages only on badParameters, never on a 401, a timeout or a refused connection. No rung can lose a punch: the end cap stops at now plus a day (a later punch is a broken clock the platform discards anyway), the start floor only moves a pre-2010 bound, an offset rung widens the window by a day on each side, and the exact (since, until] is re-applied locally.

The unit that beat the four guesses latched on rung 9, then showed two more bugs. The latch was thrown away every minute: an instance field on a reader built per pull, so five pulls made 45 rejected requests instead of 9. What a machine teaches the connector now lives in a package-level map keyed by serial (not on disk: re-learning costs a few requests, a file would need invalidating on every firmware upgrade). The roster broke three days later, because the ladder lived inside the event search. Every ISAPI search carries searchID, searchResultPosition and maxResults, so they now come from one envelope builder, and a rung proven on either path latches for both.

Ask the firmware instead of guessing

Rungs 5 and 9 differ in one field. The 36-character dashed UUID in searchID was the whole problem. And the unit, a DS-K1A8503EF-B, publishes that itself:

GET /ISAPI/AccessControl/AcsEvent/capabilities?format=json
  searchID    @max 20
  maxResults  @max 10

Two support cycles of elimination had been one GET away since the day it was installed. The envelope now reads /capabilities once per serial and clamps the searchID shape and page size to the published bounds, so an affected unit accepts rung 0 on the first request; a unit without /capabilities behaves exactly as before. The parse is lenient on purpose. A bound may be written {"@max": 20}, {"max": 20} or a bare 20, and the document is read hop by hop, so one field of an odd shape counts as "not stated" instead of throwing away the bounds that did parse.

The same unit taught two smaller things. The roster read asks UserInfo/Count first, because an empty roster and an unreadable one were both an empty list. And when every rung fails, the message says firmware mismatch, not password, and asks for the model, instead of raw JSON that reads as "the machine is broken".

Port 4370 on a web machine means "not set"

The production audit found about four in five Hikvision rows carrying port 4370, ZKTeco's port, left by the older setup's default. The older agent never saw the platform's port, so it did not matter. The connector dialled it. No web machine answers there, so for Hikvision and Dahua a stored 4370 now counts as unset, like 0, and the port falls back to 80 or 443. The older agent also fell back from HTTPS to HTTP when one scheme refused. The connector does not: quietly moving a machine to plain HTTP would send its password in the clear because one connection failed.

Backfill one day at a time

"Bring in older punches" was one search with one time budget. A terminal that pages ten rows at a time needs about 1,200 requests for 120 days, so a long backfill could time out and keep nothing. Now each day is its own search and budget, uploaded before the next, and a failure keeps every finished day. A ZKTeco pays nothing extra: it fetches its log once and filters per day.

Dahua: CGI

Dahua terminals expose CGI over HTTP with digest: recordFinder.cgi for punches, global.cgi for the clock, magicBox.cgi for identity. recordFinder has no result offset, so paging moves StartTime to the newest CreateTime seen, inclusive, and a set of record numbers stops that boundary second being added twice.

To be plain: this reader was built from the API contract, not on a real unit. The contract has no confirmed in/out field, so every Dahua punch is a plain "check" and the platform pairs the day by time. There is no roster read. And it shipped with an embarrassing gap: nothing registered it until the auto-detect change did.

Hikvision or Dahua? Ask the machine

People adding a machine often cannot tell these two apart, so one tile sends the vendor auto_ipc. That reader tries each brand's Info, Hikvision first, keeps and remembers the winner, forwards every call to it, and reports it through Detected.

Sintech: a machine that only pushes

Sintech terminals (a Realand OEM, going by the MAC's vendor block) answered no pull on any port we scanned. With "Real Time Push" on, each punch leaves as a 32-byte UDP datagram for the server and port set in the keypad menu, which defaults to the vendor's own cloud. Pointing it at a listener of ours, here a PC in the office, is how that hardware's attendance reaches the platform.

The handshake storm

The first real capture was 61 frames in one minute, sixty of them byte-identical, without pause. That is a device announcing itself and waiting for a reply nobody sent. The first storm guard required every frame to be identical, but the capture held two distinct payloads, so a dry-run replay put two handshakes into ingest as "punches". A storm is now one payload making up 90% of arrivals, and it is never returned as attendance. What the terminal wanted was its own bytes back. Once it got its echo, it uploaded its stored backlog of 459 records.

The ST-F41B decode, verified on real hardware

Those records, plus live punches whose arrival time is the ground truth, settled the layout. Every live anchor decodes within two seconds of arrival.

The order of discovery matters more than the table. A layout that passed every statistical test over stored records was still an hour wrong on a live punch, so the test fixtures are the live frames themselves.

Byte 0x12 is not check-in/check-out. Briefly its high nibble was read as the direction, 0 in and 5 out. Then a real check-in on staging was filed as a check-out: it carried 5, like an earlier check-out, while the check-in before it carried 0, another punch carried 6, and every stored record carried 0. Nothing shown so far puts in/out on the wire. Every Sintech punch is now a plain check, the nibble is kept for diagnosis, and the platform pairs by time.

What the connector does with a datagram: nothing clever

In the connector, Sintech is not a Reader but a UDP pass-through on port 5055. It never decodes a frame or composes a reply: each datagram goes to the platform as it arrived, and the only bytes sent back are the ones the platform returned, verbatim. As the PC's one inbound attendance socket it is treated as an exposure: non-private sources are dropped, each source gets 120 datagrams a minute, anything over 512 bytes is dropped, and frames the platform cannot take yet go to a bounded disk store.

Lessons for anyone reading the same machines

• Ask the device before you guess. The capabilities document and the user count each replaced a support cycle.
• Make each fallback give something up safely. Widen, never narrow; re-apply the exact bounds locally; engage only on the error it exists for.
• Keep what a device teaches you outside the object that learned it.
• Check identity on every read. Addresses move without telling anyone.
• Treat in/out labels as unproven until live hardware proves them.
• Keep the protocol read only where you can. Then nothing can harm a machine.

The machines' clocks get a post of their own: time is the hardest field in a punch.