feat: window-diff-based Type Agent locating for Open Dental appointment flow

Replaces fragile whole-screen text matching with a screenshot-diff step that
finds each dialog's actual pixel bounds, then restricts every subsequent AI
locate/click to that cropped region — eliminating false matches from text
elsewhere on screen (title bars, side panels). Adds column-boundary and
row-height detection so the patient row and Exam procedure click positions
are computed geometrically instead of relying on repeated fuzzy AI guesses
for visually similar neighbors. Also adds per-run screenshot/debug-crop
backups and a structured run.log for diagnosing failed runs, plus a
cmd:move primitive on the Windows agent for pre-click confirmation crops.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 08:50:47 -04:00
parent 59f064583a
commit ad2e15ec62
8 changed files with 817 additions and 96 deletions

View File

@@ -29,6 +29,11 @@ socket.on("cmd:screenshot", (_payload, ack) => {
ack({ image: FAKE_SCREENSHOT });
});
socket.on("cmd:move", ({ x, y }, ack) => {
console.log(`🖱️ move to (${x}, ${y}) (no click)`);
ack({ ok: true });
});
socket.on("cmd:click", ({ x, y }, ack) => {
console.log(`🖱️ click at (${x}, ${y})`);
ack({ ok: true });