feat: window-diff-based Type Agent locating for Open Dental appointment flow

Replaces fragile whole-screen text matching with a screenshot-diff step that
finds each dialog's actual pixel bounds, then restricts every subsequent AI
locate/click to that cropped region — eliminating false matches from text
elsewhere on screen (title bars, side panels). Adds column-boundary and
row-height detection so the patient row and Exam procedure click positions
are computed geometrically instead of relying on repeated fuzzy AI guesses
for visually similar neighbors. Also adds per-run screenshot/debug-crop
backups and a structured run.log for diagnosing failed runs, plus a
cmd:move primitive on the Windows agent for pre-click confirmation crops.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 08:50:47 -04:00
parent 59f064583a
commit ad2e15ec62
8 changed files with 817 additions and 96 deletions

View File

@@ -107,6 +107,13 @@ class DentalAgent:
encoded = base64.b64encode(buf.getvalue()).decode("ascii")
return {"image": encoded}
@sio.on("cmd:move", namespace="/agent")
def on_move(data):
# Moves the cursor without clicking — used to double-check a location (via a
# follow-up cropped screenshot) before committing to a click.
pyautogui.moveTo(data["x"], data["y"])
return {"ok": True}
@sio.on("cmd:click", namespace="/agent")
def on_click(data):
pyautogui.click(data["x"], data["y"])