feat: window-diff-based Type Agent locating for Open Dental appointment flow
Replaces fragile whole-screen text matching with a screenshot-diff step that finds each dialog's actual pixel bounds, then restricts every subsequent AI locate/click to that cropped region — eliminating false matches from text elsewhere on screen (title bars, side panels). Adds column-boundary and row-height detection so the patient row and Exam procedure click positions are computed geometrically instead of relying on repeated fuzzy AI guesses for visually similar neighbors. Also adds per-run screenshot/debug-crop backups and a structured run.log for diagnosing failed runs, plus a cmd:move primitive on the Windows agent for pre-click confirmation crops. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -107,6 +107,13 @@ class DentalAgent:
|
||||
encoded = base64.b64encode(buf.getvalue()).decode("ascii")
|
||||
return {"image": encoded}
|
||||
|
||||
@sio.on("cmd:move", namespace="/agent")
|
||||
def on_move(data):
|
||||
# Moves the cursor without clicking — used to double-check a location (via a
|
||||
# follow-up cropped screenshot) before committing to a click.
|
||||
pyautogui.moveTo(data["x"], data["y"])
|
||||
return {"ok": True}
|
||||
|
||||
@sio.on("cmd:click", namespace="/agent")
|
||||
def on_click(data):
|
||||
pyautogui.click(data["x"], data["y"])
|
||||
|
||||
Reference in New Issue
Block a user