Gitead
ad2e15ec62
Replaces fragile whole-screen text matching with a screenshot-diff step that finds each dialog's actual pixel bounds, then restricts every subsequent AI locate/click to that cropped region — eliminating false matches from text elsewhere on screen (title bars, side panels). Adds column-boundary and row-height detection so the patient row and Exam procedure click positions are computed geometrically instead of relying on repeated fuzzy AI guesses for visually similar neighbors. Also adds per-run screenshot/debug-crop backups and a structured run.log for diagnosing failed runs, plus a cmd:move primitive on the Windows agent for pre-click confirmation crops. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dental Agent (Windows)
agent.py runs on a staff front-desk Windows PC. It connects to the app's server over the
/agent socket.io namespace and executes low-level commands — screenshot, click,
double-click, type, key press — sent by the server. It has no knowledge of dental software
or workflows; the app's AI reads the screenshots this agent sends back and decides what to
click or type next.
Not wired into the repo's root npm install on purpose — it targets Windows (pyautogui,
tkinter) and doesn't need to install on every contributor's machine.
Command protocol
Sent by the server, over the /agent namespace, as socket.io events with an ack callback:
| Event | Payload | Ack response |
|---|---|---|
cmd:screenshot |
{} |
{ image: <base64 PNG> } |
cmd:click |
{ x, y } |
{ ok: true } |
cmd:double_click |
{ x, y } |
{ ok: true } |
cmd:double_click_current |
{} |
{ ok: true } |
cmd:type |
{ text } |
{ ok: true } |
cmd:key |
{ key } |
{ ok: true } |
Auth on connect: { auth: { token } }, where token must match the server's
WINDOWS_AGENT_TOKEN. The token is a constant baked into agent.py (AGENT_TOKEN near the
top) — set it before building for a real office, staff never see or type it. The server tells
front-desk PCs apart by connection IP address (shown on the Type Agent page), not anything
the agent sends.
Local dev / testing
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python agent.py
On first run it asks only for the Server URL, then saves it to agent_config.json next to
the script so it reconnects automatically after that. After that it has no window — just a
system tray icon (green = Connected, gray = Connecting, red = Disconnected). Click it (or
right-click for the same menu) for status, the server URL, "Settings..." (reopens the setup
window pre-filled with the current URL — close without submitting to leave it unchanged), and
Quit.
apps/Backend/scripts/fake-windows-agent.js is a Node stand-in for this agent — useful for
testing the server-side bridge without a Windows PC or a real screen to click on.
Building DentalAgent.exe
Must be run on Windows (PyInstaller doesn't cross-compile):
py -m venv .venv
.venv\Scripts\pip install -r requirements.txt
.venv\Scripts\pyinstaller --onefile --windowed --name DentalAgent agent.py
Output: dist\DentalAgent.exe. Staff download and run it once — no other setup.