Gitead
59f064583a
Adds a Windows-side agent (screenshot/click/type over HTTP) plus backend services to locate UI elements via vision and drive an existing-patient appointment flow in Open Dental, wired into the Copy/Type Agent page and socket progress updates. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
60 lines
2.7 KiB
Markdown
60 lines
2.7 KiB
Markdown
# Dental Agent (Windows)
|
|
|
|
`agent.py` runs on a staff front-desk Windows PC. It connects to the app's server over the
|
|
`/agent` socket.io namespace and executes low-level commands — screenshot, click,
|
|
double-click, type, key press — sent by the server. It has no knowledge of dental software
|
|
or workflows; the app's AI reads the screenshots this agent sends back and decides what to
|
|
click or type next.
|
|
|
|
Not wired into the repo's root `npm install` on purpose — it targets Windows (`pyautogui`,
|
|
`tkinter`) and doesn't need to install on every contributor's machine.
|
|
|
|
## Command protocol
|
|
|
|
Sent by the server, over the `/agent` namespace, as socket.io events with an ack callback:
|
|
|
|
| Event | Payload | Ack response |
|
|
|---------------------|-----------------------|--------------------|
|
|
| `cmd:screenshot` | `{}` | `{ image: <base64 PNG> }` |
|
|
| `cmd:click` | `{ x, y }` | `{ ok: true }` |
|
|
| `cmd:double_click` | `{ x, y }` | `{ ok: true }` |
|
|
| `cmd:double_click_current` | `{}` | `{ ok: true }` |
|
|
| `cmd:type` | `{ text }` | `{ ok: true }` |
|
|
| `cmd:key` | `{ key }` | `{ ok: true }` |
|
|
|
|
Auth on connect: `{ auth: { token } }`, where `token` must match the server's
|
|
`WINDOWS_AGENT_TOKEN`. The token is a constant baked into `agent.py` (`AGENT_TOKEN` near the
|
|
top) — set it before building for a real office, staff never see or type it. The server tells
|
|
front-desk PCs apart by connection IP address (shown on the Type Agent page), not anything
|
|
the agent sends.
|
|
|
|
## Local dev / testing
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
.venv/bin/pip install -r requirements.txt
|
|
.venv/bin/python agent.py
|
|
```
|
|
|
|
On first run it asks only for the Server URL, then saves it to `agent_config.json` next to
|
|
the script so it reconnects automatically after that. After that it has no window — just a
|
|
system tray icon (green = Connected, gray = Connecting, red = Disconnected). Click it (or
|
|
right-click for the same menu) for status, the server URL, "Settings..." (reopens the setup
|
|
window pre-filled with the current URL — close without submitting to leave it unchanged), and
|
|
Quit.
|
|
|
|
`apps/Backend/scripts/fake-windows-agent.js` is a Node stand-in for this agent — useful for
|
|
testing the server-side bridge without a Windows PC or a real screen to click on.
|
|
|
|
## Building DentalAgent.exe
|
|
|
|
Must be run **on Windows** (PyInstaller doesn't cross-compile):
|
|
|
|
```bat
|
|
py -m venv .venv
|
|
.venv\Scripts\pip install -r requirements.txt
|
|
.venv\Scripts\pyinstaller --onefile --windowed --name DentalAgent agent.py
|
|
```
|
|
|
|
Output: `dist\DentalAgent.exe`. Staff download and run it once — no other setup.
|