feat: AI Type Agent - vision-based Open Dental automation via Windows agent
Adds a Windows-side agent (screenshot/click/type over HTTP) plus backend services to locate UI elements via vision and drive an existing-patient appointment flow in Open Dental, wired into the Copy/Type Agent page and socket progress updates. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
59
apps/WindowsAgent/README.md
Normal file
59
apps/WindowsAgent/README.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Dental Agent (Windows)
|
||||
|
||||
`agent.py` runs on a staff front-desk Windows PC. It connects to the app's server over the
|
||||
`/agent` socket.io namespace and executes low-level commands — screenshot, click,
|
||||
double-click, type, key press — sent by the server. It has no knowledge of dental software
|
||||
or workflows; the app's AI reads the screenshots this agent sends back and decides what to
|
||||
click or type next.
|
||||
|
||||
Not wired into the repo's root `npm install` on purpose — it targets Windows (`pyautogui`,
|
||||
`tkinter`) and doesn't need to install on every contributor's machine.
|
||||
|
||||
## Command protocol
|
||||
|
||||
Sent by the server, over the `/agent` namespace, as socket.io events with an ack callback:
|
||||
|
||||
| Event | Payload | Ack response |
|
||||
|---------------------|-----------------------|--------------------|
|
||||
| `cmd:screenshot` | `{}` | `{ image: <base64 PNG> }` |
|
||||
| `cmd:click` | `{ x, y }` | `{ ok: true }` |
|
||||
| `cmd:double_click` | `{ x, y }` | `{ ok: true }` |
|
||||
| `cmd:double_click_current` | `{}` | `{ ok: true }` |
|
||||
| `cmd:type` | `{ text }` | `{ ok: true }` |
|
||||
| `cmd:key` | `{ key }` | `{ ok: true }` |
|
||||
|
||||
Auth on connect: `{ auth: { token } }`, where `token` must match the server's
|
||||
`WINDOWS_AGENT_TOKEN`. The token is a constant baked into `agent.py` (`AGENT_TOKEN` near the
|
||||
top) — set it before building for a real office, staff never see or type it. The server tells
|
||||
front-desk PCs apart by connection IP address (shown on the Type Agent page), not anything
|
||||
the agent sends.
|
||||
|
||||
## Local dev / testing
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt
|
||||
.venv/bin/python agent.py
|
||||
```
|
||||
|
||||
On first run it asks only for the Server URL, then saves it to `agent_config.json` next to
|
||||
the script so it reconnects automatically after that. After that it has no window — just a
|
||||
system tray icon (green = Connected, gray = Connecting, red = Disconnected). Click it (or
|
||||
right-click for the same menu) for status, the server URL, "Settings..." (reopens the setup
|
||||
window pre-filled with the current URL — close without submitting to leave it unchanged), and
|
||||
Quit.
|
||||
|
||||
`apps/Backend/scripts/fake-windows-agent.js` is a Node stand-in for this agent — useful for
|
||||
testing the server-side bridge without a Windows PC or a real screen to click on.
|
||||
|
||||
## Building DentalAgent.exe
|
||||
|
||||
Must be run **on Windows** (PyInstaller doesn't cross-compile):
|
||||
|
||||
```bat
|
||||
py -m venv .venv
|
||||
.venv\Scripts\pip install -r requirements.txt
|
||||
.venv\Scripts\pyinstaller --onefile --windowed --name DentalAgent agent.py
|
||||
```
|
||||
|
||||
Output: `dist\DentalAgent.exe`. Staff download and run it once — no other setup.
|
||||
Reference in New Issue
Block a user