Gitead/DentalManagementMH07
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: window-diff-based Type Agent locating for Open Dental appointment flow

Browse Source 
Replaces fragile whole-screen text matching with a screenshot-diff step that
finds each dialog's actual pixel bounds, then restricts every subsequent AI
locate/click to that cropped region — eliminating false matches from text
elsewhere on screen (title bars, side panels). Adds column-boundary and
row-height detection so the patient row and Exam procedure click positions
are computed geometrically instead of relying on repeated fuzzy AI guesses
for visually similar neighbors. Also adds per-run screenshot/debug-crop
backups and a structured run.log for diagnosing failed runs, plus a
cmd:move primitive on the Windows agent for pre-click confirmation crops.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Gitead
2026-07-27 08:50:47 -04:00
co-authored by Claude Sonnet 5
parent 59f064583a
commit ad2e15ec62
8 changed files with 817 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/Backend/src/routes/type-agent.ts
+4 -3
View File
@@ -2,7 +2,7 @@ import crypto from "crypto";
import { Router, Request, Response } from "express";
import { getAgentStatus, io } from "../socket";
import { captureScreenshot, click, disconnectAgent, doubleClick, doubleClickCurrent, pressKey, typeText } from "../services/windowsAgentBridge";
import { runExistingPatientOpenDental } from "../services/typeAgentRunner";
import { runExistingPatientOpenDental, runExistingPatientOpenDentalRatioTest } from "../services/typeAgentRunner";
import { storage } from "../storage";
const router = Router();
@@ -80,7 +80,7 @@ router.post("/run", async (req: Request, res: Response) => {
return;
}
const { software, action, ip, patientId } = req.body ?? {};
const { software, action, ip, patientId, ratioTest } = req.body ?? {};
if (software !== "open-dental" || action !== "existing-patient") {
res.status(400).json({ error: "This software/action combination isn't automated yet." });
return;
@@ -99,7 +99,8 @@ router.post("/run", async (req: Request, res: Response) => {
const runId = crypto.randomUUID();
res.json({ runId });
runExistingPatientOpenDental(userId, ip, patient.lastName, patient.firstName, (index, total, label, status, error) => {
const runFn = ratioTest ? runExistingPatientOpenDentalRatioTest : runExistingPatientOpenDental;
runFn(userId, runId, ip, patient.lastName, patient.firstName, (index, total, label, status, error) => {
io?.emit(`type-agent:run:${runId}`, { index, total, label, status, error });
}).catch((err) => {
console.error("[type-agent/run] failed:", err);