OraScan - Oral Disease Detection

The problem

Over 3.5 billion people worldwide have some form of oral disease, but most never get diagnosed because seeing a specialist is too expensive or too far away. A small camera at a dental kiosk should be enough to flag problems early - if the AI behind it is reliable enough to trust with real patients.

The challenge

Oral disease affects more than 3.5 billion people globally, yet it often goes undiagnosed in low-resource settings. The model had to be accurate enough for clinical use, small enough to run on kiosk hardware without a GPU, and consistent across different lighting conditions, camera angles, and skin tones. It also had to handle class imbalance across 11 categories, including common conditions such as caries and calculus and rare ones such as mucocele and hypodontia, without losing recall on oral cancer.

Architecture & System Design

Desktop scanning application captures oral images and sends them to inference engine for disease classification. Inference system processes images locally without cloud connectivity. Analysis results and images stored in cloud database and file storage system for historical tracking and specialist review.

A custom EfficientNet-B0 backbone (4.67M parameters) was fine-tuned on a combined DENTEX + SMART-OM dataset of 78,058 labelled oral images. Training used PyTorch with AMD GPU acceleration (DirectML on Windows). The data pipeline applies aggressive augmentation: random crops, HSV jitter, horizontal flip, and cutmix. Post-training, the model is exported to ONNX and quantised to INT8 for edge deployment. A FastAPI inference server wraps the ONNX runtime and exposes a REST endpoint consumed by the kiosk scanning application. The Go backend stores session results in PostgreSQL and uploads image artefacts to AWS S3.

Code Walkthrough

3-step walk-through of the production implementation. File paths and intent appear above each block.

1. 01

   Step 1 of 3

   Mouth-open detection as a capture trigger

   OraScan-Facemesh/mouth.py

   The scanner shouldn't shoot at random - it should fire the moment the patient's mouth is open enough for a clean intraoral view. MediaPipe's face mesh gives us the inner upper/lower lip landmarks as normalised (y) coordinates, and the per-frame pixel delta is a clean-enough signal to gate acquisition without any ML at all.

   pythonCopy

   UPPER_LIP_IDX = 13   # MediaPipe face mesh: inner upper lip
   LOWER_LIP_IDX = 14   # MediaPipe face mesh: inner lower lip
   OPEN_THRESHOLD_PX = int(os.getenv("MOUTH_OPEN_THRESHOLD_PX", "45"))
   
   def mouth_open_trigger(on_open: Callable[[np.ndarray], None]) -> None:
       cap  = cv2.VideoCapture(0)
       mesh = mp.solutions.face_mesh.FaceMesh(
           min_detection_confidence=0.5,
           min_tracking_confidence=0.5,
       )
       try:
           while cap.isOpened():
               ok, frame = cap.read()
               if not ok:
                   continue
               result = mesh.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
               if not result.multi_face_landmarks:
                   continue
   
               lm     = result.multi_face_landmarks[0].landmark
               height = frame.shape[0]
               gap_px = int((lm[LOWER_LIP_IDX].y - lm[UPPER_LIP_IDX].y) * height)
   
               if gap_px > OPEN_THRESHOLD_PX:
                   on_open(frame)           # hand the frame to the acquisition pipeline
       finally:
           cap.release()
           mesh.close()

   Takeaway

   Two landmarks and a threshold replace an entire 'press to capture' UX. The CV runs in a tight loop; the rest of the pipeline stays frame-agnostic.

2. 02

   Step 2 of 3

   Bounding hardware calls with a timeout decorator

   OraScan_Automated_Scanning/motor_driver.py

   GPIO and servo drivers hang. Not crash - hang, indefinitely, when the bus misbehaves or a pin is stuck. An ordinary Python call into a blocked driver freezes the scanner UI and the operator has to hard-reboot. A one-line decorator submits every hardware call into a thread, waits up to N seconds, and returns None on timeout, so the UI always stays responsive.

   pythonCopy

   def hardware_timeout(seconds: int = 5):
       """Wrap a hardware call so a hung device can never block the UI."""
       def decorator(fn):
           @functools.wraps(fn)
           def wrapper(*args, **kwargs):
               with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
                   future = executor.submit(fn, *args, **kwargs)
                   try:
                       return future.result(timeout=seconds)
                   except concurrent.futures.TimeoutError:
                       logging.warning(
                           "hardware: %s timed out after %ds", fn.__name__, seconds
                       )
                       return None
           return wrapper
       return decorator
   
   
   @hardware_timeout(seconds=3)
   def move_servo(axis: str, angle: int) -> None:
       """Move a servo to an absolute angle; blocks until the pulse is sent."""
       servo = _servos[axis]
       servo.move_to_angle(max(0, min(180, angle)))

   Takeaway

   Hardware is the one place you should never trust a library's own timeout handling - wrap every motor/sensor call in a bounded executor, treat None as 'try again', and the UI is freed from the physical layer.

3. 03

   Step 3 of 3

   Desktop ↔ cloud patient reconciliation

   OraScan_backend/sync_handler.go

   The desktop scanner stores every patient locally in MySQL so scans work offline. When the operator reconnects, each record is POSTed to this handler for reconciliation against the cloud store. The important design decision: we refuse to auto-create patients from sync data - sync is reconciliation, not a registration shortcut. Bypassing the signup flow would skip consent capture and identity checks.

   goCopy

   type SyncPatientInput struct {
       Email         string `json:"email" binding:"required"`
       FirstName     string `json:"first_name"`
       LastName      string `json:"last_name"`
       Contact       string `json:"contact"`
       Place         string `json:"place"`
       LocalID       int    `json:"local_id"`       // desktop app's row ID
       SyncTimestamp string `json:"sync_timestamp"` // ISO 8601
   }
   
   func (s *ApiServer) SyncPatient(c *gin.Context) {
       userID, ok := c.Get("userID")
       if !ok {
           c.JSON(http.StatusUnauthorized, gin.H{"error": "missing user context"})
           return
       }
   
       var in SyncPatientInput
       if err := c.ShouldBindJSON(&in); err != nil {
           c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
           return
       }
   
       existing, err := s.store.GetUserByEmail(in.Email)
       if err == nil && existing != nil {
           s.audit.SyncMerge(userID, in.Email, in.LocalID, existing.ID)
           c.JSON(http.StatusOK, gin.H{
               "status":   "merged",
               "cloud_id": existing.ID,
               "at":       time.Now().UTC().Format(time.RFC3339),
           })
           return
       }
   
       // Strict mode: never auto-create on sync. If the patient isn't
       // already in the cloud, the desktop operator must route them through
       // the normal registration flow first.
       c.JSON(http.StatusNotFound, gin.H{
           "error": "patient not found in cloud; please register first",
       })
   }

   Takeaway

   Offline-first apps have their own IDs; the cloud has its own. Sync handlers merge on a stable external key, audit every reconciliation, and reject rather than invent records that didn't come through the proper signup flow.

Results

The final model achieves 94.7% test accuracy and 95.2% best validation accuracy across all 11 disease classes. ONNX INT8 quantisation reduces model size by ~70% while keeping accuracy drop under 1%. Inference latency on the kiosk CPU is under 200ms per frame. The model is integrated into both the automated scanning application (MediaPipe FaceMesh-triggered) and the manual scanning desktop interface.

Gallery & Demos

AI Analysis Dashboard

Real-time disease classification results showing confidence scores across 11 oral disease categories (caries, gingivitis, cancer, etc.).

Desktop Sync Workflow

Offline-first architecture: local MySQL stores patient scans, cloud sync reconciles on reconnect matching by email, enforcing signup flow for consent.

Kiosk Hardware Layout

MediaPipe FaceMesh detects mouth-open trigger, motorized camera captures intraoral image, ONNX INT8 inference runs on CPU in <200ms.

Click any image or video to expand · ← → keys navigate