OraScan — Oral Disease Detection

The Challenge

Oral diseases (caries, gingivitis, periodontal disease, oral cancer) affect over 3.5 billion people globally yet go undiagnosed in low-resource settings due to the cost of specialist consultations. The challenge was to build a classification model accurate enough to be clinically useful, compact enough to run inference on kiosk hardware without a GPU, and robust enough to generalise across varied lighting conditions, camera angles, and skin tones. The model had to handle class imbalance across 11 categories — from common (caries, calculus) to rare (mucocele, hypodontia) — without sacrificing recall on clinically dangerous classes like oral cancer.

Architecture & System Design

A custom EfficientNet-B0 backbone (4.67M parameters) was fine-tuned on a combined DENTEX + SMART-OM dataset of 78,058 labelled oral images. Training used PyTorch with AMD GPU acceleration (DirectML on Windows). The data pipeline applies aggressive augmentation: random crops, HSV jitter, horizontal flip, and cutmix. Post-training, the model is exported to ONNX and quantised to INT8 for edge deployment. A FastAPI inference server wraps the ONNX runtime and exposes a REST endpoint consumed by the kiosk scanning application. The Go backend stores session results in PostgreSQL and uploads image artefacts to AWS S3.

Code Walkthrough

3-step walk-through of the production implementation — file paths and intent shown above each block.

1. 01

   Step 1 of 3

   Mouth-open detection as a capture trigger

   OraScan-Facemesh/mouth.py

   The scanner shouldn't shoot at random — it should fire the moment the patient's mouth is open enough for a clean intraoral view. MediaPipe's face mesh gives us the inner upper/lower lip landmarks as normalised (y) coordinates, and the per-frame pixel delta is a clean-enough signal to gate acquisition without any ML at all.

   pythonCopy

   UPPER_LIP_IDX = 13   # MediaPipe face mesh: inner upper lip
   LOWER_LIP_IDX = 14   # MediaPipe face mesh: inner lower lip
   OPEN_THRESHOLD_PX = int(os.getenv("MOUTH_OPEN_THRESHOLD_PX", "45"))
   
   def mouth_open_trigger(on_open: Callable[[np.ndarray], None]) -> None:
       cap  = cv2.VideoCapture(0)
       mesh = mp.solutions.face_mesh.FaceMesh(
           min_detection_confidence=0.5,
           min_tracking_confidence=0.5,
       )
       try:
           while cap.isOpened():
               ok, frame = cap.read()
               if not ok:
                   continue
               result = mesh.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
               if not result.multi_face_landmarks:
                   continue
   
               lm     = result.multi_face_landmarks[0].landmark
               height = frame.shape[0]
               gap_px = int((lm[LOWER_LIP_IDX].y - lm[UPPER_LIP_IDX].y) * height)
   
               if gap_px > OPEN_THRESHOLD_PX:
                   on_open(frame)           # hand the frame to the acquisition pipeline
       finally:
           cap.release()
           mesh.close()

   Takeaway

   Two landmarks and a threshold replace an entire 'press to capture' UX. The CV runs in a tight loop; the rest of the pipeline stays frame-agnostic.

2. 02

   Step 2 of 3

   Bounding hardware calls with a timeout decorator

   OraScan_Automated_Scanning/motor_driver.py

   GPIO and servo drivers hang. Not crash — hang, indefinitely, when the bus misbehaves or a pin is stuck. An ordinary Python call into a blocked driver freezes the scanner UI and the operator has to hard-reboot. A one-line decorator submits every hardware call into a thread, waits up to N seconds, and returns None on timeout, so the UI always stays responsive.

   pythonCopy

   def hardware_timeout(seconds: int = 5):
       """Wrap a hardware call so a hung device can never block the UI."""
       def decorator(fn):
           @functools.wraps(fn)
           def wrapper(*args, **kwargs):
               with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
                   future = executor.submit(fn, *args, **kwargs)
                   try:
                       return future.result(timeout=seconds)
                   except concurrent.futures.TimeoutError:
                       logging.warning(
                           "hardware: %s timed out after %ds", fn.__name__, seconds
                       )
                       return None
           return wrapper
       return decorator
   
   
   @hardware_timeout(seconds=3)
   def move_servo(axis: str, angle: int) -> None:
       """Move a servo to an absolute angle; blocks until the pulse is sent."""
       servo = _servos[axis]
       servo.move_to_angle(max(0, min(180, angle)))

   Takeaway

   Hardware is the one place you should never trust a library's own timeout handling — wrap every motor/sensor call in a bounded executor, treat None as 'try again', and the UI is freed from the physical layer.

3. 03

   Step 3 of 3

   Desktop ↔ cloud patient reconciliation

   OraScan_backend/sync_handler.go

   The desktop scanner stores every patient locally in MySQL so scans work offline. When the operator reconnects, each record is POSTed to this handler for reconciliation against the cloud store. The important design decision: we refuse to auto-create patients from sync data — sync is reconciliation, not a registration shortcut. Bypassing the signup flow would skip consent capture and identity checks.

   goCopy

   type SyncPatientInput struct {
       Email         string `json:"email" binding:"required"`
       FirstName     string `json:"first_name"`
       LastName      string `json:"last_name"`
       Contact       string `json:"contact"`
       Place         string `json:"place"`
       LocalID       int    `json:"local_id"`       // desktop app's row ID
       SyncTimestamp string `json:"sync_timestamp"` // ISO 8601
   }
   
   func (s *ApiServer) SyncPatient(c *gin.Context) {
       userID, ok := c.Get("userID")
       if !ok {
           c.JSON(http.StatusUnauthorized, gin.H{"error": "missing user context"})
           return
       }
   
       var in SyncPatientInput
       if err := c.ShouldBindJSON(&in); err != nil {
           c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
           return
       }
   
       existing, err := s.store.GetUserByEmail(in.Email)
       if err == nil && existing != nil {
           s.audit.SyncMerge(userID, in.Email, in.LocalID, existing.ID)
           c.JSON(http.StatusOK, gin.H{
               "status":   "merged",
               "cloud_id": existing.ID,
               "at":       time.Now().UTC().Format(time.RFC3339),
           })
           return
       }
   
       // Strict mode: never auto-create on sync. If the patient isn't
       // already in the cloud, the desktop operator must route them through
       // the normal registration flow first.
       c.JSON(http.StatusNotFound, gin.H{
           "error": "patient not found in cloud; please register first",
       })
   }

   Takeaway

   Offline-first apps have their own IDs; the cloud has its own. Sync handlers merge on a stable external key, audit every reconciliation, and reject rather than invent records that didn't come through the proper signup flow.

Results

The final model achieves 94.7% test accuracy and 95.2% best validation accuracy across all 11 disease classes. ONNX INT8 quantisation reduces model size by ~70% while keeping accuracy drop under 1%. Inference latency on the kiosk CPU is under 200ms per frame. The model is integrated into both the automated scanning application (MediaPipe FaceMesh-triggered) and the manual scanning desktop interface.

Gallery & Demos

Click any image or video to expand · ← → keys navigate