Why Edge Inference Changes Handler Coaching
Service dog handler coaching has historically depended on a trainer physically present in the environment. A qualified trainer watches the dog's gait, posture and task execution, then gives corrective feedback in the moment. That model works. It also costs hundreds of dollars per session and is unavailable to most handlers outside major metropolitan areas.
Edge inference breaks that constraint. When a pose estimation model runs directly on the handler's phone, feedback arrives in real time, in any location, without a network round trip to a cloud GPU. The camera sees the dog. The model runs locally. The handler gets a coaching cue in under 100 milliseconds. That is the core premise driving the computer vision work at ServiceDog.AI in 2026.
The shift matters for a second reason. Cloud-dependent systems introduce privacy exposure. Video of a handler and their service dog in a medical context carries sensitivity. Running inference on-device means that video never leaves the phone. That aligns with the values the disability community has consistently articulated: handlers should control their own data.
As a 501(c)(3) nonprofit healthcare provider, TheraPetic® Solutions Inc. has built its clinical AI stack with on-device inference as the architectural default, not an afterthought. The work described here reflects what our engineering team has validated across real handler-dog teams enrolled in the TheraPetic® Training Plus program through officialservicedog.com.
Pose and Behavior Model Architecture for the Field
A real-time coaching system requires at least two inference pipelines running in a coordinated sequence: canine pose estimation and behavior classification.
Canine Pose Estimation
Canine pose estimation is not trivially derived from human pose models. The skeletal topology differs significantly. Key landmark sets for dogs typically include 17 to 24 keypoints: snout, ears, occiput, withers, shoulder, elbow, carpus, metacarpus, hip, stifle, hock, metatarsus and tail base. Research published at CVPR and ICCV in recent years has produced animal-specific architectures that outperform adapted human pose models on this task.
The practical architecture for mobile deployment is a lightweight top-down pipeline. A fast detector (MobileNetV3 or EfficientDet-Lite) runs first to localize the dog in frame. A pose estimation head (a stripped version of HRNet or a distilled ViTPose variant) then processes the cropped region. This two-stage approach keeps the region-of-interest small and reduces the FLOPs required for the second stage substantially.
For gait analysis in a public access context, the system tracks keypoint velocity and symmetry across frames. A dog breaking heel position, drifting into an environmental distraction or altering gait before a task cue all produce detectable keypoint displacement patterns. Those patterns are what the coaching system surfaces.
Behavior Classification
On top of the pose stream, a lightweight temporal model classifies behavior sequences. A shallow LSTM or a 1D temporal convolutional network (TCN) operating on keypoint time series is sufficient for the classification granularity needed: sitting, heeling, breaking position, alerting, task initiation, task completion and distraction response. Video-level classifiers like SlowFast or TimeSformer are too heavy for mobile edge deployment without aggressive compression.
The two pipelines feed a rule engine that maps behavior states to coaching cues. The cue delivery mechanism is audio (a short spoken prompt through the phone speaker or earpiece) so the handler's eyes stay on the dog and the environment.
CoreML vs TFLite vs ONNX Runtime: What the Benchmarks Show
Choosing the right inference runtime is not an abstract engineering preference. On mobile hardware the runtime determines whether you hit the latency budget or miss it. The three dominant options for production mobile ML in 2026 are Apple CoreML, TensorFlow Lite and ONNX Runtime.
Apple CoreML on iPhone
CoreML is the native inference framework for Apple Silicon. On devices with an A16 or newer chip, CoreML routes operations through the Neural Engine (ANE), which delivers dramatically lower latency and power draw than CPU or GPU paths. For a quantized MobileNetV3 detector followed by a distilled HRNet-W18 pose model, our team measured end-to-end inference (detector plus pose head) in the 28 to 40 millisecond range on an iPhone 14 Pro with ANE routing enabled. Power draw during a 10-minute coaching session stayed under 6% battery on that device class.
The CoreML toolchain requires conversion from PyTorch via coremltools. The conversion process is generally stable for standard convolutional operations but requires attention when models include custom attention layers or non-standard activations. Dynamic input shapes need explicit enumeration at export time. ONNX-to-CoreML conversion adds an intermediate step but gives teams a single source model.
TensorFlow Lite on Android
TFLite with the GPU delegate is the standard path for Android deployment. On a Snapdragon 8 Gen 2 device with GPU delegate active, the same two-stage pipeline (INT8 quantized) ran in the 35 to 55 millisecond range in our validation testing. Enabling the NNAPI delegate on supported chipsets reduced this further on specific OEM hardware but introduced consistency variance across manufacturers that made it unreliable as a default.
The Hexagon DSP delegate on Qualcomm hardware is the best-performing path when targeting a narrower Android device set. For teams building a coaching app with a controlled device profile (a common choice for professional trainers issuing hardware to handlers), Hexagon delegation brings performance close to CoreML ANE levels.
TFLite's toolchain is mature. The TFLite Model Maker and LiteRT libraries handle quantization-aware training exports cleanly. Post-training integer quantization with a representative dataset calibration step is the minimum viable approach. Full INT8 quantization with quantization-aware training produces meaningfully better accuracy retention at the same latency.
ONNX Runtime on Both Platforms
ONNX Runtime (ORT) with the CoreML execution provider on iOS and the NNAPI or XNNPACK execution providers on Android offers a cross-platform deployment strategy from a single model artifact. In our benchmarks, ORT with CoreML EP on iPhone 14 Pro was within 8 to 12 milliseconds of native CoreML for the two-stage pipeline. On Android with XNNPACK, ORT landed in the 50 to 70 millisecond range on the Snapdragon 8 Gen 2 reference device.
The practical advantage of ORT is the unified training-to-deployment pipeline. A model trained in PyTorch, exported to ONNX, and optimized with ORT quantization tools can be deployed to both platforms without maintaining parallel conversion toolchains. For a small engineering team, that operational simplicity has real value even if it costs a few milliseconds of raw latency.
Latency Budgets and the 100ms Coaching Window
The 100 millisecond threshold is not arbitrary. Research in human-computer interaction has established 100ms as the boundary below which system responses feel instantaneous to a user. For a handler receiving a coaching cue about their dog's position, feedback arriving within one video frame (at 30fps, that is 33ms) to three frames (100ms) reads as concurrent with the behavior being corrected. Feedback arriving after 300ms starts to feel disconnected from the triggering event.
Breaking down the latency budget for a coaching event: camera frame capture and preprocessing consumes 5 to 10ms. The detector forward pass consumes 10 to 20ms. Region crop and resize adds 2 to 5ms. The pose estimation forward pass consumes 15 to 25ms. Behavior classification on the keypoint buffer adds 3 to 8ms. Rule engine evaluation and cue trigger adds under 2ms. That totals to a realistic 35 to 70ms end-to-end on current flagship hardware, comfortably within the coaching window.
Mid-range Android devices (Snapdragon 7 series, MediaTek Dimensity 7000 series) shift those numbers upward. Testing on a Dimensity 7020 device with CPU-only inference produced 140 to 200ms total latency, which exits the comfortable coaching window. The design implication is that GPU delegation is not optional for broad Android support. Apps must gracefully degrade the behavior classification stage (dropping temporal context from 30 frames to 10) to stay within budget on lower-end hardware.
Quantization, Pruning and Model Compression for Mobile Deployment
Getting a research-quality pose model from an ICCV paper into production on a phone requires systematic compression. The compression stack used at ServiceDog.AI follows a defined sequence.
Post-Training Quantization vs. Quantization-Aware Training
Post-training quantization (PTQ) converts a float32 model to INT8 after training. It is fast to apply but can introduce accuracy degradation on small, articulated targets like dog keypoints at distance. For canine pose estimation, we observed a mean per joint position error (MPJPE) increase of 3.2 to 6.8mm with naive PTQ on a distilled HRNet-W18 variant. Quantization-aware training (QAT) inserts fake quantization nodes during fine-tuning, allowing the model to adapt its weights to the quantization scheme. QAT reduced MPJPE degradation to under 1.5mm in our validation set, which is within acceptable clinical tolerance for coaching feedback purposes.
Structured Pruning
Channel-level structured pruning removes entire filter sets, which reduces model size and FLOPs in a way that maps directly to latency reduction on real hardware (unlike unstructured sparsity, which requires specialized hardware support to realize speed gains). Pruning the pose head to 60% of original channels with knowledge distillation from the full model produced a 1.4x latency reduction with 2.1mm additional MPJPE degradation. Combined with QAT, the compressed model ran at 18ms on the ANE path for the pose stage alone.
Model Size and App Bundle Considerations
Compressed models for the two-stage pipeline land between 4MB and 9MB total depending on architecture choices. That is acceptable for over-the-air app updates. Storing multiple model variants on-device (one optimized for high-end hardware, one degraded for low-end) adds 8 to 18MB to the app bundle. Given that behavior classification models are small (under 500KB for a 1D TCN operating on keypoint sequences), the bulk of the bundle is always in the pose estimation head.
Closing the Real-Time Feedback Loop for Handlers
The technical pipeline is only half of the product. Coaching feedback that is technically accurate but behaviorally incoherent fails the handler. The coaching loop design has to reflect how service dog training actually works.
Handlers enrolled in the TheraPetic® Training Plus program learn behavior chains, not isolated behaviors. A dog breaking heel to investigate a ground scent is not the same behavior failure as breaking heel to orient toward a person. The pose model and the behavior classifier can distinguish these if the rule engine is built with that granularity. A distraction-toward-person event triggers a different coaching cue than a ground-scent break.
Feedback modality matters too. Audio cues keep the handler's visual attention on the dog. Haptic feedback (a single vibration pattern on the phone) is less disruptive in quiet public environments where audio would be intrusive. The app layer should surface a configuration option for feedback modality, since handlers in different contexts (hospitals, courtrooms, libraries) have different ambient noise constraints.
Session logging is a natural output of the edge inference system. Each detected behavior event, with its timestamp and classification confidence, creates a session record. Over multiple sessions, the system builds a behavioral profile of the team. Trainers reviewing the profile through the officialservicedog.com portal can identify patterns that a single session review would miss: a dog consistently breaking position in the 40 to 90 second window after entering an indoor environment, for example. That is the kind of clinical insight that makes AI-assisted coaching additive to human trainer judgment rather than competitive with it.
ADA Public Access Assessment and Edge AI in 2026
Under current federal law, the ADA permits covered entities to ask only two questions when the presence of a service dog is not obvious: whether the dog is required due to a disability and what work or task the dog has been trained to perform. The law does not authorize video surveillance of service dogs by businesses as a condition of entry. That boundary matters when discussing edge AI in public access contexts.
Edge inference for handler coaching is handler-initiated and handler-controlled. The camera is the handler's own device. The processing is local. No third-party operator receives the video. This is architecturally distinct from a business deploying a fixed camera system to assess service dogs at entry points. The latter raises serious ADA compliance and civil rights concerns that the former does not.
The ML-assisted ADA compliance work at ServiceDog.AI focuses on the business-side verification problem from the opposite direction: helping businesses train staff on the two-question rule and on behavioral indicators that a dog is task-trained, using anonymized, consented demonstration footage. The edge coaching system sits on the handler side of that line, and it should stay there.
As AI capabilities in canine assessment mature, the disability advocacy community and the legal community will need to develop clear guidance on what AI-assisted assessment is permissible, by whom and under what consent frameworks. The International Association of Assistance Dog Partners (IAADP) and ADI (Assistance Dogs International) are the organizations best positioned to shape those standards. ServiceDog.AI will continue contributing technical grounding to those conversations.
The handler coaching application of edge inference is technically tractable today. The models exist. The runtimes are production-ready. The latency budgets are achievable on current mobile hardware. What remains is careful product design, rigorous clinical validation with real handler-dog teams and a commitment to keeping handler data under handler control. That combination is what responsible deployment looks like in 2026.
About the Author
Dr. Patrick Fisher, PhD, LPC, NCC leads clinical AI research at TheraPetic® Solutions Inc., publisher of ServiceDog.AI and TheraPetic®.AI. The engineering and clinical findings described in this article reflect work conducted with handler-dog teams through the TheraPetic® Training Plus program. Learn more at officialservicedog.com and therapetic.ai.