A professional service dog trainer stands in a crowded shopping mall. Her client, a veteran with PTSD, is working a two-year-old Belgian Malinois through a distraction-heavy environment. She needs feedback on the dog's heel position, task readiness posture and handler synchrony right now, not after she uploads video to a desktop application tonight. This is the problem edge inference solves. Running pose estimation and behavior classification models directly on a smartphone or wearable device, with no round-trip to the cloud, delivers the sub-200ms latency window that makes real-time coaching possible during a live public access evaluation.
At ServiceDog.AI, our engineering team has spent the past three years deploying canine pose and behavior models across iOS and Android hardware. This article documents what we have learned about runtime selection, model architecture trade-offs and the UX decisions that determine whether a handler actually uses the feedback or ignores it.
Why Edge Inference Changes the Training Loop
Traditional video-based assessment has a built-in delay. A trainer records a session, reviews footage, annotates behavior events and delivers written or verbal feedback at a later date. That delay is the enemy of operant conditioning. Dogs learn fastest when the consequence follows the behavior within milliseconds. Handlers learn fastest when feedback is equally immediate.
Cloud-based inference closes part of that gap, but not enough. Even with a low-latency 5G connection, a round-trip to a GPU inference server adds 80 to 300 milliseconds of network overhead before the model computation even begins. In a crowded mall or a courthouse lobby, connectivity is unreliable. Data privacy regulations under state law increasingly restrict streaming identifiable video of individuals in public spaces to remote servers.
Edge inference eliminates all three problems. The model runs on the Neural Engine of an iPhone 15 Pro or the NPU of a Google Tensor chip. Latency drops to 15 to 60 milliseconds for a well-optimized MobileNet-based pose model. Connectivity becomes irrelevant. Video never leaves the device, which resolves most privacy concerns related to filming in public accommodation spaces governed by ADA Title III.
The training loop becomes genuinely closed. A handler gets a haptic pulse when heel position degrades. A trainer gets an on-screen alert when the dog's weight distribution signals pre-distraction arousal. That density of feedback, delivered in real time, compresses months of correction into weeks.
The Model Pipeline: From Pose to Behavior Classification
Effective real-time handler coaching requires a two-stage inference pipeline. The first stage is pose estimation. The second stage is behavior classification operating on the pose sequence over time.
Stage One: Canine Pose Estimation
Human pose estimation on mobile is a solved problem. Canine pose estimation is not. The skeletal topology differs substantially. Dogs are quadrupeds with a spine that flexes in three dimensions during gait. Standard human pose models trained on COCO keypoints do not transfer. Our team fine-tuned a MobileNetV3-based backbone on the Stanford Dogs dataset augmented with frame-level keypoint annotations covering 20 joint landmarks: nose, occiput, C7 vertebra, shoulders, elbows, carpi, forepaws, T10 vertebra, pelvis, hips, stifles, tarsi and hindpaws.
At inference time on an iPhone 15 Pro using CoreML with the Neural Engine backend, this model runs at 28 to 34 frames per second at 640x480 input resolution. That is enough to capture meaningful gait events including weight shift during a sit, paw placement during a heel transition and head carriage during approach to a distraction stimulus.
Stage Two: Temporal Behavior Classification
Raw pose keypoints are not behavior. A dog whose left forepaw is 14 centimeters outside the handler's left knee is out of position, but knowing that requires a spatial reference frame anchored to the handler's own pose. Our pipeline runs a lightweight MediaPipe-based human pose model in parallel to establish the handler skeleton. The relative geometry between handler and dog keypoints becomes the feature vector for a 1D temporal convolutional network that classifies behavior across a 2-second sliding window.
Behavior classes relevant to public access assessment include: heel-position compliant, heel-position degraded, sit on command, down on command, stay under distraction, forward momentum without handler cue and approach to food distraction. Each class has a confidence score that drives the feedback layer.
CoreML vs TFLite vs ONNX Runtime: What the Benchmarks Actually Show
Runtime selection is the single biggest performance lever available before you touch the model architecture. Our team ran controlled benchmarks across three runtimes on four devices in 2026: iPhone 15 Pro, iPhone 13, Google Pixel 8 Pro and Samsung Galaxy S24. All models were the same MobileNetV3 pose backbone at INT8 quantization. Input resolution was 416x416. Batch size was 1. We measured wall-clock latency from camera frame capture to keypoint output, averaged across 500 consecutive frames.
CoreML on iOS
CoreML with the Neural Engine delegate was the fastest option on Apple silicon across all iPhone variants tested. On the iPhone 15 Pro, mean latency was 18ms with a 95th-percentile of 24ms. On the iPhone 13, mean latency was 31ms with a 95th-percentile of 41ms. CoreML's advantage comes from the tight integration between the compiler, the ANE scheduler and the A-series chip's memory architecture. The trade-off is ecosystem lock-in: CoreML models are not portable to Android and require conversion via coremltools, which introduces a conversion validation step that occasionally surfaces operator support gaps for non-standard activations.
TensorFlow Lite on Android
TFLite with the GPU delegate on Pixel 8 Pro delivered mean latency of 22ms with a 95th-percentile of 29ms. On the Galaxy S24 using the NNAPI delegate targeting the Snapdragon NPU, mean latency was 19ms with a 95th-percentile of 26ms. TFLite's delegate architecture is more complex to configure than CoreML because the optimal delegate varies by chipset vendor. An application shipping to a broad Android audience must either implement delegate auto-selection or accept the lowest-common-denominator CPU fallback, which on the same Pixel 8 Pro produced mean latency of 78ms, still usable but noticeably less smooth.
ONNX Runtime on Both Platforms
ONNX Runtime with the CoreML execution provider on iOS produced mean latency of 23ms on iPhone 15 Pro, a 5ms penalty versus native CoreML. On Android with the NNAPI execution provider, ONNX Runtime matched TFLite within measurement noise: 21ms mean on Pixel 8 Pro. ONNX Runtime's compelling advantage is the single model format across platforms. A team maintaining one ONNX graph can target iOS, Android and Windows ARM tablets without per-platform conversion pipelines. For teams with limited ML engineering capacity, that operational simplicity often outweighs the small latency penalty.
Our recommendation for 2026 deployments: use CoreML natively on iOS if your team has iOS-first focus and can afford separate conversion pipelines. Use ONNX Runtime with platform-specific execution providers if you ship cross-platform and value a single model artifact.
Quantization and Pruning for Sub-100ms Latency
Even a well-chosen runtime will not save a poorly sized model. The canine pose pipeline plus the temporal behavior classifier must both complete within a 33ms frame budget at 30fps. That budget is tight when the device is also rendering UI, processing audio and managing Bluetooth for a connected haptic band.
INT8 post-training quantization is the minimum optimization step for any production deployment. It reduces model size by approximately 75 percent compared to FP32 and typically degrades keypoint localization accuracy by less than 2 percent on our internal validation set. We use TensorFlow's representative dataset quantization flow, feeding 200 representative frames of actual service dog footage to calibrate activation ranges. The resulting calibration avoids the accuracy cliff that appears when quantization is applied without a domain-representative calibration set.
Structured pruning at 30 to 40 percent sparsity applied during fine-tuning, before quantization, provides an additional 15 to 20 percent latency reduction with minimal accuracy impact. We use magnitude-based filter pruning on the depthwise separable convolutions in the MobileNetV3 backbone. The combination of structured pruning plus INT8 quantization brings the full two-stage pipeline to a combined 44ms mean latency on iPhone 15 Pro, leaving the remaining frame budget for UI rendering and feedback dispatch.
Dynamic shape inference is worth disabling for fixed-resolution inputs. Both CoreML and TFLite incur small but non-trivial overhead for shape dispatch when input dimensions are declared dynamic. Fixing the input to 416x416 at export time removes that overhead entirely.
Designing the Feedback Layer for Handlers in the Field
A model that achieves 28ms inference latency delivers no value if the handler cannot process the feedback signal. This is where most ML teams building for this use case fail. They produce a technically impressive pipeline and attach it to a UI that overwhelms the user at exactly the moment their cognitive load is highest, namely during a live distraction sequence in a public space.
Our feedback design follows three principles drawn from sports science research on real-time performance coaching.
First: feedback must be perceptible without redirecting attention. Visual overlays on a phone screen are appropriate for a trainer reviewing footage. They are inappropriate for a handler who must maintain situational awareness in a public space. Our primary feedback channel for handlers is haptic. A connected wristband delivers distinct vibration patterns: a single short pulse for minor position drift, a double pulse for significant deviation and a long continuous pulse for a flagged behavior event that requires handler response. The handler feels the feedback without looking away from the dog.
Second: feedback must be actionable, not diagnostic. Telling a handler "left forepaw is 18 centimeters lateral" requires them to perform geometry in real time. Telling a handler "tighten left" via a paired audio cue through a single earbud is actionable in under a second. Our text-to-speech layer converts classifier output to plain-language cues using a fixed vocabulary of fewer than 30 phrases.
Third: feedback frequency must be governed. A classifier that flags heel-position degradation at 30 frames per second will produce 1,800 feedback events per minute. That is sensory flooding, not coaching. Our feedback governor applies a minimum inter-event interval of 4 seconds per behavior class and a session-level fatigue model that increases the confidence threshold required to trigger feedback as session duration exceeds 45 minutes. These parameters are configurable by the trainer through the companion trainer dashboard.
Applying Edge Inference to Public Access Test Assessment
The Public Access Test, as defined by Assistance Dogs International and widely referenced in professional training programs including the TheraPetic® Training Plus program available through officialservicedog.com, covers a standardized set of behaviors in real-world environments: building entry, elevator navigation, crowd work, distraction response and restaurant seating. These behaviors map cleanly to the behavior classes in our temporal classification model.
A PAT-aligned edge inference assessment session captures a structured 45-minute walk-through and produces an objective behavior log timestamped to video clips. Each behavior class receives a pass rate percentage across observed opportunities. A heel-position compliance rate below 85 percent across 20 or more scored intervals flags the skill for additional training focus. A distraction approach event that results in forward momentum without handler cue is flagged as a PAT failure event with the corresponding video clip attached.
This objective documentation is valuable for trainers working with programs that require graduation assessments. It is also valuable for handlers preparing their own documentation. Under current federal law governing service dogs in public accommodations, staff at a business may ask only two questions: whether the dog is a service dog required for a disability and what work or task the dog is trained to perform. An edge inference assessment log does not substitute for those legal standards, but it provides trainers with the objective training record that supports responsible certification practices. For verification-related questions, officialserviceanimal.com provides current guidance on federal standards.
The computer vision assessment does not make legal determinations. It makes training determinations. That distinction matters and is built into every piece of handler-facing language in the ServiceDog.AI application.
Deployment Considerations for ADA-Aligned Applications
Deploying an AI assessment tool in public spaces used by people with disabilities requires more than technical correctness. It requires awareness of the legal and ethical context in which the tool operates.
Video capture in public accommodations is legally permissible in most U.S. jurisdictions when performed by the handler or their trainer for personal training purposes. Streaming that video to a server changes the privacy calculus. On-device processing under the edge inference architecture keeps the data local and reduces legal exposure substantially.
Model bias toward specific dog breeds, coat colors or handler body types is a real engineering risk. Our validation set for the canine pose model covers 47 breeds across the working dog, sporting and herding groups. We specifically over-sampled dark-coated dogs, which are systematically harder for RGB-based pose models due to reduced contrast at joint landmarks. Handlers of color were specifically included in the handler pose validation set to avoid the well-documented performance gaps in human pose models trained on non-diverse datasets.
Any assessment score produced by the model must be presented with appropriate uncertainty quantification. A confidence score below 0.65 triggers a "low confidence, manual review recommended" flag rather than a definitive pass or fail. Trainers using the ServiceDog.AI platform agree in their terms of service that all AI-generated assessment outputs are tools to support professional judgment, not replacements for it.
The clinical and training science underlying the behavior classification model was developed in collaboration with the team at TheraPetic®.AI, whose Licensed Clinical Doctors and veterinary behavioral consultants reviewed the behavior taxonomy and validated the operationalization of each PAT behavior class. That cross-disciplinary review is what separates a technically impressive demo from a tool that can responsibly be deployed in real training contexts.
Edge inference is not a future capability. It is a deployable technology available today on hardware that most professional service dog trainers already carry. The engineering challenge in 2026 is not making it work. It is making it work well enough, and responsibly enough, to earn the trust of handlers whose independence depends on the dogs these tools help train.