Deploying a trained canine pose estimation model to the cloud is straightforward. Getting that same model to run at 15-30 frames per second on a handler's phone, outdoors, in variable lighting, while providing sub-second coaching cues is a genuinely hard engineering problem. At ServiceDog.AI, our applied research into edge inference for service dog assessment has surfaced performance patterns, runtime quirks and quantization pitfalls that are not well documented in mainstream mobile ML literature. This article consolidates what we have learned about edge inference for real-time handler feedback across CoreML, TensorFlow Lite and ONNX Runtime deployments on current iPhone and Android hardware.
Why Edge Inference Changes Handler Coaching
Cloud-based inference introduces round-trip latency that destroys the value of real-time coaching. A handler and service dog executing a "forward" command in a crowded environment cannot wait 400-800 milliseconds for a server response before receiving corrective feedback. By the moment feedback arrives, the behavioral window has closed.
Edge inference eliminates that latency floor. When a pose estimation model runs on-device, the inference-to-feedback cycle can drop below 80 milliseconds, which is within the neurologically meaningful window for associative learning reinforcement. That gap matters enormously during early-stage task training.
Privacy is the second driver. Handlers with disabilities have a reasonable expectation that video of their daily routines and medical support workflows is not transmitted to third-party servers. On-device inference means frames never leave the phone. This aligns with the disability community's broader concerns about surveillance and data sovereignty, concerns that IAADP has raised in the context of digital verification systems.
Offline capability is the third factor. Service dogs work in subways, rural hospitals, wilderness environments and buildings with Faraday-cage-level signal attenuation. An edge-first architecture means the coaching system works regardless of connectivity.
Model Requirements for Canine Pose and Behavior Assessment
Before choosing a runtime, the model architecture dictates what is even feasible on device. Canine pose estimation presents specific challenges that human pose models do not face: quadruped topology, high occlusion during task execution and the need to track both handler and dog simultaneously within a single frame.
Most production-viable canine keypoint models in 2026 use lightweight backbone architectures derived from MobileNetV3 or EfficientDet-Lite, with heatmap-based keypoint heads rather than regression-based heads. Heatmap approaches generalize better across dog breeds and sizes. Regression heads can overfit to the training distribution, which in practice means they perform well on Labrador Retrievers and collapse on Borzoi or Chihuahua body plans.
The canine skeleton graph used internally at ServiceDog.AI tracks 18 keypoints covering nose, occiput, withers, spine, hip, shoulder, elbow, carpus, stifle, hock and tail base. This graph is sufficient to derive:
- Gait symmetry index across all four limbs
- Head position relative to handler (heel work compliance)
- Sit and down posture classification (task execution verification)
- Stress signal detection via tail carriage angle and ear vector
Behavior classification runs as a second-stage temporal model, typically a lightweight 1D convolutional classifier or GRU operating over a 2-second sliding window of keypoint velocities. This two-stage approach keeps the real-time pose backbone under 4MB, which is the practical ceiling for models targeting consistent 30fps on mid-range Android hardware.
CoreML, TFLite and ONNX Runtime: Platform Trade-offs
Each inference runtime has a distinct execution model, hardware delegation strategy and operator coverage profile. Choosing the wrong runtime for a given model architecture is a common source of latency regressions that look like model problems but are actually deployment problems.
CoreML on Apple Silicon
CoreML 8 (shipping with iOS 18 in 2026) provides the most seamless path to Neural Engine acceleration on iPhone 15 and later hardware. The Neural Engine on A17 Pro and M-series chips delivers exceptional throughput for convolutional operations, achieving sub-20ms inference on MobileNetV3-based keypoint models at 256x256 input resolution.
The CoreML constraint that engineers consistently underestimate is operator coverage. Custom ops, especially those introduced by newer ONNX opsets, frequently fall back to CPU execution. A model with even one unsupported op in a critical path can see a 3-5x latency increase as the executor breaks the graph across Neural Engine and CPU execution contexts. Profile with the CoreML Performance Report in Xcode before committing to any architecture.
CoreML also enforces strict model input/output typing. Dynamic input shapes, which are common in pose models that accept variable-resolution crops, require explicit multi-shape specification at conversion time using the coremltools.EnumeratedShapes API. Skipping this step produces models that silently fall back to CPU for any input shape not seen during conversion.
TensorFlow Lite on Android
TFLite remains the dominant runtime for Android deployment in 2026. The GPU delegate via OpenCL provides strong performance on Snapdragon 8 Gen 3 and Dimensity 9300 chipsets. The NNAPI delegate remains inconsistently implemented across OEM firmware stacks. In production deployments tested by the ServiceDog.AI engineering team, NNAPI delegation improved latency on Pixel 8 but introduced a 40% latency regression on certain Samsung Galaxy S24 configurations due to firmware-specific NNAPI implementations.
The practical recommendation for Android is to attempt GPU delegate first, fall back to NNAPI only on verified Pixel and reference hardware, and default to CPU with 4-thread execution on unknown OEM configurations. Latency variance across Android fragmentation is the single largest reliability challenge in TFLite deployment.
ONNX Runtime Mobile
ONNX Runtime Mobile (ORT Mobile) provides a cross-platform path that targets both iOS and Android from a single model file. The execution provider ecosystem in ORT supports CoreML on iOS and NNAPI/GPU on Android through a unified API surface. For teams maintaining a single codebase, ORT reduces the per-platform engineering overhead significantly.
The trade-off is peak performance. ORT Mobile consistently runs 15-25% slower than a natively converted CoreML model on Apple Silicon, because the ORT CoreML execution provider adds a graph partitioning and session initialization overhead not present in direct CoreML deployment. For a 30fps target, that margin often matters.
Real-World Performance Benchmarks on iPhone and Android
The following benchmarks were collected by the ServiceDog.AI applied research team using our 18-keypoint canine pose model (MobileNetV3-Small backbone, 256x192 input, INT8 quantized) and a behavior classifier GRU (64 hidden units, 2-second window at 15fps keypoint rate). All measurements reflect end-to-end inference time from frame capture to keypoint output, excluding UI rendering.
iPhone 15 Pro (A17 Pro), CoreML 8, Neural Engine delegation:
Pose model: 14ms average, 19ms 95th percentile
Behavior classifier: 3ms average
Combined pipeline: 18ms average
iPhone 13 (A15 Bionic), CoreML 8, Neural Engine delegation:
Pose model: 21ms average, 28ms 95th percentile
Combined pipeline: 26ms average
Pixel 8 Pro (Tensor G3), TFLite GPU delegate:
Pose model: 23ms average, 31ms 95th percentile
Combined pipeline: 28ms average
Samsung Galaxy S24 (Snapdragon 8 Gen 3), TFLite GPU delegate:
Pose model: 19ms average, 25ms 95th percentile
Combined pipeline: 24ms average
Mid-range Android (Snapdragon 695), TFLite CPU 4 threads:
Pose model: 67ms average, 89ms 95th percentile
Combined pipeline: 72ms average
The mid-range Android figure highlights the accessibility challenge. A 72ms combined pipeline still supports real-time feedback at 13fps, which is above the threshold for meaningful coaching cues. Dropping to 8fps, which occurs on sub-695 chipsets, produces visible discontinuities in keypoint tracking that degrade feedback quality.
Quantization and Pruning Strategies for Canine Keypoint Models
INT8 post-training quantization (PTQ) is the baseline compression step for any edge deployment. For canine pose models, the mean per-joint position error (MPJPE) degradation from FP32 to INT8 is typically 1.8-2.4 pixels at 256x192 resolution when using a representative dataset calibration set of at least 500 diverse canine images. That error delta is operationally acceptable for behavior classification, which operates on normalized keypoint velocities rather than absolute pixel positions.
Quantization-aware training (QAT) recovers most of that error. In internal experiments, QAT reduced the MPJPE delta to under 0.8 pixels relative to the FP32 baseline, while maintaining the INT8 inference speed advantage. QAT requires access to the training pipeline and typically adds 20-30 additional training epochs, but the accuracy recovery is consistently worth the cost for production deployments.
Structured pruning applied to the convolutional backbone before quantization can reduce model size by an additional 30-40% without meaningful accuracy loss, provided the pruning ratio stays below 40% of channels per layer. Beyond that threshold, the heatmap quality degrades noticeably on small dogs and in high-occlusion scenarios. Channel pruning using L1-norm magnitude criteria tends to outperform random structured pruning for pose estimation backbones specifically.
Knowledge distillation from a larger teacher model (EfficientDet-D1 or ViTPose-Small) to the mobile student consistently produces better INT8 keypoint accuracy than PTQ alone on the same student architecture. If the training pipeline supports it, distillation-then-QAT is the highest-performing compression path available in 2026 tooling.
Integrating Edge Models into Handler Feedback Loops
A low-latency inference pipeline is necessary but not sufficient for effective handler coaching. The feedback modality and timing logic require as much engineering attention as the model itself.
Haptic feedback on mobile devices provides the most non-intrusive coaching channel in public environments. A brief 30ms haptic pulse delivered within 100ms of a detected task-execution failure gives the handler a private, socially neutral cue that does not draw attention in a restaurant or transit environment. Audio cues via earpiece remain the most information-dense channel for complex corrections, but they require the handler to divide attention between the auditory cue and the environment.
The coaching logic layer above the behavior classifier should implement a hysteresis filter to avoid feedback chatter. Triggering a coaching event requires the behavior classifier confidence to exceed a threshold for at least 3 consecutive frames (at 15fps, roughly 200ms of continuous detection). Single-frame anomalies from motion blur or occlusion should not trigger coaching events. The cost of false-positive coaching, which undermines handler confidence and can confuse the dog through inadvertent cue timing, exceeds the cost of occasional false negatives.
The TheraPetic® Training Plus program uses a structured session logging schema that pairs timestamped behavior classifications with handler-confirmed task annotations. This creates a labeled dataset of real-world handler-dog interactions that closes the training data loop. Edge-collected session data, with appropriate handler consent, becomes the calibration corpus for next-model-generation training.
Aligning Real-Time Feedback with Public Access Test Standards
The technical capability to detect behavioral anomalies in real time only creates value if the detection taxonomy aligns with what actually matters in service dog deployment. The Public Access Test (PAT), as defined by Assistance Dogs International, provides the most widely accepted behavioral competency framework for evaluating service dogs in community settings.
PAT assessment criteria that are directly addressable via edge pose and behavior models include:
- Controlled entry through automatic and manual doors (gait continuity under environmental distraction)
- Heel position maintenance at handler left or right (head-to-hip vector relative to handler pelvis keypoints)
- Sit and down on command without repeated cuing (task completion latency measurement)
- Calm response to crowd, noise and surface variation (stress signal absence via tail and ear keypoints)
- Ignoring food and neutral objects on floor (head orientation tracking relative to floor plane)
PAT criteria that are not addressable via current edge models include behavioral responses requiring smell or tactile assessment, and veterinary health evaluations. The ADA two-question rule framework discussed on ADA.gov applies to business interactions rather than training assessment, but understanding its constraints matters when designing verification products in this space. The broader TheraPetic®.AI clinical intelligence platform integrates the behavioral data generated by edge models with handler health documentation workflows where clinically appropriate.
Real-time PAT-aligned feedback turns every outing into a training and documentation event. A handler walking their service dog through a grocery store can receive session-level summaries indicating which PAT criteria were demonstrated successfully and which showed anomalies warranting attention. Over time, this produces longitudinal behavioral profiles that have genuine evidentiary value for training certification and, in some jurisdictions, dispute resolution under the ADA Title II and Title III access rights framework.
The engineering challenge of edge inference is real. The performance targets are achievable on current hardware. The gap that remains in 2026 is not compute capacity but training data quality, specifically diverse multi-breed canine keypoint datasets with verified behavioral labels in authentic public access environments. Closing that gap is the primary research priority for the ServiceDog.AI team and the prerequisite for any deployment that genuinely serves handlers with disabilities rather than approximating their needs.