Desert Ant Labs Who and Eye: Fast, Private AI for Audio and Image Analysis
Sending a meeting recording or a folder of personal photos to a cloud AI service is not always necessary. Some tasks, such as identifying when different people speak or sorting images by content, can be handled by smaller models running directly on your device. That can reduce network delays, avoid per-request API charges, and keep sensitive media under your control.
Desert Ant Labs takes this specialized approach with Who, a speaker diarization model, and Eye, an image classification and photo organization model. Who identifies which voice is speaking at each point in a recording, while Eye analyzes images for categories, visual quality, faces, and similar shots. Neither needs to send the media to a remote inference service when running locally.
The interesting question is not simply whether these models are small. It is how much useful work each can do, how their reported performance compares with alternatives, and where their limitations matter in real applications.
Why specialized on-device models matter
Cloud-based AI is convenient because the server handles model execution, but it also introduces a network dependency. Audio and images must travel to the service, inference may incur usage charges, and the application must account for the provider's data-handling practices.
An on-device model changes that arrangement. Once its files and runtime are available locally, inference can happen on the user's computer or in a supported browser without uploading the input to a model API. This can be useful for private recordings, photo libraries, and features that should continue working without a reliable connection.
There are still trade-offs. A model must fit within the device's memory and compute limits, initial downloads can take time, and local processing consumes battery and hardware resources. Keeping inference local also does not automatically make the entire application private: analytics, synchronization, backups, and license checks have separate data flows.
Who and Eye illustrate two different uses for this approach. One turns an audio recording into a timeline of speakers. The other turns a collection of images into searchable categories and quality signals.
Who: speaker diarization without transcription
Speaker diarization answers a specific question: who spoke, and when? It splits an audio recording into time ranges and assigns a consistent speaker label to each detected voice. This is different from speech recognition, which converts spoken words into text.
A podcast with three participants, for example, might produce a simplified timeline like this:
The labels identify recurring voices, not real-world identities. Speaker 1 might be the host, but the model does not know their name. Likewise, the example contains no transcript of what anyone said.
This distinction matters because diarization and transcription solve complementary problems. A transcription system can produce words without reliably assigning each word to a person; diarization can assign speaking intervals without knowing the words. Combining the two supports speaker-labeled transcripts.
How Who represents overlapping speech
People frequently interrupt one another, especially in meetings and interviews. A diarization model must account for intervals when more than one person speaks rather than forcing every moment into a single-speaker track.
Who is described as producing speaker intervals with 10-millisecond timestamp granularity and supporting overlapping speech. Timestamp granularity describes how finely the output can mark boundaries; it does not mean every detected boundary is accurate to within 10 milliseconds.
Separate speaker tracks make an important difference in applications such as podcast editing and meeting analysis. When two people talk at once, both intervals can be represented, making the resulting timeline more faithful to the conversation.
Speed and diarization accuracy
Desert Ant Labs reports that Who can process an hour of audio in approximately four seconds on a MacBook Pro with an M5 Pro chip, a claimed speed of roughly 950 times real time. The source material also describes a 57-minute podcast processed in 6.4 seconds, including MP3 decoding. These are reported demonstrations, not independent benchmarks, and performance will depend on hardware, recording characteristics, and processing overhead.
Speed alone is not enough. The usual quality metric is diarization error rate (DER), which captures missed speech, incorrectly detected speech, and speaker assignment mistakes. Lower scores are better.
In the provided AMI meetings comparison, Who reports a 12.6% DER versus 13.1% for pyannote.audio. That is a small advantage on this dataset, not evidence that Who is more accurate for every recording. The source also notes cases, including recordings with many speakers, where alternatives perform better.
Who is described as particularly well suited to close-miked recordings, where participants have clear, relatively isolated voices. Room echoes, background noise, frequent interruptions, and many similar-sounding speakers can make diarization harder.
| Measure | Who | Comparison or context |
|---|---|---|
| Main output | Speaker labels and time intervals | Not speech-to-text |
| Timestamp granularity | 10 ms | Not a guarantee of boundary accuracy |
| AMI meetings DER | 12.6% | pyannote.audio: 13.1%, as reported in the source |
| Reported speed | About 950x real time | M5 Pro test claimed by the developer |
| Best fit | Meetings, podcasts, and interviews | Especially clear, close-miked speech |
Combining diarization with transcription
A speaker-aware transcript illustrates how a focused model can contribute to a larger application without performing every task itself. The example application, Who Said What, processes a recording using Who for speaker labels and Apple's on-device speech transcription for spoken text.
Those two results are aligned into a transcript with timestamps, speaker labels, and an interactive playback timeline. The application also permits generic labels such as Speaker 2 to be renamed manually. This naming step is significant: assigning an identity is separate from detecting that the same voice has returned.
The source describes a nearly hour-long recording for which diarization ran at around 393 times real time and speech transcription at about 48 times real time. Because the two processes operate in parallel, the slower transcription task largely determines the overall completion time. The full pipeline was reported at approximately 48.2 times real time.
This combination is useful for searchable interviews, meeting records, and editing tools. It also shows the boundary of Who's role: the model supplies speaker timing, while another system supplies the words.
Eye: image classification and photo organization on-device
Eye applies the same focused-model approach to visual data. Instead of producing open-ended descriptions of an image, it is designed to provide structured signals that an application can use to sort and filter photo collections.
The source describes a single analysis interface, eye.analyze(), returning information such as image type, content tags, aesthetic scoring, and face-related attributes. A photo manager can use these signals to group screenshots separately from camera photos or identify likely duplicates without sending the images to a hosted AI endpoint.
What Eye can recognize
Eye's reported capabilities include several related but distinct tasks:
- Image type detection: distinguishes images such as photographs, screenshots, receipts, and documents.
- Scene and object tags: assigns labels drawn from a reported vocabulary of 558 categories, including examples such as
beach,dog, andfood. - Aesthetic scoring: estimates visual appeal to help rank candidate photos.
- Face attributes: detects faces and signals such as
smilingandeyes closed. - Near-duplicate grouping: identifies similar images, including burst sequences, so an application can propose a preferred shot.
These outputs are best understood as model predictions rather than definitive descriptions. An aesthetic score is subjective, an object tag may be wrong, and a recommended photo from a burst should remain reviewable before anything is deleted.
Confidence scores and image sorting
The source describes confidence values from 0 to 1 for generated tags, with 0.5 suggested as a positive-match threshold. A value above the threshold can make a tag eligible for automatic grouping, but confidence is not the same as a guaranteed probability of correctness. Thresholds should be evaluated against the kinds of images an application actually handles.
For instance, a photo library might place high-confidence screenshots in a separate collection but leave uncertain documents in a general review queue. Similarly, detecting closed eyes may help surface better photos from a burst, though it should not automatically decide which memories a user wants to keep.
The source cites 98% accuracy for screenshot detection and 100% for receipt detection. Without the underlying test-set composition, these figures should be treated as reported benchmark results rather than guarantees across arbitrary photo libraries.
Performance in native apps and browsers
Eye is described as running on Apple's Neural Engine and in browsers through Google's LiteRT.js runtime with WebGPU support. The source reports 5.5 milliseconds per photo on an M6 Mac Neural Engine and about 41 milliseconds per photo in a WebGPU-enabled browser. These figures refer to model processing in the cited environments; total user-perceived time may also include decoding, model initialization, rendering, and file access.
The browser example, called Eye in the browser, classifies a mixed folder of 105 images into groups such as receipts, screenshots, documents, pets, people, duplicates, and preferred shots. Its reported end-to-end completion time is 16.1 seconds. That result is not directly comparable to single-image inference latency because the complete application performs additional work.
Browser inference can be appealing when images should stay on the user's machine. The browser still needs to acquire the runtime and model files, and support for WebGPU varies by device and environment. Keeping photo inference local also does not, by itself, establish what other data the surrounding web application collects.
| Capability | Reported behavior | Practical consideration |
|---|---|---|
| Image classification | Types, scenes, and objects | Incorrect tags need review |
| Aesthetic scoring | Ranks candidate photos | Scores are subjective |
| Burst grouping | Identifies similar photos | Selection should not imply automatic deletion |
| Native inference | 5.5 ms per image | Reported M6 Neural Engine result |
| Browser inference | 41 ms per image | Reported WebGPU result |
| Folder example | 105 images in 16.1 seconds | Includes work beyond a single inference call |
Where Who and Eye fit together
Who and Eye do not compete with each other. They address separate media types, but share an architectural idea: use a small model for a defined prediction task and let the application handle the rest.
A media-management product could use Who to create speaker timelines for recordings and Eye to organize still images. Neither model needs to generate an essay or reason across an entire software project. That narrow scope is the reason specialized on-device inference can be practical.
The clearest distinction is what each model does not provide. Who does not transcribe speech or identify speakers by name. Eye does not replace a complete photo editor or guarantee the quality of its classifications. Both are components that can support larger workflows.
| Model | Input | Main output | Example use |
|---|---|---|---|
| Who | Audio | Timestamped speaker segments | Speaker-labeled interview transcripts |
| Eye | Images | Tags, scores, and grouping signals | Private photo sorting and search |
Licensing and limitations
The provided source describes Who and Eye as beta releases distributed under the Desert Ant Labs Source-Available License, not an open-source license. It states that free use covers applications with up to 100,000 monthly active devices per platform and that larger deployments require commercial licensing. It also describes an anonymous monthly SDK license check that does not include user media, inputs, or outputs.
These licensing terms, supported platforms, model availability, and beta status should be checked against Desert Ant Labs' current official documentation before a production release. Source-available code is not equivalent to permission for unrestricted modification, distribution, or commercial use.
There are also product-level limitations to consider. On-device inference transfers computational work to user hardware, so performance and battery use will vary. Model accuracy can shift with recording conditions or image types. Applications handling sensitive media still need appropriate safeguards for local storage, exported files, and any optional synchronization.
Final thoughts
Who and Eye show how useful AI features can come from narrow, local models rather than a single large cloud model. Who provides speaker timing for audio, while Eye supplies structured signals for organizing photographs. Their value is in doing those tasks efficiently enough to become part of everyday applications.
The most important difference is in what they make possible. Who can support speaker-aware transcripts without sending recordings to an inference API, provided it is paired with speech recognition. Eye can support image sorting and quality filtering directly on a device or in a compatible browser, while leaving the final decisions about tags and duplicate photos to the application and its user.
The reported speed and accuracy figures are promising, but they are not universal performance guarantees. Hardware, input quality, integration overhead, and licensing all affect whether these models are a good fit. For products that need focused media analysis with greater control over where data is processed, the approach is worth examining.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.