Abstract
A voice app without a cloud in the middle.
Most intelligent recorders are organized around a remote service: capture here, upload there, compute somewhere else, then return to an account-bound library. Harbor is organized around a different object—the iPhone itself. The phone captures the sound, performs speech recognition, separates speakers, generates useful notes, continues eligible work in the background, and writes the result into an ordinary Apple Files folder.
This is not privacy added after the product was designed. It is the consequence of choosing Apple’s on-device frameworks as the product architecture. Harbor has no developer-operated account, ingestion server, transcription endpoint, or AI minute meter. The user’s device is the compute boundary; recording.json is the canonical state; the selected Files location is the library.
Harbor treats the iPhone not as a microphone for somebody else’s cloud, but as a complete personal computing system.
01 · Architecture
One device, one coherent stack.
Harbor’s pipeline is intentionally vertical. Each layer hands work to another capability already native to iOS, so audio does not need to cross a developer-controlled network boundary to become useful.
SwiftUI
Recording, library, transcript correction, speaker naming, maps, settings, and export.
App Intents · WidgetKit · ActivityKit
Siri and Shortcuts entry points, widgets, Lock Screen status, and Dynamic Island controls.
AVFoundation · AVAudioEngine
Microphone sessions, one-tap M4A recording, waveform data, and optional voice processing.
Speech · Core ML · Foundation Models
Time-indexed transcription, speaker diarization, summaries, titles, and evidence-grounded suggestions.
Background Tasks
Visible, cancellable continued processing with foreground recovery when scheduling is unavailable.
Files · iCloud Drive · FileManager
Security-scoped folder access, one folder per recording, transactional writes, and optional Apple-managed sync.
The file outlives the interface.
Harbor can improve, move, or disappear; the user’s audio and structured data remain inspectable files.
Record
AVAudioEngine writes audio and feeds live analysis.
Transcribe
SpeechAnalyzer produces time-indexed text.
Separate
Core ML diarization finds speaker turns.
Shape
Foundation Models drafts title and notes.
Keep
Files receives canonical JSON and exports.
02 · Capture
AVFoundation begins the chain.
Harbor’s recorder is built around AVAudioSession and AVAudioEngine. A single input tap writes high-quality mono AAC audio to M4A while emitting the same captured buffers to the live-analysis pipeline. That shared origin matters: the waveform, provisional transcript, and durable recording all describe the same physical capture session.
The audio session uses Apple’s recording system rather than inventing its own device layer. It can accommodate built-in microphones and Bluetooth hands-free routes. Harbor’s Live Focus mode enables Apple voice processing when compatible hardware and routes make it available, reducing noise and echo before the sound enters later stages.
Harbor also preserves the original recording when optional enhancement is enabled. Enhancement runs through an offline AVAudioEngine graph and Apple audio units, producing a separate copy. The original remains the immutable point of reference.
03 · Speech and speakers
Language becomes a timeline—on device.
On iOS 26, Harbor uses Apple’s SpeechAnalyzer and SpeechTranscriber APIs. Live recording uses time-indexed progressive transcription; completed audio uses time-indexed transcription with alternatives. Harbor asks AssetInventory for the Apple-managed language assets needed for the current locale and installs them on first use when necessary.
Audio rarely arrives in exactly the format a speech model prefers. Harbor asks SpeechAnalyzer for the best available format, then uses AVAudioConverter to transform captured buffers before sending them through an asynchronous analyzer stream. Longer recordings are processed in bounded chunks so memory use remains predictable.
Speaker separation through Core ML
Apple’s Speech framework identifies words and time ranges, but it does not provide the speaker diarization Harbor needs. For that layer, Harbor uses FluidAudio 0.15.5, an open-source diarization pipeline executed locally with Core ML. Audio is analyzed in overlapping windows, speaker identities are reconciled across boundaries, and the resulting turns are aligned with Apple’s time-indexed transcript.
This is an important architectural distinction: FluidAudio is not a remote AI service and user audio is not sent to its developer. It is a model pipeline running inside Harbor’s process, accelerated by the Apple compute stack available on the device.
04 · Foundation Models
Apple Intelligence, constrained by evidence.
Once Harbor has a transcript and speaker timeline, it uses Apple’s on-device Foundation Models framework. SystemLanguageModel and LanguageModelSession draft titles, summaries, action items, and speaker-name suggestions without routing the conversation to a third-party language-model API.
On-device generation is only useful if it remains subordinate to the recording. Harbor therefore treats model output as a proposal, not authority. Structured generation is guided into known fields; temperatures are kept deterministic where appropriate; suggested quotations and speaker evidence are checked against the transcript; invalid or unsupported output is rejected. When the system model is unavailable—because of device capability, Apple Intelligence configuration, language, or another system condition—Harbor falls back to deterministic local logic and keeps the recording usable.
- The transcript is evidence. Generated notes may summarize it, but cannot silently replace it.
- The user is the editor. Titles, speaker names, transcript sections, and summaries remain correctable.
- Availability is a state, not a crash. Model absence degrades a feature; it does not strand the recording.
05 · Continuity and resilience
Save early. Enrich later. Retry only what failed.
Speech and note generation can outlast the moment when a user leaves the app. Harbor integrates BGContinuedProcessingTask so eligible post-record work can continue with system-visible progress, a useful title and subtitle, cancellation, and expiration handling. Core processing is submitted with failure semantics when immediacy matters; enhancement can be queued. If the operating system cannot accept a continued-processing request, Harbor retains a foreground-owned path instead of pretending the work was scheduled.
Durability is staged. Live finalized transcript segments are checkpointed while recording. After capture ends, useful transcript data is promoted into canonical state before optional enrichment begins. If title generation fails, the transcript is not discarded. If diarization needs another pass, the audio is not re-recorded. Harbor retries the failed stage rather than collapsing the whole pipeline into one fragile transaction.
Artifact writes use staging files, backups, and recovery journals. A read problem from a Files provider does not immediately erase the last known library snapshot. These choices are less visible than a waveform, but they are the difference between an AI demo and a dependable personal archive.
06 · Files and ownership
The library is a folder, not an account.
On first launch, Harbor asks the user to choose a parent location with Apple’s Files picker. That can be On My iPhone or iCloud Drive. Harbor retains permission through a security-scoped bookmark and creates one self-contained folder for each recording.
recording.json is the source of truth for identity, title, dates, optional location, transcript sections, speaker suggestions, notes, processing status, and timeline data. Markdown, RTF, and DOCX are projections that can be regenerated. This prevents a polished export from becoming more authoritative than the structured record that produced it.
If the user chooses iCloud Drive, Apple handles account identity, transfer, synchronization, recovery, and storage. Harbor’s developer still does not receive the content. That is the Apple ecosystem at its best: a user can choose cloud continuity without turning Harbor into a cloud service.
07 · The Apple ecosystem
One app, present wherever the moment begins.
Harbor’s on-device design is not isolation. Apple’s frameworks let the same private workflow appear across the system without creating a parallel platform.
App Intents
A discoverable Start Recording intent makes capture available to Siri and Shortcuts while preserving Harbor’s own recording lifecycle.
WidgetKit
Accessory widgets make Harbor visible at a glance and provide a lightweight route into recording.
ActivityKit
A Live Activity carries recording status to the Lock Screen and Dynamic Island, including a direct App Intent control to stop and save.
Core Location + MapKit
With permission, Harbor can attach an approximate place to a recording and render the personal archive geographically.
Files + iCloud
The library participates in the same document system users already trust across iPhone, iPad, and Mac.
SwiftUI
The app, settings, editing surfaces, and system-adjacent experiences share a native language and accessibility model.
Apple’s stack lets Harbor be deeply integrated without becoming centrally hosted.
That is the core celebration here. Apple silicon makes serious local inference practical. Speech turns sound into indexed language. Core ML opens the device’s heterogeneous compute to specialized models. Foundation Models brings a system language model into the app without adding an AI account. ActivityKit and App Intents make the workflow part of iOS. Files and iCloud give the result somewhere durable to live. Each framework is useful alone; together, they make a different product category possible.
08 · Honest boundaries
On-device does not mean unconditional.
Harbor’s architecture removes a developer-operated cloud from the intelligence path, but it does not remove the realities of system software.
- Apple speech assets and FluidAudio model files may need to download before first use.
- Foundation Models availability depends on supported hardware, Apple Intelligence settings, language, region, and current system state.
- Background execution remains scheduled and governed by iOS; Harbor can request and recover, not overrule the operating system.
- Transcription, diarization, and generated notes are probabilistic. Harbor keeps them editable because no model is perfect.
- Choosing iCloud Drive asks Apple’s service to sync the files. Choosing On My iPhone keeps the library local to the device.
These are not exceptions to Harbor’s philosophy. They are why the app makes state visible, preserves source artifacts, validates model output, and gives the final word to the user.
References
Apple technologies in Harbor.
- AVAudioEngine and Using voice processing, Apple Developer Documentation.
- SpeechAnalyzer and Bring advanced speech-to-text to your app with SpeechAnalyzer, Apple.
- AssetInventory, Apple Developer Documentation.
- Core ML, Apple Developer Documentation.
- Foundation Models, SystemLanguageModel, and LanguageModelSession, Apple Developer Documentation.
- Performing long-running tasks on iOS and iPadOS, Apple Developer Documentation.
- Making actions and content discoverable and widely available, Apple Developer Documentation.
- Dynamic Island, Apple Developer Documentation.
- FileManager and UIDocumentPickerViewController, Apple Developer Documentation.
- FluidAudio, Fluid Inference, Apache License 2.0.