imlistening

I built imlistening because I think faster than I type, and I wanted a dictation tool that simply gets out of the way. It’s a sub-700 ms real-time AI dictation pipeline engineered as a tight, two-part system: a native macOS client written in Swift, backed by a local Python server powered by FastAPI. It delivers clean, highly polished text without ever sending a single byte of your data to the cloud.

---

How it works

The pipeline is designed to be frictionless, transitioning from raw audio to polished prose in four steps:

1. Record — You tap the mic. The app grabs raw Float32 PCM audio directly from your selected input via AVAudioEngine and streams it to the backend as binary WebSocket frames in real time. A second tap sends a "done" signal to gracefully end the stream.
2. Transcribe — The backend catches the raw audio buffer. Instead of relying on external dependencies like ffmpeg, it uses PyTorch interpolation to instantly convert the buffer into a 16 kHz mono numpy array. This is fed directly to a locally-cached OpenAI Whisper model. You dictate the speed vs. accuracy tradeoff by selecting anything from the tiny to large model.
3. Polish — This is where the magic happens. The raw transcript is piped into an Ollama-hosted LLM (I default to Gemma 3 4B) using a completely customizable system prompt. The model strips out filler words, fixes your grammar, and restructures your thoughts into clear, articulate text.
4. Deliver — As the backend works, it streams stage-progress events back over the WebSocket. The Swift client catches these messages via an asynchronous receive() loop safely running on the @MainActor, instantly updating the UI's live pipeline tracker before dropping the final text into a persistent JSON history.

---

Features

Minimal, fluid UI

I designed a two-pane SwiftUI window that feels alive but unobtrusive. It features an animated gradient background—drifting blue, purple, and teal blobs—paired with a frameless, transparent title bar where the macOS traffic lights float cleanly over the dark content.

Live pipeline tracker

You're never left guessing what the backend is doing. A three-step indicator (Record → Whisper → Polish) tracks the exact state of the system in real time. You watch each stage transition from pending, to a spinning progress wheel, right through to a satisfying green checkmark.

Instant transcription history

Every completed recording is saved to a local history.json file and surfaced immediately in the right-hand pane. Each card displays a relative timestamp and your polished text, complete with a one-click copy button.

Total prompt control

Want the LLM to write differently? You can rewrite the system instructions directly from the settings panel to change the tone, output format, or correction style. The app persists your prompt in UserDefaults and injects it right into the WebSocket session header.

Granular hardware selection

The app enumerates all of your Core Audio input devices, letting you pinpoint exactly which microphone to use. Once selected, the device is explicitly set on AVAudioEngine via AudioUnitSetProperty before the recording session even begins.

Hot-swappable models

Both the Whisper and LLM models can be swapped out at runtime. The backend dynamically fetches available models from Ollama's /models endpoint to populate the UI pickers. Better yet, Whisper models are kept in memory, so switching sizes means zero reload penalty when you speak again.

Uncompromisingly local

Privacy is the default. No audio ever leaves your machine beyond the localhost WebSocket. Whisper runs entirely on-device utilizing your CPU or Apple Silicon (MPS), and Ollama hosts the LLM locally. There are no accounts, no subscriptions, and absolutely no API keys to manage.

---

Architecture under the hood

macOS client (Swift)
  │  AVAudioEngine tap → Float32 PCM
  │  URLSessionWebSocketTask (binary frames)
  ▼
ws://localhost:8000/ws/audio
  │
  ▼
FastAPI backend (Python)
  │  PCM → numpy float32 → Whisper ASR
  │  raw transcript → Ollama LLM → polished text
  │  stage events + result over WebSocket
  ▼
Swift client receives on @MainActor async loop
  │  updates pipeline UI + history pane

The WebSocket Protocol

Rather than abstracting the network layer away, I kept the WebSocket communication brutally simple and strictly typed. Here is the exact flow of data over the wire during a session:

• Initialization (Client → Server): The session opens with a JSON text payload detailing the configuration: {"sampleRate":N, "channels":N, "whisperModel":"...", "llmModel":"...", "systemPrompt":"..."}.
• Audio Streaming (Client → Server): As you speak, the app pushes raw PCM audio frames continuously as binary data.
• Termination (Client → Server): A simple text string "done" is fired to signal the end of the capture.
• Stage Updates (Server → Client): The backend pushes text status events, like {"type":"stage", "value":"transcribing"} or "polishing", to drive the UI tracker.
• Final Output (Server → Client): The fully polished transcript arrives as a text payload: {"type":"result", "text":"..."}.