Augment

Where it started

An ear and a brain, glued together by a web page.

The first version was a single HTML file. You pressed Record, talked, and pressed Stop. The page sent the audio to Whisper running on the Mac (the ears), passed the words to Llama 3.1 8B (the brain) to remove the ums and add punctuation, and showed both. Whisper and Llama never spoke to each other; the page was the glue.

Then it became an app

Wrapping it in Electron gave it a global hotkey, a menu bar icon and a window of floating cards: a record button, the live transcript, and what came out the other end. Both servers ship inside the app, Llama starts when you begin talking so it's warm by the time you stop, and it frees its 5 GB after five quiet minutes.

It worked, but it still ended with "now go and paste it". The rest of this project was about removing that last step and everything that made it feel like a tool you visit instead of something that's just there.

Meet Augie

A face made of two brackets and two squares.

Augment needed a way to say "I'm listening" and "I'm thinking" without words, at 28 pixels tall. Augie is the product's name drawn as a face: [··]. Every state is a keyframe animation (seven tracks: the whole body, each bracket, each eye) that I could tune in a small editor built into the app. Move your pointer around: the eyes follow it, and every so often Augie blinks.

Words as you speak

Ink settles as Whisper stops changing its mind.

Whisper doesn't stream; it transcribes a clip. So every 1.5 seconds Augment sends everything said so far and redraws it. Words that came back the same as last time are drawn in full ink; the newest stay light until Whisper stops revising them. You can watch it correct "check list" into "checklist" below. When you stop, one final pass over the whole recording produces the real transcript.

Typing where you are

One note, written the way each app reads.

The clipboard step went first. When you press ⌥Space, Augment notes which app is in front, and when the note is ready it presses ⌘V there for you. Knowing the app turned out to matter for more than pasting: the same checklist has to land differently in each one.

NotesClaudeMailTerminal

Trip to Paris

Book flights

Renew my passport

Find a pet sitter

Built with Notes' own Format menu: a real heading, then a real checklist you can tick.

Teaching an 8B model to format

Rails, not a longer prompt.

Formatting meant the model had to decide what's a heading, what's a list and what's just a sentence, and drop the words people use to dictate a list ("first one is", "let's make it a checklist") without dropping anything else. A small model asked to do all of that in one go behaved like one:

1. One prompt, Markdown outMade a list once in eight tries, kept "make this a checklist" in the note, and answered "Can you summarise this?" instead of copying it.
2. Worked examplesLists improved, but it quietly dropped words ("on the way home", "Hey, just wanted to say") and invented headings from questions.
3. Fixed fields, checked in codeLlama fills a JSON shape the server enforces; code removes the spoken instructions first and checks everything Llama returns. Every real dictation I threw at it came out right.

{
  "heading": "Trip to Paris",
  "kind": "checklist",
  "items": ["Book flights", "Renew my passport", "Find a pet sitter"],
  "before": "",
  "after": ""
}

From floating pill to notch

The pill learned where it lives.

When you start talking from another app, the full window would steal focus. So Augment shows a small pill instead: it never takes focus, mirrors what's happening, and disappears a few seconds after the note lands. Its design changed four times.

1. v1

   A floating pill

   Translucent grey, centred near the top of the screen, sliding in and fading out. It read well over any wallpaper but floated free of everything around it.

   

2. v2

   Knowing where it's going

   Once Augment knew which app it was writing into, the pill showed that app's icon. The first attempt drew a grey placeholder; macOS only hands over the real icon through Quick Look.

   Trip to Paris. Let's make it a checklist.0:04
3. v3

   Out of the notch

   Pure black, attached to the top edge, growing out of the MacBook notch and tucking back into it with the same motion reversed. macOS keeps windows below the menu bar, so this one has to ask to sit above it.

   NotesFileEdit12:31

   Trip to Paris. Let's make it a checklist.

   0:04
4. v4

   Two rows, one job each

   The top is only for Augie and your words; the app and the timer moved to a bar underneath. Words got the full width, so a finished note fits on two lines.

   Trip to Paris ☐ Book flights ☐ Renew my passport ☐ Find a pet sitter

   NotesPasted

   • Ears concave curves where it meets the screen edge, so it reads as the notch growing
   • Below the notch content starts at the menu bar's height, which on a notched Mac is the notch's
   • Bottom bar where the note is going, and what happened to it

What broke, and what it taught me

Learnings