TRADE-OFFS & DECISIONS

Engineering the Agentic Loop

Streaming Intent Parsing vs. Batch Processing

Traditional voice commands rely on a VAD (Voice Activity Detection) -> Stop -> Transcribe -> Process pipeline. For a real-time UI design tool, this latency is unacceptable.

The Trade-off: By using streaming intent parsing, we feed partial transcripts into the LLM. The LLM attempts to deduce intent before the user finishes speaking. This significantly reduces apparent latency, making the tool feel like an extension of the designer's mind.

The Cost: High token usage and potential hallucination on incomplete sentences. If the user says "Make the background red... no, wait, blue," the streaming parser might eagerly execute the "red" command before the correction arrives.

The Resolution: We implemented a debounce mechanism tied to confidence scores. If the LLM's confidence in the inferred semantic action is below a threshold, it buffers the intent. If it's high, it executes optimistically, relying on the Semantic Buffer's diff engine to easily revert or patch the state when the final transcript arrives.

Semantic Buffering vs. Pure UI Generation

Pure UI Gen (v0, Midjourney approach)

  • Prompt generates a complete component or image from scratch.
  • Stateless. No memory of previous specific pixel values.
  • "Change the padding" requires regenerating the entire component, often changing unrelated details.
  • Fast to implement, terrible UX for precise design.

Semantic Buffering (Dizzy approach)

  • Maintains a JSON AST (Abstract Syntax Tree) of the Figma document state.
  • Agents mutate specific nodes in the AST.
  • Stateful. "Change the padding" only modifies the padding property of the target node AST.
  • Complex to orchestrate, requires strict schema validation, but enables perfect precision and iterative design.

AST Mutation Trace

// 1. Initial State
{ id: "node_1", type: "FRAME", padding: 16, children: [...] }

// 2. Voice Input: "Make it roomier"
// 3. Agent parses intent -> INCREASE_PADDING
// 4. Mutation Applied to Buffer
{
  "op": "UPDATE",
  "target": "node_1",
  "path": ["padding"],
  "value": 32, // calculated based on context
  "previousValue": 16
}

// 5. Diff Engine pushes to Figma via MCP
Figma.getNodeById("node_1").padding = 32;

The Figma MCP Bottleneck

Interfacing with Figma's plugin API natively from an external agentic loop requires a bridge. We utilize the Model Context Protocol (MCP) to standardize this communication.

A major challenge is rate limiting and the synchronous nature of Figma's API updates when touching many nodes. By batching operations through the Semantic Buffer's diff engine, we reduce 50 individual node property updates into a single atomic transaction sent over the MCP WebSocket connection, preventing UI freezing in the Figma client.