ML in Production · 4 min read

Netflix Details AI Discover: LLM Planning With Code Guardrails

Netflix engineers detail the agent harness behind AI Discover: plan-then-execute agents, async TV serving, code-enforced safety and MemberLLM taste summaries in a memory store.

By Rebecca Stone845 words

Key takeaways

  1. AI Discover is a new TV-app tab in testing with a small group of Netflix members
  2. The system uses a plan-and-execute agent: an LLM proposes a plan but code decides whether it runs
  3. MemberLLM, an in-house model, generates per-member taste summaries ahead of the request path
  4. Safety rules are enforced in code, with placeholder-based planning to counter prompt injection
  5. Models can be swapped live via per-task settings without a TV app release
Engineering AI Discover
Engineering AI Discover — MustangJoe / pixabay

By Rebecca Stone

Netflix has published the full engineering account of AI Discover, a new TV-app tab now in limited member testing, and the architecture behind it is a deliberate rebuke to the let-the-LLM-do-everything school. The post, written by a ten-engineer team led by Varun Khaitan, Peng Ren and Gary Tang, describes an "intelligence layer" that treats LLM output as untrusted input, plans every request before execution, and enforces safety rules in code rather than prompts.

The reasoning is scale. "At our scale, even rare failures happen many times a day," the engineers write. "A member sees every slow call, unavailable title or off-brand line." Every turn ends on a screen in someone's living room.

What is AI Discover?

The tab moves Netflix from silent ranking toward steerable discovery. A member picks a theme, such as "Quick Watches," and receives a row of matching titles, a short note on why they fit, and narrower themes to try next. Members navigate with the remote; there is no free-text input or open-ended chat in this version. That constraint keeps the LLM's job narrow by design, and profile settings plus safety and privacy policies still apply.

How does the intelligence layer work?

The layer sits behind Netflix's Page Server, which builds pages for all Netflix apps and hands requests over asynchronously. Two components do the work:

  • Controller. It gathers context, including real-time signals such as time of day, settles on a plan, validates it, and retrieves only the member details the plan requires from Memory Hub, a persistent, low-latency per-member memory store.
  • Executor. It runs the plan as a graph of approved steps. Steps pass typed outputs to each other, independent steps run in parallel with individual deadlines and fallbacks, and each action stays within limits set in code.

A plan is a graph of steps, each running a capability (a named unit of work, versioned, belonging to families such as retrieval, ranking, generation or packaging). Before execution the controller verifies that every step is approved, every input has a producer, nothing loops, and the plan ends in exactly one page.

Plan first, then execute

Netflix rejected the familiar agent loop of reason, call a tool, reason again, because it cannot be bounded in advance. Instead the system is a plan-and-execute agent: an LLM can propose the path, but code decides whether it runs. A router the company controls chooses between established, pre-tested plans and LLM-drafted ones. If the planner is slow or its plan fails validation, the system falls back to a tested plan.

The design stays deliberately model-agnostic. All LLMs sit behind one common interface, tasks carry per-task model settings that engineers can override live, and candidates are scored by automated checks, LLM judges and human evaluation before any swap. Because most behavior lives in server-side prompts and configuration, Netflix can change models, tools and capabilities without shipping a TV app release — significant, since TV apps update slowly and run on widely varying hardware.

Why asynchronous serving?

Most of a TV app is synchronous: press a button, get a page. Multi-step generation can take several seconds, and holding requests open would tie up too many threads. So generative requests return an immediate acknowledgement, the TV switches to a "thinking" state, and the server notifies the device when the result is ready. Slow generation no longer slows anything else on the serving path.

Where does safety live?

In code, not prompts. Content and maturity standards, per-step access limits and member "do not show" choices are enforced programmatically; generated text is cleaned before it reaches the screen. The planner sees a stripped-down request and uses placeholders rather than parameter values, which the controller fills after validation — a structure that also defends against prompt injection. Each request leaves a structured record of the plan, each step's outcome and which rules applied, enough to attribute a failed page to the plan, the data, a rule or the model.

What should data teams take from it?

Several decisions translate directly:

  • Treat LLM output as untrusted input; govern behavior in code with typed contracts.
  • Wrap proven retrieval and ranking systems as composable capabilities with clear interfaces, so LLM-drafted plans stay grounded in the real catalog.
  • Plan before executing; validate the plan graph and keep a tested fallback.
  • Precompute expensive personalization: MemberLLM, Netflix's in-house model, generates taste summaries from viewing history ahead of time, keeping that work off the request path.
  • Log structured request records now; they are the raw material for offline evaluation, replay, and eventually self-improvement.

Netflix says its longer-term goal is a system that proposes its own improvements, tests them safely and keeps only what proves better, within its privacy and data-use policies. Two open questions frame the next phase: how much members actually want to shape their recommendations, and how cost and latency hold up as traffic ramps. The company promises a follow-up post on MemberLLM.

linkedin.com (Original)

Price tags

Share this article:

More from Rebecca Stone

Rebecca Stone

Show full bio

Senior reporter covering regulation, policy and their impact on Analytics Digest is a professional daily at Analytics Digest.

12 articles

Read also

Next article »