Skip to content
LLM gateway diagram with apps, auth keys, rate limits, cache, model router, prompt log and model providers on a card over a photo

LLM Gateway Architecture

Presentation template for Google Slides and PowerPoint

Make a copy in Google Slides
Price
Free
Slides
5
Difficulty
Intermediate
Updated
November 8, 2026

Every slide (5)

Overview

An application or batch job sends a chat or completion call to the gateway with its own gateway key. It checks the key and the team's model aliases, then looks up the response cache. On a miss, the rate limiter checks the team quota and returns HTTP 429 over the limit. The model router sends the call to the primary provider behind the alias. If it times out, the router retries once, then falls back. The gateway logs the prompt, token counts and cost, then returns the answer.

Provider keys stay inside the gateway, so application code never holds them. Changing a model alias is a router config change rather than an application release. Cache hits never reach a provider, which keeps repeated calls cheap, but streaming calls skip the cache. The self-hosted model gives sensitive data a route that stays on the company network. A single-provider deployment would drop the fallback rules and keep the key checks, quotas and prompt logging.

Useful for

  • Giving several product teams one shared model endpoint
  • Capping monthly LLM spend per team with request and token quotas
  • Keeping customer prompts on a self-hosted model for sensitive data

What's included

  • A designed cover slide
  • The diagram: 11 components and 10 connections as editable shapes
  • A components table and the 7-step data flow
  • A legend with notes
  • The same diagram as a draw.io (diagrams.net) file

How to use it

  1. 1

    Click Make a copy in Google Slides, then confirm on the Google page. The copy is saved to your own Google Drive.

  2. 2

    Rename Web and mobile apps, Batch jobs, LLM gateway and the other boxes inside Gateway platform (our cloud account) to your own components, and delete the ones your system does not have.

  3. 3

    Rewrite the 7-step data flow and the 7-row components table so they match the arrows you keep.

  4. 4

    For the draw.io version, download llm-gateway-architecture-diagram.drawio and open it at app.diagrams.net.

What you need

Requirements

  • A Google account for Google Slides, or the free draw.io (diagrams.net) editor

Works with

  • Google Slides
  • Microsoft PowerPoint (File → Download → .pptx)
  • draw.io / diagrams.net (.drawio file)

Questions

Why do cache hits not get billed?

A cache hit returns a stored answer without calling a model provider, so no provider charge is made for that request. The gateway still records the call in its logs. An exact-match cache only helps when prompts repeat exactly, so the hit rate is worth tracking on the cost dashboard.

What is model fallback in an LLM gateway?

Fallback sends a request to a second provider when the first one fails or times out. Routing rules set the order and how many retries happen first. The caller gets an answer from the backup without changing its code, though the wording may differ.

Why should prompts in a gateway log be redacted?

Prompts often contain personal data, customer details or secrets pasted by users. Redaction removes or masks those values before the log is stored. Without it, the log becomes a sensitive data store that needs the same protection as the source systems.

  • #ai-architecture-diagram
  • #architecture-diagram
  • #google-slides
  • #drawio
  • #diagram-template
  • #free-template

More ai architecture diagram templates

See all 12

Also in Developer workflows

All developer workflows templates
Get this template

Preparing your template…