Skip to content

skillData and AI

LLM Cost and Latency

Finds where an app wastes LLM tokens and time, then cuts both with prompt caching, smaller models, batching and streaming.

npx workspacetemplates add llm-cost-and-latency --tool claude-code

Writes 1 file to .claude/skills/llm-cost-and-latency/

.claude/skills/llm-cost-and-latency/SKILL.md28 lines · 1.6 KBFile
---
name: llm-cost-and-latency
description: Finds where an app wastes LLM tokens and time, then cuts both with prompt caching, smaller models, batching and streaming.
---

# LLM cost and latency

Measure before changing anything. Optimizing the wrong call wastes a week.

## Steps

1. Find every model call site. For each, record the model, input tokens, output tokens, calls per day and whether a user is waiting. Use existing logs or add a temporary counter.
2. Rank call sites by daily cost and by user-facing latency. Work only on the top three.
3. Cache stable prefixes: put system prompt, tool definitions and reference documents first and user input last, then enable the provider's prompt caching. Check the cache hit field in responses to confirm it works.
4. Route by difficulty: classification, extraction and formatting usually run fine on a small model. Prove it on at least 50 real examples before switching.
5. Batch anything nobody is waiting for (nightly summaries, backfills) through the provider's batch API at a discount.
6. Cut tokens: trim retrieved context to what the question needs, drop chat history beyond a rolling window plus a summary, and set a tight max output.
7. For latency, stream tokens, run independent calls in parallel, and avoid chained calls that could be one.
8. Cache whole responses for identical inputs when answers do not need to be fresh.

## Rules

- Never switch models without a before and after quality check.
- Keep a kill switch to roll back.

## Report

A table of call sites with cost and latency before and after, the change made to each, and the quality check result.

What it does

A bill that doubled or a chat that feels slow usually traces back to a handful of calls. This skill makes the agent measure first, then change the biggest cost, instead of guessing at tweaks.

The fixes are ordered by payoff and risk: cache stable prompt prefixes, route easy work to a smaller model, batch what is not interactive, and trim what you send and ask for.

Example

you › Our summarizer costs 4,000 USD a month, cut it

agent › Ranked three call sites. The nightly digest job (62% of spend) moved to the batch API and a small model, checked on 80 digests with no drop in rated quality. The 9k-token system prompt now sits first with caching on, with a 91% hit rate. Estimated spend: 1,350 USD a month.

Questions

Is prompt caching safe for private data?
It caches on the provider side for a short time per account. Check your provider terms, and keep per-user data after the cached prefix, never inside it.
How do I know a smaller model is good enough?
Run both on the same real inputs and compare against a fixed rubric or labeled answers. Switch only when the gap is below what users would notice.
  • Data Pipeline Design

    SkillData and AI

    Free
  • dbt Modeling

    SkillData and AI

    Free