AI and LLMs

An LLM is a neural network, usually of the transformer type, trained to predict the next token (a chunk of a word) across a huge collection of text. Doing that well at enormous scale teaches it grammar, facts, coding patterns and a working form of reasoning. Chat models are then tuned further on examples and human feedback so that they follow instructions and decline harmful requests.

A model does not remember earlier calls: every request carries the whole conversation, and it only knows its training data (up to a cutoff date) plus whatever is in the prompt. Many models are multimodal, reading images, audio or PDFs as well as text. Providers sell them in tiers, from small, fast, cheap models to large flagship ones that are slower and cost more per token.

Before a model sees your text, a tokeniser splits it into tokens: common words are one token, rarer words are broken into pieces, and spaces and punctuation count too. In English a token averages about four characters, or three quarters of a word; many other languages, and code, need more tokens for the same content.

Providers bill per million tokens, and output tokens usually cost several times more than input. The context window is the total a model can take in one request (instructions, history, documents and the reply), and current models range from tens of thousands to about a million tokens. A full window is slower and dearer, and models can overlook details buried in the middle, so sending only what is relevant still pays off.

Chat APIs take a list of messages. The system (or developer) message says who the model is and how it must behave, for example 'You are a support assistant for a shoe shop. Answer only from the policy below, in under 100 words.' User messages carry the questions and assistant messages hold earlier replies. Clear instructions, a few worked examples (few-shot prompting) and an explicit output format do more for quality than clever wording.

Treat prompts like code: keep them in version control and test them against known cases (often called evals) before changing them. Long prompts that repeat on every call can be cached by most providers at a large discount. Anything a user types, or any page or document the model reads, may try to override the rules (prompt injection), so the system prompt on its own is never a security boundary.

At each step a model scores every possible next token. Temperature reshapes those odds before one is picked: near 0 the likeliest token almost always wins, while higher values give less likely tokens a real chance. Top-p (nucleus sampling) is a related setting that limits the choice to the most probable tokens that together reach a given share.

Use a low temperature for extraction, classification, code and anything that must be consistent, and a higher one for brainstorming or creative writing. Even at 0, repeated answers are not guaranteed to match word for word. Ranges differ between providers (some accept 0 to 1, others 0 to 2), and some reasoning models fix the value or ignore it.

A model produces text that is statistically plausible, not text that has been checked. When the facts are missing (never in its training data, too recent or too obscure), it can fill the gap with something that sounds right: a product feature that does not exist, a court case that was never heard, a link that leads nowhere.

The risk can be reduced but not removed. Ground answers in your own documents (RAG) and ask for citations, allow the model to say it does not know, validate structured output, let tools do arithmetic and lookups, and keep a person in the loop for anything legal, medical or financial. Treat model output as a draft to check, not as a source.

You send messages (plus optional images, files or tool definitions) and get a reply, streamed token by token if you like. The line-up runs from small, low-cost models for simple high-volume work to flagship reasoning models for hard problems, alongside models for image generation, speech-to-text, text-to-speech, realtime voice and embeddings. The Responses API is the main interface today; the older Chat Completions format is so widely copied that many other providers accept it too.

There is no free tier: you prepay credit and can set budgets per project. Batch jobs that can wait up to a day cost half as much, and repeated prompt prefixes are billed at a discount when cached. API data is not used to train models by default. The same models are also sold through Microsoft Azure and Amazon Bedrock, for companies that prefer to buy through their existing cloud.

You create a key in Google AI Studio and call models in three broad tiers: Flash-Lite for low-cost, high-volume work, Flash for everyday tasks and Pro for harder reasoning. Many accept inputs of up to about a million tokens and take images, audio, video and PDFs directly. Extras include grounding answers in Google Search, context caching, a Batch API at half price and an OpenAI-compatible endpoint, so existing OpenAI code can often switch with a new base URL and key.

The free tier needs no card but has tight rate limits, covers only some models, and lets Google use your prompts and responses to improve its products. The paid tier raises the limits and does not use your content that way. Larger companies can reach the same models through Google Cloud, which adds enterprise security controls, data residency and reserved capacity.

Claude models come in tiers: Haiku for fast, low-cost work, Sonnet as the balanced middle, and Opus and larger tiers for demanding reasoning and long-running agent tasks. They read text, images and PDFs, and recent models take up to about a million tokens of context at the standard rate. The Messages API supports tool use, structured outputs, citations, web search, code execution and computer use (operating a browser or desktop on the user's behalf).

There is no free API tier: you buy credit in the Claude Console and pay per token. Prompt caching makes repeated long prompts much cheaper (a cache hit costs a tenth of the normal input price or less), and the Batch API halves prices for work that can wait. Anthropic does not train on commercial API data by default. The models are also sold through Amazon Bedrock, Google Cloud and Microsoft Foundry, and Anthropic created MCP, which its apps and SDKs support closely.

OpenRouter offers an OpenAI-compatible API in front of models from OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek and many more, so trying a different model means changing one name. For open models served by several hosts it can route by price, speed or uptime and fall back to another host when one fails, and account settings can exclude hosts that log prompts or train on them.

Model prices are passed through at each provider's own rates, and OpenRouter charges a fee when you buy credits. Some models are free with low daily request limits, which suits experiments rather than production, and you can bring your own provider keys. The trade-off is one more company in the path of every request, and hosts of the same open model can differ in speed and quality, for example in how heavily they compress it.

A NIM is a ready-made container that bundles a model with an inference server tuned for NVIDIA GPUs and an OpenAI-compatible API. The catalogue at build.nvidia.com hosts many of them, including open models such as Llama, Mistral, DeepSeek, Qwen and NVIDIA's own Nemotron family, plus speech, vision and embedding models. Members of the free NVIDIA Developer Program get an API key to call them.

The hosted endpoints are meant for prototyping: free, but rate limited and without an uptime promise. For production you download the containers and run them on your own GPUs or in a cloud account; production use generally needs an NVIDIA AI Enterprise licence, though some NIMs are free to deploy. The appeal is control: the same model and API on a workstation, in a data centre or in any major cloud, with data staying where you run it.

You send text with a target language and get the translation back; the source language can be detected at no extra cost. The Basic edition is a simple REST call to Google's neural machine translation model. The Advanced edition adds glossaries that fix how brand names and terms are translated, translation of Word, PowerPoint and PDF files that keeps their layout, batch jobs over files in Cloud Storage, custom models trained on your own examples, and a translation-tuned LLM.

Compared with asking a general chat model to translate, a dedicated API is predictable, keeps document formatting, honours glossaries and is priced simply per character. A chat model adapts tone and context better, at the cost of more variation between runs and per-token billing. DeepL, Microsoft's Azure Translator and Amazon Translate are the main alternatives.

After installing Ollama on macOS, Windows or Linux, a command such as 'ollama run' followed by a model name downloads that model from its library and starts a chat. In the background it runs a server on your machine (port 11434) with its own REST API and an OpenAI-compatible one, so apps and editors can point at it instead of a cloud provider. Models are stored in compressed (quantised) form so that smaller ones fit in an ordinary laptop's memory.

Speed and quality depend on the hardware: small models run on a typical laptop, while large ones need a strong GPU or plenty of unified memory. Nothing leaves the machine and there is no per-token bill, which suits private data and offline use. For models too big to run at home, Ollama also sells a cloud service that runs larger open models with the same commands and API.

The weights are the billions of numbers a model learns in training. When they are published, usually on Hugging Face, you can run the model on your own hardware, host it with any provider, inspect it and fine-tune it. Families from Meta (Llama), Google (Gemma), Mistral, Alibaba (Qwen), DeepSeek, Microsoft (Phi) and OpenAI (gpt-oss) come in sizes from a few billion parameters, which run on a laptop, to hundreds of billions or more, which need a cluster of GPUs.

Open weight is not the same as open source: the training data and code are rarely released, and licences range from permissive (Apache 2.0, MIT) to custom ones with conditions for very large companies or on certain uses. Quantisation stores each weight in 4 or 8 bits so models fit in less memory, at some cost in quality. Tools such as Ollama, llama.cpp and vLLM run them, and hosts such as OpenRouter serve them per token.

ONNX (Open Neural Network Exchange) is an open file format for trained models, so a model built in PyTorch or TensorFlow can be exported once and run elsewhere. ONNX Runtime loads that file and runs it efficiently on whatever hardware is present, through plug-in execution providers for CPUs, NVIDIA GPUs (CUDA, TensorRT), Windows GPUs (DirectML), Apple devices (Core ML) and more.

It suits smaller, focused models rather than big chat models: text embeddings for search, classifiers, image recognition, OCR and speech. onnxruntime-node runs them inside a Node.js server, and onnxruntime-web runs them in the browser with WebAssembly or WebGPU, so data can stay on the user's device. Hugging Face's Transformers.js is built on it and handles downloading and preparing models for you.

An embedding model turns an input into a vector, a list of a few hundred to a few thousand numbers. Inputs with similar meaning land close together, and closeness is measured with cosine similarity or a dot product. This powers semantic search, 'more like this' recommendations, grouping similar support tickets, spotting duplicates and the retrieval step of RAG.

Hosted models from OpenAI, Google, Cohere and Voyage charge per token, while open models such as BGE, E5 or MiniLM run locally, even in a browser with ONNX Runtime. Documents and queries must be embedded with the same model, and switching models means re-embedding everything. The vectors are stored in a vector index, often inside a database you already run.

Each document chunk, product or image is stored with its embedding. At query time the question is embedded too, and the database returns the nearest vectors, usually filtered by metadata such as language, owner or date. Comparing against every vector is exact but gets slow as a collection grows, so most systems build an approximate nearest neighbour (ANN) index such as HNSW, giving up a little accuracy for a large gain in speed.

You rarely need a separate product to start. pgvector adds vector columns and HNSW indexes to PostgreSQL (Supabase and Neon include it), sqlite-vec adds vector search to SQLite, and MongoDB Atlas, Redis and Elasticsearch have it built in. Dedicated services such as Pinecone, Qdrant, Weaviate, Milvus and Cloudflare Vectorize run the index for you and add features for very large collections.

Keyword search (usually ranked with BM25) is precise for names, codes and rare words, while vector search handles paraphrases and synonyms. Hybrid search runs both and fuses the lists. The common method is reciprocal rank fusion (RRF): each result scores 1 / (k + rank) in every list it appears in, with k often set to 60, and the scores are added. Because it uses rank positions rather than raw scores on different scales, it needs little tuning.

A reranker, a small model that reads the query next to each candidate, can then reorder the top few dozen results for extra precision. Elasticsearch, OpenSearch, Weaviate, Qdrant and MongoDB Atlas support hybrid queries directly, and in PostgreSQL it can be written as one SQL query that combines tsvector full-text search with pgvector. It is the usual retrieval setup for RAG over real-world documents.

Documents are split into chunks, embedded and indexed ahead of time. When a question arrives, the app retrieves the most relevant chunks (with vector, keyword or hybrid search), places them in the prompt with instructions such as 'answer only from these sources and cite them', and the model writes an answer grounded in that text. The name comes from a 2020 research paper by a team at Facebook AI Research.

RAG keeps answers current without retraining: update a document and the next answer reflects it. It also allows citations and per-user permissions, since you decide what each user's search may return. Most of the quality comes from retrieval rather than the model, so chunking, search method, reranking and good source documents matter more than prompt wording. It reduces hallucination without removing it, so answers still need spot checks and a fallback when nothing relevant is found.

You prepare pairs of example inputs and ideal outputs, from dozens to thousands of them, and the provider (or your own GPUs) trains the model on them for a few passes, called epochs. Full fine-tuning updates all the weights; parameter-efficient methods such as LoRA train a small add-on instead, which is far cheaper and is how most open models are tuned. Preference and reinforcement tuning go further by rewarding better answers over worse ones.

Fine-tuning is good at behaviour: a consistent tone, a strict output format, sorting tickets into your categories, or getting a small, cheap model to match a bigger one on one task. It is a poor way to teach facts that change, which is RAG's job. A tuned model is tied to its base model and must be retrained when that model is retired. Availability shifts too: OpenAI closed its self-serve fine-tuning to new customers in 2026, while Google Cloud still offers it and open models can be tuned on your own GPUs.

You describe each tool with a name, a plain-language description and a JSON Schema for its inputs. Instead of answering, the model can reply with a request such as get_order_status with order_id 1042. Your code runs the function, sends the result back and the model continues, possibly calling more tools, until it can give a final answer. The model never runs anything itself: your code decides what actually executes.

An agent wraps this in a loop with a goal, a budget and a stopping rule, and may plan, browse, write code or hand work to other agents. Agents fail in new ways: they can loop, run up token bills, act on a misunderstanding, or follow instructions hidden in a web page or email (prompt injection). Keep tools narrow, require confirmation for anything destructive or costly, log every step and cap the number of turns.

Anthropic introduced MCP in November 2024 and in December 2025 donated it to the Agentic AI Foundation, a Linux Foundation fund that Anthropic co-founded with OpenAI and Block. An MCP server exposes tools (actions the model can call), resources (data it can read) and prompts (reusable templates). An MCP client inside a host app, such as a chat app, an editor or an agent, discovers them and offers them to the model, exchanging JSON-RPC messages.

Local servers run as a process on your machine and talk over standard input and output; remote servers are reached over HTTP (Streamable HTTP) and usually sign users in with OAuth. Thousands of servers exist, for GitHub, databases, file systems, browsers, Slack and more. A server can do anything its credentials allow, and its tool descriptions go straight into the model's context, so install only servers you trust and give them the narrowest access that works.

Rather than parsing free text, you pass a JSON Schema (or a Zod or Pydantic model that the SDK converts) describing the fields you want, such as name, email and a priority from a fixed list. With strict structured outputs, offered by OpenAI, Google and Anthropic, the provider constrains generation so the reply always parses and matches the schema. Older 'JSON mode' only promised valid JSON, not the right fields.

It turns a model into a dependable component: extracting invoice fields, tagging support tickets, filling a form from an email, or returning a plan an agent can carry out. The shape is guaranteed but the content is not, so values still need validation and business rules (an extracted total can be wrong even when it is a valid number). Keep schemas small and descriptive, since field names and descriptions guide the model.