# UniLM.jl

> A Julian, type-safe interface to LLM providers with first-class native backends — OpenAI (Chat + Responses), Anthropic (Messages), Google Gemini (generateContent + agentic Interactions), and local models through Ollama's native API — plus any OpenAI-compatible provider (Azure, DeepSeek, Mistral, vLLM, LM Studio). One cross-provider agentic `respond` verb, an MCP client and server, and built-in token/cost accounting.
> It also has one System One surface, which answers enumerated questions about a piece of state with a typed, calibrated judgment instead of generated text — and can select a Julia method by natural-language meaning — over TypeSafe's Jev, OpenAI's Decisions API (beta) and local decision models through Ollama.

Install: `using Pkg; Pkg.add("UniLM")` (registered in Julia's General registry; requires Julia 1.13+). Source: https://github.com/algunion/UniLM.jl

## Start here (for LLMs / coding agents)

- [Single-file LLM reference](https://algunion.github.io/UniLM.jl/dev/llm/): the whole API — types, signatures, defaults, and the complete exports list — on one page. Read this first.

## Guides

- [Getting Started](https://algunion.github.io/UniLM.jl/dev/getting_started/): install, API keys, first request
- [Chat Completions](https://algunion.github.io/UniLM.jl/dev/guide/chat_completions/): stateful `Chat` + `chatrequest!`
- [Responses API](https://algunion.github.io/UniLM.jl/dev/guide/responses_api/): the newer `respond` API
- [Agentic Workflows](https://algunion.github.io/UniLM.jl/dev/guide/agentic/): cross-provider `respond` (OpenAI Responses + Gemini Interactions)
- [Tool Calling](https://algunion.github.io/UniLM.jl/dev/guide/tool_calling/): function tools + automated `tool_loop`
- [Retrieval & File Search](https://algunion.github.io/UniLM.jl/dev/guide/retrieval/): end-to-end RAG (files → vector store → `file_search`)
- [Streaming](https://algunion.github.io/UniLM.jl/dev/guide/streaming/): callback / `do`-block, across providers
- [Timeouts & Retries](https://algunion.github.io/UniLM.jl/dev/guide/timeouts/): `RequestConfig`, the typed timeout failures, retry contracts
- [Concurrency, Tasks and Cancellation](https://algunion.github.io/UniLM.jl/dev/guide/concurrency/): what a concurrent call may share (`fork` for `Chat`), fan-out patterns, streaming into a `Channel`, `CancelToken` / `with_cancel`, MCP sessions under load
- [Structured Output](https://algunion.github.io/UniLM.jl/dev/guide/structured_output/): JSON-schema-constrained generation
- [Multi-Backend](https://algunion.github.io/UniLM.jl/dev/guide/multi_backend/): native + OpenAI-compatible providers
- [Local Models with Ollama](https://algunion.github.io/UniLM.jl/dev/guide/ollama/): `OllamaEndpoint` on Ollama's native API with Gemma 4 — thinking (`reasoning_text`), tools, structured output, `ImageAttachment`/`AudioAttachment`, embeddings, `num_ctx`/`keep_alive`, model management (`list_models`, `pull_model`, `model_info`, …)
- [Custom Backends](https://algunion.github.io/UniLM.jl/dev/guide/custom_backends/): implement a new `ServiceEndpoint`
- [MCP](https://algunion.github.io/UniLM.jl/dev/guide/mcp/): Model Context Protocol client & server
- [Start Here: Decisions in Five Minutes](https://algunion.github.io/UniLM.jl/dev/guide/jev_start/): what TypeSafe's System One model Jev decides, a first `nl_classify` call, and which Decisions page answers which question
- [Backends: Jev, OpenAI and Local Models](https://algunion.github.io/UniLM.jl/dev/guide/system_one_backends/): one System One surface on TypeSafe's Jev (default), OpenAI's Decisions API (`OPENAIServiceEndpoint`, beta) and Ollama's decision models (`OllamaEndpoint()`); `service =` per call or `with_system_one` for a scope; what each refuses before sending; tune `min_confidence` per backend
- [Decisions with LLMs](https://algunion.github.io/UniLM.jl/dev/guide/jev_with_llms/): Jev before an LLM call (route the message, pick the model, screen what it reads, shortlist its tools) and after it (check the draft and its claims against the source, approve its tool calls)
- [Route and Decide](https://algunion.github.io/UniLM.jl/dev/guide/system_one/): `ask` with `choice`/`score`/`noul`, many questions in one request, images (`ImageAttachment`) and `box` questions, `long_context`, a struct filled in one request, confidence gates, loss-matrix decisions, a catch-all option
- [Dispatch on Meaning](https://algunion.github.io/UniLM.jl/dev/guide/natural_language_dispatch/): `nl_classify`, `nl_dispatch` with a keyed table (`texts =`) or `nl"..."` meanings in method signatures, type hierarchies and state machines, `decide`, `on_response`, the `@branch` switch
- [Many Items at Once](https://algunion.github.io/UniLM.jl/dev/guide/semantic_algorithms/): `ask_each` (one request per item, concurrent, results in order), many items in one request (keyed, not indexed), ranking, finding an event in a long sequence, matching records, stopping early
- [Test and Develop](https://algunion.github.io/UniLM.jl/dev/guide/jev_testing/): previews with `meanings`/`meaning_gaps`, `with_recorded_answers` record/replay for Jev and LLM calls, a Jev + LLM pipeline tested offline, an audit trail
- [Cost Tracking](https://algunion.github.io/UniLM.jl/dev/guide/cost_tracking/): token usage + USD estimation
- [Embeddings](https://algunion.github.io/UniLM.jl/dev/guide/embeddings/)
- [Image Generation](https://algunion.github.io/UniLM.jl/dev/guide/image_generation/): create + edit
- [FIM & Prefix Completion](https://algunion.github.io/UniLM.jl/dev/guide/completions/)

## Gotchas for code generation

- A `Chat` must start with a **system** message; `push!` **throws** `InvalidConversationError` for a non-system message pushed onto an empty `Chat` (and for consecutive same-role messages, except `tool`). Use `respond("...")` for a single turn without a system prompt.
- Real defaults: OpenAI `gpt-5.6-sol`; native Gemini `gemini-3.8-flash`; native Anthropic `claude-opus-5-5` (auto `max_tokens` 16000); DeepSeek `deepseek-flash`; System One: TypeSafe `jev-latest`, OpenAI's Decisions API `gpt-6-luna`, Ollama none (name a decision model, e.g. `tev1:0.8b`); image `gpt-image-2`; embeddings `text-embedding-3-small`.
- Prefer the Responses API / `respond` for new code. Native Anthropic = `ANTHROPICServiceEndpoint`; native Gemini = `GEMINIServiceEndpoint`; the OpenAI-compat Gemini shim (needed for embeddings) = `GEMINIOpenAIServiceEndpoint`.
- Local validation throws before any request: an unsupported capability, an option the provider/model cannot express, or a `callback`/`on_tool_call` given without `stream=true` is an `ArgumentError`, and sending a history-keeping `Chat` that ends with an assistant message is an `InvalidConversationError`. Request failures — HTTP errors, timeouts (`cause::UniLMTimeout`), cancellations (`cause::UniLMCancelled`) — come back as typed results (`*Success` / `*Failure` / `*CallError`). The accessors `text`, `output_text`, `embedding_vectors`, `image_data` and `fim_text` throw `LLMResultError` on a non-success result, so branch on `issuccess` (or the result type) first. `estimated_cost` returns `0.0` (logging one warning per model id) for any model not in `DEFAULT_PRICING` (pass `pricing=`, or add a row with `DEFAULT_PRICING[model] = row`), except that a dated snapshot uses its base model's row and a versioned `jev-X.Y.Z` id without its own row uses the `jev-latest` row.
- `Chat(n=…)` must be 1. A `Chat` (and an `Embeddings`) is mutated by its request: use one per in-flight call and `fork(chat, n)` to fan out. Cancel a call from any task with a `CancelToken` (`cancel=tok`, or `with_cancel(tok) do … end`) and `cancel!(tok)`; a token stays cancelled, so use a fresh one per unit of work.
- `TYPESAFEServiceEndpoint` is **not** a chat backend: it declares only `:system_one` and `:models`, so the verbs `chatrequest!`, `respond`, `embeddingrequest!` and the other platform verbs reject it with an `ArgumentError` — constructing a `Chat` or an `Embeddings` that names it is fine, and omitting `model=` is refused the same way. `ask` and `list_models` follow the package convention and **return** `SystemOneSuccess`/`SystemOneFailure`/`SystemOneCallError`, but the natural-language control flow built on them **throws**: `@branch`, `nl_dispatch` and `nl_classify` raise `SystemOneError` on a failed call, and — when there is no `_` line / `fallback` — `LowConfidenceError` for an answer below `min_confidence` and `DecisionDeclinedError` when a `decide` policy returns `nothing`. `nl_dispatch` raises an `ArgumentError` before any request when no natural-language method accepts the types of its ordinary arguments, or when a combination of the meanings it would offer has no method (a gap; `meaning_gaps(f, argtypes)` lists them). With `texts = TABLE`, `nl_dispatch` dispatches on keys instead of `nl"..."` meanings: a `NamedTuple` of sentences (each `Symbol` key reaches its method as `Val(key)`) or a vector of `key => sentence` pairs (your own instances, types or enum values, passed as written); only the sentences are sent, in table order, a `Dict` is refused (no order), and a key no method takes is an `ArgumentError` before any request. `nl_classify(text, TABLE)` returns the chosen key as the table writes it (`:billing`, not `Val(:billing)`). `on_response = r -> …` on `nl_dispatch`, `nl_classify` and `@branch` is called once with the `SystemOneSuccess` (request id, model, raw body) before the decision policy runs, never for a failed call. `with_recorded_answers` records and replays `ask` and `list_models` and the non-streaming `chatrequest!`, `respond` and `embeddingrequest!` (and so the tool loops); streamed calls and every other verb reach the network as usual. In its replay scope a request with no recording throws `ReplayMissError` out of the verb itself — and so out of `@branch`, `nl_dispatch`, `nl_classify` and the tool loops — never a call-error result. Default model `jev-latest` (`TYPESAFE_API_KEY`; `TYPESAFE_DEFAULT_MODEL` overrides).
- System One has three backends behind one surface: `TYPESAFEServiceEndpoint` (default), `OPENAIServiceEndpoint` (OpenAI's Decisions API, public beta, `gpt-6-luna`, `OPENAI_API_KEY`) and `OllamaEndpoint()` (local decision models; no default model: `model = "tev1:0.8b"`). Pass `service = …` per call, or `with_system_one(service; model = …) do … end` for every call inside (`ask`, `ask_each`, `list_models`, the NL verbs; spawned tasks too). The question types set no upper bound; each backend's wrapper refuses what its wire cannot carry with an `ArgumentError` before any request — TypeSafe: more than 255 options or 10 levels, images that are not PNG/JPEG of 280–1568 px a side and ≤ 16 MiB, a `box` question without an image; OpenAI: a question without string `instructions`, a description or level that is not a string, `noul` yes/no criteria, Box questions, images outside the state; Ollama: images, Box questions, more than 26 options, a question without instructions. Provider options are `ask` keywords (TypeSafe `long_context`, OpenAI `prompt_cache_key`/`safety_identifier`, Ollama `keep_alive`); the NL verbs take them as `options = (long_context = true,)`, and `with_system_one(service; model, options...) do … end` sets them for the calls inside that name no service and no options. A reply that does not fit the request is a `SystemOneCallError`. A `RefusalAnswer` (OpenAI may decline a question) is a decline for `nl_dispatch`, `nl_classify` and `@branch` (fallback, else `DecisionDeclinedError`). `min_confidence` must be a finite number in 0 … 1, and a `min_confidence` tuned on one backend does not carry over to another. `ask_each(states, questions...)` returns one typed result per state, in input order (16 in flight on TypeSafe and OpenAI, 1 on Ollama). `nl_dispatch` refuses, before any request, a default state holding a function, type, module, `Ref`, `Task`, `Channel`, lock, `IO`, `Expr` or a value JSON.jl cannot write (an `AudioAttachment`); an `ImageAttachment` argument is data (sent as a TypeSafe image); every field of a struct argument is sent — pass `state = …` to choose what the model reads. A `Meaning` type prints as `Meaning{Symbol("…")}` (a one-word sentence as `Meaning{:x}`).

## Reference

- [API reference (start)](https://algunion.github.io/UniLM.jl/dev/api/chat/)
- [Repository](https://github.com/algunion/UniLM.jl)
