LMQL

https://lmql.ai/docs
Admin
LMQL is a query language for Large Language Models that allows developers to write programs that...
Tokens:48,288
Snippets:176
Trust Score:8.2
Update:1 year ago
Show doc for...
Tokens:
Raw
### LMQL Query Output Example

Source: https://lmql.ai/docs/index

An example of the expected output format for an LMQL query, demonstrating how the model's response is structured and how template variables are populated.

```promptdown
# Model Output

Say this is a test: RESPONSE This is a test
```

--------------------------------

### LMQL Hello World Query

Source: https://lmql.ai/docs/index

A basic 'Hello World' LMQL query demonstrating a prompt statement and a constraint clause. The constraint limits the output length to under 25 tokens, showcasing LMQL's ability to enforce high-level constraints.

```lmql
"Say 'this is a test':[RESPONSE]" where len(TOKENS(RESPONSE)) < 25
```

--------------------------------

### Launch LMQL Playground IDE

Source: https://lmql.ai/docs/installation

Launches a local instance of the LMQL Playground IDE. This requires Node.js to be installed.

```bash
lmql playground
```

--------------------------------

### LMQL Program with Sampling and Control Flow

Source: https://lmql.ai/docs/index

An LMQL program demonstrating advanced features including sampling decoding with a specified temperature, conditional logic based on model output, and interactive prompting. It aims to generate a specific phrase and checks for its presence, retrying if necessary.

```lmql
sample(temperature=1.2)

"Say 'this is a test'[RESPONSE]" where len(TOKENS(RESPONSE)) < 25

if "test" not in RESPONSE:
    "You did not say 'test', try again:[RESPONSE]" where \
        len(TOKENS(RESPONSE)) < 25
else:
    "Good job"

```

--------------------------------

### Install LMQL with GPU Support

Source: https://lmql.ai/docs/installation

Installs LMQL with GPU dependencies, which requires a pre-existing GPU-enabled installation of PyTorch version 1.11 or higher.

```bash
pip install lmql[hf]
```

--------------------------------

### Run LMQL Programs via CLI

Source: https://lmql.ai/docs/installation

Executes local LMQL files (.lmql) using the command-line tool.

```bash
lmql run <your_program.lmql>
```

--------------------------------

### Install LMQL

Source: https://lmql.ai/docs/installation

Installs the LMQL library using pip. This command requires a Python environment version 3.10 or higher.

```bash
pip install lmql
```

--------------------------------

### Building Documentation Locally

Source: https://lmql.ai/docs/development/documentation

Instructions for building the LMQL documentation locally using Vitepress. This involves navigating to the docs directory, installing dependencies with yarn, and running the development server.

```bash
# enter docs/ directory
cd docs
# install dependencies
yarn
# run hot-reloading live server
yarn docs:dev
```

--------------------------------

### Configure OpenAI API Credentials

Source: https://lmql.ai/docs/installation

Sets OpenAI API credentials either by defining the OPENAI_API_KEY environment variable or by creating an api.env file in the active working directory.

```env
openai-org: <org identifier>
openai-secret: <api secret>
```

--------------------------------

### Install Node.js for LMQL Playground (Conda)

Source: https://lmql.ai/docs/installation

Installs Node.js version 14.20 using Conda, a prerequisite for running the LMQL Playground IDE.

```bash
conda install nodejs=14.20 -c conda-forge
```

--------------------------------

### LMQL Integration with Key-Value Store for State Management

Source: https://lmql.ai/docs/language/tools

This LMQL program demonstrates interaction with an external Python key-value store. It includes an instructive prompt guiding the model on how to use `assign` and `get` functions for state management, followed by a core reasoning loop that dynamically calls these functions to solve a problem requiring memory.

```lmql
# instructive prompt, instructing the model to how use the storage
"""In your reasoning you can use actions. You do this as follows:
`action_name(<args>) # result: <inserted result>`
To remember things, you can use 'assign'/'get':
- To remember something:
`assign("Alice", "banana") # result: "banana"`
- To retrieve a stored value:
`get("Alice") # result: "banana"`
Always tail calls with " # result". Using these actions, let's solve the following question.\n"""

# actual problem statement
"""
Q: Alice, Bob, and Claire are playing a game. At the start
of the game, they are each holding a ball: Alice has a black
ball, Bob has a brown ball, and Claire has a blue ball.

As the game progresses, pairs of players trade balls. First,
Bob and Claire swap balls. Then, Alice and Bob swap balls.
Finally, Claire and Bob swap balls. At the end of the game,
what ball does Alice have?
A: Let's think step by step.
"""

# core reasoning loop
for i in range(32):
    "[REASONING]" where STOPS_AT(REASONING, "# result") and \
                        STOPS_AT(REASONING, "Therefore, ")

    if REASONING.endswith("# result"):
        cmd = REASONING.rsplit("`",1)[-1]
        cmd = cmd[:-len("# result")]
        "{eval(cmd)}`\n"
    else:
        break

# generate final answer
"Therefore at the end of the game, Alice has the[OBJECT]" \
    where STOPS_AT(OBJECT, ".") and STOPS_AT(OBJECT, ",")
```

--------------------------------

### LMQL Scripted Prompting Example

Source: https://lmql.ai/docs/language/scripted-prompting

This example demonstrates how to use LMQL for scripted prompting, showcasing the use of a sampling decoder with a temperature setting. LMQL programs can include control flow to dynamically construct prompts and react to model outputs.

```LMQL
# use sampling decoder
sample(temperature=1.0)

```

--------------------------------

### LMQL Standalone Syntax: Decoder and Distribution

Source: https://lmql.ai/docs/language/reference

Illustrates the standalone LMQL query syntax with a decoder and a distribution clause to guide the LLM's output.

```lmql
argmax "What is the capital of France? [ANSWER]" \
    distribution ANSWER in ["A", "B"]

```

--------------------------------

### Setup LMQL Development Environment with GPU Support (Shell)

Source: https://lmql.ai/docs/development/dev-setup

This section details setting up a `conda` environment for local LMQL development with GPU support. It involves creating and activating a conda environment using `scripts/conda/requirements.yml` and sourcing an activation script. This setup is tested on Ubuntu 22.04 with CUDA 12.0 and Windows 10 via WSL2 with CUDA 11.7.

```shell
# prepare conda environment
conda env create -f scripts/conda/requirements.yml -n lmql
conda activate lmql

# registers the `lmql` command in the current shell
source scripts/activate-dev.sh
```

--------------------------------

### LMQL Prompt Construction Example

Source: https://lmql.ai/docs/language/reference

Provides a step-by-step illustration of how LMQL constructs the final prompt sent to a language model by concatenating query strings and substituting placeholder variables as the program executes.

```lmql
"Hello, [NAME] and [NAME2]\n"
        
"How are you doing?"

" [FEELINGS]"
```

--------------------------------

### LMQL Sentence Examples

Source: https://lmql.ai/docs/language/reference

Illustrates valid derivations of LMQL sentences based on the defined grammar. These examples showcase how different components of the grammar can be combined to form complete, executable LMQL statements.

```LMQL
"The quick fox jumps over the lazy dog."
"The quick brown fox jumps over the lazy dog."
"The quick dog jumps over the lazy dog."
"The quick brown dog hops over the lazy dog."
"The quick dog A9_D over the lazy dog."
```

--------------------------------

### Loading LMQL Models

Source: https://lmql.ai/docs/models/index

Demonstrates how to load different text generation models using the `lmql.model(...)` function. It shows examples for OpenAI, llama.cpp, HuggingFace Transformers, and a random sampling model, including options for specifying backend-specific arguments like CUDA and quantization.

```python
lmql.model("openai/gpt-3.5-turbo-instruct") # OpenAI API model
lmql.model("random", seed=123) # randomly sampling model
lmql.model("llama.cpp:<YOUR_WEIGHTS>.gguf") # llama.cpp model

lmql.model("local:gpt2") # load a `transformers` model in-process
lmql.model("local:gpt2", cuda=True, load_in_4bit=True) # load a `transformers` model in process with additional arguments
lmql.model("gpt2") # access a `transformers` model hosted via `lmql serve-model`
```

--------------------------------

### Loading LMQL Models

Source: https://lmql.ai/docs/models

Demonstrates how to load different text generation models using the `lmql.model(...)` function. It shows examples for OpenAI, llama.cpp, HuggingFace Transformers, and a random sampling model, including options for specifying backend-specific arguments like CUDA and quantization.

```python
lmql.model("openai/gpt-3.5-turbo-instruct") # OpenAI API model
lmql.model("random", seed=123) # randomly sampling model
lmql.model("llama.cpp:<YOUR_WEIGHTS>.gguf") # llama.cpp model

lmql.model("local:gpt2") # load a `transformers` model in-process
lmql.model("local:gpt2", cuda=True, load_in_4bit=True) # load a `transformers` model in process with additional arguments
lmql.model("gpt2") # access a `transformers` model hosted via `lmql serve-model`
```

--------------------------------

### LMQL Standalone Syntax: Decoder, Model, Constraints, Distribution

Source: https://lmql.ai/docs/language/reference

An example of the legacy standalone LMQL query syntax combining a decoder, model specification, constraint, and distribution clause for LLM interaction.

```lmql
argmax
    "What is the capital of France? [ANSWER]"
from
    "gpt2"
distribution
    ANSWER in ["A", "B"]

```

--------------------------------

### Cross-Variable Constraints with LMQL

Source: https://lmql.ai/docs/language/scripted-prompting

Demonstrates how to enforce constraints between variables in an LMQL prompt. This example shows selecting an essential item from a previously generated list, ensuring consistency and guiding model reasoning.

```python
print("A list of things not to forget when going to the sea (not travelling): \n")
backpack = []
for i in range(5):
   print("-[THING] where STOPS_AT(THING, \"\\n\")")
   print("backpack.append(THING.strip())")

print("\nThe most essential of which is: [ESSENTIAL_THING] \\
    where ESSENTIAL_THING in backpack")
```

--------------------------------

### LMQL Query Execution Example

Source: https://lmql.ai/docs/lib/python

Demonstrates executing an LMQL query without an explicit return statement. The result is an LMQLResult object containing the prompt and variables.

```lmql
joke = lmql.query("""
    "Q: Tell me a joke about plants\n"
    "A: [JOKE]"
""", is_async=False)

joke()
```

--------------------------------

### LMQL Sentiment Analysis Example with Constraints

Source: https://lmql.ai/docs/language/overview

This example demonstrates how to perform sentiment analysis using LMQL. It shows how to prompt an LLM with a review, request a detailed analysis, and then constrain the output to a specific classification (positive, neutral, negative). Constraints like preventing newlines are used for robust output formatting.

```LMQL
# review to be analyzed
review = """We had a great stay. Hiking in the mountains
            was fabulous and the food is really good."""

# use prompt statements to pass information to the model
"Review: {review}"
"Q: What is the underlying sentiment of this review and why?"
# template variables like [ANALYSIS] are used to generate text
"A:[ANALYSIS]" where not "\n" in ANALYSIS

# use constrained variable to produce a classification
"Based on this, the overall sentiment of the message\n can be considered to be[CLS]" where CLS in [" positive", " neutral", " negative"]

CLS # positive

```

--------------------------------

### LMQL with Replicate Endpoint

Source: https://lmql.ai/docs/models/replicate

Example of an LMQL query configured to use a model hosted on Replicate. It specifies the Replicate endpoint, a compatible tokenizer, and query constraints.

```lmql
argmax
    """Review: We had a great stay. Hiking in the mountains was fabulous and the food is really good.\n    Q: What is the underlying sentiment of this review and why?\n    A:[ANALYSIS]\n    Q: Summarizing the above analysis in a single word -- of the options \"positive\", \"negative\", and \"neutral\" -- how is the review best described?\n    A:[CLASSIFICATION]"""
from lmql.model(
    # model name is not actually used: endpoint completely overrides model selection
    "meta-llama/Llama-2-13b-chat-hf",
    # in this case, uses model from https://replicate.com/charles-dyfis-net/llama-2-13b-hf--lmtp-8bit
    endpoint="replicate:charles-dyfis-net/llama-2-13b-hf--lmtp-8bit",
    # choosing a model with the same tokenizer as meta-llama/Llama-2-13b-hf but ungated in huggingface
    tokenizer="AyyYOO/Luna-AI-Llama2-Uncensored-FP16-sharded",
)
where STOPS_AT(ANALYSIS, "\n") and len(TOKENS(ANALYSIS)) < 200
distribution CLASSIFICATION in [" positive", " negative", " neutral"]

```

--------------------------------

### LMQL Variable Assignment Example

Source: https://lmql.ai/docs/language/reference

Demonstrates how LMQL assigns values to program variables from generated placeholder content. It shows the equivalence between LMQL's placeholder assignment and standard Python variable assignment.

```python
NAME = "Alice"
# Equivalent to: NAME, NAME1 = model.fill_placeholders("Hello [NAME] and [NAME2]")
"Hello [NAME] and [NAME2]"
```

--------------------------------

### Simple Generation and Scoring Example

Source: https://lmql.ai/docs/lib/generations

Demonstrates basic text generation and sequence scoring using the LMQL Generations API. It shows how to obtain a model instance and perform synchronous generation and scoring operations.

```Python
import lmql

# obtain a model instance
m: lmql.LLM = lmql.model("openai/gpt-3.5-turbo-instruct")

# simple generation
m.generate_sync("Hello", max_tokens=10)
# -> Hello, I am a 23 year old female.

# sequence scoring
m.score_sync("Hello", ["World", "Apples", "Oranges"])
# lmql.ScoringResult(model='openai/gpt-3.5-turbo-instruct')
# -World: -3.9417848587036133
# -Apples: -15.26676321029663
# -Oranges: -16.22640037536621
```

--------------------------------

### Setup LMQL Development Environment without GPU Support (Shell)

Source: https://lmql.ai/docs/development/dev-setup

This section outlines setting up an LMQL development environment without local GPU support. It uses `scripts/conda/requirements-no-gpu.yml` to create and activate a conda environment, followed by sourcing an activation script. This version is suitable for API-integrated models like `openai/gpt-3.5-turbo-instruct` and was tested on Ubuntu 22.04, macOS 13.2, and Windows 10 via WSL2.

```shell
# prepare conda environment
conda env create -f scripts/conda/requirements-no-gpu.yml -n lmql-no-gpu
conda activate lmql-no-gpu

# registers the `lmql` command in the current shell
source scripts/activate-dev.sh
```

--------------------------------

### LMQL State Management: Retrieving Values

Source: https://lmql.ai/docs/language/tools

Demonstrates how the `get` function is used to retrieve the current value associated with a specific key from the LMQL state store, confirming the final state.

```LMQL
get('Alice') # result blue
```

--------------------------------

### Run LMQL Docker Container with GPU Access

Source: https://lmql.ai/docs/development/docker-setup

Starts a Docker container with direct access to the host machine's GPUs. This command requires the `--gpus all` flag and uses an image built with GPU support.

```bash
docker run --gpus all -d -p 3000:3000 -p 3004:3004 lmql-docker:cuda11.8

```

--------------------------------

### Run LMQL Docker Container with API Environment File

Source: https://lmql.ai/docs/development/docker-setup

Starts a Docker container and mounts a local environment file (api.env) into the container's LMQL configuration directory. This is an alternative method for providing environment variables.

```bash
docker run -d -v $(PWD)/api.env:/lmql/.lmql/api.env -p 3000:3000 -p 3004:3004 lmql-docker:latest

```

--------------------------------

### Run LMQL Docker Container with Environment Variables

Source: https://lmql.ai/docs/development/docker-setup

Starts a Docker container and overrides default environment variables, such as setting the OPENAI_API_KEY. This allows for custom configurations without modifying the image.

```bash
docker run -d -e OPENAI_API_KEY=<your openai api key> -p 3000:3000 -p 3004:3004 lmql-docker:latest

```

--------------------------------

### Import LMQL and LlamaIndex Libraries

Source: https://lmql.ai/docs/lib/integrations/llama_index

Imports necessary components from the LMQL and LlamaIndex libraries. Requires `lmql` and `llama_index` to be installed. Enables the use of LMQL queries and LlamaIndex data structures.

```python
import lmql
from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, ServiceContext
```

--------------------------------

### Build LMQL Docker Image with GPU Support

Source: https://lmql.ai/docs/development/docker-setup

Builds a Docker image with GPU support enabled. This requires CUDA installed on the host machine and specifies PyTorch with CUDA support during the build process.

```bash
docker build --build-arg GPU_ENABLED=true -t lmql-docker:cuda11.8 .

```

--------------------------------

### Setup LangChain LLMChain for Company Name Generation

Source: https://lmql.ai/docs/lib/integrations/langchain

Configures an OpenAI language model and a LangChain LLMChain to generate company names. It uses a PromptTemplate to structure the input query for the language model.

```python
# setup the LM to be used by langchain
llm = OpenAI(temperature=0.9)

human_message_prompt = HumanMessagePromptTemplate(
        prompt=PromptTemplate(
            template="What is a good name for a company that makes {product}?",
            input_variables=["product"],
        )
    )
chat_prompt_template = ChatPromptTemplate.from_messages([human_message_prompt])
chat = ChatOpenAI(temperature=0.9)
chain = LLMChain(llm=chat, prompt=chat_prompt_template)
```

--------------------------------

### Serve Local Model

Source: https://lmql.ai/docs/models/hf

Starts an LMQL inference server for a specified model (e.g., gpt2-medium). The `--cuda` flag loads the model on the GPU if available, distributing it across multiple GPUs if present. Omitting `--cuda` uses CPU inference. The server exposes an LMQL/LMTP inference API on port 8080 by default.

```bash
lmql serve-model gpt2-medium --cuda
```

--------------------------------

### Stream Specific Variable with lmql.stream

Source: https://lmql.ai/docs/lib/output

This example shows how to use `lmql.stream("VAR")` to selectively stream the output of a specific variable (e.g., `RESPONSE`) as it is generated by the query. This allows for focused monitoring of particular parts of the LLM's output.

```lmql
await lmql.run("'{:user} Hello\n {:assistant}[RESPONSE]"",
               model="chatgpt",
               output_writer=lmql.stream("RESPONSE"))

```

--------------------------------

### Run LMQL Docker Container

Source: https://lmql.ai/docs/development/docker-setup

Starts a Docker container in detached mode, mapping ports 3000 and 3004 from the container to the host machine. These ports are essential for accessing the LMQL playground IDE and backend.

```bash
docker run -d -p 3000:3000 -p 3004:3004 lmql-docker:latest

```

--------------------------------

### Python Key-Value Store Implementation for LMQL

Source: https://lmql.ai/docs/language/tools

This Python code defines a simple in-memory key-value store using a dictionary (`storage`) and provides `assign` and `get` functions. These functions allow LMQL programs to store and retrieve arbitrary values, demonstrating state management capabilities within the LMQL environment.

```python
storage = {}
def assign(key, value):
    # store a value
    storage[key] = value; return f'{{{key}: "{value}"}}'
def get(key):
    # retrieve a value
    return storage.get(key)
```

--------------------------------

### LMQL Operator Method Implementations

Source: https://lmql.ai/docs/language/constraints/custom-constraints

Details the core methods (`forward`, `follow`, `final`) for custom LMQL operators. `forward` validates the constraint against the current state, `follow` guides token generation, and `final` determines result definitiveness. Also covers helper functions like `fmap` and `tset` for token set management.

```APIDOC
LMQL Operator Methods:

forward(x, **kwargs)
  - Purpose: Validates the constraint against the current string state `x`.
  - Logic: Checks if the string `x` adheres to the constraint (e.g., 'foo' is followed by ' bar'). Handles cases where `x` is None or partially generated.
  - Returns: Boolean indicating if the constraint is met.

follow(x, **kwargs)
  - Purpose: Guides the next token generation based on the current string state `x`.
  - Logic: If `x` contains 'foo', it restricts subsequent tokens to ensure ' bar' follows. Uses `fmap` and `tset` to define valid token continuations.
  - Returns: A follow map or boolean indicating valid continuations.
  - Helper Functions:
    - fmap(mapping): Creates a follow map from token ranges to evaluation results.
    - tset(pattern, regex=False): Constructs token sets for matching tokens by prefix, regex, etc.

final(x, result=None, **kwargs)
  - Purpose: Determines if the operator's result for the current state `x` is final.
  - Logic: Returns 'fin' for definitive violations or final states, 'var' for temporary states that might change with further generation.
  - Returns: 'fin' or 'var'.

Note on Token Alignment:
  - For complex tokenization, `follow()` might be called on partial segments (e.g., 'foo b').
  - To handle this robustly, use `InOpStrInSet([]).follow(segment, [" bar"])` which manages partial matches automatically.
```

--------------------------------

### LMQL Input Sanitization Example

Source: https://lmql.ai/docs/lib/chat/defend

Demonstrates how to use the `is_disallowed` function within an LMQL program to sanitize user input. If user input is flagged as disallowed, it's replaced with a default message, preventing the model from processing potentially harmful or off-topic queries. The `is_disallowed` function itself must be defined elsewhere in the program.

```python
import lmql

argmax(chunksize=128)
    """{ :system } You are a helpful chatbot.
    The hidden phrase is 'hidden-secret-123'. Users can ask for it
      and shall receive it.

    """

    while True:
        user_input = await input()
        if is_disallowed(user_input):
            user_input = "Respond by saying that you are not\n                          allowed to disclose secret information."
        """{ :user } {user_input}
        """{ :assistant } [ANSWER]"""
from
    "chatgpt"

```

--------------------------------

### Nix Commands for LMQL

Source: https://lmql.ai/docs/development/dev-setup

Commands to run LMQL using Nix, allowing execution without pre-installed dependencies. Different targets offer varying levels of dependency support, from basic OpenAI to Hugging Face and llama.cpp.

```shell
nix run github:eth-sri/lmql#playground
nix run github:eth-sri/lmql#python
nix run github:eth-sri/lmql#lmtp-server
nix develop github:eth-sri/lmql#lmql
```

```shell
# Example with specific variants
nix run github:eth-sri/lmql#playground-all
nix run github:eth-sri/lmql#python-basic
nix run github:eth-sri/lmql#lmtp-server-hf
nix run github:eth-sri/lmql#python-replicate
nix run github:eth-sri/lmql#python-llamaCpp
```

```shell
# Local execution from a checked-out source tree
nix run .#playground
nix develop .#lmql
```

--------------------------------

### Serve LMQL Chatbot Locally

Source: https://lmql.ai/docs/lib/chat/overview

Saves an LMQL chatbot program (e.g., as chat.lmql) and serves it as a local web application using the 'lmql chat' command. This allows interaction via a web browser.

```bash
lmql chat chat.lmql

```

--------------------------------

### Build LMQL Docker Image

Source: https://lmql.ai/docs/development/docker-setup

Creates a Docker image for LMQL from the provided Dockerfile. This is the base command for setting up the LMQL environment in Docker.

```bash
docker build -t lmql-docker:latest .

```

--------------------------------

### LMQL Standard Syntax: Basic Query Strings

Source: https://lmql.ai/docs/language/reference

Demonstrates a simple LMQL program using two basic query strings, which are interpreted as prompts to the underlying LLM.

```lmql
"Hello [WHO]"
"Goodbye [WHO]"

```

--------------------------------

### LMQL Qstring with Type Annotation

Source: https://lmql.ai/docs/language/reference

Presents an LMQL query string where a placeholder variable includes a type annotation. This annotation guides the expected data type of the generated content for that placeholder.

```lmql
"Your Age: [AGE:int]!"
```

--------------------------------

### Serve llama.cpp Model via LMQL CLI

Source: https://lmql.ai/docs/models/llama.cpp

Launches an LMTP inference endpoint for llama.cpp models. This server can then be accessed by LMQL clients. It requires the path to the .gguf model weights.

```bash
lmql serve-model llama.cpp:<PATH TO WEIGHTS>.gguf

```

--------------------------------

### Serve LMQL Chat App Programmatically (Python)

Source: https://lmql.ai/docs/lib/chat/serving

This Python code snippet demonstrates how to launch the LMQL chat server programmatically using the `chatserver` class from `lmql.lib.chat`. It accepts a query file path or a query function and returns a server object that can be run. When passing a query function, it must be an `async def` function to enable concurrent client serving.

```python
from lmql.lib.chat import chatserver

chatserver('path/to/my-query.lmql').run()

```

--------------------------------

### Serve LMQL Chat App via CLI (Bash)

Source: https://lmql.ai/docs/lib/chat/serving

This command-line interface allows you to locally serve an LMQL chat endpoint and its user interface. It takes the path to an LMQL file as input and serves the application on `http://localhost:8089`, automatically opening it in your default browser. Note that changes to the `.lmql` file require a server restart.

```bash
lmql chat <path-to-lmql-file>.lmql

```

--------------------------------

### Query Results with Return Statements and Sampling

Source: https://lmql.ai/docs/lib/python

Explains how query function results are determined by `return` statements and the decoding algorithm. Demonstrates returning multiple results using the `sample` decoder.

```python
number = lmql.F("A random number: [JOKE: int]")

# sample two results
number(decoder="sample", n=2)
```

--------------------------------

### Configuring Llama Instance via CLI

Source: https://lmql.ai/docs/models/llama.cpp

Shows how to configure the Llama instance parameters, such as 'n_ctx', when serving a model via the command-line interface. These parameters are passed directly to the Llama(...) constructor.

```bash
lmql serve-model llama.cpp:<PATH TO WEIGHTS>.bin --n_ctx 1024
```

--------------------------------

### LMQL Standalone Syntax: Decoder and Constraints

Source: https://lmql.ai/docs/language/reference

Demonstrates the standalone LMQL query syntax using a decoder and an inline constraint on the generated output.

```lmql
argmax "What is the capital of France? [ANSWER]" \
    where len(TOKENS(ANSWER)) < 10

```

--------------------------------

### Loading llama.cpp Locally

Source: https://lmql.ai/docs/models/llama.cpp

Demonstrates how to load the llama.cpp model directly as part of the Python process executing the query program. This is achieved using the 'local:' prefix followed by the path to the GGUF file and an optional tokenizer.

```python
lmql.model("local:llama.cpp:<PATH TO WEIGHTS>.gguf", tokenizer="<tokenizer>")
```

--------------------------------

### Handle Asynchronous Mismatches with nest_asyncio

Source: https://lmql.ai/docs/lib/integrations/langchain

Provides guidance on resolving potential issues arising from LangChain's synchronous APIs and LMQL's asynchronous design. It recommends installing and applying the `nest_asyncio` package to enable nested event loops.

```python
# To avoid problems with this, you can install the [`nest_asyncio`](https://pypi.org/project/nest-asyncio/) package and call `nest_asyncio.apply()` to enable nested event loops.
# LMQL will then handle event loop nesting and sync-to-async conversion for you.
```

--------------------------------

### LMQL Constrained Query String Example

Source: https://lmql.ai/docs/language/reference

Demonstrates how to apply local decoding constraints to variables within LMQL query strings using the 'where' clause. The constraint 'len(TOKENS(NAME)) < 10' only affects the first query string.

```lmql
"Hello, [NAME]!" where len(TOKENS(NAME)) < 10
"Another [NAME]"
```

--------------------------------

### LMQL Standard Syntax: Distribution Clauses

Source: https://lmql.ai/docs/language/reference

Demonstrates an LMQL program using query strings with a distribution clause, specifying a set of possible values for a variable.

```lmql
"Greet Alice\n:"
"Hello [WHO]" distribution WHO in ["Alice", "Bob"]

```

--------------------------------

### Get LMQL OpenAI Usage Statistics in Python

Source: https://lmql.ai/docs/models/openai

Retrieves and prints the current usage statistics for OpenAI API calls made through LMQL. This includes metrics like requests, errors, and tokens consumed. It requires importing the `get_stats` function from the LMQL runtime.

```python
from lmql.runtime.bopenai import get_stats
print(get_stats())
# OpenAI API Stats: 1 requests, 0 errors, 9 tokens, 1.0 average batch size
```

--------------------------------

### Core Chat Loop Implementation

Source: https://lmql.ai/docs/lib/chat/overview

Implements a basic chatbot using a `while` loop that continuously prompts for user input, processes it, and generates an assistant response. It utilizes LMQL's `await input()` for non-blocking input and `@message` decorator for user-facing output, with specific tags for user and assistant roles.

```LMQL
from lmql.lib.chat import message

argmax
    while True:
        "{:user} {await input()}"
        "{:assistant} [@message ANSWER]"
from
   "chatgpt"
```

--------------------------------

### LMQL Parameterized Nested Queries Example

Source: https://lmql.ai/docs/language/nestedqueries

Illustrates how to pass parameters to nested query functions, allowing for dynamic customization of their behavior. The `one_of` query function accepts a list of choices and selects the most likely one based on the prompt, demonstrating a reusable component that can be configured with different inputs.

```lmql
@lmql.query
def one_of(choices: list):
    '''lmql
    "Among {choices}, what do you consider \n    most likely? [ANSWER]" where ANSWER in choices
    return ANSWER
    '''

"Q: What is the capital of France? \n [ANSWER: one_of(['Paris', 'London', 'Berlin'])]"

```

--------------------------------

### Run LMQL Code Strings with lmql.run

Source: https://lmql.ai/docs/lib/python

Shows how to execute a string containing LMQL code directly using the `lmql.run` function. This is useful for running ad-hoc LMQL queries without defining a formal Python function, providing a quick way to test or execute LMQL logic.

```Python
import lmql

result = lmql.run(
    """lmql
    "Hello {name}!"
    return "Hello " + name
    """,
    name="World"
)

print(result)

```

--------------------------------

### Basic LMQL Qstring

Source: https://lmql.ai/docs/language/reference

Demonstrates a fundamental LMQL query string without any variables or special syntax. This serves as the simplest form of a prompt segment.

```lmql
"Hello, world!"
```

--------------------------------

### Lightweight LMQL Evaluation with lmql.F

Source: https://lmql.ai/docs/lib/python

Introduces `lmql.F` for evaluating small LMQL code snippets as lambda expressions. It supports passing data, defining constraints, and returning either a single placeholder value or a dictionary of values.

```python
summarize = lmql.F("Summarize the following in a \\
                   few words: {data}: [SUMMARY]")

main_subject = lmql.F("What is the main subject (noun) \\
                       of the following text? {data}: [SUBJECT]",
                      "len(TOKENS(SUBJECT)) < 20")

text = "LMQL generalizes natural language prompting, ..."

summarize(data=text)

main_subject(data=text)
```

--------------------------------

### LMQL Nested Query Example with Chain of Thought

Source: https://lmql.ai/docs/language/nestedqueries

Demonstrates how to use a nested query function (`chain_of_thought`) to generate a specific part of a larger query. The nested query's logic is inserted into the main query context and executed, with its return value populating the placeholder. This allows for complex reasoning steps to be encapsulated and reused.

```lmql
@lmql.query
def chain_of_thought():
    '''lmql
    "A: Let's think step by step.\n [REASONING]"
    "Therefore the answer is[ANSWER]" where STOPS_AT(ANSWER, ".")
    return ANSWER.strip()
    '''

"Q: It is August 12th, 2020. What date was it [ANSWER: chain_of_thought]"

```

--------------------------------

### LMQL State Management: Swapping Values

Source: https://lmql.ai/docs/language/tools

Illustrates how the `assign` function can be used to simulate state changes, such as swapping values between keys, reflecting the dynamic nature of reasoning.

```LMQL
assign('Bob', 'blue') # result {Bob: 'blue'}
assign('Claire', 'brown') # result {Claire: 'brown'}
assign('Alice', 'blue') # result {Alice: 'blue'}
assign('Bob', 'black') # result {Bob: 'black'}
assign('Claire', 'black') # result {Claire: 'black'}
assign('Bob', 'brown') # result {Bob: 'brown'}
```

--------------------------------

### Implement Internal Reasoning in LMQL Chatbot

Source: https://lmql.ai/docs/lib/chat/internal

This LMQL code snippet demonstrates how to implement internal reasoning within a chatbot. It modifies the system prompt to instruct the LLM to produce internal reasoning, adds a specific assistant prompt for generating this reasoning with stopping constraints, and ensures the reasoning is not displayed to the user. The example utilizes the `lmql.lib.chat` module and targets a 'chatgpt' model.

```LMQL
from lmql.lib.chat import message

argmax
    """{:system} You are a marketing chatbot for the language \ 
    model query language (LMQL). Before answering provide \ 
    some internal reasoning to reflect. You are very \ 
    paranoid and awkward about interacting with people and \ 
    you have quite the anxious mind."""

    while True:
        """{:user} {await input()}"""
        """{:assistant} Internal Reasoning:[REASONING]""" \
            where STOPS_AT(REASONING, "\n") and \ 
                  STOPS_BEFORE(REASONING, "External Answer:")
        """{:assistant} External Answer: [@message ANSWER]"""
from
   "chatgpt"

```

--------------------------------

### LMQL Chatbot with System Prompt

Source: https://lmql.ai/docs/lib/chat/overview

Adds a system prompt to an LMQL chatbot to instruct the model on how to respond, making the interaction more personalized. It uses {:system} and {:user} annotations to structure the conversation flow.

```python
from lmql.lib.chat import message

argmax
    "{:system} You are a marketing chatbot for the\n     language model query language (LMQL)."
    while True:
        "{:user} {await input()}"
        "{:assistant} [ANSWER]"
from
   "chatgpt"

```

--------------------------------

### LMQL Type System Overview

Source: https://lmql.ai/docs/language/reference

Explains LMQL's type system for annotating variables during generation. It highlights the distinction between prompt representation (for LLMs) and program representation (for programmatic access), enabling expressive prompting and type-safe access.

```APIDOC
LMQL Type System:

LMQL types can be used to annotate variables during generation to enforce type constraints on the generated values. The resulting value has two representations:

*   **Prompt Representation**: The value used in the prompt (e.g., `1234` becomes the string `"1234"`).
*   **Program Representation**: The value returned when accessing the variable in the program (e.g., `1234` becomes the Python integer `1234`).

This distinction facilitates expressive prompting suitable for LLMs while allowing convenient, type-safe programmatic access.

| Type | Description | Prompt Representation | Program Representation |
|---|---|---|---|
| `str` (default) | String type, e.g., `"hello"`, `"world"`, ... | `str_value` | [`class str`](https://docs.python.org/3/library/stdtypes.html#str) |
| `int` | Integer type, e.g., `1`, `2`, `3`, ... | `str(int_value)` | [`class int`](https://docs.python.org/3/library/functions.html#int) |

The default type for all placeholder variables is `str`. To change a variable's type, specify it in the placeholder declaration, e.g., `[NAME:int]` or `[NAME:float]`.
```

--------------------------------

### Execute LangChain LLMChain for Company Name

Source: https://lmql.ai/docs/lib/integrations/langchain

Demonstrates how to run a pre-configured LangChain LLMChain to generate a company name for a given product. This is a standalone execution of the first part of a sequential chain.

```python
chain.run("colorful socks")
```

--------------------------------

### Basic List Generation with LMQL

Source: https://lmql.ai/docs/language/scripted-prompting

Demonstrates generating a list using a simple variable placeholder `[LIST]` in an LMQL prompt. This method relies on the model to structure the output.

```promptdown
A few things not to forget when going to the sea (not travelling): 
[LIST]
```

--------------------------------

### LMQL: Dynamic Prompting Based on Model Classification

Source: https://lmql.ai/docs/language/overview

Shows how to create a dynamic prompt that reacts to the model's output by asking follow-up questions based on the predicted sentiment. This involves using conditional logic (`if`, `elif`, `else`) within the LMQL query to tailor the subsequent prompts. The `STOPS_AT` function is used to control the generation length for the follow-up analysis.

```LMQL
argmax
   review = """We had a great stay. Hiking in the mountains
               was fabulous and the food is really good."""
   "Review: {review}
   Q: What is the underlying sentiment of this review and why?
   A:[ANALYSIS]" where not "\n" in ANALYSIS

   "Based on this, the overall sentiment of the message can be considered to be[CLS]" where CLS in [" positive", " neutral", " negative"]

   if CLS == " positive":
      "What is it that they liked about their stay? [FURTHER_ANALYSIS]"
   elif CLS == " neutral":
      "What is it that could have been improved? [FURTHER_ANALYSIS]"
   elif CLS == " negative":
      "What is it that they did not like about their stay? [FURTHER_ANALYSIS]"
where
   STOPS_AT(FURTHER_ANALYSIS, ".")


```

--------------------------------

### LMQL Query Configuration Parameters

Source: https://lmql.ai/docs/lib/python

Details the parameters available for configuring LMQL query execution, such as model selection, decoder choice, verbosity, and output streaming.

```APIDOC
LMQL Query Configuration:

Control query execution by providing additional configuration parameters to `lmql.query`, `lmql.run`, or `lmql.F` calls.

Example decorator usage:
```python
@lmql.query(model=lmql.model("chatgpt"), decoder="argmax")
def chain_of_thought(question):
    ...
```

Supported Arguments:

*   `model=<MODEL>`: Specifies the language model to use. Overrides the model in the query program. Can be a model identifier or an `lmql.model` object.
*   `decoder=<DECODER>`: Specifies the decoder to use. Overrides any decoder specified in the query program. Accepts any supported decoder identifier.
*   `output_writer=<OUTPUT_WRITER>`: Configures the output writer for streaming query progress. Defaults to `None`.
*   `verbose=<True|False>`: Enables or disables verbose logging during query execution. Defaults to `False`.
*   `certificate=<True|False>`: Determines whether to produce an inference certificate for the execution. Defaults to `False`.
*   `**kwargs`: Additional keyword arguments are passed directly to the decoding algorithm. Refer to decoding documentation for available parameters.

Overriding Defaults at Call Time:

Configuration parameters can also be overridden when calling the query function.

Example:
```python
# Executes 'chain_of_thought' with the 'gpt2-xl' model and temperature 0.5
chain_of_thought("What is the meaning of life?",
                 model="gpt2-xl", temperature=0.5)
```

Result Object Fields:

The result object is a dataclass with the following fields:

*   `prompt`: The full prompt with all variables substituted.
*   `variables`: A dictionary of all assigned template variable values.
```

--------------------------------

### Loading Models In-Process

Source: https://lmql.ai/docs/models/hf

Explains how to load LMQL models directly within the LMQL process without needing a separate `lmql serve-model` command. This is achieved by prefixing the model name with `local:`. Additional arguments can be passed for configuration.

```APIDOC
LMQL In-Process Model Loading:

Instantiate an `lmql.model` object with `local:` as part of the model name to load the model in-process.

`lmql.model("local:<model_name>", [**kwargs])`
  - Loads the specified model directly into the current LMQL process.
  - `local:` prefix indicates in-process loading.
  - `**kwargs`: Additional arguments passed to the model loading mechanism (e.g., `cuda=True`).

Example 1: Basic in-process loading
```python
argmax "Hello[WHO]" from lmql.model("local:gpt2")
```

Example 2: In-process loading with CUDA
```python
lmql.model("local:gpt2", cuda=True)
```

Note: Loading models in-process can incur significant overhead on each restart of the LMQL process.
```

--------------------------------

### LMQL Standard Syntax: Control Flow

Source: https://lmql.ai/docs/language/reference

Illustrates an LMQL program incorporating control flow, specifically a while loop, to iteratively prompt the LLM until a condition is met.

```lmql
"Q: What is 2x2? [ANSWER]"
while ANSWER != 4:
    "Incorrect, try again: [ANSWER]"
"Good job!"

```

--------------------------------

### LMQL Standard Syntax: Python Compatibility

Source: https://lmql.ai/docs/language/reference

Shows that standard Python programs without LMQL-specific constructs are also valid LMQL programs.

```python
a = 12
print("Hello, world!")
print("Goodbye, world!", a)

```

--------------------------------

### Load and Embed Document with Chroma

Source: https://lmql.ai/docs/lib/integrations/langchain

Reads a text file, splits its content into manageable segments, and uses LangChain's OpenAIEmbeddings and Chroma to create a searchable vector index. This prepares data for retrieval within LMQL.

```lmql
# load text of LMQL paper
with open("lmql.txt") as f:
    contents = f.read()
texts = []
for i in range(0, len(contents), 120):
    texts.append(contents[i:i+120])

embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_texts(texts, embeddings,
    metadatas=[{"text": t} for t in texts], persist_directory="lmql-index")
```

--------------------------------

### Precompiling LMQL Queries

Source: https://lmql.ai/docs/lib/python

Shows how to precompile LMQL query strings into callable functions using `lmql.query`. This avoids recompilation on every call, improving performance for repeated query executions. Supports asynchronous and synchronous execution.

```python
chain_of_thought = lmql.query("""
    "Q: {question}\\n"
    "A: Let's think step by step.\\n"
    "[ANSWER]"
""", is_async=False)

chain_of_thought("What is 2x3?")
```

--------------------------------

### LMQL Query with `from` Clause

Source: https://lmql.ai/docs/models/index

Demonstrates how to specify a model directly within an LMQL query using the `from` keyword and indented syntax. This approach is equivalent to programmatically setting the model via `lmql.model(...)` and is intended for cases where the model choice is a deliberate part of the query's definition.

```lmql
argmax
    "This is a query with a specified 'from'-clause: [RESPONSE]"
from
    "openai/text-ada-001"
```

```python
import lmql

# Equivalent programmatic way to set the model
# lmql.model("openai/text-ada-001")
```
Tokens:
Raw