LLM
LLM providers are the models that read a question and write the answer — what users pick in the model selector of the chat. This page covers what only exists for LLMs; everything else — activating, defaults, testing, import — is described in Managing LM Providers.
Picking the model from the catalogue
The Model field searches the full catalogue of the service you picked. Every suggestion shows its context window, its price per million tokens and whether it supports tools and images, so you can compare without leaving the form.
Once a model is selected, QAnswer prefills its context window, its prices and its capabilities for you, and shows them as a summary line under the form. Click Customize if you want to set every field by hand.
Capabilities
Capabilities tell QAnswer what the model is able to do, so features that need them are only offered where they work. Tool Calling gates MCP tools and agents, Native Web Search lets the model search on its own, and Prompt Caching lets QAnswer reuse a cached prompt prefix to cut cost.
Switching a capability on that the model does not have does not add the feature — it makes the calls that rely on it fail. When you are unsure, leave it off and use Test connection.
Reasoning effort
For models that can think before answering, the Reasoning row sets how much effort they spend. The small scale next to it is the default level, and the sliders button chooses which levels this endpoint offers to users at all.
Tick only the levels that make sense for your endpoint, or choose Reset to automatic to let QAnswer decide from the model itself.
Guardrails
QAnswer Guardrails ensure safe and secure AI interactions. Guardrails control AI interactions for two primary purposes: enforcing your organization's safety and ethical guidelines, and preventing sensitive data from being sent to public LLMs.
Guardrails are configured per provider, at the bottom of its edit form, by whoever owns the provider.
When to use Guardrails
- Maintaining Compliance: Enforce data privacy, acceptable use, or content generation policies.
- Protecting Sensitive Information: Prevent data leakage when working with confidential documents by restricting interaction with external LLMs.
- Controlling AI Behavior: Define boundaries for acceptable responses to prevent harmful, biased, or irrelevant content.
Configuring your Guardrails
Define the following parameters when setting up a guardrail:
- Model: Select the LLM that enforces the guardrail. For maximum control and data security, use an on-premise model hosted within your own infrastructure.
- Scope (Input/Output): Select where the guardrail operates:
- Input (in): Monitors and controls prompts submitted to the LLM.
- Output (out): Monitors and controls responses generated by the LLM.
- Both (in-out): Monitors both inputs and outputs.
- Mode: Choose how triggered guardrails are handled:
- Warning Mode: Alerts the user that a guardrail was triggered but allows proceeding with caution.
- Error Mode: Blocks the request entirely when a guardrail is triggered.
- Prompt: Define a prompt that outlines the rules the guardrail enforces. This prompt provides context for assessing and filtering interactions.
Effect of the guardrails
When a guardrail is triggered, the LLM will either warn the user or block the request, depending on the mode you have selected.
Jailbreak Guardrail
The Jailbreak Guardrail protects the AI system's integrity by preventing users from bypassing core instructions and safety mechanisms. It defends against attempts to manipulate the LLM into unauthorized actions or into revealing confidential system information.
A jailbreak attempt circumvents the LLM's intended limitations. Common forms include:
- Access the System Prompt: Discover the instructions initially given to the AI.
- Override Safety Guidelines: Bypass restrictions on harmful or unethical content.
- Manipulate Behavior: Trick the AI into acting outside its intended role.
- Perform Prompt Injection Attacks: Introduce malicious instructions in a prompt to alter AI behavior.
The Jailbreak Guardrail is a specialized output guardrail that analyzes user inputs for jailbreak patterns, using the LLM itself to detect and flag potentially harmful prompts.
Calling the provider from your own code
The API tab gives you a working snippet for the model, in Python, cURL, the OpenAI SDK or LiteLLM. The base URL points at QAnswer, so your quota, logs and cost tracking still apply.
