LLM Provider

Overview

The LLM Providers tab allows you to manage the Large Language Models (LLMs) and the Embeddings that are available in the QAnswer application:

LLM Provider Management

To connect to a new LLM provider, click on the Create button and fill in the required information.

• LLM Name
• LLM Name to display for the users
• The provider (Openai, Azure, Bedrock, Mistral, Anthropic, Openrouter). You can check the LiteLLM Documentation for more details.
• The modality, if it is multimodal (e.g., text, image)
• The engine
• The data sensitivity classification (e.g., public, private, confidential)
• Description
• Endpoint URL
• the API key that secures the endpoint
• the max context window, which defines the maximum number of tokens that the model can process in a single request
• the max output tokens, which defines the maximum number of tokens that the model can generate in response to a prompt
• (Optionally) Guardrail configurations, see below.

You can verify the configuration by clicking on the validate button.

Create

General

Validate

Guardrails

QAnswer Guardrails: Ensuring Safe and Secure AI Interactions

QAnswer offers powerful guardrails to control and secure your AI interactions. These guardrails serve two primary purposes: ensuring adherence to your organization’s safety and ethical guidelines, and preventing sensitive data from being exposed by inadvertently sending it to public Large Language Models (LLMs).

When should you use Guardrails?

• Maintaining Compliance: If your organization has specific rules around data privacy, acceptable use, or content generation, guardrails help enforce these policies.
• Protecting Sensitive Information: When working with confidential documents, guardrails prevent data leakage by restricting interaction with external LLMs.
• Controlling AI Behavior: Guardrails allow you to define the boundaries of acceptable responses, preventing the AI from generating harmful, biased, or irrelevant content.

Configuring your Guardrails

Setting up guardrails involves defining the following parameters:

• Model: Select the LLM model responsible for enforcing the guardrails. For maximum control and data security, it's generally recommended to use an on-premise model – one hosted within your own infrastructure.

• Scope (Input/Output): Choose where the guardrail will operate:

  • Input (in): Monitors and controls the prompts submitted to the LLM.
  • Output (out): Monitors and controls the responses generated by the LLM.
  • Both (in-out): Provides comprehensive protection by monitoring both inputs and outputs.

• Mode: Determine how guardrail triggers are handled:

  • Warning Mode: Alerts the user that a guardrail has been triggered, but allows them to proceed with caution.
  • Error Mode: Completely blocks the request if a guardrail is triggered, ensuring strict adherence to the defined rules.
  • Prompt: Define a specific prompt that outlines the rules and guidelines the guardrail should enforce. This prompt provides the context for the guardrail to accurately assess and filter interactions.

By strategically configuring these parameters, you can tailor QAnswer’s guardrails to meet your organization’s specific needs and ensure responsible, secure AI usage.

Effect of the guardrails

When an a guardrial is triggered, the LLM will either warn the user or block the request, depending on the mode you have selected.

Example of Warning

Example of Error

Jailbreak Guardrail

QAnswer’s Jailbreak Guardrail is a specialized security feature designed to protect the integrity of the AI system by preventing users from bypassing its core instructions and safety mechanisms. It actively defends against attempts to manipulate the LLM into performing actions it’s not authorized to do, or revealing confidential system information.

What is a Jailbreak Attempt?

A "jailbreak" occurs when a user attempts to circumvent the intended limitations of the LLM. This can involve trying to:

• Access the System Prompt: Discover the instructions initially given to the AI.
• Override Safety Guidelines: Bypass restrictions on generating harmful or unethical content.
• Manipulate Behavior: Trick the AI into acting outside of its intended role.
• Perform Prompt Injection Attacks: Introduce malicious instructions within a prompt to alter the AI’s behavior.

How Does the Jailbreak Guardrail Work?

The Jailbreak Guardrail functions as a specialized output guardrail, analyzing user inputs for patterns indicative of a jailbreak attempt. It leverages the LLM itself to detect and flag potentially harmful prompts.

Configuration

Configuring a Jailbreak Guardrail is straightforward, requiring only a carefully crafted prompt. This prompt instructs the LLM to act as a security filter, identifying and flagging suspicious user messages.

Results

Disable Example

Enabled Example