Inferable is open source (MIT license) and can be self-hosted on your own infrastructure. This guide will walk you through setting up a self-hosted instance of Inferable using Docker Compose.

Requirements:

  • Docker
  • Docker Compose
1

Download Docker Compose definition

We provide a Docker Compose definition within the Inferable repository.

Use curl to download the file:

curl -O https://raw.githubusercontent.com/inferablehq/inferable/refs/heads/main/control-plane/docker-compose.yml
2

Define environment variables

When running using docker-compose.yml, most environment variables have defaults but the following need to be provided explicitly:

  • MANAGEMENT_API_SECRET (This can be generated using openssl rand -hex 32, take note of the value)
  • ANTHROPIC_API_KEY (With access to claude-3-5-sonnet / claude-3-haiku - our default reasoning models)
  • COHERE_API_KEY (With access to embed-english-v3 - our default embedding model)

You can alternatively (or in addition) use AWS Bedrock as the model provider. This will require you to provide AWS credentials. Please see the configuration section.

export MANAGEMENT_API_SECRET=$(openssl rand -hex 32)
export ANTHROPIC_API_KEY=XXXXXXX
export COHERE_API_KEY=XXXXXXX
3

Start Docker Compose

docker-compose up

This will start The Inferable control plane at http://localhost:4000 and the following auxiliary resources:

  • PostgreSQL database with pgvector
  • Redis for caching
  • ElasticMQ for a local SQS-compatible queue implementation
4

Manage via the CLI

The Inferable CLI can be used to manage your Inferable instance. Export INFERABLE_API_ENDPOINT to target your instance, rather than Inferable Cloud.

npm install -g @inferable/cli
export INFERABLE_API_ENDPOINT=http://localhost:4000

# If running in headless mode, you will be prompted for the `MANAGEMENT_API_SECRET` generated above
inf auth login

# Create a new cluster
inf clusters create

# Create a cluster key (for registering machines)
inf auth keys create "MY_KEY"
5

Register functions

const client = new Inferable({
  apiSecret: "", // Generated in the previous step
  baseUrl: "http://localhost:4000",
});

client.default.register({
  //....
});

​
Configuration

The Inferable control-plane accepts the following environment variables:

VariableDescriptionRequired
MANAGEMENT_API_SECRETA deployment secret for the management API.Yes (If running in headless mode)
ANTHROPIC_API_KEYAPI key for the Anthropic API.Yes (If Bedrock is not available)
COHERE_API_KEYAPI key for the Cohere API.Yes (If Bedrock is not available)
BEDROCK_AVAILABLEIf set to true, Bedrock will be used as the model provider (You must also provide AWS credentials).No
DATABASE_URLThe connection string for the PostgreSQL database.Yes
DATABASE_SSL_DISABLEDIf set to true, SSL will be disabled for the PostgreSQL database.No
REDIS_URLThe connection string for the Redis instance.Yes
SQS_RUN_PROCESS_QUEUE_URLThe connection string for the SQS queue.Yes
SQS_RUN_GENERATE_NAME_QUEUE_URLThe connection string for the SQS queue.Yes
SQS_CUSTOMER_TELEMETRY_QUEUE_URLThe connection string for the SQS queue.Yes
SQS_EXTERNAL_TOOL_CALL_QUEUE_URLThe connection string for the SQS queue.Yes
APP_ORIGINThe origin of the Inferable Web UI for CORS.No
JWKS_URLThe URL of the Clerk JWKS url.No

​
Headless Mode

This self-hosting guide will produce a “headless” instance of Inferable, meaning the the control plane will be running but not the Web UI. Web UI is not essential to interacting with the control-plane.

The Web UI relies on Clerk for authentication with Cluster being associated with a Clerk Organization. In headless mode, a MANAGEMENT_API_SECRET is used instead.

The MANAGEMENT_API_SECRET should only be used for managing clusters. You can use it to create API Keys which are used by the SDK to register functions.

inf auth keys create "MY_KEY"

The Web UI is also open source. We will be adding a guide for self-hosting the Web UI in the future.

Was this page helpful?

Architecture Deep DiveShould I use Inferable?