Connect AI models to apps

Connect AI models to apps

Point LiteLLM, Kestra, Langflow or your own code at an AI model with three variables.

An AI model answers on an OpenAI-compatible URL, so any app that speaks the OpenAI API can use it. The AI-ready templates all read the same three variables:

VariableValue
MODEL_URLThe model's base URL, ending in /v1: https://api.stackblaze.cloud/api/inference/<id>/v1
MODEL_API_KEYThe model's API key (dashboard, or stackblaze models key <name>).
MODEL_NAMEThe model id, for example meta-llama/Llama-3.1-8B-Instruct.

Set them under the app's Variables. Saving a variable restarts the app.

Which to use

SetupConnectWhy
One model, one appThe app straight to the modelNothing in between.
Several models or several appsApps to LiteLLM, LiteLLM to the modelsOne URL for every model, a key per app, usage per key and model.

LiteLLM

The LiteLLM template is a gateway in front of your models. Apps call one URL with their own key.

  1. 1

    Register the first model

    Set MODEL_URL, MODEL_API_KEY and MODEL_NAME. After the restart the model is listed under Models. Optional: MODEL_ALIAS is the name clients send; empty means MODEL_NAME.
  2. 2

    Add more models

    In the LiteLLM admin UI, add each further model under Models with its URL, key and name.
  3. 3

    Give each app a key

    Create a virtual key per app under Virtual Keys. Apps then use base URL https://<litellm app URL>/v1, their virtual key, and the model name.

LiteLLM tracks spend per key and per model, so you can see which app uses what.

Kestra

In the Kestra template, set MODEL_URL to the model's URL or to LiteLLM's /v1, plus MODEL_API_KEY and MODEL_NAME. Flows read them as {{ envs.model_url }}, {{ envs.model_api_key }} and {{ envs.model_name }}.

This flow sends a question to the model and logs the answer. Kestra 2 uses the toJson filter and the fromJson() function:

ask-model.yaml
id: ask-model
namespace: demo
inputs:
  - id: question
    type: STRING
    defaults: "In one sentence, what is quantum annealing?"
tasks:
  - id: ask
    type: io.kestra.plugin.core.http.Request
    uri: "{{ envs.model_url }}/chat/completions"
    method: POST
    contentType: application/json
    headers:
      Authorization: "Bearer {{ envs.model_api_key }}"
    body: |
      {"model": "{{ envs.model_name }}", "max_tokens": 800,
       "messages": [{"role": "user", "content": {{ inputs.question | toJson }}}]}
  - id: answer
    type: io.kestra.plugin.core.log.Log
    message: "{{ fromJson(outputs.ask.body).choices[0].message.content | trim }}"

Kestra's built-in AI Copilot is separate: it uses Google Gemini, not the MODEL_* model.

Langflow

In the Langflow template, MODEL_URL, MODEL_API_KEY and MODEL_NAME become Global Variables at start. Langflow allows environment variables only in secret fields, so in an OpenAI component:

FieldValue
OpenAI API BaseType the base URL (the value of MODEL_URL).
API KeyPick MODEL_API_KEY from the globe icon.
Model NameType the model name (the value of MODEL_NAME).

Note

Langflow only sends a key from the environment to hosts listed in LANGFLOW_PROVIDER_CREDENTIAL_ALLOWED_HOSTS. The Langflow template adds the host of MODEL_URL to that list automatically. To call any other host with MODEL_API_KEY, add it to the variable, separated by commas. A Langflow app deployed before this was added needs the variable set by hand.

vLLM gateway

The vLLM template is a small gateway for one model. It gives your apps a stable name and their own key, so the model can change without touching them.

SideSettings
GatewaySet MODEL_URL, MODEL_API_KEY and MODEL_NAME.
ClientsBase URL https://<app URL>/v1, the key generated in API_KEY, and the model name in MODEL_ALIAS (default model).

Any app or code

Use any OpenAI SDK. Set the base URL to the model's URL or to LiteLLM's /v1, with the matching key and model name.

Python

ask.py
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MODEL_URL"],     # ends in /v1
    api_key=os.environ["MODEL_API_KEY"],
)

reply = client.chat.completions.create(
    model=os.environ["MODEL_NAME"],
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

curl

terminal
curl "$MODEL_URL/chat/completions" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL_NAME\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"

Tip

An on-demand model sleeps when idle, and the first request after that waits while it wakes. Give your client a timeout of a few minutes and retry on 503. See On demand and always on.