Kafka

Managed Apache Kafka: a distributed event-streaming platform with single-broker or multi-broker high availability, a built-in catalog of source and sink connectors, topic and consumer tooling, and Prometheus metrics.

Kafka is not a traditional database. It is a distributed, append-only event log. Producers write messages to named topics, each topic is split into partitions for parallelism, and consumers read at their own pace. Messages are retained on disk so multiple consumers can replay the same stream independently. StackBlaze runs Kafka with the Strimzi operator in KRaft mode (no ZooKeeper) and adds a managed layer on top: a connector catalog, topic and consumer-group tooling, a message viewer, and metrics.

Creating a Kafka cluster

From the dashboard, click New service, then Database, then Kafka. Pick a plan, a storage size, and whether you want a single broker (Standard) or a multi-broker cluster (high availability). Connecting the cluster to a service injects KAFKA_BOOTSTRAP_SERVERS into that service, so there is nothing to hardcode.

Connecting

Kafka clients connect through bootstrap servers. A client contacts a bootstrap address, discovers the full set of brokers, then talks directly to whichever broker hosts the partitions it needs. The internal plaintext listener is on port 9092, and a TLS listener is available on port 9093.

[service-name].internal:9092

Producing and consuming

Any standard Kafka client works. This example uses kafkajs for Node.js, reading brokers from the injected variable:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: process.env.KAFKA_BOOTSTRAP_SERVERS.split(','),
})

async function produce() {
  const producer = kafka.producer()
  await producer.connect()
  await producer.send({
    topic: 'events',
    messages: [{ value: 'hello kafka' }],
  })
  await producer.disconnect()
}

async function consume() {
  const consumer = kafka.consumer({ groupId: 'my-group' })
  await consumer.connect()
  await consumer.subscribe({ topic: 'events', fromBeginning: true })
  await consumer.run({
    eachMessage: async ({ message }) => console.log(message.value.toString()),
  })
}

produce().then(consume)

Standard vs high availability

You choose the topology when you create the cluster. Both modes run in KRaft mode, where each broker is a combined controller and broker (there is no separate ZooKeeper to operate).

Standard

A single broker with a replication factor of 1 and min.insync.replicas=1. It is the simplest, lowest-cost option and a good fit for development and lower-stakes workloads. There is no replication, so a broker restart briefly interrupts the stream.

High availability

Multiple brokers, each on its own node with its own persistent volume. Topics use a replication factor greater than 1 (3 by default), so every partition has a leader plus in-sync replicas on other brokers. Writes are acknowledged once they reach the minimum in-sync replicas (min.insync.replicas=2 by default), which protects against data loss. If the broker holding a partition leader fails, Kafka elects a new leader from the in-sync replicas automatically, and producers and consumers continue without intervention.

Scaling from Standard to HA raises the broker count and replication factor; StackBlaze performs a rolling change. The internal topics that back consumer offsets and transactions are replicated to match.

Kafka Connect: source and sink connectors

Most data does not start or end in Kafka. Kafka Connect moves it in and out without writing custom producers or consumers. Source connectors stream changes from another system into topics, and sink connectors stream topic records out to another system. StackBlaze ships a curated catalog with the connector plugins pre-baked into the runtime image, so there is nothing to install.

Connector catalog

Zero-config setup

Each connector is paired with a sibling add-on in your project. When you pick, for example, the Postgres CDC source and point it at one of your PostgreSQL add-ons, StackBlaze auto-fills the host, port, database, and credentials from that add-on. The S3 sink auto-fills the endpoint and access keys from a paired RustFS add-on. You only provide the connector-specific fields: the topic prefix or topic list, and which tables or collections to include. A test step validates the connection before the connector is created.

A common pattern is change data capture: a Debezium source streams every change from your Postgres add-on into topics, and a sink fans those records out to analytics (ClickHouse), to a search index, or to object storage for archival, all from the same stream.

Topics, consumer groups, and message inspection

The Kafka add-on page is a full management console, so you rarely need a separate desktop tool.

Metrics and monitoring

Broker and connector metrics are exported from JMX to Prometheus automatically. You get broker health, per-topic message and partition counts, and per-partition consumer lag, plus connector throughput and error rates for both source and sink tasks. Lag and failed connector tasks surface directly on the topology view so problems are easy to spot.

Security

Traffic inside your project uses the private network. A TLS listener is available on port 9093, and the Strimzi user operator manages SCRAM credentials for authenticated clients. You can also drive topics and messages from the chat assistant, which talks to your cluster over an authenticated connection.

Backups and retention

Kafka retains messages on disk per each topic's retention settings, so consumers can replay history within that window. The persistent volumes backing the brokers follow the standard platform backup policy. For schedules and restores, see /docs/databases/backups.