feat(hosting): default self-hosted realtime streams to v2 (s2-lite)

docs/self-hosting/docker.mdx

@@ -354,6 +354,27 @@ EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2
 
 This only affects new runs; existing runs continue to read from wherever their events were originally stored.
 
+## Realtime streams
+
+Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` service — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. It stores stream data in a persistent volume and is preconfigured in the webapp compose file, so no setup is required.
+
+To fall back to the Redis-backed **v1** streams, set on the webapp in your `.env`:
+
+```bash
+REALTIME_STREAMS_DEFAULT_VERSION=v1
+```
+
+To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, point the endpoint at your basin and supply an access token:
+
+```bash
+REALTIME_STREAMS_S2_BASIN=your-basin
+REALTIME_STREAMS_S2_ENDPOINT=https://your-basin.b.aws.s2.dev/v1
+REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false
+REALTIME_STREAMS_S2_ACCESS_TOKEN=your-access-token
+```
+
+See the [webapp environment variables](/self-hosting/env/webapp) for the full list of realtime stream settings.
+
 ## Troubleshooting
 
 - **Deployment fails at the push step.** The machine running `deploy` needs registry access. See the [registry setup](#registry-setup) section for more details.

docs/self-hosting/env/webapp.mdx

@@ -139,9 +139,14 @@ mode: "wide"
 | **Task events**                                  |          |                       |                                                                                                                    |
 | `EVENT_REPOSITORY_DEFAULT_STORE`                 | No       | postgres              | Where to store task events. Set to `clickhouse_v2` to store in ClickHouse (recommended for production).            |
 | **Realtime**                                     |          |                       |                                                                                                                    |
-| `REALTIME_STREAM_VERSION`                        | No       | v1                    | Realtime stream protocol version. One of `v1`, `v2`.                                                               |
+| `REALTIME_STREAM_VERSION`                        | No       | v1                    | Stream version exposed to tasks via the `TRIGGER_REALTIME_STREAM_VERSION` variable. Distinct from `REALTIME_STREAMS_DEFAULT_VERSION`. One of `v1`, `v2`. |
 | `REALTIME_STREAM_MAX_LENGTH`                     | No       | 1000                  | Realtime stream max length.                                                                                        |
 | `REALTIME_STREAM_TTL`                            | No       | 86400 (1d)            | Realtime stream TTL (s).                                                                                           |
+| `REALTIME_STREAMS_DEFAULT_VERSION`               | No       | v1                    | Server-side default the webapp uses when a stream request doesn't pin a version (modern SDKs request `v2`). The Docker and Helm self-hosting defaults set this to `v2`. One of `v1`, `v2`. |
+| `REALTIME_STREAMS_S2_BASIN`                       | No       | —                     | S2 basin that holds v2 realtime streams. Required for `v2`. Must be at least 8 characters.                          |
+| `REALTIME_STREAMS_S2_ENDPOINT`                    | No       | —                     | Custom S2 API endpoint, including the `/v1` suffix (e.g. `http://s2/v1` for the bundled s2-lite). Omit to use hosted S2 at s2.dev. |
+| `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS`         | No       | false                 | Skip minting per-stream access tokens. Set to `true` for s2-lite, which needs no authentication.                    |
+| `REALTIME_STREAMS_S2_ACCESS_TOKEN`               | No       | —                     | S2 access token. Required for hosted S2 unless `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` is `true`.                  |
 | **Bootstrap**                                    |          |                       |                                                                                                                    |
 | `TRIGGER_BOOTSTRAP_ENABLED`                      | No       | 0                     | Trigger bootstrap enabled.                                                                                         |
 | `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME`            | No       | —                     | Trigger bootstrap worker group name.                                                                               |

docs/self-hosting/kubernetes.mdx

@@ -375,6 +375,32 @@ webapp:
 
 This only affects new runs; existing runs continue to read from wherever their events were originally stored.
 
+## Realtime streams
+
+Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` deployment — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. The chart deploys it with a persistent volume, so no extra services are required.
+
+To fall back to the Redis-backed **v1** streams, set the default version to `v1`:
+
+```yaml
+s2:
+  defaultStreamVersion: "v1"
+```
+
+To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, disable the bundled deployment and point at your basin. Supply the access token via an existing secret:
+
+```yaml
+s2:
+  deploy: false
+  skipAccessTokens: false
+  external:
+    endpoint: "https://your-basin.b.aws.s2.dev/v1"
+    existingSecret: "s2-credentials"
+    existingSecretAccessTokenKey: "access-token"
+  basin: "your-basin"
+```
+
+To disable realtime streams v2 entirely and use v1, set `s2.deploy: false` with no external endpoint. See `helm show values` for all `s2` options.
+
 ## Worker token
 
 When using the default bootstrap configuration, worker creation and authentication is handled automatically. The webapp generates a worker token and makes it available to the supervisor via a shared volume.

hosting/docker/.env.example

@@ -106,6 +106,22 @@ OBJECT_STORE_SECRET_ACCESS_KEY=very-safe-password
 # MINIO_ROOT_USER=admin
 # MINIO_ROOT_PASSWORD=very-safe-password
 
+# Realtime streams
+# - Realtime streams power AI-agent token streaming and run streams
+# - They default to v2, backed by the bundled s2-lite service (open-source S2, https://s2.dev)
+# - To fall back to the Redis-backed v1 streams, set REALTIME_STREAMS_DEFAULT_VERSION=v1
+# REALTIME_STREAMS_DEFAULT_VERSION=v2
+# The bundled s2-lite creates whatever basin you set here (min 8 characters)
+# REALTIME_STREAMS_S2_BASIN=trigger-realtime
+# REALTIME_STREAMS_S2_ENDPOINT=http://s2/v1
+# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=true
+# To use a hosted S2 (https://s2.dev) instead of the bundled s2-lite:
+# - point the endpoint at your basin, disable token-skipping, and set an access token
+# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false
+# REALTIME_STREAMS_S2_ACCESS_TOKEN=
+# Pin the s2-lite image in production (full image reference, digest recommended)
+# S2_IMAGE=ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c
+
 # Other image tags
 # - These are the versions of the other images to use
 # - You should lock these to a specific version in production

hosting/docker/webapp/docker-compose.yml

@@ -18,6 +18,7 @@ services:
       - postgres
       - redis
       - clickhouse
+      - s2
     networks:
       - webapp
       - supervisor
@@ -44,6 +45,15 @@ services:
       LOGIN_ORIGIN: ${LOGIN_ORIGIN:-http://localhost:8030}
       API_ORIGIN: ${API_ORIGIN:-http://localhost:8030}
       ELECTRIC_ORIGIN: http://electric:3000
+      # Realtime streams v2, backed by the bundled s2-lite service. This powers
+      # AI-agent token streaming and run streams. Point the endpoint at a hosted
+      # S2 (https://s2.dev) instead by overriding these and setting an access token.
+      # To fall back to the Redis-backed v1 streams, set the version to `v1`.
+      REALTIME_STREAMS_DEFAULT_VERSION: ${REALTIME_STREAMS_DEFAULT_VERSION:-v2}
+      REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime}
+      REALTIME_STREAMS_S2_ENDPOINT: ${REALTIME_STREAMS_S2_ENDPOINT:-http://s2/v1}
+      REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS: ${REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS:-true}
+      REALTIME_STREAMS_S2_ACCESS_TOKEN: ${REALTIME_STREAMS_S2_ACCESS_TOKEN:-}
       DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable}
       DIRECT_URL: ${DIRECT_URL:-postgresql://postgres:postgres@postgres:5432/main?schema=public&sslmode=disable}
       SESSION_SECRET: ${SESSION_SECRET}
@@ -225,12 +235,62 @@ services:
       retries: 5
       start_period: 10s
 
+  # Generates the s2-lite basin spec from REALTIME_STREAMS_S2_BASIN so the bundled
+  # basin always matches the basin the webapp targets. s2-lite is distroless (no
+  # shell), so a tiny init container writes the spec into a shared volume first.
+  s2-init:
+    image: busybox:${BUSYBOX_IMAGE_TAG:-1.37}
+    restart: "no"
+    logging: *logging-config
+    environment:
+      REALTIME_STREAMS_S2_BASIN: ${REALTIME_STREAMS_S2_BASIN:-trigger-realtime}
+    command:
+      - sh
+      - -c
+      - |
+        cat > /config/s2-spec.json <<JSON
+        {
+          "basins": [
+            {
+              "name": "$$REALTIME_STREAMS_S2_BASIN",
+              "config": { "create_stream_on_append": true, "create_stream_on_read": true }
+            }
+          ]
+        }
+        JSON
+    volumes:
+      - s2-config:/config
+    networks:
+      - webapp
+
+  # s2-lite: the open-source, self-hostable S2 (https://s2.dev) server that backs
+  # realtime streams v2. It's distroless (no shell), so a wget/curl healthcheck
+  # always reports unhealthy even when the API is up — hence no healthcheck.
+  # Runs as root so it can write to the named volume; storage lives in SlateDB
+  # under /data, and the basin is created on startup from the generated spec.
+  s2:
+    image: ${S2_IMAGE:-ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c}
+    restart: ${RESTART_POLICY:-unless-stopped}
+    logging: *logging-config
+    depends_on:
+      s2-init:
+        condition: service_completed_successfully
+    command: ["lite", "--init-file", "/config/s2-spec.json", "--local-root", "/data"]
+    user: "0:0"
+    volumes:
+      - s2-config:/config
+      - s2:/data
+    networks:
+      - webapp
+
 volumes:
   clickhouse:
   postgres:
   redis:
   shared:
   minio:
+  s2:
+  s2-config:
 
 networks:
   docker-proxy:

hosting/k8s/helm/templates/_helpers.tpl

@@ -387,6 +387,27 @@ http://{{ include "trigger-v4.fullname" . }}-electric:{{ .Values.electric.servic
 {{- end -}}
 {{- end }}
 
+{{/*
+Whether realtime streams v2 (S2) is wired up: either the bundled s2-lite is
+deployed, or an external S2 endpoint has been configured.
+*/}}
+{{- define "trigger-v4.s2.enabled" -}}
+{{- if or .Values.s2.deploy .Values.s2.external.endpoint -}}
+true
+{{- end -}}
+{{- end }}
+
+{{/*
+S2 API endpoint URL, including the /v1 suffix expected by the client.
+*/}}
+{{- define "trigger-v4.s2.url" -}}
+{{- if .Values.s2.deploy -}}
+http://{{ include "trigger-v4.fullname" . }}-s2:{{ .Values.s2.service.port }}/v1
+{{- else -}}
+{{ .Values.s2.external.endpoint }}
+{{- end -}}
+{{- end }}
+
 {{/*
 ClickHouse hostname
 */}}

hosting/k8s/helm/templates/s2.yaml

@@ -0,0 +1,142 @@
+{{- if .Values.s2.deploy }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "trigger-v4.fullname" . }}-s2-spec
+  labels:
+    {{- $component := "s2" }}
+    {{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
+data:
+  s2-spec.json: |
+    {
+      "$schema": "https://raw.githubusercontent.com/s2-streamstore/s2/main/cli/schema.json",
+      "basins": [
+        {
+          "name": {{ .Values.s2.basin | quote }},
+          "config": {
+            "create_stream_on_append": true,
+            "create_stream_on_read": true
+          }
+        }
+      ]
+    }
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: {{ include "trigger-v4.fullname" . }}-s2
+  labels:
+    {{- $component := "s2" }}
+    {{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
+spec:
+  replicas: 1
+  serviceName: {{ include "trigger-v4.fullname" . }}-s2
+  selector:
+    matchLabels:
+      {{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 6 }}
+  template:
+    metadata:
+      annotations:
+        checksum/spec: {{ .Values.s2.basin | sha256sum }}
+        {{- with .Values.s2.podAnnotations }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      labels:
+        {{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 8 }}
+    spec:
+      {{- with .Values.s2.podSecurityContext }}
+      securityContext:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+        - name: s2
+          {{- with .Values.s2.securityContext }}
+          securityContext:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
+          image: "{{ .Values.s2.image.registry }}/{{ .Values.s2.image.repository }}:{{ .Values.s2.image.tag }}{{ with .Values.s2.image.digest }}@{{ . }}{{ end }}"
+          imagePullPolicy: {{ .Values.s2.image.pullPolicy }}
+          command:
+            - "lite"
+            - "--init-file"
+            - "/config/s2-spec.json"
+            - "--local-root"
+            - "/data"
+            - "--port"
+            - {{ .Values.s2.service.targetPort | quote }}
+          ports:
+            - name: http
+              containerPort: {{ .Values.s2.service.targetPort }}
+              protocol: TCP
+          {{- if .Values.s2.livenessProbe.enabled }}
+          livenessProbe:
+            tcpSocket:
+              port: http
+            initialDelaySeconds: {{ .Values.s2.livenessProbe.initialDelaySeconds }}
+            periodSeconds: {{ .Values.s2.livenessProbe.periodSeconds }}
+            timeoutSeconds: {{ .Values.s2.livenessProbe.timeoutSeconds }}
+            failureThreshold: {{ .Values.s2.livenessProbe.failureThreshold }}
+            successThreshold: {{ .Values.s2.livenessProbe.successThreshold }}
+          {{- end }}
+          {{- if .Values.s2.readinessProbe.enabled }}
+          readinessProbe:
+            tcpSocket:
+              port: http
+            initialDelaySeconds: {{ .Values.s2.readinessProbe.initialDelaySeconds }}
+            periodSeconds: {{ .Values.s2.readinessProbe.periodSeconds }}
+            timeoutSeconds: {{ .Values.s2.readinessProbe.timeoutSeconds }}
+            failureThreshold: {{ .Values.s2.readinessProbe.failureThreshold }}
+            successThreshold: {{ .Values.s2.readinessProbe.successThreshold }}
+          {{- end }}
+          resources:
+            {{- toYaml .Values.s2.resources | nindent 12 }}
+          volumeMounts:
+            - name: spec
+              mountPath: /config
+              readOnly: true
+            - name: data
+              mountPath: /data
+      volumes:
+        - name: spec
+          configMap:
+            name: {{ include "trigger-v4.fullname" . }}-s2-spec
+        {{- if not .Values.s2.persistence.enabled }}
+        - name: data
+          emptyDir: {}
+        {{- end }}
+  {{- if .Values.s2.persistence.enabled }}
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+        {{- if .Values.s2.persistence.retain }}
+        annotations:
+          helm.sh/resource-policy: keep
+        {{- end }}
+      spec:
+        accessModes:
+          - {{ .Values.s2.persistence.accessMode }}
+        resources:
+          requests:
+            storage: {{ .Values.s2.persistence.size }}
+        {{- if .Values.s2.persistence.storageClass }}
+        storageClassName: {{ .Values.s2.persistence.storageClass }}
+        {{- end }}
+  {{- end }}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "trigger-v4.fullname" . }}-s2
+  labels:
+    {{- $component := "s2" }}
+    {{- include "trigger-v4.componentLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
+spec:
+  type: {{ .Values.s2.service.type }}
+  ports:
+    - port: {{ .Values.s2.service.port }}
+      targetPort: http
+      protocol: TCP
+      name: http
+  selector:
+    {{- include "trigger-v4.componentSelectorLabels" (dict "Chart" .Chart "Release" .Release "Values" .Values "component" $component) | nindent 4 }}
+{{- end }}

hosting/k8s/helm/templates/validate-external-config.yaml

@@ -39,6 +39,10 @@ This template will fail the Helm deployment if external config is missing for re
 {{- end }}
 {{- end }}
 
+{{- if and .Values.s2.deploy .Values.s2.external.endpoint }}
+{{- fail "S2 configuration conflict: s2.external.endpoint is set but s2.deploy=true, so the external endpoint would be ignored. Set s2.deploy=false to use an external S2 endpoint, or clear s2.external.endpoint to use the bundled s2-lite." }}
+{{- end }}
+
 {{- if not .Values.registry.deploy }}
 {{- if or (not .Values.registry.external.host) }}
 {{- fail "Registry external configuration is required when registry.deploy=false. Please provide registry.external.host" }}

hosting/k8s/helm/templates/webapp.yaml

@@ -188,6 +188,27 @@ spec:
               value: {{ .Values.webapp.apiOrigin | quote }}
             - name: ELECTRIC_ORIGIN
               value: {{ include "trigger-v4.electric.url" . | quote }}
+            {{- if include "trigger-v4.s2.enabled" . }}
+            # Realtime streams v2, backed by S2 (bundled s2-lite or an external endpoint)
+            - name: REALTIME_STREAMS_DEFAULT_VERSION
+              value: {{ .Values.s2.defaultStreamVersion | quote }}
+            - name: REALTIME_STREAMS_S2_ENDPOINT
+              value: {{ include "trigger-v4.s2.url" . | quote }}
+            - name: REALTIME_STREAMS_S2_BASIN
+              value: {{ .Values.s2.basin | quote }}
+            - name: REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS
+              value: {{ .Values.s2.skipAccessTokens | quote }}
+            {{- if and .Values.s2.external.endpoint .Values.s2.external.existingSecret }}
+            - name: REALTIME_STREAMS_S2_ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: {{ .Values.s2.external.existingSecret }}
+                  key: {{ .Values.s2.external.existingSecretAccessTokenKey }}
+            {{- else if .Values.s2.external.accessToken }}
+            - name: REALTIME_STREAMS_S2_ACCESS_TOKEN
+              value: {{ .Values.s2.external.accessToken | quote }}
+            {{- end }}
+            {{- end }}
             {{- if include "trigger-v4.postgres.useSecretUrl" . }}
             - name: DATABASE_URL
               valueFrom: