feat(elevenlabs): secondary languages and language detection for realtime STT

[captainbanan]

What

Two additions to the ElevenLabs realtime STT plugin, and the fix that makes the second one observable:

1. secondary_languages — passed through to Scribe v2 realtime. Declares extra languages that may appear in the audio while keeping the primary one hinted. Codes are normalized to what the API accepts: it takes ISO-639-1/639-3 and rejects everything else, including the region-tagged tags LanguageCode produces (ru-RU), so the region comes off before the connect URL. The primary language_code goes through the same normalization — it was sent verbatim before, and the API rejects a region-tagged primary code too, so language_code="en-US" produced a session that stayed connected and never transcribed.
2. include_language_detection — now an explicit option. It was implicit: on when no language_code was set, off otherwise, with no way to change either.
3. Fix: the detected language never reached SpeechData.language. Detail below.

Why

Sessions where the speaker switches language mid-call — a tutor whose learner drops into their native language for a sentence, support lines in bilingual regions. Pinning language_code biases recognition toward the expected language, which you want, but there was no way to say "and they might also speak Russian".

The detected language matters beyond the transcript text: MultilingualModel.unlikely_threshold() keys the per-language EOU threshold on SpeechData.language, so a wrong label silently picks the wrong turn-detection threshold (or, for a language outside languages.json, supports_language() returns False and the EOU model is skipped entirely).

The bug

The plugin asks for language detection whenever no language_code is pinned, but the server puts the detected language only on the delayed committed_transcript_with_timestamps copy of the commit — and the plugin ignores that copy unless include_timestamps=True. So the detection it requested was thrown away, and every transcript came out labelled with the pinned language, or the hardcoded LanguageCode("en") fallback in auto-detect mode.

Measured against the live API with scribe_v2_realtime, one English utterance followed by one Russian one:

session config	committed_transcript	committed_transcript_with_timestamps
language_code=en	lang absent	lang absent
language_code=en&include_language_detection=true	lang absent	lang='en', then lang='ru'
no language_code (what the plugin sends today)	lang absent	lang='en', then lang='ru'

After the change, the final comes from whichever copy carries what the session asked for: include_timestamps or include_language_detection. Word timings are still only attached when include_timestamps was requested, so enabling detection doesn't start handing out TimedStrings the caller never asked for and the aligned_transcript capability stays truthful.

Latency: none for the detection-only path — the timestamped copy arrives within ~1ms of the plain one (0.855s vs 0.856s, 1.888s vs 1.889s in the run above). The ~80ms lag people associate with that copy comes from include_timestamps=true, whose behavior is unchanged here.

Serialization

secondary_languages is a repeated query param. Verified against the live API — secondary_languages=ru&secondary_languages=es is echoed back as "secondary_languages": ["ru", "es"] in session_started.config, while a comma-joined or JSON-encoded value is rejected with invalid_request: Invalid language code received: 'ru,es'.

Both new options are realtime-only and warn + no-op on batch models, matching how server_vad and previous_text behave.

Verification

End-to-end against the live API with the patched plugin, same two utterances:

{'language_code': 'en'}                                                   -> en / en
{'language_code': 'en', 'secondary_languages': ['ru']}                    -> en / en
{..., 'secondary_languages': ['ru'], 'include_language_detection': True}  -> en / ru

uv run pytest --unit → 1987 passed, make check clean. 12 new tests in tests/test_plugin_elevenlabs_stt.py cover the query params, the normalization of both the primary and the secondary codes, the batch-model warning, and both the pinned-language and auto-detect final-transcript paths — uv run pytest tests/test_plugin_elevenlabs_stt.py → 37 passed. That module is marked plugin("elevenlabs"), so it runs in the livekit-plugins-elevenlabs job rather than under --unit, and that job is gated on head.repo.fork == false — it is skipped on this PR.

Out of scope on purpose

One neighbouring problem this PR deliberately leaves alone, found while testing it and ready to send separately if you want it:

• The error branch in _process_stream_event never reaches the caller — recv_task catches everything it raises and only logs, so a session the server rejects stays connected and silent for the rest of the call.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

captainbanana seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}