feat(room-io): listen to several participants through one audio input - #6884
Open
Darshak03 wants to merge 8 commits into
Open
feat(room-io): listen to several participants through one audio input#6884Darshak03 wants to merge 8 commits into
Darshak03 wants to merge 8 commits into
Conversation
A single AgentSession could only ever hear the linked participant, so a room
with two humans in it produced a one-sided chat context.
The composition point is an AudioInput rather than RoomIO: `_RoomAudioInput` is
what RoomIO drives, and `_ParticipantAudioInputGroup` holds one unchanged
`_ParticipantAudioInputStream` per participant. `AudioInputOptions.participants`
selects how they are combined:
- "linked" (default): unchanged, a single participant stream.
- "mix": summed by `rtc.AudioMixer`, keeps overlapping speech.
- "pick": only the participant the server reports as speaking, preceded by
their held pre-roll and separated by a silence gap.
RoomIO keeps owning the participant lifecycle and routes it through
`add_participant`/`remove_participant`, which do nothing for an input that only
listens to the linked participant. `participant_identity` and `set_participant`
still choose the linked participant, which now only drives the outputs.
Mixing an idle stream would pace the mix below real time and warn on every
block, so a participant is registered with the mixer only while a live, unmuted
track is being read: `_ParticipantInputStream` now reports when that changes,
including a track that stops delivering without being unpublished.
- guard the end-of-stream `_set_streaming(False)` on the publication it was reading, so a forward task winding down after a track swap cannot de-register the live stream its replacement just registered, whatever the task order. - close a noise cancellation processor shared between participants once, from the group, instead of once per child stream in `_ParticipantInputStream`. - turn off pre-connect audio when mixing: the mixer paces every stream at real time, so the buffer is never caught up on and leaves that participant seconds behind the mix for the whole session. - defer the end-of-turn silence by a tick. replacing a track empties the mix until the new stream registers later in the same tick, which is not the end of a turn and must not flush the transcript. - re-select the next reported speaker when the picked one is removed, rather than dropping their turn until the next server update.
The pre-connect buffer is audio from before the participant joined, so it is not concurrent with anyone else in the room. Mixing paces it at real time and leaves that participant seconds behind for the session; picking drops all but the last two seconds of it into a pre-roll that is only replayed if the server later names them the speaker. RoomIO now only registers the handler for `participants="linked"` and says so, rather than decoding and buffering byte streams nothing will read. Mute events also move to `_ParticipantAudioInputGroup`: a muted track is never unpublished and its stream never ends, so without them a muted speaker the server still reports would hold the floor in pick mode while delivering nothing. Selection now skips participants that are not delivering.
Replacing a microphone closes the old stream and opens its replacement in the same tick, so the speaker looks like they stopped for an instant. The pick path took that as the end of the turn: it pushed a silence gap, handed the floor to somebody else, and left the original speaker unable to take it back until the next server update, with the rest of their sentence held as pre-roll. The floor change is now deferred a tick and re-checked, the same way the mixing path already defers closing a turn. Also drops two dead statements left after the group hook docstrings, and documents that close_on_disconnect still only watches the linked participant when listening to everyone.
The mixer is the only reader of a child stream, so unregistering one left its forwarding task filling an unbounded channel nobody drains. A track that keeps delivering while reported muted would grow that backlog for as long as the participant stayed out of the mix. The child is now detached when it leaves the mix and attached again when it rejoins, which drops the frames at the source instead of relying on a muted track going quiet. Also: - keep the reported speakers unfiltered, so a participant added after the last `active_speakers_changed` can take the floor as soon as their stream is live instead of waiting for the server to send another update. - validate `frame_size_ms` in the group constructor, where the linked mode already rejects it, rather than failing later from a room event handler. - close over a snapshot of the streams, so a disconnect landing mid-close cannot mutate the dict being iterated. - log the pre-connect incompatibility at info: it is on by default, so a warning fires for everyone selecting a group mode without asking for it.
… those leaving the mix A track subscribed while already muted never enters the mix, so `_stop_mixing` returned early and the child kept filling a channel with no reader. The gate now lives where the decision is made, covering a child that was never registered as well as one that just left. The gate is also its own flag rather than the child's `on_attached`/ `on_detached`: those are the session's side of the `AudioInput` contract, and a group's children are not the session's input, so driving them from mixer bookkeeping would fire session semantics on every mute. Mute events for cameras and screenshares no longer re-evaluate a participant: only the microphone decides whether a child can feed a group.
The window is there to cover how late `active_speakers_changed` is, not to keep as much audio as possible: everything held is replayed on selection, so a long one re-sends more of what the previous speaker was talking over and hands the stt a larger burst at every switch.
Each child keeps its own `track_subscribed` handler until its own close finishes, and those closes are awaited one at a time. Closing the mixer first meant a track subscribed during that window registered with a mixer that was already gone, and `AudioMixer.add_stream` raises once closed, so the error surfaced from inside the room's event dispatch. The group also refuses to react at all once closed, so ordering is not the only thing keeping it safe. A group segments its own turns, so its children no longer push the end-of-track silence: in the mixing group it landed in a channel nobody drains, and in the active speaker group it filled the pre-roll of a participant who was not speaking, replacing the first syllables it exists to keep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A single AgentSession could only ever hear the linked participant, so a room with two humans in it produced a one-sided chat context.
The composition point is an AudioInput rather than RoomIO:
_RoomAudioInputis what RoomIO drives, and_ParticipantAudioInputGroupholds one unchanged_ParticipantAudioInputStreamper participant.AudioInputOptions.participantsselects how they are combined:rtc.AudioMixer, keeps overlapping speech.their held pre-roll and separated by a silence gap.
RoomIO keeps owning the participant lifecycle and routes it through
add_participant/remove_participant, which do nothing for an input that only listens to the linked participant.participant_identityandset_participantstill choose the linked participant, which now only drives the outputs.Mixing an idle stream would pace the mix below real time and warn on every block, so a participant is registered with the mixer only while a live, unmuted track is being read:
_ParticipantInputStreamnow reports when that changes, including a track that stops delivering without being unpublished.