feat(web-ui): automatic GPU deinterlacing with bwdif and shader-based content detection#610
Merged
Conversation
…ction Render-side deinterlacing on top of the existing MSE pipeline: a requestVideoFrameCallback + WebGL2 overlay draws deinterlaced frames to a per-slot canvas while the video element keeps driving playback and audio. - Heuristic detection (no metadata): periodic luma sampling with a classic comb metric, M-of-N rolling window, sticky verdict, reset on stream switch - Gated to 1080-class content (height 1080/1088, width <= 1920) - Pluggable algorithm registry; initial bob renders at field rate (25i->50p) with separate chroma low-pass to remove field-interleaved color combing - Settings toggle (auto/off) persisted to player storage - devlab: interlaced 1080i scan channel plus 1080p/2160p gating controls
GLSL port of FFmpeg's bwdif filter_line: per-pixel motion adaptation keeps both fields in still areas (full vertical resolution, no bob shimmer on static detail like logos/subtitles) and reconstructs moving areas with the edge-preserving spatio-temporal filter, clamped to the temporal neighborhood. Runs one frame behind the video (3-frame texture ring) at field rate for 50p motion. Chroma keeps the separate vertical low-pass since field-interleaved chroma combing from progressive 4:2:0 upsampling also sits on kept lines.
BFF sources played with a TFF assumption produce two-forward-one-back motion judder at field rate. Add a field-order vote to the detector: capture two consecutive frames via requestVideoFrameCallback and compare TFF vs BFF hypotheses by their temporal-midpoint interpolation error (the wrong order averages fields two field-times apart, so with motion its error is larger). Majority vote with abstention on still/ambiguous pairs; activation starts immediately with the TFF default and re-emits if BFF wins. Also lower the combed-frame ratio to 0.01 (in-browser measurements: interlaced 0.018-0.022 vs progressive 0.004-0.005; the old 0.02 threshold made detection flaky on frames with moderate motion) and add a 1080i BFF devlab channel. Verified in Chrome against devlab: tff=4/0 on the TFF channel, bff=4/0 on the BFF channel, no false positive on 1080p.
Review against libavfilter/bwdifdsp.c + vf_bwdif.c found three gaps: - Frame-boundary rows: FFmpeg dispatches y<4 / y+5>h to FILTER_EDGE (plain (c+e)/2 with the spatial check only where its +-2 taps fit) instead of running the +-3/+-4-tap filters into duplicated edge rows; the shader now mirrors that row dispatch instead of relying on texture clamping. - Chroma was an unconditional vertical low-pass, softening static color detail that FFmpeg's per-plane weave path preserves. Now motion-adaptive: static pixels pass original chroma through, moving pixels ramp toward the [1,2,2,2,1]/8 low-pass (true per-plane bwdif is impossible on RGB frames -- the progressive 4:2:0 upsample bakes field-mixed chroma into all rows). - Documented the weave-threshold float mapping (FFmpeg's integer !diff test == 0.5/255) and the intentional RGB-vs-YUV deviations. Verified in Chrome on the 1080i TFF channel: moving comb 0.0026, paused still 0.0000, static chroma edges preserved.
…e paused frame - Parse H.264 frame_mbs_only_flag and H.265 field_seq/interlaced_source flags, plumb them demuxer -> remuxer -> worker -> player 'video-info' event -> deinterlace pipeline. Eligible interlaced streams now start deinterlaced immediately instead of waiting for the combing heuristic warm-up; the heuristic remains for streams with progressive metadata but combed content. - Prime the renderer with the current video frame on start(), so re-enabling deinterlacing while paused replaces the stale canvas frame from a previous run. Until the texture ring holds real history the shader falls back to spatial-only interpolation (temporal terms would compare duplicated frames).
The React player no longer manages deinterlace pipelines, video-info hinting, or verdict resets. It passes an overlay canvas and mode via PlayerConfig (deinterlaceCanvas / deinterlaceMode), toggles mode at runtime with Player.setDeinterlaceMode(), and reacts to the new deinterlace-active-change event for visibility styling. The core resets the interlace verdict on loadSegments/stop and hints the detector from codec metadata internally. The canvas is stripped from the config posted to the transmux worker (not structured-cloneable). Also raise COMB_PIXEL_THRESHOLD 121 -> 400 (~±20 luma levels): sharp progressive 1080p detail scored up to 0.011 at the old threshold and false-positived the detector; measured in-browser the new threshold keeps progressive <=0.005 while weaved interlaced stays >=0.011.
…rlace detection Weaved TFF content encoded and flagged as progressive (no fieldorder filter, no interlaced encoder flags), so the codec-metadata hint stays silent and only the heuristic comb detector can activate deinterlacing.
…pipeline to boolean
Contributor
Documentation previewThe documentation preview has been deployed for this pull request. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa706bc0db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an automatic, render-side deinterlacing pipeline to the Web UI MPEG-TS player. It detects interlaced streams via codec metadata hints and/or a heuristic combing detector, then renders deinterlaced frames to a WebGL2 overlay canvas while leaving the MSE pipeline and audio timing driven by the original <video> element.
Changes:
- Introduces a new WebGL2 deinterlacing pipeline (detector + renderer + bwdif GLSL algorithm) and plumbs codec-level “may be interlaced” metadata from demux → remux → worker → player events.
- Adds runtime user control (auto/on vs off) persisted in player storage and surfaced in the player settings UI.
- Extends devlab with multicast “scan” channels to exercise interlace detection, field order, and resolution gating.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| web-ui/src/pages/player.tsx | Wires persisted “deinterlace” setting into the player page and settings dropdown props. |
| web-ui/src/mpegts/worker/transmux-worker.ts | Emits a new video-info worker event derived from init segment metadata. |
| web-ui/src/mpegts/worker/messages.ts | Adds video-info worker event type (width/height + interlace hint). |
| web-ui/src/mpegts/types.ts | Defines VideoTrackInfo, new player events, and setDeinterlace() API. |
| web-ui/src/mpegts/remux/mp4-remuxer.ts | Attaches videoInfo (dimensions + interlace hint) to video init segments. |
| web-ui/src/mpegts/player/mpegts-player.ts | Forwards video-info messages to the player impl without breaking init batching. |
| web-ui/src/mpegts/index.ts | Creates/owns the deinterlace pipeline, forwards hints, exposes events and setDeinterlace(). |
| web-ui/src/mpegts/demux/ts-demuxer.ts | Sets mayBeInterlaced in track metadata from H.264/H.265 parsed flags. |
| web-ui/src/mpegts/demux/sps-parser.ts | Exposes H.264 frame_mbs_only_flag for interlace-capability hinting. |
| web-ui/src/mpegts/demux/h265-parser.ts | Exposes an HEVC “interlaced-capable” hint from VUI/PTL flags. |
| web-ui/src/mpegts/deinterlace/renderer.ts | New: WebGL2 renderer driven by requestVideoFrameCallback with field-rate output. |
| web-ui/src/mpegts/deinterlace/index.ts | New: Pipeline wiring detector verdicts to renderer activation and events. |
| web-ui/src/mpegts/deinterlace/detector.ts | New: Comb-metric heuristic detector + field-order voting logic. |
| web-ui/src/mpegts/deinterlace/algorithms/types.ts | New: Algorithm registry + interface for pluggable GPU deinterlacers. |
| web-ui/src/mpegts/deinterlace/algorithms/gl-utils.ts | New: Shared shader/program helpers and fullscreen triangle vertex shader. |
| web-ui/src/mpegts/deinterlace/algorithms/bwdif.ts | New: GLSL bwdif implementation (motion-adaptive deinterlacing). |
| web-ui/src/mpegts/config.ts | Adds deinterlaceCanvas and deinterlace config knobs with defaults. |
| web-ui/src/lib/player-storage.ts | Persists the deinterlacing setting in local storage. |
| web-ui/src/i18n/player.ts | Adds i18n strings for the deinterlacing toggle. |
| web-ui/src/components/player/video-player.tsx | Adds overlay canvases per slot, hooks player events, toggles runtime deinterlacing. |
| web-ui/src/components/player/settings-dropdown.tsx | Adds the deinterlacing toggle UI control. |
| tools/devlab/README.md | Documents new “interlace-scan” multicast channels for deinterlacing testing. |
| tools/devlab/devlab.py | Adds scan-channel generation (1080i TFF/BFF, combed progressive, 2160p gate). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- detector: codec metadata hint now requires at least one combed frame before activating (hintPending flag), preventing progressive streams with permissive frame_mbs_only_flag from being permanently processed by bwdif; hint is discarded after a full window with no combing - detector: guard hintInterlaced() so field-order votes are never scheduled while the detector is stopped (resolved via hintPending path which defers all work to the timer-driven sample() callback) - detector: start() now resumes scheduleFieldOrderVote() when re-enabled if interlaced is already true but field order is not yet settled - pipeline: defer onActiveChange(true) until the first WebGL frame is drawn (pendingFirstFrame flag + notifyFirstFrameRendered callback), preventing a black flash on metadata-hinted streams where the canvas has not been painted when the verdict arrives - video-player: hidden background slot uses opacity-0 instead of invisible when deinterlacing is active, keeping rVFC firing for seamless-switch warm-up
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…nal detection - Remove codec metadata hint path (hintInterlaced); detection is now purely heuristic, avoiding false-positive locks on permissive encoders - Add motion-gated reversion: sustained progressive-looking frames under MOTION_FLOOR motion guard revert the interlaced verdict without resetting the channel session - Replace fixed-offset fast-sample setTimeout with chained rVFC calls anchored to the first rendered frame, ensuring pre-samples land on real decoded frames instead of firing during stream buffering - Fix rVFC quota bug: sample() now returns boolean; onFrame only decrements remaining when a frame was actually processed, so wasted fires (readyState < HAVE_CURRENT_DATA) do not exhaust the fast-sample quota
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Owner
Author
|
@codex[agent] review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 090cd7be2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ounter reset - Renderer: accept onContextLost/onContextRestored callbacks, stop the render loop on context loss and notify the pipeline so raw video is revealed as a fallback. On context restore, re-establish the entire pipeline lazily via apply() rather than rebuilding in-place. - Pipeline: wire context callbacks — emit onActiveChange(false) on loss to restore raw video visibility, and re-apply the deinterlace verdict on restore to restart shader/texture setup. - Detector: reset reversionConsecutiveCount on motion-abstained samples so that the reversion-to-progressive path requires truly consecutive qualifying frames, as documented.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Owner
Author
|
@codex[agent] review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Move the Canvas 2D getImageData detection path to a WebGL2 fragment shader pipeline that reuses the renderer's texture ring: - Marker pass: computes comb score (R) and abs-diff motion (G) per pixel in a 256×H RGBA16F FBO using the same BT.709 luma weights as bwdif. - Reduction chain: 2×2 box filter passes down to ≤8×8, shared by the marker and field-order passes. - Field-order pass: half-height FBO computing per-row errTff/errBff prediction errors, replacing the rVFC drawImage pair in CPU mode. - PBO async readback: readPixels into a PIXEL_PACK_BUFFER; result is retrieved 500 ms later via getBufferSubData — no pipeline stall on the bwdif render path, and GPU→CPU transfer drops from ~1.1 MB to 512 bytes per sample cycle. Detection automatically falls back to the existing Canvas 2D path when EXT_color_buffer_float is unavailable or before the renderer's GL context is created (initial progressive channel scan). All verdict logic (rolling window, motion-gated reversion, field-order voting) is shared between both paths via processSampleMetrics(). renderer.ts: add onDetectionFrame callback hook and currentGl getter. index.ts: drive GPU sampling cadence (3-frame fast phase + 500 ms steady state) via onDetectionFrame; wire GL context lifecycle.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Owner
Author
|
@codex[agent] review d07c40c |
Co-authored-by: stackia <5107241+stackia@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d07c40c96e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…any algorithm runs Drop the legacy CPU detection path entirely so only the GPU shader chain remains, and merge the duplicated CPU/GPU verdict logic into a single engine. The resolution gate now lives in the pipeline: the whole GPU chain (frame uploads, bwdif, detection passes) only starts once the source size is known (via the video `resize` event) and within the SD/HD gate, so oversized streams run no algorithm at all.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…tion readback fully async - Split the renderer into detection-only and full-render modes driven by the detector verdict: progressive content within the gate now uploads frames only on the sampling cadence and never draws, instead of paying per-frame uploads + field-rate bwdif with a hidden canvas - Replace the sync/async readback split with a fence-gated PBO pool drained non-blocking on every frame; orphan each PBO before readPixels and defer getBufferSubData one frame so Chrome's readback shadow copy stays on the accelerated path (no performance warnings) - Fix field-order voting: sample every frame while voting is open (was one vote per 500 ms), sample at texel centers so LINEAR filtering no longer blends the two fields (TFF was misdetected as BFF), and keep the pass scheduled on static-scene abstain instead of stalling forever - Cache detection shader uniform locations once per program compile and hoist loop-invariant GL state out of the reduction chain - Skip transiently oversized frames at the rVFC entry to close the stream-switch race before the resize event re-evaluates the gate - Background video slot now uses opacity-0 unconditionally so rVFC keeps firing during seamless-switch warm-up (visibility:hidden stopped it) - Document the 1080i-tff/1080i-bff devlab scan channels
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
stackia
changed the title
Owner
Author
|
花了一千多块钱的 token (Fable 5)，终于做出来了。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds render-side automatic deinterlacing to the web player for interlaced IPTV streams (typically broadcast 1080i content). Deinterlacing runs as a WebGL2 overlay that intercepts each decoded frame via
requestVideoFrameCallback, draws the result to a per-slot canvas, and lets the<video>element continue driving playback and audio — the MSE pipeline is untouched.How it works
Detection — GPU shader pipeline: Content detection runs entirely on the GPU. A marker pass computes a per-pixel comb metric ((above−cur)·(below−cur) thresholding) and an abs-diff motion measure; a multi-pass 2×2 box-filter reduction collapses the result to 8×8, and only those few hundred bytes cross the GPU→CPU boundary. The CPU side just applies thresholds: an M-of-N rolling window (3 combed of 12 samples) declares the source interlaced, and 4 consecutive clean+moving samples revert the verdict when content changes back to progressive. An earlier CPU-sampling implementation of the same heuristic was fully replaced by this pipeline.
Fully async readback: Sample results return through a fence-gated PBO pool drained non-blocking on every frame —
readPixelsnever blocks andgetBufferSubDatais deferred until one frame after the fence signals, keeping Chrome's accelerated readback shadow copy on the fast path (each PBO is orphaned before reuse). No synchronous GPU→CPU reads anywhere.Detection cadence: The first 3 frames after start/reset are sampled back-to-back so a verdict lands within the first few frames of channel start; after that a 500 ms steady-state interval picks up mid-stream content changes. While field-order voting is open, sampling temporarily runs per-frame so the vote converges in frames, not seconds.
Two render modes, switched by the verdict: While the verdict is progressive the renderer stays in detection-only mode — frames are uploaded only when a sample is due and nothing is drawn, so progressive content within the gate costs almost nothing between samples. An interlaced verdict switches to full mode: per-frame uploads into a 3-frame texture ring, field-rate bwdif onto the canvas, and the canvas is revealed. Reversion hides the canvas and drops back to detection-only.
Field order (TFF/BFF): A half-height GPU pass compares each row pair's temporal-midpoint interpolation error under both TFF and BFF hypotheses — the wrong assumption averages fields two field-times apart, so motion amplifies the error. A majority vote (min 4 votes, margin 2, max 10 rounds) with abstention on static scenes drives the decision; sampling happens at texel centers so LINEAR filtering never blends the two fields into both hypotheses.
Algorithm — bwdif (GLSL port of FFmpeg's
vf_bwdif): Per-pixel motion adaptation keeps both fields in still areas (full vertical resolution, no bob shimmer on static content like logos and subtitles) and reconstructs moving pixels with the edge-preserving spatio-temporal filter, clamped to the temporal neighbourhood. Runs one frame behind via the texture ring at field rate for 50p motion output from 25i input. Chroma keeps a separate motion-adaptive vertical low-pass to remove field-interleaved color combing from progressive 4:2:0 upsampling. The shader was reviewed againstlibavfilter/bwdifdsp.candvf_bwdif.c.Resolution gate: Active for all content up to 1080 (width ≤ 1920, height ≤ 1088), covering 480i, 576i, 720i, and 1080i. Above the gate no algorithm runs at all — no WebGL context, no uploads, no detection. The gate is evaluated on the video element's
resizeevent (no per-frame polling), with a cheap per-frame guard that skips transiently oversized frames during stream switches before the resize event lands.Lifecycle: The pipeline handles the deinterlace settings toggle, seamless-switch slot swapping (background slots stay at
opacity-0so rVFC keeps firing and detection warms up while hidden — an interlaced verdict is ready the moment the switch completes), rapid channel zapping (verdict + voting reset per source, stale in-flight readbacks are dropped), and WebGL context loss/restore.Codec metadata plumbing: The demuxer still parses H.264
frame_mbs_only_flag/ H.265field_seq_flag+interlaced_source_flaginto amayBeInterlacedhint exposed via thevideo-infoevent. It no longer gates activation (many progressive streams carry interlaced-capable flags); it's retained for a future stream-metadata UI.User control: A settings dropdown toggle (auto / off) is persisted to player storage.
Testing
devlab gains scan channels for exercising the detector (
MCAST_SCAN_CHANNELS):Verified in Chrome against devlab: TFF and BFF channels detect within the first frames and resolve the correct field order; 1080p stays in detection-only mode with no false positive; 2160p never starts the GPU chain; seamless and non-seamless channel zapping across all of the above behaves; console is clean (no readback performance warnings).
Files changed
web-ui/src/mpegts/deinterlace/— new module: GPU detector, two-mode renderer, bwdif algorithm, GL utilities, type registryweb-ui/src/mpegts/demux/— H.264/H.265 SPS parsing for interlace flagsweb-ui/src/mpegts/— plumbing through remuxer, worker messages, player coreweb-ui/src/components/player/— settings toggle, video-player slot/canvas integrationtools/devlab/— scan channels and README documentation🤖 Generated with Claude Code