fix(linux/kms): correct multi-plane monitor indexing and hardware-scaled capture - #5532
Draft
T-Gander wants to merge 5 commits into
Draft
fix(linux/kms): correct multi-plane monitor indexing and hardware-scaled capture#5532T-Gander wants to merge 5 commits into
T-Gander wants to merge 5 commits into
Conversation
kms_display_names() and display_t::init() each independently counted every active, non-cursor plane as one "monitor" when building/matching the display index. A CRTC with more than one simultaneously-active plane (e.g. gamescope's base + overlay layers) was counted twice by both passes, throwing their indices out of sync with each other and producing duplicate/misnumbered display names. Track counted CRTCs per card in both passes so a CRTC contributes to the index exactly once, regardless of how many active planes it has.
import_source() still logged at error level on every failed frame even after the round-1 fallback started rate-limiting the caller-side warnings, since the function has no memory of prior calls. Downgrade those internal logs to debug (callers already surface a rate-limited, actionable warning) and add a debug-level dump of the exact surface descriptor (fourcc, modifier, per-plane fd/offset/pitch) on failure, to compare against the real buffer's attribs when the import genuinely fails. Also check eglGetError() after a successful eglCreateImage(), not just after a null return. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
display_ram_t::snapshot() read GetTextureSubImage() using the CRTC's output viewport dimensions, not the imported plane texture's actual size. These differ when the display controller hardware-scales a plane at scanout (e.g. a game's native-resolution exclusive-fullscreen swapchain stretched to fill a higher-resolution output) -- kmsgrab imports the plane's raw pre-scale buffer via DMA-BUF, bypassing the CRTC scaler entirely, so the texture is the smaller native size while the destination buffer is sized for the full output. Reading past the texture's real bounds triggered GL_INVALID_VALUE every frame, which manifested as a black screen for the whole scaled-plane session via the existing import-failure fallback (the "Failed to bind EGLImage" log line was a stray error from this call, not the import itself). Clamps the read to the texture's real dimensions and uses GL_PACK_ROW_LENGTH so a smaller read still lands correctly in the destination buffer's top-left corner instead of shearing across rows. This does not reproduce the hardware scaling itself -- the rest of the frame is left as whatever the buffer previously contained -- but it resolves the crash/black-screen and captures the correct content at its native resolution. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aled Replaces the clamp-and-crop fix from 14a6f2a with the real fix: when the display controller is hardware-scaling a plane at scanout (its CRTC destination rect differs from the plane's native buffer size -- e.g. a game's native-resolution exclusive-fullscreen swapchain stretched to fill a higher-resolution output), kmsgrab's raw DMA-BUF import bypasses that scaler entirely and only ever sees the pre-scale buffer. Reproduce the same upscale with a linear-filtered glBlitFramebuffer into a scratch texture sized to the configured capture resolution, before the existing GetTextureSubImage readback, so the captured frame matches what the display actually shows instead of being cropped to the plane's native corner. The scratch texture/FBOs are lazily created on first use and reused across frames; the common case (plane already fills the output, no scaling needed) is unaffected and takes the same path as before. Untested against real hardware in this session -- built by cross-referencing this project's own glad config (gl:compatibility=4.6, confirming BlitFramebuffer/GL_READ_FRAMEBUFFER/GL_DRAW_FRAMEBUFFER are generated) and the existing FBO helper patterns already used elsewhere in this file, but needs a live test to confirm the blit against an EGLImage-backed source texture behaves as expected on this driver. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
T-Gander
marked this pull request as draft
August 20, 2026 07:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Human Overview of changes:
This bug fix allows my steam lxc running Sunshine + Gamescope with a dummy display port connector to successfully render and stream. I'll be happy to run this custom patched version for just myself, however I thought I would post this fix so for if anyone else in the community is looking for it or wants to clean it up, they can or they can use it too. I've spent a month of troubleshooting other avenues before ending up looking at Sunshines code and ending up with this solution.
This PRs code changes are entirely AI generated, but has been tested on my hardware and is working for me (RX 9060 XT). YMMV.
This PR comprises of two bug fixes.
AI Overview of changes:
Fixes several bugs in the Linux KMS capture backend (
kmsgrab) related to multi-plane monitors and hardware-scaled scanout.kms_display_names()anddisplay_t::init()each counted every active, non-cursor plane as a separate monitor. A CRTC with more than one simultaneously-active plane (e.g. gamescope's base + overlay layers) was counted twice by both passes, throwing their indices out of sync and producing duplicate/misnumbered display names. Both passes now count active CRTCs instead, so a CRTC contributes to the index exactly once regardless of how many planes are active on it.kmsgrab's raw DMA-BUF import bypasses that scaler and only ever sees the pre-scale buffer. The texture readback was sized to the CRTC's output viewport rather than the actual imported texture, which triggeredGL_INVALID_VALUEevery frame and manifested as a black screen for the whole session. The capture path now reproduces the hardware upscale itself via a linear-filteredglBlitFramebufferinto a scratch texture sized to the configured capture resolution before readback, so the captured frame matches what the display actually shows. The scratch texture/FBOs are lazily created and reused; the common unscaled case is unaffected.Tested against real hardware with a hardware-scaled plane at scanout (native-resolution exclusive-fullscreen swapchain stretched to a higher-resolution output).
Screenshot
N/A — not a UI change.
Issues Fixed or Closed
None referenced.
Roadmap Issues
None.
Type of Change
Checklist
AI Usage
See our AI usage policy.