DO NOT MERGE: continuous prompt-toolkit application - does not fix the toolbar flicker (findings recorded)

CHANGELOG.md

@@ -1,10 +1,23 @@
-## 4.2.4 (TBD)
+## 4.3.0 (TBD)
 
 - Bug Fixes
     - Fixed the right prompt being redrawn beside every accepted command line in the scrollback.
       prompt-toolkit includes it in the final frame of each prompt, which is the frame left on the
       terminal; it is now hidden there, as the bottom toolbar already was, and stays on the live
       prompt line only
+    - Fixed an intermittent bottom toolbar flicker when running a command. The toolbar's
+      prompt-toolkit application now stays running across both the prompt and command execution
+      instead of stopping and restarting for each one, since prompt-toolkit drops the bottom toolbar
+      from the final frame of every application run.
+- Enhancements
+    - `enable_bottom_toolbar=True` now keeps the toolbar visible and refreshing during command
+      execution
+    - Added an embedded pager which `Cmd.ppaged()` uses while the bottom toolbar is running, so the
+      toolbar stays visible and refreshing instead of the terminal being handed to an external
+      pager. It supports vertical and horizontal scrolling, incremental search, and chopped lines on
+      every platform, including Windows where the default external pager (`more`) always wraps.
+      Output which already fits on the screen is printed directly rather than paged. Set
+      `self.use_builtin_pager = False` to keep using the external `pager`/`pager_chop` commands.
 
 ## 4.2.3 (September 2, 2026)
 

cmd2/pager.py

@@ -0,0 +1,229 @@
+"""A pager view hosted by the main prompt-toolkit application."""
+
+import re
+import threading
+from collections.abc import Callable
+from functools import partial
+
+from prompt_toolkit.document import Document
+from prompt_toolkit.filters import has_focus, is_searching, to_filter
+from prompt_toolkit.formatted_text import ANSI, fragment_list_to_text, to_formatted_text
+from prompt_toolkit.formatted_text.base import StyleAndTextTuples
+from prompt_toolkit.formatted_text.utils import split_lines
+from prompt_toolkit.key_binding import KeyBindings
+from prompt_toolkit.key_binding.bindings import search
+from prompt_toolkit.key_binding.key_processor import KeyPressEvent
+from prompt_toolkit.layout import HSplit, Window
+from prompt_toolkit.layout.containers import ConditionalContainer
+from prompt_toolkit.layout.controls import FormattedTextControl, UIContent
+from prompt_toolkit.lexers import Lexer
+from prompt_toolkit.search import SearchDirection, start_search
+from prompt_toolkit.utils import get_cwidth
+from prompt_toolkit.widgets import SearchToolbar, TextArea
+
+_OSC_RE = re.compile(r"\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)")
+
+
+def _fragments(text: str) -> StyleAndTextTuples:
+    """Parse captured output, dropping the trailing newline that ends its last line."""
+    # prompt-toolkit's ANSI parser does not handle OSC sequences, including Rich
+    # hyperlinks. Remove their metadata while retaining visible text and SGR styles.
+    text = _OSC_RE.sub("", text)
+    return to_formatted_text(ANSI(text.removesuffix("\n")))
+
+
+def output_fits(text: str, columns: int, rows: int, *, chop: bool) -> bool:
+    """Check rendered line heights, including wrapping and wide Unicode characters.
+
+    Measuring the text itself keeps output that needs no scrolling from paying for a
+    Pager's widgets and key bindings, which would only be built to be thrown away.
+    """
+    lines = fragment_list_to_text(_fragments(text)).split("\n")
+    if chop:
+        # Even one wide line needs a pager so its hidden columns remain
+        # accessible through horizontal scrolling.
+        return len(lines) <= rows and all(get_cwidth(line) <= columns for line in lines)
+    # Reuse prompt-toolkit's own wrapping arithmetic so this matches what a Pager
+    # would render, without building a control to ask on the UI thread.
+    content = UIContent(get_line=lambda number: [("", lines[number])], line_count=len(lines))
+    height = 0
+    for line in range(content.line_count):
+        height += content.get_height_for_line(line, columns, None)
+        if height > rows:
+            return False
+    return True
+
+
+class _AnsiLexer(Lexer):
+    """Preserve captured Rich styles while searching and scrolling plain text."""
+
+    def __init__(self, fragments: StyleAndTextTuples) -> None:
+        self.lines = list(split_lines(fragments))
+
+    def lex_document(self, document: Document) -> Callable[[int], StyleAndTextTuples]:  # noqa: ARG002
+        def get_line(number: int) -> StyleAndTextTuples:
+            return self.lines[number] if number < len(self.lines) else []
+
+        return get_line
+
+
+class Pager:
+    """Scrollable, searchable output; the host supplies the persistent toolbar."""
+
+    def __init__(self, text: str, *, chop: bool) -> None:
+        """Build an independent view without creating an application or input reader."""
+        self.closed = threading.Event()
+        self.on_close: Callable[[], None] = self.closed.set
+        self.chop = chop
+        fragments = _fragments(text)
+        self.search = SearchToolbar()
+        self.text = TextArea(
+            text=fragment_list_to_text(fragments),
+            lexer=_AnsiLexer(fragments),
+            read_only=True,
+            wrap_lines=not chop,
+            search_field=self.search,
+        )
+        self.text.window.always_hide_cursor = to_filter(True)
+        self.container = HSplit(
+            [
+                self.text,
+                self.search,
+                ConditionalContainer(
+                    Window(
+                        FormattedTextControl(" Space/PgDn: next  b/PgUp: back  /: search  n: next match  q: quit"),
+                        height=1,
+                        style="class:bottom-toolbar",
+                    ),
+                    filter=~is_searching,
+                ),
+            ]
+        )
+        self.bindings = bindings = KeyBindings()
+        focused = has_focus(self.text)
+
+        @bindings.add("<any>", filter=focused)
+        def ignore(event: KeyPressEvent) -> None:
+            # Pager keystrokes must not become commands at the next prompt.
+            pass
+
+        @bindings.add("q", filter=focused)
+        @bindings.add("escape", filter=focused, eager=True)
+        @bindings.add("c-c", filter=focused)
+        def close(event: KeyPressEvent) -> None:  # noqa: ARG001
+            self.on_close()
+
+        for keys, pages in (
+            ((" ", "pagedown", "f", "c-f"), 1.0),
+            (("b", "pageup", "c-b"), -1.0),
+            (("d", "c-d"), 0.5),
+            (("u", "c-u"), -0.5),
+        ):
+            for key in keys:
+                bindings.add(key, filter=focused)(partial(self._scroll_page, pages=pages))
+
+        @bindings.add("j", filter=focused)
+        @bindings.add("down", filter=focused)
+        @bindings.add("enter", filter=focused)
+        def down(event: KeyPressEvent) -> None:
+            self._scroll(event, 1)
+
+        @bindings.add("k", filter=focused)
+        @bindings.add("up", filter=focused)
+        def up(event: KeyPressEvent) -> None:
+            self._scroll(event, -1)
+
+        @bindings.add("right", filter=focused)
+        @bindings.add("l", filter=focused)
+        def right(event: KeyPressEvent) -> None:
+            self._scroll_horizontal(event, 1)
+
+        @bindings.add("left", filter=focused)
+        @bindings.add("h", filter=focused)
+        def left(event: KeyPressEvent) -> None:
+            self._scroll_horizontal(event, -1)
+
+        @bindings.add("g", filter=focused)
+        @bindings.add("home", filter=focused)
+        def first(event: KeyPressEvent) -> None:
+            event.current_buffer.cursor_position = 0
+
+        @bindings.add("G", filter=focused)
+        @bindings.add("end", filter=focused)
+        def last(event: KeyPressEvent) -> None:
+            event.current_buffer.cursor_position = len(event.current_buffer.text)
+
+        @bindings.add("/", filter=focused)
+        def find(event: KeyPressEvent) -> None:  # noqa: ARG001
+            start_search(direction=SearchDirection.FORWARD)
+
+        @bindings.add("?", filter=focused)
+        def find_backwards(event: KeyPressEvent) -> None:  # noqa: ARG001
+            start_search(direction=SearchDirection.BACKWARD)
+
+        @bindings.add("n", filter=focused)
+        def next_match(event: KeyPressEvent) -> None:
+            event.current_buffer.apply_search(event.app.current_search_state, include_current_position=False)
+
+        @bindings.add("N", filter=focused)
+        def previous_match(event: KeyPressEvent) -> None:
+            event.current_buffer.apply_search(~event.app.current_search_state, include_current_position=False)
+
+        # Explicit search bindings also work when the main prompt uses Vi mode.
+        bindings.add("enter", filter=is_searching)(search.accept_search)
+        bindings.add("escape", filter=is_searching, eager=True)(search.abort_search)
+        bindings.add("c-c", filter=is_searching)(search.abort_search)
+
+    @staticmethod
+    def _column_at_width(line: str, width: int) -> int:
+        used = 0
+        for column, char in enumerate(line):
+            if used >= width:
+                return column
+            used += get_cwidth(char)
+        return len(line)
+
+    def _scroll_page(self, event: KeyPressEvent, *, pages: float) -> None:
+        info = self.text.window.render_info
+        if info is not None:
+            amount = max(1, int(max(1, info.window_height - 1) * abs(pages)))
+            self._scroll(event, amount if pages > 0 else -amount)
+
+    def _scroll(self, event: KeyPressEvent, rows: int) -> None:
+        """Move by display rows, including within lines taller than the viewport."""
+        info = self.text.window.render_info
+        if info is None or info.window_width == 0:
+            return
+        document = event.current_buffer.document
+        line = document.cursor_position_row
+        wrapped_row = 0 if self.chop else get_cwidth(document.current_line_before_cursor) // info.window_width
+        target = wrapped_row + rows
+
+        def height(number: int) -> int:
+            return 1 if self.chop else info.get_height_for_line(number)
+
+        while target < 0 and line > 0:
+            line -= 1
+            target += height(line)
+        while target >= height(line) and line < document.line_count - 1:
+            target -= height(line)
+            line += 1
+        target = max(0, min(target, height(line) - 1))
+        # Chopped lines never wrap, so leave the cursor at the column the reader
+        # scrolled to. Moving it to the start of the line would drag the view back
+        # with it, since the window scrolls horizontally to keep the cursor visible.
+        width = self.text.window.horizontal_scroll if self.chop else target * info.window_width
+        column = self._column_at_width(document.lines[line], width)
+        event.current_buffer.cursor_position = document.translate_row_col_to_index(line, column)
+        self.text.window.vertical_scroll = line
+        self.text.window.vertical_scroll_2 = target if height(line) > info.window_height else 0
+
+    def _scroll_horizontal(self, event: KeyPressEvent, direction: int) -> None:
+        info = self.text.window.render_info
+        if info is None or not self.chop:
+            return
+        document = event.current_buffer.document
+        target = max(0, self.text.window.horizontal_scroll + direction * max(1, info.window_width // 2))
+        column = self._column_at_width(document.current_line, target)
+        event.current_buffer.cursor_position = document.translate_row_col_to_index(document.cursor_position_row, column)
+        self.text.window.horizontal_scroll = get_cwidth(document.current_line[:column])

cmd2/utils.py

@@ -690,6 +690,9 @@ def __init__(
         self.saved_pipe_proc_reader = pipe_proc_reader
         self.saved_redirecting = saved_redirecting
 
+        # Holds the bottom toolbar's suspension while a pipe process owns the terminal
+        self.toolbar_suspension: contextlib.ExitStack | None = None
+
 
 def categorize(func: Callable[..., Any] | Iterable[Callable[..., Any]], category: str) -> None:
     """Categorize a function.

docs/features/initialization.md

@@ -52,6 +52,7 @@ Here are instance attributes of `cmd2.Cmd` which developers might wish to overri
 - **max_completion_table_items**: The maximum number of completion results allowed for a completion table to appear (Default: 50)
 - **pager**: sets the pager command used by the `Cmd.ppaged()` method for displaying wrapped output using a pager
 - **pager_chop**: sets the pager command used by the `Cmd.ppaged()` method for displaying chopped/truncated output using a pager
+- **use_builtin_pager**: defaults to `enable_bottom_toolbar`. While the command toolbar is running, `Cmd.ppaged()` uses an embedded pager that keeps that toolbar visible. Set to `False` to always use the external `pager`/`pager_chop` commands. This is an opt-*out* only: setting it to `True` does nothing unless `enable_bottom_toolbar` is also set, because the embedded pager shares the toolbar's display.
 - **py_bridge_name**: name by which embedded Python environments and scripts refer to the `cmd2` application by in order to call commands (Default: `app`)
 - **py_locals**: dictionary that defines specific variables/functions available in Python shells and scripts (provides more fine-grained control than making everything available with **self_in_py**)
 - **quiet**: if `True`, then completely suppress nonessential output (Default: `False`)

docs/features/os.md

@@ -42,7 +42,48 @@ system.
 
 ## Terminal pagers
 
-Output of any command can be displayed one page at a time using the [cmd2.Cmd.ppaged][] method.
+Output of any command can be displayed one page at a time using the [cmd2.Cmd.ppaged][] method. A
+pager is only used when the terminal is interactive. Inside a script, or when the command's output
+is redirected or piped, `ppaged()` falls back to `poutput()`.
+
+### Embedded pager
+
+While the [bottom toolbar](./prompt.md#bottom-toolbar) is running, `ppaged()` displays its output in
+an embedded pager rather than handing the terminal to an external program. The toolbar stays visible
+and keeps refreshing beneath the paged output, and `cmd2` never gives up control of the terminal.
+
+Output which already fits in the space above the toolbar is printed directly, so there is no pager
+to dismiss.
+
+| Keys                               | Action                                            |
+| ---------------------------------- | ------------------------------------------------- |
+| `Space`, `PageDown`, `f`, `Ctrl-F` | Forward one page                                  |
+| `b`, `PageUp`, `Ctrl-B`            | Back one page                                     |
+| `d` / `u`, `Ctrl-D` / `Ctrl-U`     | Forward / back half a page                        |
+| `j` / `k`, `Down` / `Up`, `Enter`  | Down / up one line                                |
+| `h` / `l`, `Left` / `Right`        | Scroll left / right (chopped output only)         |
+| `g` / `G`, `Home` / `End`          | Jump to the beginning / end                       |
+| `/` / `?`                          | Search forward / backward                         |
+| `n` / `N`                          | Jump to the next / previous match                 |
+| `q`, `Escape`, `Ctrl-C`            | Close the pager and return to the running command |
+| `Ctrl-Z`                           | Suspend to the background, where supported        |
+
+Passing `chop=True` truncates lines longer than the screen width instead of wrapping them, and the
+hidden columns remain reachable with the left and right arrow keys. Unlike the default external
+pager on Windows (`more`), the embedded pager supports chopping on every platform.
+
+Rich styles in the paged output are preserved: colors are rendered while scrolling and searching,
+and the visible text of a hyperlink is shown without its escape sequence metadata.
+
+### External pagers
+
+Set `self.use_builtin_pager = False` to always use the external commands configured in `self.pager`
+and `self.pager_chop`. This is the opt-out for applications which need `less` features the embedded
+pager does not provide. External pagers temporarily hide the toolbar and restore it on exit.
+
+The setting only ever turns the embedded pager off. Because the embedded pager shares the toolbar's
+display, it is unavailable anywhere a toolbar is not running, such as a command invoked outside the
+command loop, and `ppaged()` uses the external pager there regardless of this setting.
 
 Alternatively, a terminal pager can be invoked directly using the ability to run shell commands with
 the `!` shortcut like so:
@@ -51,7 +92,7 @@ the `!` shortcut like so:
 
 !!! warning
 
-    Once you are in a terminal pager, that program temporarily has control of your terminal,
+    Once you are in an external terminal pager, that program temporarily has control of your terminal,
     **NOT** `cmd2`. Typically you can use either the arrow keys or `<PageUp>`/`<PageDown>` keys to
     scroll around or type `q` to quit the pager and return control to your `cmd2` application.