Logging and speech to text refactoring - #18
Open
vinniefalco wants to merge 14 commits into
Open
Conversation
Move asset and gateway-config route tests into child modules so the production modules stay below their ratchet ceilings without changing behavior. Record the measured parent and test-module sizes in `module-ceilings.toml`.
The gateway command line has one job, to serve, so the `serve` subcommand and the positional config argument come out: the bare invocation serves with boot discovery, and `--config PATH` names an explicit config, winning over `PROMPTFORGE_GATEWAY_CONFIG`. `parse_args` drops its subcommand match and rejects any non-flag argument as a usage error; every repository-owned caller - the systemd unit, the installer, the Workshop launcher, the tray autostart entries, the release-test workflow, and the guides - moves to the new shape in the same change. - `init_logging` now runs after the already-running handoff check, so `--help`, `--version`, and a second-instance handoff never rotate the running gateway's log; the handoff's browser-open failure reports through `eprintln!` because no subscriber is installed yet. - `--version` is accepted at any position in the argument list, and a second `--config` is a usage error. - On the handoff path in `relaunch.rs`, the connection-file resolution warning is dropped with no subscriber installed; the boot that follows logs its own failure once logging is live. Plan: 2026-09-05-1-gateway-logging-cli
Move the log pipeline out of `crates/gateway/src/main.rs` into a new `gateway-logging` crate so the queue, rotation, sink, and worker lifecycle are owned and tested in one place. The crate exports `LogConfig`, `LogRuntime`, `LogWriter`, and the opaque `LogError`; `LogRuntime::start` rotates and opens `gateway.log`, spawns one worker thread, and `shutdown` closes admission, drains, flushes, and joins. `main.rs` keeps global subscriber installation, holds the returned `LogRuntime`, and shuts the logger down last so fatal error chains logged through `log_error_chain` reach the disk. - Queue policy is fixed in `queue.rs`: `CAPACITY` of 8192 records, drain `BATCH` of 256, one deque per `LogPriority` under one mutex. A full queue evicts the oldest Debug, then Trace, then Info; Warn and Error records are never evicted, and a producer with no eligible record blocks on a condition variable. - `LogEventWriter` buffers every `Write` call for one event and enqueues on `Drop`, moving the buffer through `String::from_utf8` and paying the lossy copy only for invalid UTF-8. It is public but `#[doc(hidden)]` because `MakeWriter::Writer` cannot name a private type. - A failed write or flush on the file sink falls back to synchronous stderr, and a worker panic surfaces from `shutdown` as a `LogError` that `is_io` classifies separately from filesystem and spawn failures. - Rotation keeps one previous run: an existing `gateway.log` renames to `gateway.log.1`, overwriting the older rotation. Plan: 2026-09-05-1-gateway-logging-cli
A failed gateway run must be discoverable without config knowledge, and no log record may carry secret material. The gateway gains a `diagnostics` subcommand that prints a read-only JSON report of the state dir, config, logs, and connection file, the log rotation retains five previous runs, and every queued record crosses a redaction pass that masks bearer tokens, authorization and cookie header values, and `api_key` assignments. - The log layout gets one owner: `LogConfig::log_path` and `LogConfig::retained_log_paths` name every path, so `diagnostics_json` enumerates the logs without starting a runtime and `open_log_file` rotates the same chain. - Redaction sits at the one chokepoint every record crosses: `LogEventWriter::drop` masks the formatted line with `redact_line` before the record enters the queue. - `is_running` in `shared-sidecar` is read-only: a stale or corrupt connection file reads as not-running and stays on disk for the next launch to clean. - `discover_in(explicit, gather)` splits the report's config discovery into a testable inner in the `resolve_in` pattern; unit tests pin the explicit, discovered, profile-fallback, and gather-failure branches, and both `diagnostics` integration tests assert `config.path` and `config.exists`. - `diagnostics` runs before the handoff check and before logging starts; it never serves, rotates a log, parses a config, or mutates the state directory, and `parse_diagnostics_args` accepts only `--config PATH`. - The generated config carries `# Diagnostics: promptforge-gateway diagnostics` as a comment, so the file stays parseable. - New tests pin the sink's stderr fallback on rejected writes and flushes, saturation that never evicts or duplicates Warn or Error records, and a shutdown that writes every record in enqueue order before the join returns. - `Sink::Null` and `Sink::is_stderr` are `#[cfg(test)]` seams for the fallback and latency tests. - `production_logging_stays_within_latency_budget` is `#[ignore]`d; it runs only through `cargo test -p gateway-logging --release -- --ignored`. Plan: 2026-09-05-1-gateway-logging-cli
The public `serve` docs linked the private `GRACEFUL_DRAIN_TIMEOUT` and `WORKER_JOIN_TIMEOUT` constants, which `RUSTDOCFLAGS="-D warnings" cargo doc` rejects as private intra-doc links. The constants are now plain backticked names. The break was introduced in 7f24bb0 and predates the logging work.
The logging contract now lives in the documentation, and the dependency boundary has a test that enforces it. A new integration test `the_manifest_declares_only_the_tracing_dependencies` reads the crate's own `Cargo.toml` and fails when any dependency other than `tracing` and `tracing-subscriber` appears. The `gateway-logging` `AGENTS.md`, the gateway `README.md`, and both gateway guides now describe the `gateway.log` rotation, the five-run retention, the redaction pass, and the `promptforge-gateway diagnostics` report. - The boundary test rejects build, dev, and target-specific dependency tables in addition to extra `[dependencies]` entries, so the allowlist covers every way a crate can enter the build. It parses the manifest line by line and adds no TOML parser dependency. - `AGENTS.md` records that `LogEventWriter` is public but `#[doc(hidden)]` because `MakeWriter::Writer` cannot name a private type, and that the test seams `Sink::Null` and `Sink::is_stderr` exist only under `cfg(test)`. Plan: 2026-09-05-1-gateway-logging-cli
Plan: 2026-09-05-1-gateway-logging-cli
Characterize current batch routing and realtime transcription before the speech subsystem changes. Separate physical-model routing checks from legacy socket cases, and add deterministic coverage for stream policy, generation, origin, ordering, shutdown, and final-model authority. - `crates/gateway-stt/tests/it/main.rs` now separates batch model selection from legacy socket characterization. - `fixture_runtime_with_models` starts caller-selected interim and final fixture models on a dedicated thread, while `TestServer::shutdown` stops the server before blocking runtime shutdown. - `batch_selects_each_loaded_physical_model_by_name` changes one vocabulary token to verify direct routing to each loaded physical model. - `final_model_segments_and_tail_are_authoritative_at_stop` verifies that interim text stays provisional and that the final worker produces committed segments and the remaining tail. - `crates/gateway-stt/tests/it/batch.rs` keeps its physical-model case ignored because it requires `tests/fixtures/`. Other native speech cases remain ignored for the same reason. Design: extends oversized-unit @ crates/gateway-stt/tests/common/mod.rs Design: new flag-parameter @ crates/gateway-stt/tests/common/mod.rs::fixture_runtime deps: bool Design: replaces flag-parameter @ crates/gateway-stt/tests/common/mod.rs::fixture_server deps: bool was: crates/gateway-stt/tests/it/stt.rs::fixture_server Design: new stringly-typed @ crates/gateway-stt/tests/common/mod.rs::multipart_body deps: &[u8],&str boundary: wire Design: new stringly-typed @ crates/gateway-stt/tests/common/mod.rs::transcribe_batch deps: &[f32],&str,SttState boundary: wire Design: replaces oversized-unit @ crates/gateway-stt/tests/it/legacy_stream.rs was: crates/gateway-stt/tests/it/stt.rs Design: new pure-function @ crates/gateway-stt/tests/it/legacy_stream.rs::transcript_words deps: &str Design: new pure-function @ crates/gateway-stt/tests/it/legacy_stream.rs::distinguishing_word deps: &str,&str Deferred: physical-model characterizations remain ignored without whisper fixtures Plan: 2026-09-05-2-generic-realtime-stt
Add an ignored native integration target that loads the packaged runtime and exact model fixture. It fixes interim, final, transcript-conditioning, glossary-prompt, silence-gating, and cleanup behavior so the upcoming engine split can be checked against one baseline. - `packaged_runtime_preserves_native_transcription_contract` copies the exact tiny model into a temporary directory, loads the packaged library, and configures the same model for interim and final decoding. - `JFK_TRANSCRIPT` anchors assertions for the full interim transcription, unprompted and transcript-conditioned final output, glossary bias, silence gating, and conditioning divergence. - `std::fs::remove_file` verifies that dropping both engines releases the copied model. - `#[ignore = "requires whisper test fixtures (tests/fixtures/)"]` keeps native characterization outside default test runs because it requires packaged runtime fixtures. Design: new oversized-unit @ crates/gateway-transcribe/tests/native_whisper.rs::packaged_runtime_preserves_native_transcription_contract Plan: 2026-09-05-2-generic-realtime-stt
Add canonical client, server, session, error, and sequence fixtures for realtime transcription. Validate one shared fixture set in Rust and the Workshop UI to pin strict fields, event ordering, capacity errors, item isolation, and hypothesis semantics before the subsystem changes. - `realtime-wire-fixtures.mjs` consumes the Gateway-owned fixtures directly, which makes Rust and Workshop share one canonical corpus. - `canonical_realtime_events_are_complete_strict_and_round_trip` enforces exact case sets, strict field sets, identifier separation, session defaults, and hypothesis composition. - `canonical_realtime_sequences_cover_valid_and_invalid_contract_paths` validates event order, error correlation, minimum audio, and recovery metadata across all declared sequences. - `crates/gateway-stt/tests/it/realtime_fixtures.rs` does not call production realtime parsers or session handlers, so these tests pin fixture consistency rather than implementation conformance. Design: new oversized-unit @ crates/gateway-stt/tests/it/realtime_fixtures.rs::assert_server_event_fields deps: &Value,&str Design: new oversized-unit @ crates/gateway-stt/tests/it/realtime_fixtures.rs::canonical_realtime_events_are_complete_strict_and_round_trip Design: new oversized-unit @ crates/gateway-stt/tests/it/realtime_fixtures.rs::canonical_realtime_sequences_cover_valid_and_invalid_contract_paths Violates: A2 - not determinable from diff Violates: A96 - not determinable from diff Deferred: batch transcription characterization is absent Deferred: native two-model characterization is absent Plan: 2026-09-05-2-generic-realtime-stt
Give the backend-neutral engine its intended role-specific identity. Update workspace metadata, runtime references, tests, fixture exclusions, and documentation while preserving the engine implementation. - `crates/gateway-stt-engine` changes the package, path, and Rust import identity. - `crates/gateway-stt-engine/src/engine.rs` and the other engine source files move with 100 percent similarity. - `crates/gateway-stt-engine/tests/native_whisper.rs` changes only its import path. The commit adds no test assertions or compatibility crate. Design: new shotgun-surgery @ crates/gateway-stt-engine/Cargo.toml Violates: A2 - not determinable from diff Violates: A96 - not determinable from diff Plan: 2026-09-05-2-generic-realtime-stt
Move per-take speech state out of decode workers so each decode job is independent and every take owns its lifecycle. Carry immutable guidance and finalized history with each job, aggregate segment results in one ordered pipeline, and preserve interim fallback after failures. - `Take` consolidates guidance, finalized history, segmentation, local agreement, transcript aggregation, completion, and failure behind one per-take state object. - `FinalJob` carries all decode inputs and the reply channel, so final-model workers retain no take identity or transcript between jobs. - `Segmenter` moves from the engine surface to the gateway speech surface with take orchestration. - `run_final_pipeline` serializes closed segments and the closing tail, records only successful sample boundaries, and stops decode work after the first failure. - `next_interim` promotes token prefixes confirmed by two hypotheses and keeps committed text append-only. - `TestServer` now bounds fixture server and runtime cleanup at 30 seconds. - `guidance` has no end-to-end assertion from runtime activation through both batch and streaming decode paths. Design: new surface-growth @ crates/gateway-stt-engine/src/engine.rs::SttEngine::transcribe boundary: pub Design: new surface-growth @ crates/gateway-stt-engine/src/engine.rs::SttEngine::transcribe_final boundary: pub Design: new shared-parameter-cluster @ crates/gateway-stt-engine/src/engine.rs::SttEngine::transcribe_final boundary: pub Design: new shared-parameter-cluster @ crates/gateway-stt-engine/src/final_pass.rs::FinalTranscriber::transcribe Design: new surface-growth @ crates/gateway-stt/src/lib.rs::Segmenter boundary: pub Design: new value-object @ crates/gateway-stt/src/take.rs::AgreementSnapshot Design: new parameter-object @ crates/gateway-stt/src/take.rs::Take Design: new shared-mutable-state @ crates/gateway-stt/src/take.rs::TakeState Design: new message-passing @ crates/gateway-stt/src/take.rs::FinalPipeline Design: new pure-function @ crates/gateway-stt/src/take.rs::matching_token_prefix_end deps: &str,&str Design: new pure-function @ crates/gateway-stt/src/take.rs::token_spans deps: &str Design: new pure-function @ crates/gateway-stt/src/take.rs::after_token_prefix deps: &str,usize Design: new oversized-unit @ crates/gateway-stt/src/take.rs Design: extends oversized-unit @ crates/gateway-stt/tests/common/mod.rs Design: extends oversized-unit @ crates/gateway-stt/tests/it/legacy_stream.rs Violates: A2 - not determinable from diff Violates: A96 - not determinable from diff Deferred: configured guidance propagation lacks an end-to-end assertion Plan: 2026-09-05-2-generic-realtime-stt
Make speech decoding backend-neutral while keeping Whisper construction, prompting, progress, and error translation in a safe adapter. Inject model factories into dedicated workers so model creation and decoding stay on their owning threads. Preserve batch and native transcription behavior through relocated and expanded tests. - `Decoder` and `ModelFactory` establish backend strategy contracts, while `WhisperModelFactory` contains safe model construction and decode policy. - `SttEngine::new` constructs each decoder on its owning worker and returns initialization errors before activation. - `native_whisper.rs` relocates native characterization and adds isolation, optional-final, and load-progress checks. These fixture-dependent tests remain ignored. - `std::sync::mpsc::channel` leaves both worker job queues unbounded. - `require_fixture` duplicates native fixture loading across unit and integration test support. Design: new strategy @ crates/gateway-stt-engine/src/decoder.rs::Decoder boundary: pub Design: new surface-growth @ crates/gateway-stt-engine/src/decoder.rs::Decoder boundary: pub Design: new strategy @ crates/gateway-stt-engine/src/decoder.rs::ModelFactory boundary: pub Design: new surface-growth @ crates/gateway-stt-engine/src/decoder.rs::ModelFactory boundary: pub Design: new facade @ crates/gateway-stt-backend-whisper/src/lib.rs boundary: pub Design: new constructor-injection @ crates/gateway-stt-engine/src/engine.rs::SttEngine::new Design: new flag-parameter @ crates/gateway-stt-backend-whisper/src/model.rs::WhisperDecoder::load Design: new flag-parameter @ crates/gateway-stt-engine/src/worker.rs::worker_loop deps: &dyn ModelFactory,&std::sync::mpsc::Receiver<Job>,&std::sync::mpsc::SyncSender<Result<bool, TranscribeError>>,bool Design: replaces shared-mutable-state @ crates/gateway-stt/src/runtime.rs::SttSlot was: crates/gateway-stt-engine/src/slot.rs::SttSlot Design: new global-state @ crates/gateway-stt-backend-whisper/src/prompt.rs::NATIVE_TEST Design: new global-state @ crates/gateway-stt-backend-whisper/tests/native_whisper.rs::NATIVE_TEST Design: new clone-block @ crates/gateway-stt/src/test_fixtures.rs Design: new clone-block @ crates/gateway-stt/tests/common/mod.rs Violates: A2 - crates/gateway-stt/src/runtime.rs is not determinable from diff Pending: N9 - compounds Deferred: model worker queues remain unbounded Deferred: native Whisper characterization remains ignored behind external fixtures Plan: 2026-09-05-2-generic-realtime-stt
Add mandatory checks for workspace edges, module cycles, public exports, source size, migration targets, and unsafe isolation. Split compiler-resolved checks from strict policy checks and run both in the normal continuous integration path. - `parseCargoModulesDot` is a 110-line parser that collapses item edges into module edges and rejects malformed graph output. - `module-ceilings.toml` files set strict source and public-root ceilings and name each planned migration target. - `architecture` runs with pinned tool versions in the normal continuous integration job. Design: new global-state @ crates/gateway-stt/tests/it/architecture.rs::workspace_metadata Design: new oversized-unit @ tools/check-stt-architecture.mjs::parseCargoModulesDot deps: output Violates: A2 - not determinable from diff Plan: vibe/2026-09-05-2-generic-realtime-stt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the commit log and plans