feat(streaming): add Kafka two-pass analysis service

[phodal]

Summary by CodeRabbit

• New Features

  • Added Kafka-based streaming analysis for code changes.
  • Produces raw AST, resolved AST, and diagnostic events.
  • Supports project state tracking, file updates and deletions, duplicate suppression, and exactly-once processing.
  • Added Java source analysis with same-package reference resolution.
  • Added Docker Compose configuration for local Kafka setup.

• Documentation

  • Added setup instructions, event formats, topic details, configuration options, and operational guidance.

• Tests

  • Added coverage for event routing, deduplication, state retention, resolution, and diagnostics.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8cde375d-6091-4615-bea3-534305e73d67

📥 Commits

Reviewing files that changed from the base of the PR and between 83083ef and 03a20cf.

📒 Files selected for processing (15)

• README.md
• chapi-streaming/README.md
• chapi-streaming/build.gradle.kts
• chapi-streaming/compose.yaml
• chapi-streaming/src/main/kotlin/chapi/streaming/ChapiStreamingApplication.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/analysis/SourceAnalyser.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/analysis/StreamingAnalysisEngine.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/analysis/StreamingState.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/kafka/ChapiStreamingProcessor.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/kafka/ChapiStreamingTopology.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/kafka/KafkaStreamingState.kt
• chapi-streaming/src/main/kotlin/chapi/streaming/model/StreamingEvents.kt
• chapi-streaming/src/test/kotlin/chapi/streaming/analysis/StreamingAnalysisEngineTest.kt
• chapi-streaming/src/test/kotlin/chapi/streaming/kafka/ChapiStreamingTopologyTest.kt
• settings.gradle.kts

---

📝 Walkthrough

Walkthrough

Adds the chapi-streaming Kafka Streams module. It analyzes code-change events, maintains project state, emits raw and resolved AST events, publishes diagnostics, supports deletes and deduplication, and documents local Kafka operation and configuration.

Changes

Kafka streaming analysis pipeline

Layer / File(s)	Summary
Module contracts and runtime setup settings.gradle.kts, chapi-streaming/build.gradle.kts, chapi-streaming/compose.yaml, chapi-streaming/src/main/kotlin/chapi/streaming/model/StreamingEvents.kt, chapi-streaming/src/main/kotlin/chapi/streaming/analysis/SourceAnalyser.kt, chapi-streaming/src/main/kotlin/chapi/streaming/analysis/StreamingState.kt, chapi-streaming/src/main/kotlin/chapi/streaming/ChapiStreamingApplication.kt, README.md, chapi-streaming/README.md	Adds the Gradle module, Kafka runtime setup, serialized event models, Java analyzer registration, state abstractions, application lifecycle, local Kafka configuration, and streaming API documentation.
Stateful source analysis chapi-streaming/src/main/kotlin/chapi/streaming/analysis/StreamingAnalysisEngine.kt, chapi-streaming/src/test/kotlin/chapi/streaming/analysis/StreamingAnalysisEngineTest.kt	Validates and deduplicates events, updates project snapshots, performs raw and resolved analysis, handles deletes and failures, and emits diagnostics. Tests cover package resolution, re-resolution, package boundaries, and duplicate events.
Kafka topology and persistent processing chapi-streaming/src/main/kotlin/chapi/streaming/kafka/*, chapi-streaming/src/test/kotlin/chapi/streaming/kafka/ChapiStreamingTopologyTest.kt	Builds the Kafka topology, persists processed events and project sources, validates record keys, and routes raw AST, resolved AST, and diagnostic outputs. Topology tests cover routing, deduplication, source retention, resolution, and invalid keys.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Producer
  participant ChapiStreamingProcessor
  participant StreamingAnalysisEngine
  participant KafkaStreamingState
  participant OutputTopics
  Producer->>ChapiStreamingProcessor: publish keyed CodeChangeEvent
  ChapiStreamingProcessor->>KafkaStreamingState: check event and load project state
  ChapiStreamingProcessor->>StreamingAnalysisEngine: analyze event
  StreamingAnalysisEngine->>KafkaStreamingState: update sources and processed-event state
  StreamingAnalysisEngine-->>ChapiStreamingProcessor: return AST and diagnostic events
  ChapiStreamingProcessor->>OutputTopics: publish raw, resolved, and diagnostic events

Loading

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/issue-51-kafka-streaming

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03a20cf266

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not deduplicate blank event IDs

When the first event with a blank eventId reaches validation, it is recorded under the empty string; every subsequent blank-ID event handled by the same task then hits this check and is silently discarded, even when it belongs to a different project or file. This suppresses the required validation diagnostics after one malformed message, so blank IDs should not be stored or checked as normal deduplication keys.

Useful? React with 👍 / 👎.