From 5ab0caf6035fc785458fcdb222193baebf84f91f Mon Sep 17 00:00:00 2001 From: swinston Date: Mon, 13 Jul 2026 13:34:25 -0700 Subject: [PATCH] Refactor the LLM-Assisted Vulkan Development tutorial to better align with feedback received. --- .../02_environment_setup/01_introduction.adoc | 60 ++++---- .../02_jetbrains_clion_android_studio.adoc | 105 ++++++-------- .../03_visual_studio.adoc | 49 ++++--- .../02_environment_setup/04_xcode_apple.adoc | 72 ++++------ .../05_goose_native_agent.adoc | 116 ++++++++-------- .../06_mcp_context_bridge.adoc | 50 +++---- .../01_introduction.adoc | 42 +++--- .../02_base_models.adoc | 112 +++++++-------- .../03_hardware_vram.adoc | 114 +++++++--------- .../04_rag_mcp_specialization.adoc | 128 ++++++++---------- .../05_fine_tuning_lora.adoc | 107 +++++++-------- .../04_multimodal_ai/01_introduction.adoc | 32 ++--- .../02_multimodal_models.adoc | 55 ++++---- .../03_visual_bug_diagnosis.adoc | 69 +++++----- .../04_expectations_limits.adoc | 28 ++-- .../05_workflow/01_introduction.adoc | 34 ++--- .../05_workflow/02_phase_1_planning.adoc | 74 +++++----- .../03_phase_2_implementation.adoc | 64 +++++---- .../05_workflow/04_phase_3_refactoring.adoc | 62 ++++----- .../06_debugging/01_introduction.adoc | 42 +++--- .../06_debugging/02_vuid_autofix.adoc | 70 +++++----- .../03_renderdoc_ai_integration.adoc | 72 +++++----- .../04_shader_debugging_logs.adoc | 53 ++++---- .../06_debugging/05_gfxreconstruct_ai.adoc | 64 ++++----- .../07_advanced_mcp/01_introduction.adoc | 36 ++--- .../02_vulkan_registry_mcp.adoc | 40 +++--- .../03_agentic_automation_qa.adoc | 83 ++++++------ .../08_deployment/01_introduction.adoc | 30 ++-- .../08_deployment/02_android_ios_mobile.adoc | 46 +++---- .../03_embedded_safety_critical.adoc | 44 +++--- en/AI_Assisted_Vulkan/09_faq.adoc | 37 ++--- en/AI_Assisted_Vulkan/10_final_project.adoc | 24 ++-- en/AI_Assisted_Vulkan/introduction.adoc | 54 ++++---- en/conclusion.adoc | 10 +- 34 files changed, 968 insertions(+), 1110 deletions(-) diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc index 418b645d..c2ed5b72 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc @@ -1,44 +1,44 @@ :pp: {plus}{plus} -= The Modern AI Toolbelt: Environment Setup += Environment Setup -== Introduction: The Evolution of the "Lab" +== Introduction -In traditional graphics programming, setting up your environment was a static process: install a compiler (GCC/Clang/MSVC), configure a debugger, and point your IDE to the Vulkan SDK. Your relationship with these tools was strictly one-way—you gave commands, and the tools executed them. +In traditional graphics programming, setting up your environment is a one-time, one-way process: install a compiler (GCC/Clang/MSVC), configure a debugger, and point your IDE at the Vulkan SDK. The tools don't know anything about your project beyond what you tell them at build time. -To truly embrace **Collaborative Engineering**, we must rethink the "Development Lab" as a living ecosystem. We aren't just adding a "chat plugin" to our IDE; we are building a **Context Bridge**. This bridge allows a high-reasoning AI model to see what you see, understand your architectural intent, and even query the hardware limits of your specific GPU. +An AI-assisted setup adds another layer: a **Context Bridge** that gives a model visibility into your codebase, your build configuration, and (via MCP) the Vulkan specification and your GPU's actual limits. This chapter covers what that setup looks like and why it's worth the effort. -This chapter sets the foundation for this transformation, moving from a "Traditional Setup" to an **AI-Enhanced Hub**. +== The problem it solves: the context gap -== The Core Concept: The "Context Gap" +The most common frustration with AI coding tools is what we'll call the **context gap**. You ask an assistant to "add a new image barrier," and it hands back a generic snippet using `VkImageMemoryBarrier` (Vulkan 1.0) instead of the `VkDependencyInfo` (Vulkan 1.3) pattern your engine actually uses. It may also drop the change in the wrong file relative to your engine's conventions. Fixing that costs more time than writing it yourself would have, because now you're explaining your engine's abstractions after the fact instead of before. -The most common frustration for developers using AI is the **Context Gap**. You ask an AI to "Add a new image barrier," and it gives you a generic snippet that uses `VkImageMemoryBarrier` (Vulkan 1.0) instead of the `VkDependencyInfo` (Vulkan 1.3) pattern your engine requires. It might even do this without regard to where in the engine it should and you're left with hunting down changes in multiple files using multiple versions of Vulkan. You then spend twice as long as writing it yourself, explaining your engine's abstractions and core requirements to an AI effectively doing the AI's job for it. +A properly configured setup narrows this gap along three axes: -An AI-integrated setup aims to eliminate this friction by providing three distinct layers of context. First, **Syntactic Context** leverages the IDE's deep understanding of your codebase structure and Abstract Syntax Tree (AST). When the AI knows your `Texture` class wraps a `vk::raii::Image`, it won't suggest incompatible raw pointer operations. Second, **Semantic Context** connects the AI directly to the official Vulkan specification and `vk.xml` registry via the **Model Context Protocol (MCP)**, allowing it to "look up" exactly what a bitmask means. Finally, **Runtime Context** provides the AI with the reality of your specific hardware. Modern agents can query `vulkaninfo` or your engine's logs to understand why a pipeline creation failed and suggest a fix based on your actual hardware limits. +- **Syntactic context** — the IDE's own understanding of your codebase (its AST, macro expansions, included headers). If the AI knows your `Texture` class wraps a `vk::raii::Image`, it won't suggest raw pointer operations that don't compile. +- **Semantic context** — a live connection to the Vulkan specification and `vk.xml` registry via the Model Context Protocol (MCP), so the model can look up what a bitmask or extension actually means instead of relying on memorized training data. +- **Runtime context** — access to `vulkaninfo` output or your engine's own logs, so the model can reason about why a pipeline creation call failed on your specific hardware rather than guessing in the abstract. -== Strategy: The "Collaborative Engineering" Loop +== How much to delegate -We don't want an AI that "writes the engine for us." That leads to unmaintainable code and "Black Box" architectures. Instead, we focus on **Incremental Wins** that build trust and speed. +It's worth being explicit about the failure mode here: an agent that's told to "write the engine" with no oversight will produce code you don't understand and can't maintain. The useful middle ground is delegating specific, bounded tasks and reviewing the output, rather than delegating architecture decisions wholesale. -=== The "Boilerplate" Win -Vulkan is one of the more verbose APIs in modern graphics. A single `VkGraphicsPipelineCreateInfo` can span 200 lines. In our new environment, we describe the *intent* ("Create a pipeline for a deferred G-Buffer pass with 4x MSAA and depth testing enabled") and let the AI generate the verbose structure. We then **audit** the result, rather than typing it from scratch. +Two examples of that middle ground: -=== The Specification Assistant Win -Imagine you're debugging a synchronization issue. Instead of tabbing out to a browser, you ask your AI assistant: *"Based on our current use of `VK_IMAGE_LAYOUT_ATTACHMENT_OPTIMAL`, what are the required access masks for a transition to `SHADER_READ_ONLY_OPTIMAL` on an Adreno 750?"* Because the assistant has the **Context Bridge**, it provides a spec-accurate answer tailored to your target hardware. +**Boilerplate.** A single `VkGraphicsPipelineCreateInfo` can run 200 lines. Describing the intent — "a pipeline for a deferred G-buffer pass with 4x MSAA and depth testing enabled" — and having the model fill in the structure, then reviewing the result, is usually faster and no less correct than typing it by hand, provided you actually read what comes back. -== Choosing Your Primary Interface: IDE vs. Agent +**Spec lookups.** If you're debugging a synchronization issue, you can ask an assistant with MCP access something like "given our use of `VK_IMAGE_LAYOUT_ATTACHMENT_OPTIMAL`, what access masks does a transition to `SHADER_READ_ONLY_OPTIMAL` need on an Adreno 750?" instead of tabbing over to the spec yourself. The value here is retrieval, not reasoning — it saves you a lookup, not a decision. -A common point of confusion is whether to use an "In-IDE" assistant or an "External Agent." In a professional Vulkan workflow, you need both. Your **Integrated Assistant** (like JetBrains AI or GitHub Copilot) lives in your "Inner Loop," excelling at real-time autocomplete and providing instant explanations of the code under your cursor. Conversely, your **Task-Oriented Agent** (like Goose or Aider) lives in your "Outer Loop." You can give it high-level tasks—such as migrating descriptor management to Descriptor Buffers—and it will work across multiple files, run the compiler, and iterate until the build passes. +== IDE assistant vs. standalone agent -In the next generation version of the IDEs this demarcation is blurred, with AI agents becoming more integrated into the IDE's core functionality. However, the distinction remains useful for understanding the roles and responsibilities of each tool. +There are two broad categories of tool, and most Vulkan workflows end up using both. An **in-IDE assistant** (JetBrains AI, GitHub Copilot) sits in your inner loop — autocomplete, quick explanations of the code under your cursor. A **task-oriented agent** (Goose, Aider) sits in your outer loop — you hand it something like "migrate descriptor management to descriptor buffers" and it works across files, runs the compiler, and iterates until the build passes. -We will show you how to set up both, creating a hybrid environment where the AI is as much a part of your team as a human colleague. +The line between these two categories is blurring as IDEs add more agentic features natively, but the distinction is still a useful way to think about which tool to reach for on a given task. -== First Step: Building the Vulkan MCP Server +== First step: building the Vulkan MCP server -Before diving into any platform-specific setup, complete this one-time prerequisite. Every IDE and agent chapter that follows assumes you have already cloned and built the **mcp-Vulkan** server. The resulting binary is what you will point each tool at with the path `/path/to/mcp-Vulkan/vulkan/build/index.js`. +Before the platform-specific chapters, there's one prerequisite: build the **mcp-Vulkan** server. Every chapter that follows assumes it's already built and points each tool at `/path/to/mcp-Vulkan/vulkan/build/index.js`. -You will need **Node.js** (LTS) and **npm** installed. Download the LTS installer from link:https://nodejs.org/[nodejs.org] if you do not already have them. +You'll need **Node.js** (LTS) and **npm**. Get the LTS installer from link:https://nodejs.org/[nodejs.org] if you don't have them already. [source,bash] ---- @@ -53,22 +53,14 @@ npm install npm run build ---- -Note the absolute path to the `mcp-Vulkan` directory (run `pwd` inside it). Every platform chapter uses this path when registering the server, shown as `/path/to/mcp-Vulkan/vulkan/build/index.js`. +Note the absolute path to the `mcp-Vulkan` directory (run `pwd` inside it) — every platform chapter needs it when registering the server. -== The Environment Roadmap: From IDE to Agent +== What's next -In the following chapters, we have structured the guide to cover the major development ecosystems used by graphics engineers. You don't need to read them all—focus on the one that matches your daily "Lab." +The following chapters cover the major development environments used in graphics programming. You don't need to read all of them — pick the one that matches your setup. -We begin with **JetBrains CLion & Android Studio**, the standard-bearers for deep C{pp} indexing and mobile-first Vulkan development. You will learn how to leverage their sophisticated "semantic awareness" to provide the AI with a deep understanding of your codebase's structure. +We start with **JetBrains CLion & Android Studio**, covering their code indexing and how it reduces AI hallucinations. For Windows, there's a **Visual Studio** chapter focused on connecting its debugger state to chat-based assistants. For Apple developers, the **Xcode & Apple Silicon** chapter covers on-device inference and the memory advantages of M-series chips. -For those on Windows, we explore the **Visual Studio** experience, focusing on its comprehensive debugging tools and how to bridge them with high-reasoning models for real-time diagnostic assistance. - -For Apple developers, we address the unique capabilities of the **Xcode & Apple Silicon** ecosystem, showing you how to leverage native intelligence and the high-speed memory of M-series chips for local inference. - -After setting up your primary environment, we reach the **Common Point**: the **Goose & Agentic Workflows** chapter. You will learn how to set up autonomous agents like **Goose** that can work across multiple files, run the compiler, and iterate on your engine's architecture using a local model powered by **Ollama**. - -Finally, we dive into the **Context Bridge**. You will learn how to use the **Model Context Protocol (MCP)** to connect your AI directly to the live Vulkan Specification. - -Next, we will begin with the **Standard Bearers** of C{pp} development. +After picking a primary IDE, every path converges on the **Goose & Agentic Workflows** chapter, which covers setting up an autonomous agent backed by a local model via **Ollama**. The last chapter in this section, **The Context Bridge**, covers using MCP to connect your AI directly to the Vulkan specification. xref:AI_Assisted_Vulkan/introduction.adoc[Previous: Introduction] | xref:AI_Assisted_Vulkan/02_environment_setup/02_jetbrains_clion_android_studio.adoc[Next: JetBrains CLion & Android Studio] diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/02_jetbrains_clion_android_studio.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/02_jetbrains_clion_android_studio.adoc index 2a5b4c4d..61ebcd69 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/02_jetbrains_clion_android_studio.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/02_jetbrains_clion_android_studio.adoc @@ -1,108 +1,81 @@ :pp: {plus}{plus} -= JetBrains CLion & Android Studio: The Standard Bearers += JetBrains CLion & Android Studio -== Introduction: The Power of the Indexer +== Introduction: the indexer's role -In the world of C{pp} and Android, JetBrains has long set a high standard for tooling. **CLion** and **Android Studio** are not just text editors; they are massive, sophisticated engines designed to understand the semantic structure of your code. For a Vulkan developer, this "understanding" is the most valuable asset you can provide to an AI. +**CLion** and **Android Studio** build a full semantic model of your code, not just a text index — a complete Abstract Syntax Tree covering every macro expansion and every header pulled in from the Vulkan SDK. For AI tooling, that's the most useful thing an IDE can provide, because it's the difference between an assistant that pattern-matches on text and one that can check its answer against your actual code. -To truly master **Collaborative Engineering**, we must transform these IDEs from simple tools into **AI-Enhanced Hubs** that leverage their deep indexing to reduce AI hallucinations. +== Why indexing matters here -== The Core Concept: Indexing as the Semantic Foundation +Most AI tools, left to their own devices, treat your code as text: they see `VkImageCreateInfo` and respond based on patterns learned during training, regardless of whether your engine's version of that struct matches what's in memory. If your engine wraps it differently, or you're on an older SDK, a generic assistant will often suggest members or flags that don't exist in your build. -Most AI tools treat your code as simple text strings. They see `VkImageCreateInfo` and recognize it based on patterns they learned during training. However, if your engine uses a custom wrapper or a specific version of the Vulkan headers, a generic AI will often suggest members or flags that don't exist in your environment. +Because CLion and Android Studio index the whole project, an assistant that queries that index can check where `VkImage` is actually defined, which extensions are enabled in your `CMakeLists.txt`, and whether you're using the `vk::raii` wrappers before it answers. That doesn't make it infallible, but it does mean a "refactor our image creation logic" request is checked against your code rather than guessed from a training-data average. -CLion and Android Studio solve this through their **Indexing Engine**. They build a complete **Abstract Syntax Tree (AST)** of your project, including every macro, every template expansion, and every included header from the Vulkan SDK. +== Setting up CLion -This **Syntactic Awareness** means that when you ask the AI to "Refactor our image creation logic," it doesn't just guess. It queries the IDE's index to find exactly where `VkImage` is defined, which extensions are active in your `CMakeLists.txt`, and whether you are using the `vk::raii` wrappers. By bridging the AI to the IDE's index, you move from "Probabilistic Guessing" to "Deterministic Truth." The AI becomes a **Precision Tool** that knows your codebase as well as the compiler does. +=== JetBrains AI Assistant + MCP -== Transforming CLion into an AI Hub +JetBrains AI Assistant supports the Model Context Protocol directly, which lets it call out to the Vulkan spec instead of relying only on what it saw during training. -A standard CLion install is a "Traditional Tool." To turn it into an AI hub, we focus on three primary integration points: native assistants, autonomous agents, and hybrid model strategies. +**Adding the Vulkan MCP server:** -=== Native Integration: JetBrains AI Assistant & MCP - -JetBrains has integrated the **Model Context Protocol (MCP)** directly into their AI Assistant. This allows the assistant to use external "tools" to verify facts, which is essential for Vulkan developers. - -**Adding the Vulkan MCP to AI Assistant:** - -To provide your Cloud-based assistant (like **Claude 4.6** or **GPT-5.3**) with the ground truth of the Vulkan specification, follow these steps. This assumes you have already built the `mcp-Vulkan` server as described in xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[the Environment Setup introduction]. +This assumes you've already built `mcp-Vulkan` as described in xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[the environment setup introduction]. 1. Open **Settings** (`Ctrl+Alt+S` or `Cmd+,`). 2. Navigate to **Tools > AI Assistant > MCP Servers**. -3. Click the **+** (Add) button and select **Local Stdio**. +3. Click **+** and select **Local Stdio**. 4. **Name:** `Vulkan-Spec` -5. **Command:** `node` (Ensure Node.js is in your PATH). -6. **Arguments:** Provide the absolute path to your built MCP server: `/path/to/mcp-Vulkan/vulkan/build/index.js`. -7. **Environment Variables:** (Optional) Add any keys if required by other servers. - -Once configured, the "Tools" icon in your AI chat window will show the active MCP. Now, when you ask a question about a specific structure like `VkVideoProfileInfoKHR`, the assistant can call the MCP to fetch the exact definition from the latest registry, even if it wasn't in its training data. - -**Real-World Scenario: The Spec Check** -Imagine you are implementing **Dynamic Rendering**. You aren't sure if `vkCmdBeginRendering` requires a specific extension enabled in your instance creation. Instead of leaving the IDE, you use the AI Assistant with the **Vulkan Spec MCP**. You ask: *"Does our current instance setup support dynamic rendering?"* The Assistant uses the MCP tool to query the live Vulkan registry and your project's `init.cpp`, providing a definitive "Yes" or "No" based on your actual code. - -=== The Autonomous Agent: Junie - -For those who want to move beyond simple chat, JetBrains has introduced **Junie**. Unlike a standard assistant that lives in a side panel and waits for your questions, Junie is a true **Autonomous Agent**. - -By default, Junie uses **Gemini 3.1 Flash** as its primary reasoning engine. This model is chosen for its speed and massive context window, which allows it to "read" your entire project and the Vulkan specification simultaneously. - -**Managing Your Models: When to Switch** - -While **Gemini 3.1 Flash** is an excellent default, Junie allows you to switch to even more powerful reasoning-heavy models when you encounter a particularly difficult problem. For example, if you are asking Junie to design a complex, multi-threaded frame graph, you might want to switch to **Gemini 3.1 Pro**, **Claude 4.6**, or **GPT-5.3** in the Junie settings. These models have deeper reasoning capabilities for "High-Level Design" tasks, though they come with a higher cost. - -You can select your preferred model via the dropdown menu in the Junie tool window or the `Settings > Tools > Junie` menu. This allows you to tailor the "Reasoning Cost" and "Intelligence Depth" to the task at hand. Many developers use Gemini 3.1 Flash for 90% of their coding—such as implementing shader interfaces or writing unit tests—and only "promote" the task to Claude 4.6 when the autonomous agent needs to solve a deep logic bug or design a new engine subsystem. - -In a Vulkan project, Junie's integration with the IDE's semantic model provides a significant advantage. Because it is built directly into the IDE, it has first-class access to the indexer, the debugger, and the build system. This enables an **Autonomous Workflow** where you can assign Junie a high-level task like: *"Implement a basic `Buffer` class that handles both staging and device-local memory using VMA (Vulkan Memory Allocator)."* Junie doesn't just give you a snippet; it creates the header and source files, adds them to your `CMakeLists.txt`, and even attempts to compile the result, fixing any syntax errors it encounters. +5. **Command:** `node` (make sure Node.js is on your PATH). +6. **Arguments:** the absolute path to your built server: `/path/to/mcp-Vulkan/vulkan/build/index.js`. +7. **Environment Variables:** optional, only needed for other servers. -This **"Context Bridge" Advantage** ensures Junie understands your project's architecture because it can "walk" the code. If you've already defined a `Device` wrapper, Junie will find it and use it, rather than trying to instantiate a raw `VkDevice`. +Once configured, the tools icon in the AI chat window shows the active MCP connection. Ask about a specific struct like `VkVideoProfileInfoKHR` and the assistant can pull the definition from the current registry rather than from stale training data. -=== The Bridge: Providing Context to External Agents +**Example:** say you're implementing dynamic rendering and aren't sure whether `vkCmdBeginRendering` needs a specific extension enabled at instance creation. With the Vulkan Spec MCP connected, you can ask the assistant directly instead of checking the spec yourself — it queries the registry and your project's `init.cpp` and gives you a yes/no answer grounded in both. -Sometimes, you need to use an external agent like **Goose** for a massive, multi-file refactor. To do this effectively, the agent needs to "see" your project through CLion's eyes. This is achieved via the **JetBrains IDE MCP Server**. +=== Junie -By installing the **JetBrains IDE MCP** plugin, you expose the IDE's internal API to external agents. When you tell an agent: *"Find every usage of `VkFramebuffer` and migrate it to the new `AttachmentBuilder` class,"* it doesn't just `grep` for text. It asks CLion's indexer for the actual symbol usages, ensuring it doesn't miss a single instance, even in complex template code. +**Junie** is JetBrains' autonomous agent — it can act on your project rather than just suggesting snippets in a side panel. -== Model Strategy: Seamlessly Blending Cloud and Local +By default, Junie uses **Gemini 3.1 Flash**, chosen for speed and a large context window that lets it read most of a project at once. -A key part of the **Collaborative Engineering** paradigm is the ability to choose the right model for the task. You don't always need a multi-billion parameter cloud model to write a simple `VkImageView` creation loop. +**Switching models for harder tasks:** Gemini 3.1 Flash is a reasonable default, but for something like designing a multi-threaded frame graph, switching to **Gemini 3.1 Pro**, **Claude 4.6**, or **GPT-5.3** in Junie's settings will generally get better results at higher cost. You can switch from the dropdown in the Junie tool window or under `Settings > Tools > Junie`. Many developers leave Flash as the default for routine work (shader interfaces, unit tests) and switch to a stronger model only when Junie needs to reason through a genuinely hard bug or subsystem design. -=== The "Hybrid Hub" Strategy +Because Junie runs inside the IDE, it has direct access to the indexer, debugger, and build system. You can hand it something like "implement a `Buffer` class handling both staging and device-local memory using VMA," and it will create the header/source files, wire them into `CMakeLists.txt`, and attempt a build, fixing syntax errors it introduces along the way. It's also more likely to reuse an existing `Device` wrapper than to instantiate a raw `VkDevice`, because it can see the wrapper in your code. -JetBrains has its own AI Assistant, which is natively capable of connecting to many different external AI tools. Additionally, many developers use the **Continue** plugin or external agents to create a truly flexible environment. This allows you to configure multiple "Backends" and switch between them based on the complexity of the task. +=== Bridging external agents through the JetBrains IDE MCP -For complex architectural tasks—like designing a new synchronization-aware render graph—you can use high-reasoning **Cloud Strategy** models like **Claude 4.6** or **GPT-5.3**. These are expensive but have a deep understanding of complex C{pp} abstractions. Conversely, for a **Local Strategy** such as implementing repetitive boilerplate, writing unit tests, or refactoring simple functions, you can switch to a local model like **Qwen 3-Coder (30B)** or **Llama 4** running via **Ollama**. This is free, zero-latency, and ensures your proprietary engine code never leaves your machine. +For a large, multi-file refactor, you may want an external agent like **Goose** to see the project the way CLion does. The **JetBrains IDE MCP** plugin exposes the IDE's internal API for this. A request like "find every usage of `VkFramebuffer` and migrate it to the new `AttachmentBuilder` class" then goes through the actual indexer rather than a text search, which matters for catching usages buried in template code. -Both strategies can natively be used within the JetBrains AI task window, allowing you to easily switch between them based on the complexity of the task. This hybrid approach maximizes efficiency and cost-effectiveness, ensuring that you can leverage the best of both worlds for your Vulkan development needs. +== Choosing between cloud and local models -=== Reaching the "Common Point" +Not every task needs a large cloud model — writing a `VkImageView` creation loop doesn't call for the same model as designing a synchronization-aware render graph. -To reach the **Common Point** required for the rest of this tutorial, you should have your IDE configured with at least two endpoints: a high-reasoning **Cloud Model** for architectural planning and complex debugging, and a fast, **Local Model** for implementation and cost-effective iteration. +JetBrains AI Assistant can connect to multiple backends, and many developers also use the **Continue** plugin to add others. A reasonable split: use a high-reasoning cloud model (**Claude 4.6**, **GPT-5.3**) for architectural work and complex C{pp} abstractions, and a local model (**Qwen 3-Coder 30B**, **Llama 4**) via **Ollama** for boilerplate, tests, and simple refactors. The local option is free, has no network latency, and keeps engine code on your machine — worth factoring in if that matters for your project. -Once these are linked to your IDE's indexer (via AI Assistant, Junie, or Continue), you have reached the "Base Camp" of AI-assisted development. To complete your environment, you must now proceed to the **Common Point**: xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to set up your autonomous agent and local inference engine. +Both can be reached from the same JetBrains AI task window, so switching between them is a matter of picking a different model, not a different tool. -== Android Studio: The Gemini Advantage +=== Before moving on -For mobile developers, **Android Studio** offers a unique advantage: **Gemini**. This model is specifically tuned for the Android NDK and the intricacies of mobile GPU architectures (like Qualcomm's Adreno or ARM's Mali). Additionally, as Android Studio is based upon JetBrains, Junie and most of the rest of the tools described for CLion also work natively in Android Studio. +The rest of this tutorial assumes at least two configured endpoints: a cloud model for architectural planning and complex debugging, and a local model for implementation. Once those are wired into your IDE (via AI Assistant, Junie, or Continue), continue to xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to set up an autonomous agent and local inference. -=== The "Context Sensitivity" Workflow +== Android Studio specifics -When working on a mobile Vulkan renderer, you often deal with Android-specific extensions like `VK_ANDROID_external_memory_android_hardware_buffer`. This is a notoriously challenging area of the API. +Android Studio adds **Gemini**, tuned for the Android NDK and mobile GPU architectures (Adreno, Mali). Since Android Studio is built on the same platform as CLion, Junie and most of the CLion-specific tooling above work the same way here. -In Android Studio, Gemini has deep access to your `AndroidManifest.xml` and your C{pp} source. When you ask: *"How do I import an `AHardwareBuffer` into our Vulkan renderer?"*, Gemini doesn't just give you a snippet. It checks if you have the correct permissions enabled in your manifest, verifies your NDK version, and provides a multi-file plan that includes both the Java/Kotlin side and the JNI/Vulkan side. +Android-specific extensions like `VK_ANDROID_external_memory_android_hardware_buffer` are a known rough edge in the API. Because Gemini in Android Studio has access to both your `AndroidManifest.xml` and your C{pp} source, a question like "how do I import an `AHardwareBuffer` into our Vulkan renderer?" gets checked against your actual manifest permissions and NDK version, with a plan that spans both the Java/Kotlin and JNI/Vulkan sides — rather than a generic snippet that may not match your target API level. -== Drawbacks and Incremental Wins +== Tradeoffs -=== The "Heavy Indexer" Tax -The depth of JetBrains' understanding comes at a cost of hardware resources. For massive engines, the initial indexing and the overhead of the AI Assistant can consume significant CPU and RAM. You might notice your laptop fans spinning up during a large refactor, but this "tax" is paid upfront. Once indexed, the AI's precision increases dramatically, saving you hours of debugging state mismatches later. +**Indexing cost.** JetBrains' depth of understanding isn't free — initial indexing and the AI Assistant's overhead can be noticeable on large engines, especially on machines without much RAM to spare. That cost is paid upfront; once indexed, lookups and refactors are faster and more accurate. -=== Incremental Win: Semantic Refactoring -The biggest win in this environment is the ability to perform **Semantic Refactoring**. If you need to change a core Vulkan abstraction (e.g., moving from a custom `Buffer` class to a more modern one), you can highlight the class and ask the AI to *"Perform a semantic migration."* Because it has the AST, it can safely rename, update signatures, and fix call sites across your entire project with a level of safety that is difficult to achieve with raw-text AI models. +**Where it actually helps: semantic refactoring.** If you're migrating a core abstraction (say, a custom `Buffer` class to a newer one), having the AST means the AI can rename, update signatures, and fix call sites project-wide more safely than a text-based tool would manage. This is the strongest argument for paying the indexing cost. -== Summary: Integrated AI Development +== Summary -Using CLion and Android Studio for AI-assisted Vulkan is about leveraging **Semantic Analysis**. By ensuring the AI "sees" what the compiler sees, you reduce the "Context Gap" and transform your IDE into a powerful, collaborative lab where the AI acts as a deeply informed extension of your own engineering expertise. +The main advantage of CLion and Android Studio for AI-assisted Vulkan work is that the AI can check its answers against what the compiler actually sees, rather than guessing from patterns. That narrows the context gap described in the previous chapter, at the cost of indexing overhead you'll want to budget for on large projects. -Next, we will jump to the **goose native agent** experience. You may want to look at the **Windows** specific xref:AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc[Visual Studio] instructions to get an idea of other environments. +Next: the xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose native agent] chapter. If you're on Windows, you may also want to check the xref:AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc[Visual Studio] chapter for comparison. xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Previous: Environment Introduction] | xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Next: Goose Native Agent] diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc index 7e5ca755..b0733fb5 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/03_visual_studio.adoc @@ -2,24 +2,23 @@ = Visual Studio: Windows-Native Development -== Introduction: The Diagnostic Hub +== Introduction -For many graphics engineers, Visual Studio (MSVS) is a central tool in their workflow. Its tight integration with the Microsoft C{pp} compiler (MSVC) and its advanced debugger have made it a common choice for Windows-based Vulkan development. +For many graphics engineers on Windows, Visual Studio (MSVS) is the default. Its integration with MSVC and its debugger have made it the common choice for Windows-based Vulkan development for a long time. -However, a "Traditional" Visual Studio setup can feel like a heavyweight environment that is slow to adapt to the rapidly evolving AI landscape. To truly master **Collaborative Engineering**, we must transform Visual Studio from a passive editor into an **Active Diagnostic Hub**. +A stock Visual Studio setup, though, doesn't do anything with AI on its own. What's worth building is a link between the debugger's live state and a chat-based assistant, so you're not manually copying values out of the Locals window every time you ask a question. -== The Core Concept: The "Debug-to-Chat" Pipeline +== Debug-to-chat -The most unique advantage of Visual Studio is its **Deep Debugging Context**. While other editors treat a crash as a simple line of text in a console, Visual Studio "owns" the entire execution state. In an AI-enhanced workflow, we connect this state directly to a high-reasoning model. +Visual Studio's advantage over most editors is that it owns the entire execution state during a debug session, not just a line of text in a console. That state can be handed to an assistant directly instead of transcribed by hand. -For instance, imagine you hit a `VUID-VkBufferMemoryBarrier-buffer-01191` error due to an invalid offset. In a normal workflow, you'd pause, look at your `VkBufferMemoryBarrier` struct in the "Locals" window, and manually check the offset against your buffer's size. With the **Copilot Chat** integration, you don't just "read" the error. You highlight it and ask: *"Why is this offset invalid for our current `stagingBuffer`?"* The AI doesn't just guess; it "sees" the runtime values of your `stagingBuffer.size` and your `barrier.offset` directly from the MSVC debugger's state. It then identifies that your alignment logic is using a 16-byte constant instead of querying `minStorageBufferOffsetAlignment` from your physical device. +For example: you hit a `VUID-VkBufferMemoryBarrier-buffer-01191` error from an invalid offset. Normally you'd pause, check the `VkBufferMemoryBarrier` struct in the Locals window, and compare the offset against your buffer's size by hand. With **Copilot Chat**, you can instead highlight the error and ask why the offset is invalid for the current `stagingBuffer` — Copilot can read `stagingBuffer.size` and `barrier.offset` directly from the debugger's state, which is what lets it catch something like a hardcoded 16-byte alignment constant instead of a query against `minStorageBufferOffsetAlignment`. -== Tutorial: Setting up the Windows Diagnostic Hub +== Setting up the workflow -To transform Visual Studio from a passive editor into an active diagnostic tool, follow these steps to link your debugger and project context to your AI assistant. +=== Step 1: solution-aware planning with Copilot -=== Step 1: Solution-Aware Planning (Copilot) -Ensure **GitHub Copilot** is installed and that it has completed its initial index of your `.sln`. Then, use the `@workspace` command for architectural tasks. +Make sure **GitHub Copilot** is installed and has indexed your `.sln`. Use `@workspace` for tasks that span the whole project. [source,text] ---- @@ -28,8 +27,9 @@ It must follow the naming conventions in 'DescriptorManager.cpp' and implement the 'VK_EXT_descriptor_buffer' extension. ---- -=== Step 2: Debug-to-Chat (The Live Fix) -When your application crashes with a Vulkan error or a Validation VUID, keep the debugger active and use the **Copilot Chat** window. +=== Step 2: debug-to-chat for live errors + +When your application crashes with a Vulkan error or validation VUID, keep the debugger active and use **Copilot Chat**. [source,text] ---- @@ -39,8 +39,9 @@ for our 'stagingBuffer'? Check the 'Locals' window for the current offset and buffer size. ---- -=== Step 3: Local Private Specialist (Continue) -For privacy-sensitive rendering logic, configure the **Continue** extension to use a local **Qwen 3-Coder** model and the **Vulkan MCP**. This assumes you have already built the `mcp-Vulkan` server as described in xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[the Environment Setup introduction]. +=== Step 3: local model for sensitive code + +For rendering logic you'd rather not send to a cloud provider, configure the **Continue** extension with a local **Qwen 3-Coder** model and the Vulkan MCP server. This assumes you've already built `mcp-Vulkan` as described in xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[the environment setup introduction]. [source,json] ---- @@ -55,24 +56,20 @@ For privacy-sensitive rendering logic, configure the **Continue** extension to u } ---- -== Drawbacks and Incremental Wins - -=== The "Heavy IDE" Performance Tax -Visual Studio is a resource-intensive application. Adding multiple AI extensions and indexing processes can lead to noticeable UI lag, especially on systems with less than 32GB of RAM. You might see the "Indexing Workspace" icon spinning for several minutes after opening a large solution. To mitigate this, consider disabling legacy features like **IntelliCode** to free up resources for more modern, high-reasoning tools like Copilot and local SLMs. +== Tradeoffs -=== Incremental Win: Boilerplate as Intent -The biggest "Quality of Life" improvement is treating boilerplate as **Intent**. Instead of typing out a 20-line `VkGraphicsPipelineCreateInfo` structure, you write a single comment: `// Create a graphics pipeline state for a wireframe overlay with no culling.` The AI, seeing your existing pipeline creation functions, fills in the structure with the correct defaults for your engine (e.g., using your specific `VkRenderPass` handles). This turns "Typing" into "Reviewing," which is the core of the Collaborative Engineering shift. +**Resource cost.** Visual Studio is already heavy; adding AI extensions and their indexing on top can produce noticeable UI lag, particularly with less than 32GB of RAM. If you see the indexing icon spin for minutes after opening a large solution, consider disabling older features like IntelliCode to free up headroom for Copilot and local models. -=== Reaching the "Common Point" +**Where it helps: boilerplate as a description.** Instead of typing out a 20-line `VkGraphicsPipelineCreateInfo`, you can write a comment describing the pipeline you want — `// wireframe overlay pipeline, no culling` — and let the assistant fill in the structure based on your existing pipeline code, then check its work. That review step matters; treat the output as a draft, not a finished answer. -To reach the **Common Point** required for the rest of this tutorial, your Visual Studio environment should be configured with a **Hybrid Model Strategy**. This includes a solution-aware model like **GitHub Copilot** for high-level architectural tasks and cross-file planning within your `.sln`, as well as a local specialist via the **Continue** extension for cost-effective iteration and privacy-sensitive code generation. +=== Before moving on -Once these are linked to your IDE (via Copilot and Continue), you have reached the "Base Camp" of AI-assisted development. To complete your environment, you must now proceed to xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to set up your autonomous agent and local inference engine. +The rest of this tutorial assumes your Visual Studio setup has both a solution-aware model (Copilot, for architectural work and cross-file planning) and a local model via Continue (for cost-free iteration on sensitive code). Once both are wired in, continue to xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to set up an autonomous agent and local inference. -== Summary: The Windows Professional +== Summary -An AI-enhanced Visual Studio environment is more than just a code generator; it is a **Diagnostic Hub**. By bridging the IDE's comprehensive debugging tools with both cloud-based reasoning models and local specialized models, you create a workflow where the AI helps you not just write Vulkan code, but understand the deep "why" behind every bitmask and memory barrier. +An AI-connected Visual Studio setup is mostly about wiring the debugger's state into chat, so questions about a crash or a VUID come with the actual runtime values attached instead of a description you typed from memory. -Next, we will jump to the **goose native agent** experience. You may want to look at the **Native-First Hybrid** strategy for xref:AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc[Xcode & Apple Silicon] to get an idea of other environments. +Next: the xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose native agent] chapter. macOS users may also want to compare against the xref:AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc[Xcode & Apple Silicon] chapter. xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Previous: Environment Introduction] | xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Next: Goose Native Agent] diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc index 9af18037..a795349d 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/04_xcode_apple.adoc @@ -1,37 +1,30 @@ :pp: {plus}{plus} -= Xcode & Apple Silicon: Native Intelligence & Agentic Extensions += Xcode & Apple Silicon -== Introduction: The Translation Specialist +== Introduction -Developing for Vulkan on Apple platforms is a unique journey. Because Apple does not natively support the API, we rely on **MoltenVK** or **KosmicKrisp** to translate our calls into Metal on the fly. +Vulkan on Apple platforms always goes through a translation layer, since Apple doesn't support Vulkan natively. **MoltenVK** and **KosmicKrisp** are the two options, and they trade off differently. **KosmicKrisp** is a Mesa-based implementation built specifically for Apple Silicon, using Metal 4 to support modern Vulkan (up to 1.4); it currently targets macOS 26+ and is expanding to iOS (A14+). **MoltenVK** is the older, more broadly compatible bridge, supporting everything from Intel Macs to the newest iPhones and iPads, and is generally the safer choice for production apps that need wide device coverage. -In this "Collaborative Engineering" environment, your AI assistant needs to be more than just a coder; it must be a specialized model configuration that understands the "translation tax" and the architectural nuances of Apple Silicon. This includes the critical distinction between our two primary bridges. **KosmicKrisp** is a high-performance, Mesa-based Vulkan implementation designed specifically for Apple Silicon that leverages Metal 4 to provide modern Vulkan (up to 1.4) support. While initially focused on macOS (requiring version 26+), it is rapidly expanding to support iOS (A14+). On the other hand, **MoltenVK** is the industry-standard, universal bridge that supports all Apple devices—from the oldest Intel-based Macs to the latest iPhones and iPads—making it the most stable option for production apps that need broad compatibility. +An AI assistant is useful here mainly for managing the differences between the two and helping with the migration work when you need to switch. -Your AI-assisted setup should be configured to help you choose and migrate between these two depending on your performance targets and device compatibility requirements. +== Where AI actually helps: migration and packaging -== The Core Concept: The "Migration Specialist" AI +The tedious part of Apple Vulkan development is usually the conditional compilation and separate build targets needed to support both bridges, not the graphics logic itself. This is a reasonable place to delegate: state the constraint (e.g. "iOS target needs MoltenVK for compatibility, but keep an experimental KosmicKrisp path for A14+") and let the assistant draft the build configuration and the surface-creation code (`vkCreateMacOSSurfaceMVK` vs `vkCreateIOSSurfaceMVK` or their KosmicKrisp equivalents), then review it — this is exactly the kind of repetitive, well-specified restructuring that's easy to check and easy to get wrong by hand. -The greatest advantage of an AI-enhanced Apple workflow is reducing the "nitty-gritty" of navigating what is supported and when. Instead of manually managing conditional compilation flags and separate build targets, you leverage your AI for packaging and migration tasks. +== Native and agentic tooling in Xcode 26 -Imagine you have a high-performance desktop engine using **KosmicKrisp** to leverage Vulkan 1.4 features on macOS, but now you need to target older iPhones that require **MoltenVK**, or you want to test the emerging iOS support for KosmicKrisp. By providing your current setup to your assistant, you can state your intent: *"We are adding an iOS target. Draft the build configuration to use MoltenVK for universal compatibility, but keep an experimental path for KosmicKrisp on A14+ devices. Refactor our surface creation to handle both `vkCreateMacOSSurfaceMVK` and `vkCreateIOSSurfaceMVK` (or the KosmicKrisp equivalents) within our cross-platform view wrapper."* The AI then handles the tedious reorganization of the project structure and the subtle differences in the Metal-layer setup, allowing you to focus on the high-level graphics architecture. +Xcode 26 includes **Predictive Code Completion** and **Swift Assist**, on-device models aimed at Swift and Apple-specific code. These are useful for the Swift/SwiftUI side of a Vulkan project — for example, refactoring compute shader data structures that use `VK_EXT_scalar_block_layout` for better cache behavior on an M4 Max. -== The AI Ecosystem: Native AI Integration & Agentic Extensions +For larger, multi-file work — cross-platform refactors, compile-fix loops — an external agent like **Goose** is still worth using alongside Xcode's built-in tools rather than instead of them. Xcode handles the inner loop (Swift completions, its Instruments profiler); the external agent handles the outer loop. -With the release of **Xcode 26**, the Apple ecosystem has moved from a "Sidecar" necessity to a "Native-First" AI environment. Apple's integration of **Predictive Code Completion** and **Swift Assist** provides high-reasoning, on-device models that understand the unique constraints of the Apple Silicon architecture. +== Setting up the workflow -This native advantage is clear when implementing complex compute shaders that use `VK_EXT_scalar_block_layout`; Xcode 26's native AI can refactor your data structures for optimal caching on an M4 Max chip, leveraging its deep understanding of the Metal-layer translation. While Xcode provides integrated "Inner Loop" assistance, we still leverage standalone agents like **Goose** for the "Outer Loop." This creates a "Native-and-Agentic" setup: you use Xcode's built-in models for real-time coding and its integrated **Instruments** profiler, while calling upon an external agent for complex, multi-file refactors or autonomous "Compile-Fix" cycles that require specific models like **Qwen 3-Coder**. - -== Tutorial: Setting up the Xcode Hybrid Lab - -To build a modern graphics workflow on Apple Silicon, you must leverage the unique **Unified Memory Architecture (UMA)** and the new native AI capabilities in Xcode 26. Follow these steps to set up your hybrid lab. - -=== Step 1: Bridging Swift and C{pp} (Native-First) -Use Xcode 26's **Swift Assist** to generate the platform-specific boilerplate. +=== Step 1: Swift/C{pp} bridging with Swift Assist [source,swift] ---- -// Example SwiftUI wrapper generated by native AI +// Example SwiftUI wrapper generated with Swift Assist import SwiftUI import MetalKit @@ -47,12 +40,11 @@ struct MetalView: UIViewRepresentable { } ---- -=== Step 2: Agentic Cross-Platform Refactoring (Goose) -While the native assistant handles the "Inner Loop" of Swift code, use an autonomous agent like **Goose** for the "Outer Loop" tasks, such as migrating your engine between MoltenVK and KosmicKrisp. +=== Step 2: cross-platform refactoring with Goose [source,text] ---- -# Example command for a 'Migration Specialist' agent +# Example agent instruction for a MoltenVK/KosmicKrisp migration Goose, refactor our 'surface_creation.cpp' to support both MoltenVK and KosmicKrisp. Ensure we use the 'vkCreateIOSSurfaceMVK' call for universal iOS support, @@ -60,8 +52,9 @@ but enable a high-performance path for macOS 26+ using the new KosmicKrisp Metal 4 features. ---- -=== Step 3: Local Inference via MLX -Leverage your Mac's Unified Memory by running a high-reasoning **Qwen 3-Coder (30B)** model locally via Ollama. This allows the AI to access up to 48GB+ of VRAM on an M-series Max/Ultra chip. +=== Step 3: local inference via Ollama + +Apple Silicon's unified memory means a local model can use significantly more memory than it could on a typical discrete GPU. A 30B model like **Qwen 3-Coder** can run locally on an M-series Max/Ultra chip with 48GB+ of available memory. [source,bash] ---- @@ -69,35 +62,26 @@ Leverage your Mac's Unified Memory by running a high-reasoning **Qwen 3-Coder (3 ollama run qwen3-coder:30b ---- -By using this **Native-Agent Synergy**, you ensure that the integrated AI handles the platform-specific "Glue Code" (Swift/C{pp}) while the autonomous agents handle the heavy Vulkan lift. - -=== Step 4: Efficiency on Base Models (Mistral-Nemo) -If you are working on a base M3 or M4 with 16GB or 24GB of RAM, you might find the 30B models a bit heavy. In these cases, use **Mistral-Nemo (12B)** or **Llama 4 (17B)**. These models are highly efficient on Apple's Neural Engine and leave more Unified Memory available for your Vulkan framebuffers and texture streaming. - -== The Local Inference Advantage: UMA Power - -Apple Silicon is uniquely suited for local AI because of its **Unified Memory**. In a traditional PC, your AI model is restricted to the VRAM of your GPU. On a Mac with 64GB of RAM, your AI can access almost the entire pool of memory. +=== Step 4: smaller models for base hardware -**Technical Deep Dive:** You can run a high-reasoning **Qwen 3-Coder (30B)** model locally via Ollama while still leaving enough headroom for your Vulkan application's massive texture arrays. This is difficult on a desktop PC with a 12GB or even a 16GB GPU. By leveraging tools like **MLX** (Apple's native machine learning framework), your AI assistant can run efficiently directly on the Neural Engine, providing quick responses to your graphics queries. +On a base M3 or M4 with 16GB or 24GB of RAM, a 30B model is likely too much. **Mistral-Nemo (12B)** or **Llama 4 (17B)** run efficiently on Apple's Neural Engine and leave more memory for your Vulkan framebuffers and texture streaming. -== Drawbacks and Incremental Wins +== Why unified memory matters for local inference -=== The "Integrated vs. Agentic" Tax -The primary challenge in the modern Apple setup is managing the handoff between Xcode's native AI and your external agents. +On a typical PC, a local model is limited to whatever VRAM your GPU has. On a Mac, the model and the OS share the same memory pool, so a machine with 64GB of RAM can give a local model access to most of it — you can run a 30B model locally while still leaving headroom for large texture arrays, something that's much harder on a 12–16GB discrete GPU. Apple's **MLX** framework lets models run efficiently on the Neural Engine specifically, which helps here too. -* **The Mitigation:** Use **Terminal-based agents (Aider)** or integrated **Goose** triggers that can run alongside your Xcode window, ensuring that the integrated AI handles the Swift/C{pp} bridge while the autonomous agents handle the heavy Vulkan lifting. +== Tradeoffs -=== Incremental Win: Metal-to-SPIRV Diagnosis -The biggest "Quality of Life" win is diagnosing **Shader Translation Errors**. When MoltenVK or KosmicKrisp fails to compile your SPIR-V into Metal (MSL), the error log can be cryptic. By feeding these logs to an AI that has context on both the Vulkan spec and Metal's shading language, you can efficiently identify which specific SPIR-V instruction is causing the failure and how to refactor your GLSL/Slang code to be "Apple-friendly." KosmicKrisp users can even use the `MESA_KK_DEBUG=msl` environment variable to inspect the generated MSL directly with the AI for deep performance tuning. +**Managing the handoff between Xcode's native AI and external agents** is the main friction point — there's no single tool that covers both the Swift side and the heavy Vulkan refactoring. Terminal-based agents (Aider) or Goose triggers that run alongside Xcode are the usual workaround. -=== Reaching the "Common Point" +**Where it helps: shader translation errors.** When MoltenVK or KosmicKrisp fails to translate SPIR-V into Metal, the resulting error can be hard to parse on its own. Feeding the log to a model with context on both the Vulkan spec and MSL is a reasonable way to narrow down which SPIR-V instruction is at fault and how to adjust your GLSL/Slang source. KosmicKrisp users can also set `MESA_KK_DEBUG=msl` to inspect the generated MSL directly alongside the AI's explanation. -To reach the **Common Point** required for the rest of this tutorial, your Apple environment should be configured with a **Native-Agent Hybrid Strategy**. This setup uses Xcode 26's native AI for real-time Predictive Code Completion and Swift-to-C{pp} bridging, an autonomous agent like **Goose** (powered by Claude 4.6) for high-level reasoning and complex cross-file refactoring of translation layers, and an MLX-optimized local model like **Qwen 3-Coder** via Ollama to leverage your Mac's Unified Memory for free, high-speed iteration. +=== Before moving on -Once these are active, you have reached the "Base Camp" of AI-assisted development. To complete your environment, you must now proceed to the **Common Point**: xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to set up your autonomous agent and local inference engine. +The rest of this tutorial assumes an Apple setup with Xcode 26's native completion for Swift/C{pp} bridging, an agent like Goose (backed by a cloud model such as Claude 4.6) for cross-file refactoring, and a local model via Ollama/MLX for fast iteration. Once these are in place, continue to xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Goose & Local Intelligence] to finish setting up the agent and local inference. -== Summary: The Apple Specialist +== Summary -An AI-enhanced Apple developer lab is more than just a code generator; it is a **Native-to-Spec Bridge**. By leveraging Xcode 26's integrated intelligence alongside autonomous agents and the unique memory architecture of Apple Silicon, you transform your workflow into a cross-platform workflow that is as deeply aware of Metal's nuances as it is of the Vulkan specification. Whether you are optimizing for the broad reach of MoltenVK or the cutting-edge performance of KosmicKrisp, your AI assistant ensures you stay focused on the graphics, not the glue. +Apple Vulkan development means working across two layers — Swift/Metal and Vulkan/C{pp} — and an AI assistant is most useful for keeping the boilerplate and translation-layer differences between MoltenVK and KosmicKrisp from eating your time, whether you're targeting broad compatibility or KosmicKrisp's newer feature set. xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Previous: Environment Introduction] | xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Next: Goose Native Agent] diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc index 2eaf541b..2afb18cb 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc @@ -1,46 +1,40 @@ :pp: {plus}{plus} -= Goose: The Autonomous Agent & Local Intelligence += Goose: Autonomous Agent & Local Intelligence -== Introduction: Autonomous Agents +== Introduction -While IDE-integrated assistants like JetBrains AI and Visual Studio Copilot focus on deep, real-time code suggestions, **Goose** represents a fundamental shift in the development lifecycle: the **Autonomous AI Agent**. +IDE-integrated assistants (JetBrains AI, Copilot) focus on real-time suggestions inside your editor. **Goose** is a different category of tool: an agent that can act on your project directly — running commands, editing multiple files, checking the build — rather than just producing snippets for you to paste in. -In this chapter, we establish the **Common Point** for all platforms. Regardless of whether you are a CLion user on Linux, a Visual Studio developer on Windows, or an Xcode expert on macOS, the agentic workflow is the universal bridge that transforms you from a "Code Writer" into an "Engineering Director." +This chapter is the common point for all three IDE paths (CLion, Visual Studio, Xcode) — the agent setup here is the same regardless of which one you use. -== The Core Concept: The Task-Oriented Agent +== What makes it an agent rather than an assistant -The primary difference between a "Traditional Assistant" and an "AI Agent" is **Agency**. A passive assistant might give you a code snippet for creating a `VkDescriptorPool` that you must then copy, paste, and manually fix. In contrast, an active agent can be tasked with creating that same descriptor pool in `VulkanContext.cpp` and running the build to ensure it works. +The practical difference is whether the tool just proposes code or actually does something with it. A chat-based assistant gives you a snippet for a new `VkDescriptorPool` that you copy in and fix up yourself. An agent can be told to add that pool to `VulkanContext.cpp` and then actually run the build to check it compiles — it reads your headers, writes the code, opens a terminal, runs `cmake`, and iterates on any errors it introduced. -The Agent doesn't just provide code; it **acts**. It reads your headers, writes the C{pp}, opens a terminal, runs `cmake`, analyzes any errors, and iterates until the build passes. +This is a genuinely useful capability, but it's also one where you want to stay in the loop on what it changed, especially for anything touching synchronization or memory lifetime — the earlier chapters covered IDE-integrated agents that stay closer to your review process; a system-level agent like Goose has broader reach (it can talk to an external RenderDoc instance, tweak system configuration) and correspondingly less built-in oversight. -While in the preceeding chapters, we identified howto use the integrated AI agents in the workflow, this covers the inner workflow that typically excludes other tools or system integration steps. Having a system wide agent provides deep integration with things like an external RenderDoc instance or help with tunning system configurations. - -== Setting Up the Native Agent: Goose - -Goose is an open-source autonomous agent designed to work directly with your local tools. Setting it up "natively" means giving it the permissions to act as a real developer on your machine. +== Installing Goose === 1. Installation -Goose can be installed as a standalone CLI tool or integrated into your workspace. - -**For macOS and Linux:** +**macOS and Linux:** [source,bash] ---- curl -fsSL https://block.github.io/goose/install.sh | bash ---- -**For Windows:** -While there is a PowerShell installer, for the most control on Windows, we recommend the manual installation: +**Windows:** +A PowerShell installer exists, but for more control, install manually: 1. Go to link:https://goose-docs.ai/[goose-docs.ai] and download the Goose executable. -2. Unzip the folder and place it in a permanent location (e.g., `C:\Programs\Goose`). -3. Pin the `goose.exe` to your Start menu for easy access. -4. **Note:** You may need to locate your `config.yaml` file for advanced manual configuration. On Windows, this is typically found at: `C:\Users\[YourUserName]\AppData\Roaming\Block\goose\config\config.yaml`. +2. Unzip it into a permanent location (e.g., `C:\Programs\Goose`). +3. Pin `goose.exe` to your Start menu. +4. If you need to edit configuration manually, `config.yaml` is typically at `C:\Users\[YourUserName]\AppData\Roaming\Block\goose\config\config.yaml`. -=== 2. Giving Goose "Hands" +=== 2. Starting a session -Once installed, Goose needs access to your environment. You initiate a session within your project's root directory: +Start a session from your project root: [source,bash] ---- @@ -48,42 +42,43 @@ cd /path/to/your/vulkan_project goose session ---- -From here, you can provide Goose with **Tools** (via the Model Context Protocol) that allow it to read your filesystem, run shell commands, and even search the Vulkan specification. +From here you can add **tools** (via MCP) that let Goose read your filesystem, run shell commands, and search the Vulkan specification. -==== Adding Filesystem Access (MCP) -To allow Goose to interact with your local files reliably, you should add a custom extension: +==== Filesystem access (MCP) -1. **Install Node.js:** Download the LTS version from link:https://nodejs.org/[nodejs.org] using the Windows Installer (.msi). Verify with `node --version` and `npm --version`. -2. **Add Extension in Goose:** Go to **Extensions -> Add Custom Extension** and fill out the fields: +To give Goose reliable access to your local files: + +1. **Install Node.js:** the LTS installer from link:https://nodejs.org/[nodejs.org]. Verify with `node --version` and `npm --version`. +2. **Add the extension in Goose:** go to **Extensions -> Add Custom Extension**: * **Extension Name:** `Filesystem Access` * **Type:** `STDIO` * **Description:** `Allows goose to access project files.` - * **Command:** `npx -y @modelcontextprotocol/Filesystem_Access [filepath]` (Replace `[filepath]` with your project path using forward slashes, e.g., `C:/Users/Dev/VulkanProject`). + * **Command:** `npx -y @modelcontextprotocol/Filesystem_Access [filepath]` (replace `[filepath]` with your project path, forward slashes, e.g. `C:/Users/Dev/VulkanProject`). * **Timeout:** `300` -3. **Restart Goose** to enable the extension. +3. **Restart Goose** to load the extension. -== Model Strategy: Hybrid Cloud and Local Roles +== Splitting work between cloud and local models -To power your agent effectively, we recommend a **Hybrid Model Strategy**. This approach balances high-level architectural "Reasoning" with low-level, zero-latency "Implementation." +A hybrid setup — cloud model for planning, local model for implementation — tends to work better than picking one for everything, since the two categories are good at different things and cost very different amounts. -=== 1. The Cloud Planner (High-Reasoning) +=== 1. Planning with a cloud model -For the initial session setup and complex planning, you should use a high-reasoning cloud model like **Claude 4.6** or **GPT-5.3**. These models excel at understanding the abstract requirements of a graphics pipeline and creating a "Contract" for the work to be done. +For initial session setup and harder architectural planning, a cloud model (**Claude 4.6**, **GPT-5.3**) is generally worth the cost — this is where the model needs to actually reason about the design, not just produce plausible-looking code. To configure a cloud provider in Goose: 1. Open Goose Settings -> Models -> Configure Providers. -2. **Cloud:** Select **Anthropic**. You will need an API key from the link:https://platform.claude.com/[Claude Console]. Paste your key and submit. +2. Select **Anthropic**. You'll need an API key from the link:https://platform.claude.com/[Claude Console] — paste it and submit. -=== 2. The Local Runner (Implementation Specialist) +=== 2. Implementation with a local model -Once the high-level plan is established, you can switch the "active hands" to a local model running via **Ollama**. This is your "Implementation" specialist—fast, free, and secure. +Once you have a plan, switching to a local model via **Ollama** for the actual implementation work is free and keeps the code on your machine. **Setting up Ollama:** -1. Download the native installer from link:https://ollama.com[ollama.com]. -2. **Configuration:** In Ollama settings, choose a "Model location" on a drive with sufficient space. -3. **Download Models:** Open PowerShell and run the following commands to pull the latest specialists: +1. Download the installer from link:https://ollama.com[ollama.com]. +2. Pick a model storage location with enough free space. +3. Pull the models you'll want: + [source,bash] ---- @@ -100,48 +95,45 @@ ollama run qwen3-coder:30b # Download the 12B Mistral-Nemo model for balanced reasoning and context ollama run mistral-nemo ---- -4. **Connect to Goose:** In Goose Settings -> Models -> Configure Providers, find **Ollama** and click configure. Select your desired model (e.g., `qwen3-coder:14b`) from the dropdown. +4. **Connect to Goose:** in Goose Settings -> Models -> Configure Providers, find **Ollama**, and pick your model (e.g. `qwen3-coder:14b`) from the dropdown. -=== 3. Implementing the Hybrid Workflow +=== 3. Putting it together -In a real-world session, you leverage these models by switching profiles or starting the session with a specific target. For the **Planning Phase**, start a session with your Cloud model and ask it to analyze your current `VulkanContext` to draft a multi-step plan for a feature like shadow mapping. Once the plan is ready, you move to the **Implementation Phase** by switching to your local Ollama profile and tasking Goose with following that plan to implement the necessary boilerplate code. +In practice, this means starting a session with the cloud model to draft a plan — for example, analyzing your current `VulkanContext` for a shadow-mapping feature — and then switching to the local Ollama profile to have Goose implement that plan. This keeps VRAM free for the cloud-planning phase and reserves the (free, but slower to iterate on correctness) local model for implementation, where mistakes are cheaper to catch. -This workflow ensures your VRAM is preserved for the Vulkan application during the bulk implementation phase, while still using the high-reasoning capabilities of cloud models for the critical design phase. +==== Permissions in VS Code -==== Security & Permissions (VS Code) -If you are using Goose or other AI agents (like Claude Code) within VS Code, you may be prompted for permission for every command. To enable a more autonomous (but higher risk) workflow: +If you're using Goose (or another agent, like Claude Code) inside VS Code, you'll normally be prompted for permission on every command. To skip that: 1. Press `Ctrl + ,` to open Settings. 2. Search for `permissions`. -3. For your AI extension, set **Initial Permission Mode** to `bypassPermissions`. +3. Set **Initial Permission Mode** to `bypassPermissions` for your AI extension. 4. Enable **Allow Dangerously Skip Permissions**. -*CAUTION:* Only do this in a controlled environment (like a VM or container) as it allows the AI to execute scripts and modify files without confirmation. - -== Hardware Requirements: The VRAM Budget +*Caution:* only do this in an environment you can afford to have modified without confirmation — a VM or container, not your main machine — since it lets the agent run scripts and edit files unattended. -The biggest challenge in AI-assisted Vulkan development is **VRAM Contention**. Both your Vulkan application and your local AI model require GPU memory. +== VRAM budgeting -=== The VRAM Budgeting Formula +Running a local model alongside a Vulkan application means both are competing for the same GPU memory, so it's worth doing the arithmetic before you start. -Before starting a local agent, you must perform a "VRAM Audit." For a typical developer machine with 16GB of VRAM, you might find that your Vulkan application uses ~4.5GB (for a modern deferred renderer at 1440p) and your OS and IDE overhead consumes ~1.5GB. This leaves you with approximately 10GB available for your AI models (`16 - (4.5 + 1.5) = 10GB`). +=== A rough budget -=== Quantization Strategies +For a machine with 16GB of VRAM: a modern deferred renderer at 1440p might use ~4.5GB, and OS/IDE overhead another ~1.5GB, leaving roughly 10GB for a local model (`16 - (4.5 + 1.5) = 10GB`). -We use **Quantization** (compression) to reduce model memory footprint. Ollama handles this automatically, but you should be aware of the trade-offs. Using **4-bit (Q4_K_M)** quantization is often the "sweet spot," as it is 75% smaller with only about a 5% loss in reasoning. Stepping up to **8-bit (Q8_0)** provides higher precision reasoning but requires significantly more VRAM. +=== Quantization -NOTE: Even at 4-bit quantization, `qwen3-coder:30b` requires approximately 20 GB of VRAM — more than most consumer graphics cards provide. On a 16 GB GPU running a Vulkan application simultaneously, this will exceed your budget. Use `qwen3-coder:14b` (~9 GB at Q4_K_M) as a practical alternative that fits comfortably within a 16 GB VRAM budget. +Quantization reduces a model's memory footprint, and Ollama handles it automatically, but the tradeoff is worth knowing. **4-bit (Q4_K_M)** is usually the reasonable default — about 75% smaller than full precision, with a modest (roughly 5%) drop in output quality. **8-bit (Q8_0)** is more accurate but needs considerably more VRAM. -If your GPU runs out of VRAM, your system may trigger a **TDR (Timeout Detection and Recovery)**, crashing both your AI and your Vulkan renderer. Always leave a 1–2GB "safety margin." +NOTE: Even at 4-bit quantization, `qwen3-coder:30b` needs roughly 20GB of VRAM — more than most consumer GPUs have. On a 16GB GPU running a Vulkan application at the same time, this won't fit. Use `qwen3-coder:14b` (~9GB at Q4_K_M) instead if you're on a 16GB budget. -== Decision Framework: IDE-AI vs. Goose +If you run out of VRAM, you risk a **TDR (Timeout Detection and Recovery)** event, which can crash both the model and your Vulkan renderer. Leave a 1–2GB safety margin. -When deciding between your tools, use your **IDE Assistant** (like JetBrains AI or Copilot) for "Inner Loop" tasks such as autocomplete, explaining code, or fixing syntax errors in your current file. Switch to **Goose** for "Outer Loop" tasks that involve broader changes, like adding a new `PostProcess` class or performing engine-wide refactors from raw pointers to smart pointers. +== When to use which tool -== Summary: Reaching the Common Point +Use your **IDE assistant** (JetBrains AI, Copilot) for inner-loop work — autocomplete, explaining code, fixing syntax errors in the current file. Switch to **Goose** for outer-loop work that spans files — adding a new `PostProcess` class, or an engine-wide migration from raw pointers to smart pointers. -You have now reached the **Common Point**. Whether you arrived here from CLion, Visual Studio, or Xcode, you now have a native agent (Goose) capable of acting on your codebase, a local model (Ollama) running a specialized coding assistant, and a VRAM budget that respects your Vulkan application's needs. +== Summary -This unified environment is where the real work of **Collaborative Engineering** begins. In the following chapters, we will treat this setup as the baseline for all our Vulkan development exercises. +At this point, regardless of which IDE you started from, you should have Goose set up as a system-level agent, a local model running through Ollama, and a VRAM budget that leaves room for your Vulkan application. The remaining chapters treat this setup as the baseline. -Next, we will look at the final piece of the environment puzzle: **The Context Bridge (MCP)**. +Next: xref:AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc[The Context Bridge (MCP)]. xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Previous: Environment Introduction] | xref:AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc[Next: The Context Bridge (MCP)] diff --git a/en/AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc b/en/AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc index 304f0de4..ef80a656 100644 --- a/en/AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc +++ b/en/AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc @@ -2,35 +2,29 @@ = The Context Bridge: Model Context Protocol (MCP) -== Introduction: The Missing Link +== Introduction -We have explored the IDEs and the high-reasoning models. But even the most sophisticated AI, like **Qwen 3-Coder**, suffers from a fundamental flaw: **The Knowledge Cutoff**. +Even a strong model like **Qwen 3-Coder** has a hard limit: its training data has a cutoff date. It can't know about a Vulkan extension released after that date, it can't see your local filesystem, and it has no way of knowing your GPU's `maxSamplerAnisotropy` limit unless something tells it. The **Model Context Protocol (MCP)** is an open standard for letting an AI query that kind of ground truth — your local environment, or an external data source like the Vulkan registry — instead of guessing from memorized training data. -A model trained in 2025 cannot know the specifics of a Vulkan extension released in 2026. It cannot "see" your local file system, and it has no idea that your specific GPU has a `maxSamplerAnisotropy` limit of 16. This is where the **Model Context Protocol (MCP)** becomes a core component of AI-assisted development. MCP is the open standard that allows an AI to securely "reach out" and query the ground truth of your local environment. +== Why this matters in practice -== The Core Concept: "Context is King" +Ask a generic model "how do I implement a mesh shader for our `Renderer` class?" and, if its defaults assume Vulkan 1.0, it may not even recognize what a mesh shader is. With MCP, an assistant pointed at the **mcp-Vulkan** server can query the actual registry and pick up newer features — mesh shaders, ray tracing, descriptor buffers — that aren't reliably in its training data. The Vulkan MCP server also serves samples, guide entries, man pages, and other reference material, so the assistant isn't limited to whatever subset of the ecosystem it happened to see during training. -In **Collaborative Engineering**, we move beyond the "Chat Window" to the **Context Bridge**. +== The mcp-Vulkan server -Consider the case where you ask an AI: *"How do I implement a mesh shader for our existing `Renderer` class?"* A generic AI might give you a "best guess" snippet based on its training data, but if that model defaults to Vulkan 1.0, it won't even know what a mesh shader is! With the **Model Context Protocol (MCP)**, the AI can query the actual Vulkan Registry. By pointing your assistant to the **mcp-Vulkan** server, it gains the ability to "set its context" to the latest version of the spec, ensuring it knows about modern features like Mesh Shaders, Ray Tracing, and Descriptor Buffers. The Vulkan MCP provides up-to-date context from samples, guide entries, man pages, and technical documentation so your AI can fully understand the ecosystem you are developing in. +**mcp-Vulkan** is an open-source MCP server that exposes tools for searching Vulkan documentation and pulling specific topics from the current spec. -== The "Truth" Pipeline: The mcp-Vulkan Implementation +A common failure mode without it: an assistant suggests a deprecated call like `vkCreateDescriptorPool` without the `pNext` structures a modern feature needs. With mcp-Vulkan connected, you can ask it to set its context to a specific Vulkan version ("Vulkan 1.3, then list the extensions that are core in that version") and get an answer grounded in that version's actual spec rather than an average across versions. -The most powerful way to bridge the Vulkan ecosystem to your AI is the open-source **mcp-Vulkan** project. This is a specialized MCP server that provides tools for searching Vulkan documentation and retrieving specific topics directly from the latest spec. +=== Building the server -**Real-World Experience: The Version Drift Win** -Many developers get frustrated when an AI suggests deprecated functions like `vkCreateDescriptorPool` without including the necessary `pNext` structures for modern features. -By using **mcp-Vulkan**, you can simply tell your assistant: *"Use the Vulkan MCP to set your context to Vulkan 1.3, then list the extensions that are core in that version."* The AI calls the MCP, updates its internal representation, and provides an answer that is directly based on the 1.3 specification. +If you haven't built it yet, see xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[First Step: Building the Vulkan MCP Server]. That covers cloning the repo and running `npm install && npm run build` in `mcp-Vulkan/vulkan/`. -=== Building the Server +=== Connecting mcp-Vulkan to Goose -If you have not already built the server, see xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc#_first_step_building_the_vulkan_mcp_server[First Step: Building the Vulkan MCP Server] in the Environment Setup introduction. That section covers cloning the repository and running `npm install && npm run build` inside the `mcp-Vulkan/vulkan/` subdirectory. +Register the server as a Goose extension by editing its config file — `~/.config/goose/config.yaml` on macOS/Linux, or `%APPDATA%\goose\config.yaml` (typically `C:\Users\\AppData\Roaming\goose\config.yaml`) on Windows. -=== Connecting mcp-Vulkan to Goose (The Agentic Common Point) - -Once the `mcp-Vulkan` project is built, you must register it as a "Tool" for Goose. Goose stores its configuration in a YAML file, which varies by platform. On macOS or Linux, this is typically located at `~/.config/goose/config.yaml`, while on Windows it is found at `%APPDATA%\goose\config.yaml` (usually `C:\Users\\AppData\Roaming\goose\config.yaml`). - -To connect the bridge, open your `config.yaml` and add the following entry under the `extensions:` section. Make sure to replace `/path/to/` with the actual absolute path where you cloned the repository (you can find this by typing `pwd` in your terminal inside the `mcp-Vulkan` directory). +Add this under `extensions:`, replacing `/path/to/` with the actual path (`pwd` inside the `mcp-Vulkan` directory): [source,yaml] ---- @@ -42,26 +36,22 @@ extensions: - /path/to/mcp-Vulkan/vulkan/build/index.js ---- -Once saved, restart your Goose session (`goose session`). You can verify the connection by asking: *"List the tools you have available."* You should see `search-vulkan-docs` and `get-vulkan-topic` in the list. +Restart your Goose session (`goose session`) and confirm the connection by asking "list the tools you have available" — you should see `search-vulkan-docs` and `get-vulkan-topic`. === Connecting mcp-Vulkan to your IDE -While Goose is our autonomous agent, you can also leverage the **mcp-Vulkan** bridge directly within your primary development environment. This allows your Cloud-based assistant (like Claude or GPT) to access the Vulkan spec without leaving your editor. For instance, **JetBrains CLion** and **Android Studio** users can add the server via the MCP Servers panel in their AI Assistant settings, while **Visual Studio** and **VS Code** users can use the **Continue** or native **Copilot** extensions to register the server. - -For detailed, step-by-step instructions on these integrations, refer back to the platform-specific chapters in this section. +The same server can be registered directly in your IDE instead of (or in addition to) Goose. **JetBrains CLion** and **Android Studio** users add it through the AI Assistant's MCP Servers panel; **Visual Studio** and **VS Code** users can register it via the **Continue** or **Copilot** extensions. See the platform-specific chapters earlier in this section for the exact steps. -== Drawbacks and Incremental Wins +== Tradeoffs -=== The "Security vs. Utility" Balance -Giving an AI the ability to read your file system is a significant decision. When you first activate an MCP tool, your IDE or agent will ask for explicit permission to access a directory. To mitigate security risks, always restrict the AI's "Scope" to your specific project folder. Never give an AI access to your entire home directory or sensitive system folders. +**Filesystem access is a real permission grant, not a formality.** The first time you enable an MCP tool with filesystem access, your IDE or agent will ask which directory to allow. Scope it to your project folder — don't grant access to your home directory or anything outside the project. -=== Incremental Win: Automated System Design -The biggest win is the ability to perform **Automated System Design** for new features. Because the AI has access to your entire engine's history through MCP, you can ask it to: *"Design a new 'Post-Processing' pass that matches the coding style and memory management of our existing 'Shadow' pass."* The AI will read the shadow pass implementation and generate a post-processing system that feels like you wrote it yourself. +**Where it helps: consistency with existing code.** Because the model can pull in your engine's own history through MCP, you can ask it to design something like a new post-processing pass "matching the style and memory management of the existing shadow pass." It will actually read the shadow pass implementation rather than generating something generic — the result still needs review, but it starts from your codebase's conventions rather than a default template. -== Summary: The Complete Toolbelt +== Summary -The Model Context Protocol is the final piece of the environmental puzzle. It transforms the AI from a simple "Chatbot" into a **Graphics Engineering Assistant** that has access to the specification and your local codebase. By mastering the Context Bridge, you complete the setup of your AI-enhanced graphics lab. +MCP is what turns a model from something that answers from memory into something that can check its answers against the current spec and your own codebase. That's the last piece of the environment setup covered in this section. -Next, we move to **Section 3: Model Selection & Specialization**, where we will choose the specific model to inhabit this powerful environment. +Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc[Model Selection & Specialization], where we look at picking a model for this setup. xref:AI_Assisted_Vulkan/02_environment_setup/05_goose_native_agent.adoc[Previous: Goose & Local Intelligence] | xref:AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc[Next: Model Selection & Specialization] diff --git a/en/AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc b/en/AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc index cb0df68d..34db9abb 100644 --- a/en/AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc @@ -2,41 +2,41 @@ = Model Selection & Specialization: Introduction -== The Core Decision: Choosing Your AI Model Strategy +== Choosing a Model Strategy -In the previous section, we built the "Context Bridge"—the infrastructure that allows an AI to see your code, your build system, and the Vulkan specification. Now, we must address the most critical decision in your AI-assisted journey: **Selecting the right model for the right task.** +In the previous section we set up the context bridge: the infrastructure that lets an AI see your code, your build system, and the Vulkan specification. The next question is which model to actually use, and for what. -Not all Large Language Models (LLMs) are created equal. Some are reasoning-heavy with deep architectural capabilities but high latency; others are syntax-specialized that excel at C{pp}23 code generation but may struggle with high-level system design. In this section, we move beyond just "having an AI" to **specializing** one for the unique, high-performance world of Vulkan. +Not all LLMs are equally suited to this work. Some are strong at reasoning and architectural questions but slow; others are tuned for fast C{pp}23 code completion but weaker on high-level design. This section is about matching a model to a task rather than defaulting to whichever one is popular. -== The Concept: Base Model Weights vs. Live Context +== Base weights vs. live context -To master model selection, you must understand the difference between what a model "knows" (its training data) and what it can "see" (its live context). +There's a useful distinction between what a model knows from training and what it can see at inference time: -1. **The Base Model (Pre-trained Weights):** This is the model as it comes "out of the box." It has already read millions of lines of C{pp} and the entire Vulkan Spec as it existed when the model was trained. -2. **Specialization (The Live Context):** This is how we provide the model with your specific engine's "flavor." We achieve this through: - * **MCP (Model Context Protocol):** Real-time access to the exact version of the Vulkan Spec and your local project files. - * **RAG (Retrieval-Augmented Generation):** A searchable "memory" of your engine's internal wiki, coding standards, and past architectural decisions. - * **Fine-Tuning (LoRA):** Training a small "patch" for the model that teaches it your specific naming conventions (e.g., *"In this engine, we always use `vk::raii` and never raw pointers"*). +1. **The base model.** This is the model as shipped, trained on some snapshot of public C{pp} code and the Vulkan spec as it existed at training time. +2. **Added context.** This is how you give the model information specific to your project. The main mechanisms are: + * **MCP (Model Context Protocol):** real-time access to the current Vulkan spec and your local project files. + * **RAG (Retrieval-Augmented Generation):** a searchable index of your engine's internal docs, coding standards, and past decisions. + * **Fine-tuning (LoRA):** a small trained adjustment that teaches the model your naming conventions — for example, that this engine uses `vk::raii` and never raw pointers. -== Why Specialization Matters for Vulkan +== Why this matters for Vulkan -Vulkan is a "living" API. New extensions are added monthly, and best practices evolve (e.g., the shift from Render Passes to Dynamic Rendering, or from Descriptor Sets to Descriptor Buffers). +Vulkan changes fairly often — new extensions land regularly, and recommended practice shifts over time (dynamic rendering over render passes, descriptor buffers over descriptor sets, and so on). A couple of practical consequences follow: -* **The Hallucination Risk:** A general-purpose AI might suggest a Vulkan 1.0 pattern because it is more common in its training data. -* **The Precision Requirement:** Graphics programming requires extreme precision in memory offsets, alignment, and synchronization. A model that is 95% accurate is dangerous; a specialized model that understands your engine's specific alignment wrappers is an asset. +* A general-purpose model can suggest an older pattern simply because it was more common in its training data. +* Graphics code is unforgiving of small mistakes in memory offsets, alignment, and synchronization. A model that's usually right but occasionally wrong on these details can cost you more debugging time than it saves, unless it also knows your engine's specific wrappers. -== The Model Selection Roadmap: From Generic to Specialized +== What's ahead in this section -In the following chapters, we will guide you through the technical and strategic process of choosing and specializing your AI assistant. +The next few chapters walk through selecting and adapting a model: -We begin by **Selecting the Base Model**. We will compare the leading models for Vulkan development, including **Qwen 3-Coder**, **Mistral-Nemo**, and **Llama 4**. You will learn which models excel at the complex, high-level reasoning required for architectural planning, versus the smaller, faster models best suited for the "inner loop" of real-time autocomplete and shader implementation. +We start with **selecting a base model**, comparing options like Qwen 3-Coder, Mistral-Nemo, and Llama 4, and looking at which ones are worth the latency cost for architectural work versus which are better suited to fast autocomplete and shader iteration. -Next, we address the critical issue of **Hardware & VRAM Budgeting**. Running these models locally requires a precise strategy. We will provide detailed formulas for calculating your VRAM needs, accounting for the "KV Cache Tax" and the role of **Quantization** in ensuring your AI doesn't starve your Vulkan application of its most precious resource. +Then we cover **hardware and VRAM budgeting**. Running models locally has a real cost in GPU memory, and we'll go through the arithmetic — including the KV cache and quantization — for figuring out how much you actually have to work with alongside a Vulkan application. -We then dive deep into **Ground Truth: MCP & RAG Specialization**. You will learn how to use **MCP (Model Context Protocol)** to bridge the "Global Context Gap" by pointing your AI at the live Vulkan Registry (`vk.xml`), and how to use **RAG (Retrieval-Augmented Generation)** to index your own engine's "Local Context." Together, these ensure your assistant suggests valid API calls that match your project's unique architectural style. +After that, **MCP and RAG specialization**: using MCP to point your AI at the live Vulkan registry (`vk.xml`), and RAG to index your own engine's code and docs, so suggestions stay both spec-correct and consistent with your codebase. -Finally, for those who want a highly specialized assistant, we explore **Fine-Tuning for Your Engine**. You will learn the **Teacher-Student** paradigm, using high-reasoning Cloud models to "teach" and retrain a local assistant into a **Specialized SLM**. Using **LoRA (Low-Rank Adaptation)**, you will "bake" your engine's specific abstractions and coding standards into the model's weights, ensuring it defaults to your senior engineering patterns without constant prompting. +Finally, for anyone who wants to go further, **fine-tuning with LoRA** — using a stronger cloud model to help generate training data, then training a small adapter that bakes your engine's conventions into a local model so it doesn't need constant reminding. -By the end of this section, you will no longer be using a "generic" AI. You will have a specialized, Vulkan-aware assistant that understands both the global standards of graphics programming and the local "laws" of your specific codebase. +By the end of this section you'll have a model setup that's aware of both the Vulkan spec in general and your codebase in particular, rather than a generic assistant with no project context. Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/02_base_models.adoc[Selecting the Base Model] diff --git a/en/AI_Assisted_Vulkan/03_model_selection_specialization/02_base_models.adoc b/en/AI_Assisted_Vulkan/03_model_selection_specialization/02_base_models.adoc index 01f7786b..e4f0f683 100644 --- a/en/AI_Assisted_Vulkan/03_model_selection_specialization/02_base_models.adoc +++ b/en/AI_Assisted_Vulkan/03_model_selection_specialization/02_base_models.adoc @@ -1,112 +1,100 @@ :pp: {plus}{plus} -= Selecting the Base Model: The AI Model Strategy += Selecting the Base Model -== Introduction: The Graphics Developer's Constraint +== The graphics developer's constraint -Selecting a base model for Vulkan development is not just about choosing the "biggest" model. In the world of graphics programming, we face a unique constraint: **GPU Resource Contention.** Every gigabyte of VRAM your AI model consumes is a gigabyte that is unavailable for your Vulkan textures, vertex buffers, and framebuffers. +Choosing a base model for Vulkan development isn't just about picking the largest one available. Graphics work has a specific constraint: every gigabyte of VRAM your model uses is a gigabyte unavailable for textures, vertex buffers, and framebuffers. -To succeed, we must move beyond the generic models and choose specialized models that balance reasoning depth with a manageable hardware footprint. +So the goal is a model that balances reasoning ability against a hardware footprint you can actually afford to keep running alongside your application. -== The Concept: How to Evaluate Your Model +== How to evaluate a model -In the rapidly evolving landscape of AI, the "best" model changes every few months. To remain effective, you shouldn't just memorize a list of currently popular models. Instead, you must learn to evaluate new models based on three critical criteria for graphics development. +The "best" model changes every few months, so it's more useful to have criteria than to memorize a leaderboard. Three things matter most for graphics work: -=== 1. The VRAM Tax: Parameter Count vs. Quantization +=== 1. VRAM cost: parameters vs. quantization -The first question you must ask is: *"Does this model fit on my hardware alongside my Vulkan application?"* Generally, a higher parameter count translates to more "intelligence" and better reasoning capabilities. However, a massive 70B model is counterproductive if it starves your Vulkan application of VRAM, forcing it to use much slower system RAM. +The first question is whether the model fits on your hardware alongside your Vulkan application. Higher parameter counts generally mean better reasoning, but a 70B model isn't worth it if it forces your application into slower system RAM. -To fit these models onto consumer hardware, we rely on **Quantization** (compression), typically using formats like **GGUF** or **EXL2**. A 4-bit (Q4) quantization is often the "Goldilocks" zone—it can reduce the model's memory footprint by 75% while only incurring a negligible drop in reasoning accuracy. +Quantization (compression formats like GGUF or EXL2) makes larger models fit on consumer hardware. 4-bit (Q4) quantization is a common middle ground — it cuts memory footprint by roughly 75% with only a small drop in accuracy. -=== 2. Reasoning Depth: The "Structural Reasoning Test" +=== 2. Reasoning depth -Can the model understand the complex state machine of Vulkan? To test a new model, ask it a non-trivial architectural question, such as: *"Explain the synchronization requirements for a compute-to-graphics dependency where the compute shader writes to a storage buffer that is then used as a vertex buffer in the next pass."* +A reasonable test question: "Explain the synchronization requirements for a compute-to-graphics dependency where the compute shader writes to a storage buffer that is then used as a vertex buffer in the next pass." -A model that fails this test will often provide a generic `vkDeviceWaitIdle` or a simple execution barrier without defining the necessary access masks. In contrast, a "passing" model will correctly identify the requirement for a transition from `VK_ACCESS_SHADER_WRITE_BIT` to `VK_ACCESS_VERTEX_ATTRIBUTE_READ_BIT` and suggest the appropriate pipeline stages to ensure the data is visible and the execution is ordered correctly. +A weaker model will often answer with a blanket `vkDeviceWaitIdle` or a generic barrier with no access masks specified. A stronger one will identify the need for a transition from `VK_ACCESS_SHADER_WRITE_BIT` to `VK_ACCESS_VERTEX_ATTRIBUTE_READ_BIT` and name the correct pipeline stages. -NB: This target will change over time as AI models are constantly improving and what is true as of the writing of this tutorial, might not be current. Thus, please assume that over the course of development, you have the ability to adapt to new models and their capabilities through the tools and techniques taught in this tutorial. We're teaching an ecosystem rather than an exact version. You might have a local server able to stand in for a local to company environment and the SLM targeting information here will allow you to further refine it specifically to your needs. It is recommended to treat the model you use along with the extra training it needs as a hyperparameter to give you more or less help as your development needs dictate. +NB: This test, and any specific model recommendations, will age. Treat the criteria as durable and the named models as examples current at time of writing — expect to re-evaluate as new models ship. The model (and how much fine-tuning it gets) is really just another parameter you tune based on how much help a given task needs. -=== 3. Context Window: The "Engine Memory" +=== 3. Context window -Vulkan engines are large. A model with a small context window (e.g., 4,000 tokens) will "forget" your memory allocator by the time it finishes reading your pipeline setup. For 2026 standards, look for models with at least a **32k to 128k context window**. This allows the assistant to keep your entire renderer's header files in its active memory while it works on a specific implementation, ensuring it remains aware of your project's unique architectural patterns. +Vulkan engines have a lot of files. A model with a small context window (say, 4,000 tokens) will lose track of your memory allocator by the time it's read through your pipeline setup. Look for at least 32k–128k tokens of context if you want the assistant to hold a renderer's header files in view while working on a specific implementation. -== The 2026 best in class (as Examples) +== Current examples (as of writing) -While you should use the evaluation criteria above for future models, here are the current leaders that exemplify these traits: +These are current examples of models that illustrate the criteria above — expect the specific names to change. -=== 1. Qwen 3-Coder (30B): High-Reasoning Model +=== Qwen 3-Coder (30B): strong reasoning -The **Qwen 3-Coder (30B)** model is currently the world's most capable local non-cloud coding model for C{pp}. +Qwen 3-Coder (30B) is currently one of the stronger local, non-cloud coding models for C{pp}. -**Real-World Experience: The Bindless Transition** -Imagine you are refactoring your engine to use a **Bindless Texture** system. This requires a deep understanding of Descriptor Set Layout bindings, `descriptorIndexing` features, and the nuances of how GLSL access is translated into SPIR-V. +**Example: a bindless texture refactor.** Moving an engine to bindless textures touches descriptor set layout bindings, `descriptorIndexing` features, and how GLSL access maps to SPIR-V. Given this task, Qwen 3-Coder can look at how your shaders use `sampler2D textures[]` and suggest the specific `DESCRIPTOR_BINDING_PARTIALLY_BOUND_BIT` flags needed to avoid validation errors, and it tends to catch synchronization issues that smaller models miss. That makes it a reasonable choice for architectural work and Vulkan 1.3/1.4 features — at the cost of roughly 20GB of VRAM at Q4_K_M, which realistically means a 24GB GPU or a local server. -When you ask Qwen 3-Coder to perform this refactor, it doesn't just give you a boilerplate `VkDescriptorSetLayoutCreateInfo`. It analyzes your shader code to see how you're using `sampler2D textures[]` and suggests the exact `DESCRIPTOR_BINDING_PARTIALLY_BOUND_BIT` flags you need to prevent validation errors. Its reasoning depth allows it to catch synchronization issues that smaller models would completely overlook. This makes it the ideal choice for complex architectural planning and Vulkan 1.3/1.4 feature implementation. However, it requires significant resources—approximately 20GB of VRAM at Q4_K_M quantization—making it most suitable for developers with 24GB GPUs or those working in local server environments. +=== Llama 4 (17B): low latency -=== 2. Llama 4 (17B): High-Speed Assistant +For fast, inline feedback while typing, Llama 4 (17B) is a solid choice. It can predict the next several lines of a `VkPipelineShaderStageCreateInfo` array as you write it, correctly infer entry point names from project context, and set `pSpecializationInfo` to `nullptr` when it sees you aren't using specialization constants. -For the "inner loop" of development—where you need near-instant feedback while typing—speed is everything. **Llama 4 (17B)** is the current champion of low-latency coding. +At around 6GB of VRAM, it runs comfortably alongside a Vulkan application on mid-range hardware (12–16GB GPUs), which makes it a good fit for autocomplete, unit tests, and quick shader iteration. -**The Experience:** You are typing a new `VkPipelineShaderStageCreateInfo` array. Llama 4, running locally on your GPU, predicts the next 10 lines of code before you've even finished the first struct. It correctly identifies that you are using a vertex and fragment shader, automatically pulls in the correct entry point names from your project context, and sets the `pSpecializationInfo` to `nullptr` because it sees you aren't using specialization constants in this pass. +=== Mistral-Nemo (12B): large context -The low VRAM footprint—consuming only ~6GB—allows it to run comfortably alongside a heavy Vulkan application on mid-range hardware (12GB-16GB GPUs), making it perfect for real-time autocomplete, unit tests, and rapid shader iteration. +Mistral-Nemo (12B), a joint NVIDIA/Mistral AI release, is tuned for efficient reasoning with a 128k context window. -=== 3. Mistral-Nemo (12B): The "Context King" +**Example: an engine-wide refactor.** A global illumination system that touches the scene graph, several passes, and post-processing needs a model that can keep more than one file in view at a time. Because Mistral-Nemo can hold up to 128,000 tokens of context, it can read across a renderer's header directory and give suggestions that stay consistent with the rest of the codebase — useful for large refactors and following class hierarchies that smaller models tend to lose track of. At roughly 8–10GB VRAM (Q4), it sits between the fast 17B models and the heavier 30B+ ones. -**Mistral-Nemo (12B)**, a collaboration between NVIDIA and Mistral AI, is a powerhouse for local development, specifically tuned for high-efficiency reasoning with a massive **128k context window**. +== Interaction style: cloud vs. local -**Real-World Experience: The Engine-Wide Refactor** -Imagine you are implementing a new **Global Illumination** system that touches dozens of files—from the scene graph to the final post-processing pass. You need an assistant that doesn't just see the current file, but maintains the context of your entire architectural hierarchy. +Cloud and local models expect to be talked to differently, and knowing which is which saves some frustration. -Mistral-Nemo (12B) excels in these "Long-Context" scenarios. Because it can hold up to 128,000 tokens in its active memory, it can "read" your entire renderer's header directory and provide suggestions that are architecturally consistent across the whole codebase. It is particularly effective for large-scale refactoring and navigating complex class hierarchies where smaller models would lose the thread. With a slim VRAM footprint of only ~8GB-10GB (Q4), it provides a perfect middle-ground between the ultra-fast 17B models and the resource-heavy 30B+ models, making it an ideal choice for developers working on complex, multi-file Vulkan systems. +=== Cloud models: high-level intent -== Interaction Styles: Intent vs. Explicit Instructions +Cloud models like Claude 4.6 or GPT-5.3 handle abstraction well — you can talk to them roughly the way you'd talk to a colleague. If you say "I'm seeing Z-fighting on distant terrain, using a standard 24-bit depth buffer, what should I check first?", the model can infer you want the artifact fixed and suggest likely causes (depth precision, near/far plane ratio) without you spelling out your projection matrix setup. -A critical difference between Cloud models and Local models lies in how you "talk" to them. Understanding this distinction is key to avoiding frustration during your development workflow. +=== Local models: explicit instructions -=== The Cloud: High-Level Intent +Local models, even capable ones like Qwen 3-Coder (30B), generally need more specific instructions — they're better treated as a skilled implementer working from a spec than as a conversational partner. A more effective local-model prompt: "I'm seeing Z-fighting. Check whether the near plane is too close (currently 0.01) or the far plane too distant (10000.0). Suggest a logarithmic depth buffer if the ratio exceeds 1:1,000,000, and provide the GLSL for the vertex shader update." Being explicit like this keeps a local model on track. -Cloud models like **Claude 4.6** or **GPT-5.3** are designed to handle high levels of abstraction. You can interact with them as you would a human engineer, as they excel at "reading between the lines" and translating conversational requests into structured tasks. For example, if you prompt: *"I'm seeing some Z-fighting in the distant terrain. I'm using a standard 24-bit depth buffer. What should I look at first?"*, the model understands your intent to fix the visual artifact. It identifies likely culprits like depth buffer precision or near/far plane ratios and provides a prioritized checklist without requiring you to specify the technical details of your projection matrix setup. +== Cloud-to-local: using one to brief the other -=== The Local Model: Explicit Instructions (Instruction-Following Model) +One workflow worth knowing: use a high-reasoning cloud model to write the technical spec, then hand that spec to a local model for implementation. -Local models, even advanced ones like **Qwen 3-Coder (30B)**, generally require more specific and verbose instructions. They are less capable of inferring intent from a casual sentence and more effective when treated as a highly skilled implementer following a detailed technical specification. A more effective prompt for a local model would be: *"I am experiencing Z-fighting. Analyze the following projection matrix calculation. Check if the near plane is too close (currently 0.01) or if the far plane is too distant (10000.0). Suggest a logarithmic depth buffer implementation if the ratio exceeds 1:1,000,000. Provide the GLSL for the vertex shader update."* By providing these explicit constraints, you ensure the local model remains focused and accurate, as it thrives on technical detail rather than conversational abstraction. +=== Writing the handoff document -== The Translation Strategy: Cloud to Local Pipeline +When planning with a model like Claude 4.6, instead of asking for code directly, ask it to produce an instruction document for a local assistant. -One of the most powerful workflows in the "Collaborative Engineering" paradigm is using a high-reasoning Cloud model to prepare the technical specification for a specialized Local model. This effectively bridges the gap between conversational intent and precise implementation. +* **The workflow:** + 1. Discuss the high-level goal with the cloud model (e.g., "I want to implement a GPU-driven culling system"). + 2. Ask it to write a file, say `LLM_NOTES.md`, containing the exact structs, memory layout, and synchronization requirements just discussed, in explicit, unambiguous terms. + 3. Open `LLM_NOTES.md` in your IDE and point your local assistant (via Goose or Junie) at it. The local model now has a concrete spec to follow instead of having to infer your intent. -=== Creating the "Contract" (LLM_NOTES.md) +=== Why this is worth doing -When you are in the planning phase with a model like **Claude 4.6**, don't just ask it for code. Ask it to generate a "Contract" or an instruction set specifically optimized for a local model like **Qwen 3-Coder** or **Mistral-Nemo**. +The cloud model does the translation from loosely specified intent to a precise spec, which saves you from writing that spec by hand; the local model then does the implementation without needing your proprietary code to leave the machine. This works best as a plan-then-implement split, and it's really only worth the overhead for tasks big enough that writing the spec pays for itself — for something trivial, it may take the cloud model as long to write the spec as it would to just do the task itself. -* **The Workflow:** - 1. **Converse with the Cloud:** Discuss your high-level architectural goals (e.g., "I want to implement a GPU-driven culling system"). - 2. **Generate the Spec:** Ask the Cloud model: *"Now, write a technical specification file named `LLM_NOTES.md` that I can provide to a local coding assistant. It should include the exact Vulkan structs, memory layout, and synchronization requirements we just discussed, using verbose and explicit instructions."* - 3. **Feed the Local Model:** Open `LLM_NOTES.md` in your IDE and point your local assistant (via Goose or Junie) to that file. The local model will now have a "Ground Truth" document that it can follow with high precision, without needing to guess your high-level intent. +NB: `LLM_NOTES.md` is a working document for the current session, not project documentation — it shouldn't be checked into git. -=== Why This Works +This approach is most useful on longer tasks with multiple iterations, and it's also a reasonable way to unstick a local model that's gone in circles on a problem: a short cloud session can often get it back on track. -This strategy leverages the best of both worlds by using the **Cloud** to handle the "Human-to-Technical" translation, saving you from writing pages of verbose instructions, while the **Local Model** handles the "Technical-to-Code" implementation. This keeps your proprietary engine logic on your machine and provides low-latency execution. +== Picking a model per phase -By treating the Cloud AI as your **Translator** and the Local AI as your **Implementer**, you minimize the risk of "Instruction Drift" and ensure that your Vulkan engine remains technically sound from conception to commit. This strategy is best used in a "planning" then "implementing" workflow. This practice needs to be balanced because if the task is too tiny then the benefits of having the local LLM handle the implementation disappear as it might take the cloud LLM as much time to do the work on its' own rather than offload the implementation step to the local LLM. +A reasonable default is to match the model to the phase of work, while treating these as fluid rather than fixed roles: -NB: The `LLM_NOTES.md` file is a transient, developer-focused "Contract" used to provide context for the current AI session. Since it contains private architectural decisions and project-specific instructions that are not intended for final source code documentation, it is **not** part of the engine's source project and should never be checked into your Git repository. It is your local "Ground Truth" for the AI, not a permanent project file. +During **planning and system design**, use a high-reasoning cloud model or Qwen 3-Coder (30B) for things like a cross-platform frame graph or a thread-safe allocator, where reasoning depth matters more than speed. During **implementation and verification**, Mistral-Nemo (12B) or Qwen 3-Coder for turning the design into code that satisfies validation requirements. For **rapid iteration**, Llama 4 (17B) or a fast cloud model, where speed matters more than depth. -This strategy is best for large tasks that take multiple iterations of implementation. You often might find the local LLM get "stuck" on a problem; a quick Cloud AI session can rescue the local LLM and set it on the right track again using this strategy. +== Summary -== Decision Framework: Finding Your Balance +Model selection is mostly about matching reasoning depth to VRAM budget, and knowing when it's worth paying for a cloud model's reasoning versus running something local and fast. Using cloud models for planning, mid-sized local models for precise implementation work, and small fast models for the daily inner loop gives you a setup that's responsive without being under-powered for the harder problems. -How do you choose between these powerhouses? We recommend a **Hybrid Strategy** based on your development phase, but remember that these roles are fluid. The "best" model is the one that fits your current task without starving your Vulkan application of resources. - -During the **Planning & System Design Phase**, use high-reasoning **Cloud Models** or **Qwen 3-Coder (30B)** for architectural brainstorming—like designing a cross-platform frame graph or a thread-safe memory allocator. These provide maximum reasoning depth with zero VRAM cost (in the case of cloud models). In the **Implementation & Verification Phase**, switch to **Mistral-Nemo (12B)** or **Qwen 3-Coder** to translate your architecture into code that meets strict validation requirements. Their precision significantly reduces the time spent in the fix-loop. Finally, for the **Rapid Iteration Phase**, use **Llama 4 (17B)** or a cloud-based "Flash" model like **Gemini 3.1 Flash**. Their high speed allows you to keep your Vulkan renderer running at high frame rates while the AI assists with implementation details, unit tests, or simple refactors. - -== Summary: Integrated Model Strategy - -Selecting a base model is about managing your **"Intelligence-to-VRAM" ratio** and knowing when to leverage the cloud. By utilizing Cloud models for "Deep Planning," specialized local models like Nemo or Qwen for "Precision Refactoring," and Llama for the "Daily Inner-Loop," you create a development environment that is both deeply knowledgeable and highly responsive. - -The most successful developers are those who treat their AI models as a dynamic resource—scaling their "intelligence" up or down and switching between local and cloud providers as the complexity of the graphics task, budget, and hardware constraints dictate. - -In the next chapter, we will look at the **Hardware Requirements** and show you the exact formulas for calculating your VRAM budget. +The main skill is treating model choice as something you actively adjust — scaling up or down, switching between local and cloud — based on what the current task actually needs, rather than sticking with one model for everything. Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc[Hardware & VRAM Budgeting] diff --git a/en/AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc b/en/AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc index 30246d1d..ea8e3ca6 100644 --- a/en/AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc +++ b/en/AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc @@ -1,97 +1,83 @@ :pp: {plus}{plus} -= Hardware & VRAM Budgeting: The Resource Strategy += Hardware & VRAM Budgeting -== Introduction: The Battle for VRAM +== VRAM is a shared resource -Vulkan development is a high-performance pursuit. When you introduce a local AI assistant, you are no longer just competing with other applications for CPU cycles; you are competing with your own engine for **VRAM**. +Once you introduce a local AI assistant, you're no longer only competing with other applications for CPU time — you're competing with your own engine for VRAM. Treating this seriously means working out a budget rather than assuming things will fit. -In the "Collaborative Engineering" era, the GPU is a shared resource. To succeed, you must move beyond "plug and play" and develop a **Resource Strategy** that ensures your AI assistant doesn't starve your Vulkan application of the memory it needs to render a single frame. +== VRAM contention and swapping to system RAM -== The Core Concept: VRAM Contention & The "Swap Trap" +VRAM isn't a static bucket you fill once. With a local model running, your GPU is regularly switching between two demanding workloads: -Most developers think of VRAM as a static bucket. You have 16GB, you use 10GB, you have 6GB left. However, in an AI-assisted environment, your GPU is constantly context-switching between two "heavy" workloads: +1. **The LLM inference engine**, holding the model's weights in memory to generate output. +2. **The Vulkan application**, holding textures, buffers, and render targets. -1. **The LLM Inference Engine:** Holding billions of weights in memory to generate code. -2. **The Vulkan Application:** Holding textures, buffers, and render targets to display your graphics. +If the combined requirement exceeds your physical VRAM, the OS starts moving AI weights or Vulkan resources into much slower system RAM. This is bad for both sides: token generation can drop from ~50/s to single digits, and your application's frame rate can fall into the single digits too. -If the combined VRAM requirement exceeds your physical GPU memory, your system will hit the **Swap Trap**. Your OS will begin moving AI weights or Vulkan textures into your much slower System RAM (DDR4/5). For a graphics developer, this is catastrophic: AI generation will slow from 50 tokens per second to 2, and your Vulkan application's frame rate will drop into the single digits. +== Calculating your actual budget -== The Technical Formula: Calculating Your "True Budget" - -To avoid the Swap Trap, you must calculate your VRAM budget using a formula that accounts for more than just the model size. You must also account for the **KV Cache** (the AI's short-term memory). +The model-size number on a download page isn't the whole picture — you also need to account for the KV cache, the model's short-term memory for the current conversation. [source,text] ---- Total VRAM - (OS/Desktop + Vulkan Peak + (Model Size * Quantization) + KV Cache) = Safety Margin ---- -* **OS/Desktop:** Typically **1.5 - 2 GB** on Windows/Linux. -* **Vulkan Peak:** The maximum VRAM your engine uses during a heavy debug session. -* **Model Size * Quantization:** A **30B model** (like **Qwen 3-Coder**) at **4-bit (Q4_K_M)** quantization is ~18GB, while a **12B model** (like **Mistral-Nemo**) is approximately ~8GB. -* **KV Cache Tax:** This is the most overlooked part. As you chat with the AI, it needs memory to remember the previous parts of the conversation. At a **16k context window**, this can consume an additional **1-2 GB** of VRAM. - -**Real-World Experience: The 24GB VRAM Budget** -Imagine you are using an **RTX 4090 (24GB)**. - -* **OS + Vulkan App:** 6GB. -* **Qwen 3-Coder (30B) @ Q4_K_M:** 18GB. -* **Budget:** `24 - (6 + 18) = 0GB`. -* **The Result:** You have **zero** safety margin. As soon as you ask the AI a long question and the KV Cache grows, your system will start swapping. To fix this, you must either use a smaller model (like **Mistral-Nemo 12B** or **Llama 4 (17B)**) or reduce the AI's context window to **4,096 tokens** to reclaim that 1-2 GB of "breathing room." - -== Quantization: Optimizing the Model Footprint - -You do not need to run a model at its full, uncompressed precision (FP16). Instead, we use **Quantization** to reduce the number of bits used for each weight. Think of it like texture compression (e.g., BC7 or ASTC) for neural networks: you trade a small amount of mathematical precision for a massive reduction in memory footprint. - -**The "Sweet Spot" for Graphics Devs:** -We recommend **4-bit (Q4_K_M)** or **5-bit (Q5_K_M)** quantization. +* **OS/Desktop:** typically 1.5–2GB on Windows/Linux. +* **Vulkan peak:** the maximum VRAM your engine uses during a heavy debug session. +* **Model size × quantization:** a 30B model (e.g. Qwen 3-Coder) at 4-bit (Q4_K_M) is roughly 18GB; a 12B model (e.g. Mistral-Nemo) is roughly 8GB. +* **KV cache:** easy to overlook. At a 16k context window, this can add another 1–2GB. -* **Why?** At 4-bit, the model's size is reduced by 75%, but its ability to reason about complex C{pp} code remains nearly 98% intact. Going lower than 4-bit (e.g., 2-bit) causes the AI to "lose the plot," leading to hallucinations in your synchronization logic because the weights no longer have enough precision to represent the subtle relationships between Vulkan's state machine transitions. +**Example: a 24GB card.** On an RTX 4090 (24GB), with 6GB going to OS + Vulkan app and 18GB to Qwen 3-Coder (30B) at Q4_K_M, that's `24 - (6 + 18) = 0GB` of margin. Any long conversation that grows the KV cache will push you into swapping. The fix is either a smaller model (Mistral-Nemo 12B or Llama 4 17B) or reducing the context window to something like 4,096 tokens to free up that 1–2GB. -== Practical Tutorial: How to Quantize a Model +== Quantization -While most developers download pre-quantized models from HuggingFace, a professional graphics engineer should know how to perform this "compression" themselves. This allows you to take a "raw" model (FP16) released by a research lab and immediately optimize it for your local GPU. +Running a model at full precision (FP16) usually isn't necessary. Quantization reduces the bits used per weight — similar in spirit to texture compression (BC7, ASTC): you trade some precision for a large reduction in memory. -The primary tool for this is **llama.cpp**. Here is the step-by-step process: +4-bit (Q4_K_M) or 5-bit (Q5_K_M) is a reasonable default. At 4-bit, size drops by about 75% while reasoning ability on C{pp} code stays close to its original level. Going below 4-bit (e.g. 2-bit) tends to cause real degradation — the model starts making mistakes on the finer details of Vulkan's state machine because the weights no longer encode them precisely enough. -1. **Clone and Build the Tools:** +== Quantizing a model yourself - [source,bash] - ---- - git clone https://github.com/ggerganov/llama.cpp - cd llama.cpp - make -j # Builds the quantize and convert binaries - ---- +Most people download pre-quantized models from HuggingFace, but it's worth knowing how to do this yourself so you can take a freshly released FP16 model and fit it to your hardware. -2. **Convert to GGUF Format:** - Most models are released in the HuggingFace `SafeTensors` format. You first need to convert them into the `GGUF` format, which is optimized for local inference. +The main tool is **llama.cpp**: - [source,bash] - ---- - python3 convert_hf_to_gguf.py models/my_vulkan_model/ - ---- +1. **Clone and build:** ++ +[source,bash] +---- +git clone https://github.com/ggerganov/llama.cpp +cd llama.cpp +make -j # Builds the quantize and convert binaries +---- -3. **Run the Quantization:** - Now, you apply the compression. We will use the `Q4_K_M` method, which uses a "Medium" 4-bit quantization that balances speed and accuracy. +2. **Convert to GGUF.** Most models ship in HuggingFace `SafeTensors` format and need converting to GGUF for local inference: ++ +[source,bash] +---- +python3 convert_hf_to_gguf.py models/my_vulkan_model/ +---- - [source,bash] - ---- - ./quantize models/my_vulkan_model/model-f16.gguf models/my_vulkan_model/model-Q4_K_M.gguf Q4_K_M - ---- +3. **Quantize.** `Q4_K_M` is a reasonable balance of speed and accuracy: ++ +[source,bash] +---- +./quantize models/my_vulkan_model/model-f16.gguf models/my_vulkan_model/model-Q4_K_M.gguf Q4_K_M +---- -4. **Verification:** - The resulting file will be significantly smaller. A **32GB** FP16 model will now be approximately **18GB**. You can now load this file directly into **Ollama** or **LM Studio** and enjoy the extra VRAM "breathing room" for your Vulkan swapchain. +4. **Verify.** A 32GB FP16 model should come out to roughly 18GB. Load it into Ollama or LM Studio and check the VRAM headroom for your swapchain. -== Mitigating Contention: Professional Strategies +== If VRAM is still tight -If your hardware is limited, you must become a **Resource Manager**: +A few concrete options if your hardware is limited: -1. **Context Limiting:** In your inference engine (Ollama/LM Studio), manually set the `context_length` to **8192**. This is usually enough for a few files of Vulkan code but small enough to save 1.5GB of VRAM over the default settings. -2. **Layer Offloading (The GGUF Strategy):** If you use the **GGUF** format, you can "split" the model. If a model has 60 layers and your GPU is full, you can move 10 layers to your CPU. The AI will be 20% slower, but your Vulkan app will regain 3GB of VRAM and stop stuttering. -3. **The "Secondary GPU" Hack:** If you have an old 8GB or 12GB card, install it in your second PCIe slot. Use the **Primary GPU** for your Vulkan app and point **Ollama** exclusively at the **Secondary GPU**. This is the ultimate "No-Contention" setup for professional engine developers. +1. **Limit the context window.** Set `context_length` to something like 8192 in Ollama/LM Studio — usually enough for a few files of Vulkan code, and it saves 1.5GB or so versus the default. +2. **Offload layers to CPU.** With GGUF, you can move some layers off the GPU — e.g. 10 of 60 layers — trading roughly 20% inference speed for a few GB of VRAM back. +3. **Use a second GPU.** If you have a spare 8–12GB card, put it in a second PCIe slot and point Ollama at it exclusively, keeping your primary GPU dedicated to the Vulkan application. -== Summary: The Strategic Lab +== Summary -Hardware budgeting is the "secret sauce" of a stable AI-assisted environment. By understanding your VRAM footprint and proactively managing your KV Cache and Quantization, you ensure that your AI assistant remains a helpful co-author rather than a resource-hungry rival to your graphics engine. +Hardware budgeting is the unglamorous part of running a local AI setup, but understanding your VRAM footprint — and actively managing the KV cache and quantization — is what keeps the assistant from competing with your renderer for memory. -Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc[Ground Truth: MCP & RAG Specialization] +Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc[MCP & RAG Specialization] diff --git a/en/AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc b/en/AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc index aa547e31..06f40615 100644 --- a/en/AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc +++ b/en/AI_Assisted_Vulkan/03_model_selection_specialization/04_rag_mcp_specialization.adoc @@ -1,83 +1,75 @@ :pp: {plus}{plus} -= Ground Truth: MCP & RAG Specialization += MCP & RAG Specialization -== Introduction: The "Hallucination" Gap +== The training-cutoff problem -Even the most powerful base model, like **Qwen 3-Coder**, is trapped by its **Training Cutoff**. If the Vulkan Working Group releases a new extension today, your AI model will not know about it. This creates a dangerous "Hallucination Gap": the AI might "invent" structure members or bitmasks that don't exist, leading to hours of wasted debugging. +Even a strong base model like Qwen 3-Coder only knows what existed in its training data. If the Vulkan working group ships a new extension today, the model won't know about it — and it may confidently invent struct members or bitmasks that don't exist, which can cost real debugging time. -To bridge this gap, we move from a "Closed-Book" AI to an **"Open-Book" Specialist**. We do this through two primary methods: **MCP** and **RAG**. +MCP and RAG are the two main ways to give a model access to information beyond its training data. -== The Core Concept: Global Truth vs. Local Truth +== Two kinds of context -To build a truly effective AI-assisted workflow, you must understand the difference between the two types of knowledge: +It helps to separate two kinds of knowledge a model needs: -1. **Global Truth (The Vulkan Spec):** This is the official definition of the API. It is shared by every developer on the planet. We connect the AI to this "Global Truth" via **MCP**. -2. **Local Truth (Your Engine Architecture):** This is your engine's specific coding style, memory management, and internal abstractions. No AI model in the world knows this because it’s private. We connect the AI to this "Local Truth" via **RAG**. +1. **The Vulkan spec itself** — the same for every developer. This is what MCP connects the model to. +2. **Your engine's specific conventions** — private to your project, and something no general-purpose model was trained on. This is what RAG connects the model to. -== Strategy 1: The Context Bridge (MCP) +== MCP: connecting to the spec -As we explored in the environment section, the **Model Context Protocol (MCP)** allows your AI to "reach out" and query real-time data. +The Model Context Protocol lets your AI query real-time data instead of relying only on training data. -**Real-World Experience: The `vk.xml` Lookup** -Imagine you are implementing a complex post-processing pass and need to know the valid `VkImageUsageFlags` for a storage image on an Android device. Instead of leaving your IDE, you ask your assistant: *"Query our local Vulkan SDK's `vk.xml` for the valid usage flags of a storage image."* +**Example.** Say you need the valid `VkImageUsageFlags` for a storage image on Android. Instead of looking it up yourself, you ask the assistant to query your local SDK's `vk.xml` for it. Rather than guessing from training data, the assistant reads the actual XML file and returns the real bitmask definitions — correct even for extensions released yesterday. -The AI doesn't "guess" the flags from its training data. It uses an **MCP Tool** to read the actual XML file in your Vulkan SDK. It finds the exact bitmask definitions and provides an answer that is consistent with the specification. This is the difference between "Probabilistic Guessing" and "Ground Truth." +== RAG: connecting to your codebase -* **The Win:** By connecting the AI to the **Vulkan Registry**, you ensure your code is always syntactically correct, even for extensions that were released yesterday. +Where MCP handles structured spec data, RAG handles your own source, internal docs, and architectural conventions. -== Strategy 2: Retrieval-Augmented Generation (RAG) +**Example.** You're writing a new `Buffer` wrapper and want it to match your existing `Texture` class's style. With RAG set up, you ask the assistant to use `Texture` as a reference and generate a `Buffer` class following the same RAII and allocation patterns. Instead of a generic C{pp} class, the system retrieves your actual `Texture` implementation and generates something that matches your codebase's existing conventions, without retraining the underlying model. -While MCP is for "Structured Data" (like XML or documentation), **RAG** is for "Contextual Data"—your engine's own source code, internal wikis, and unique architectural patterns. +== How a RAG pipeline works -**The Experience: "Contextual Awareness"** -You are working on a new `Buffer` wrapper. You want it to match the coding style of your existing `Texture` class. You use a RAG-enabled tool. You ask: *"Using our `Texture` class as a reference, generate a new `Buffer` class that follows our custom RAII wrapping and memory allocation patterns."* +Understanding the pipeline is useful because it's how you turn a general-purpose model into something that knows your engine specifically, using only your local files. -Instead of giving you a generic C{pp} class, the system uses **RAG** to "Retrieve" the implementation of your `Texture` class from your local project. It then "Augments" its generation with your engine's specific "Voice." The resulting code looks exactly like something you would have written yourself. +=== Phase 1: curating source documents -* **The Win:** RAG ensures that the AI's suggestions are **Architecturally Consistent** with your specific codebase without needing to retrain the underlying model. +Identify the documents worth indexing for your engine: -== The Mechanics of Specialization: The RAG Pipeline +* **Header files** (`.h`, `.hpp`, etc.) — your internal API surface. +* **Engineering guidelines** — docs covering memory management or synchronization conventions. +* **External docs** — local copies of third-party library manuals (Slang, VMA, Volk, SDL3) or vendor optimization guides not already covered by MCP tools. -For a graphics developer, understanding the RAG pipeline is essential because it allows you to "specialize" a general-purpose model into a **Vulkan Expert** using only your local files. You are essentially providing the AI with a "Working Memory" that is far larger and more accurate than its original training data. +NB: There's a tradeoff between index breadth and retrieval speed — a wider index can be more accurate but slower to query, and it also costs tokens, leaving less budget for the actual work in a session. Keep the index focused rather than indexing everything. -To build a RAG-powered specialization for your engine, you must understand the four phases of the pipeline: +=== Phase 2: embedding and indexing -=== Phase 1: Knowledge Curation (The "Source Truth") -You must identify the "Ground Truth" documents for your engine. For a Vulkan project, this includes: +1. **Chunking:** files are split into smaller overlapping pieces (a function, a class definition). +2. **Embeddings:** a small embedding model (`bge-small-en`, `nomic-embed-text`) converts each chunk into a vector. +3. **Vector database:** vectors are stored so that conceptually similar chunks (e.g. `VkBuffer` and `VkImage` creation code) end up near each other. -* **Your Header Files (`.h, .hpp, etc`):** These define your internal API. -* **Engineering Guidelines (`GUIDELINES.md`):** Documents that explain your memory management or synchronization philosophy. -* **External Technical Documentation:** Local Markdown or HTML versions of third-party library manuals (e.g., Slang, VMA, Volk, or SDL3) and specialized GPU vendor optimization guides that aren't yet integrated into your MCP tools. +=== Phase 3: retrieval -NB: There is a trade-off between the breadth of your RAG index and the speed of retrieval. A wider index can provide more accurate answers but may slow down query times. Conversely, a narrower index can speed up queries but may sacrifice accuracy. Additionally, this can come at the cost of tokens which means your AI session would have less tokens available for the actual work session, so avoid doing too much in here. +A query is converted to a vector and matched against the nearest stored chunks. Asking about "pipeline barriers" retrieves the files in your engine that actually handle synchronization. -=== Phase 2: Embedding & Indexing (The "Mathematical Map") -Computers cannot "read" code; they understand math. +=== Phase 4: augmentation -1. **Chunking:** Your files are broken into small, overlapping pieces (e.g., a single function or a class definition). -2. **Embeddings:** A small, specialized model (an **Embedding Model** like `bge-small-en` or `nomic-embed-text`) turns these text chunks into high-dimensional mathematical vectors. -3. **The Vector Database:** These vectors are stored in a local database. Chunks that are "conceptually similar" (e.g., a `VkBuffer` creation and a `VkImage` creation) are stored "closer" to each other in this mathematical space. +The retrieved chunks get added to the prompt as context before the model generates a response: -=== Phase 3: Retrieval (Finding the Needle) -When you ask a question, the system turns your prompt into a vector and searches the database for the most relevant "neighbors." If you ask about "Pipeline Barriers," the system retrieves the specific files in your engine that handle synchronization. - -=== Phase 4: Augmentation & Generation (The Final Assembly) -The retrieved chunks are "stuffed" into the AI's prompt as hidden context. The final prompt sent to the LLM looks like this: [source,text] ---- Context: [Your engine's Buffer.cpp code] Context: [The VMA Reference for allocation] User Prompt: Create a new vertex buffer for our triangle. ---- -The model now has all the "Facts" it needs to generate a correct, specialized answer. -== Practical Implementation: Building Your Knowledge Index +== Building a knowledge index + +Many IDE plugins automate this, but it's worth setting up something more transparent using tools already covered in the environment section. + +=== Step 1: organize a docs directory -While many IDE plugins automate RAG, you can build a more powerful and transparent "Specialization Index" using the tools you've already set up in the environment section. This allows you to treat your AI as a custom-built expert tailored to your specific hardware and engine. +Index a specific, curated directory rather than your whole home folder: -=== Step 1: The "Knowledge Vault" Structure -Create a dedicated directory in your project for RAG. Do not just index your entire home folder; be precise. [source,text] ---- /vulkan_project @@ -89,11 +81,11 @@ Create a dedicated directory in your project for RAG. Do not just index your ent (Your engine code) ---- -=== Step 2: Semantic Retrieval with Goose -Instead of relying on a hidden plugin, you can use **Goose** to perform explicit RAG. When you need to implement a new feature, follow this multi-step retrieval workflow. +=== Step 2: manual retrieval with Goose -1. **Context Loading:** - Before asking for code, tell the agent to index your documents. +Instead of relying on a hidden plugin, you can do RAG explicitly with Goose. + +1. **Load context.** Before asking for code, tell the agent what to read: + [source,text] ---- @@ -102,8 +94,7 @@ Goose, read 'docs/naming_conventions.md' and a new Index Buffer class. ---- -2. **The Augmented Prompt:** - Ask for the implementation while explicitly referencing the retrieved files. +2. **Ask, referencing what you loaded:** + [source,text] ---- @@ -113,38 +104,33 @@ and 'IndexBuffer.cpp'. Ensure it uses 'vmaCreateBuffer' for allocation. ---- -**Why this works:** You are manually performing the "Retrieval" phase. Because you are using the **Model Context Protocol (MCP)** via Goose, the agent can "read" your files as ground truth, effectively augmenting its internal weights with your project's specific laws. +This works because you're doing the retrieval step yourself, and MCP (via Goose) lets the agent read your actual files as ground truth rather than guessing at your conventions. + +=== Step 3: vector indexing at scale -=== Step 3: Local Vector Indexing (Optional but Recommended) -If your project grows to thousands of files, manual retrieval becomes slow. At this stage, you should use a local **Vector Database** like **ChromaDB** or **Pinecone** (via MCP). This allows the AI to perform a "Semantic Search" across your entire engine in milliseconds, finding relevant code snippets even if you don't know the exact filename. +If your project grows to thousands of files, manual retrieval gets slow. At that point a local vector database (ChromaDB, Pinecone, via MCP) lets the AI search the whole engine in milliseconds even without knowing the exact filename. -=== Step 4: Knowledge Graphs (Structural Context) -While Vector Databases excel at finding "similar" text, a **Knowledge Graph** excels at finding "related" structures. In a Vulkan engine, this is critical for understanding the ownership hierarchy (e.g., which `VkDevice` created which `VkCommandPool`). +=== Step 4: knowledge graphs for structure -By indexing your project as a Knowledge Graph, the AI can traverse relationships: -* **Query:** "Show me all systems that depend on the `GlobalDescriptorSet`." -* **Result:** The AI identifies not just the classes that use the set, but also the shaders, pipeline layouts, and synchronization barriers that are logically linked to it. +Vector search is good at finding similar text; a knowledge graph is better at finding related structure — for instance, tracing which `VkCommandPool` was created by which `VkDevice`. Indexing your project as a graph lets you ask things like "show me all systems that depend on `GlobalDescriptorSet`" and get back not just the classes that use it, but the shaders and barriers logically linked to it. -=== Step 5: The Function Registry (Tool Capability) -A **Function Registry** acts as a "Phonebook" of capabilities that the AI can call upon. Instead of just writing code, the AI uses the registry to identify which existing engine tools or Vulkan functions are available to solve a task. +=== Step 5: a function registry -For example, if you task the AI with "Capturing a frame," the AI consults its Function Registry, identifies that your engine has a `RenderDoc::CaptureFrame()` method, and chooses to call that existing tool rather than trying to implement a capture system from scratch. +A function registry is a lookup of capabilities the AI can call on, rather than reimplement. If you ask it to capture a frame, it can check whether your engine already has a `RenderDoc::CaptureFrame()` method and use that instead of writing a new capture system from scratch. -**Advanced Tip: The "Cloud-to-Local" Specialization Pipeline** -To effectively "specialize" a 17B SLM without using a plugin, you can use a high-reasoning Cloud LLM (like **Claude 4.6**) to perform the **Phase 1: Knowledge Curation**. Ask the Cloud model to analyze your renderer and summarize the "Architectural Laws" into a single Markdown file. By providing this curated document to your local SLM via RAG, you bridge the reasoning gap, giving the smaller model the "wisdom" of the larger one without the VRAM cost. +**Tip: using a cloud model to prep the RAG source.** You can use a stronger cloud model to do the initial curation — summarizing your renderer's conventions into a single markdown file that a smaller local model can then retrieve from, getting some of the larger model's understanding without the VRAM cost. -=== The SLM Advantage: RAG as "External Memory" +=== Small models with a good RAG index can compete with far larger ones -One of the most powerful aspects of Retrieval-Augmented Generation is that it allows **Small Language Models (SLMs)**—typically models in the 7B to 30B parameter range—to outperform 400B+ parameter "Giant" models in the specific domain of your game engine. By offloading the "Memorization" of your C{pp} codebase to a RAG index, the SLM no longer needs to waste its internal parameters trying to remember your custom class names; it focuses 100% of its reasoning on the logic you provide. +A model in the 7B–30B range, given a good RAG index of your codebase, can outperform a much larger general-purpose model on tasks specific to your engine — because it isn't spending its limited attention trying to recall your class names from training; the index handles that, and the model can focus on reasoning about the logic in front of it. -== Summary: The Informed Specialist +== Summary -By combining the **Pre-trained Weights** of a base model with the **Global Truth** of MCP and the **Local Truth** of RAG, you transform your AI from a "Predictive Text Generator" into a **Graphics Engineering Assistant**. +Combining a base model's pre-trained knowledge with MCP's access to the current spec and RAG's access to your own codebase gives you an assistant that's aware of both the Vulkan spec and your specific engine. -* **MCP** provides the "Facts" (The Spec). -* **RAG** provides the "Flavor" (Your Engine). -* **Informed Context** ensures the model respects your local architectural laws. +* MCP handles the spec. +* RAG handles your project's conventions. -In the final chapter of this section, we will explore the ultimate form of specialization: **Fine-Tuning (LoRA)** to "bake" your engine's intuitive laws directly into the model's weights and how to deploy your resulting specialist to a shared team server. +The next chapter covers fine-tuning with LoRA — a further step for baking your engine's conventions directly into a model's weights, and deploying that to a shared team server. Next: xref:AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc[Fine-Tuning for Your Engine (LoRA)] diff --git a/en/AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc b/en/AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc index d8e9f6e7..c4c20039 100644 --- a/en/AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc +++ b/en/AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc @@ -1,54 +1,48 @@ :pp: {plus}{plus} -= Fine-Tuning for Your Engine: The "Holy Grail" of Style += Fine-Tuning for Your Engine (LoRA) -== Introduction: The "Digital Senior" +== The style problem -Imagine you have a new hire on your graphics team. They are a brilliant C{pp} programmer and they know the Vulkan spec by heart. However, on their first day, they start writing raw `vkCreateBuffer` calls and using `std::vector` for vertex data. Your engine, however, uses the **Vulkan Memory Allocator (VMA)** and a custom `FixedArray` type for performance. You have to spend the next week "correcting" their style to match your engine's architectural laws. +A new hire who's a strong C{pp} programmer and knows the Vulkan spec well will still, on their first day, write raw `vkCreateBuffer` calls and reach for `std::vector` when your engine actually uses the Vulkan Memory Allocator and a custom `FixedArray` type. You'd spend the first week correcting their style to match your conventions. -This is the same challenge we face with even the best base models like **Qwen 3-Coder**. They know Vulkan, but they don't know *your* Vulkan. To solve this, we use **Fine-Tuning**, specifically a technique called **Low-Rank Adaptation (LoRA)**. +Base models have the same issue: even a strong one like Qwen 3-Coder knows Vulkan in general but not your specific engine. Fine-tuning, specifically Low-Rank Adaptation (LoRA), is one way to close that gap. -== The Core Concept: The "Style Patch" +== How LoRA works, roughly -Traditional fine-tuning requires retraining all billions of parameters in a model—a process that costs thousands of dollars and requires massive GPU clusters. **LoRA** changes the game by training only a tiny "patch" of weights (typically less than 1% of the model's size) that sits on top of the base model. This setup combines the broad knowledge of the **Base Model**, providing general understanding of C{pp} and the Vulkan 1.3 spec, with the specialized voice of the **LoRA Adapter**, which injects your engine's specific naming conventions, abstractions, and "architectural laws." +Full fine-tuning retrains all of a model's parameters, which is expensive and needs real GPU clusters. LoRA instead trains a small adapter — typically under 1% of the model's size — that sits on top of the frozen base model. You get the base model's general Vulkan and C{pp} knowledge, plus an adapter carrying your engine's specific naming conventions and abstractions. -== The Teacher-Student Pipeline: Cloud-Powered Specialization +== Generating training data with a cloud model -To transform a general-purpose 17B or 12B model into a **Specialized SLM**, you need a high-quality "Instruction-Response" dataset. Manually writing thousands of these pairs is impractical for a small team. Instead, we use a high-reasoning Cloud LLM (like **Claude 4.6** or **GPT-5.3**) to "teach" the local model by synthesizing training data from your engine's source code. +Turning a general 12–17B model into something specialized to your engine needs a decent instruction/response dataset, and writing thousands of these by hand isn't practical for a small team. A common approach is to use a stronger cloud model (Claude 4.6, GPT-5.3) to generate that dataset from your own source code: -This is the **Teacher-Student** paradigm: +1. **Curate** your engine's core headers and architecture docs. +2. **Generate instructions.** Feed those files to a cloud model and ask it to produce instruction/response pairs testing understanding of your API. +3. **Fine-tune locally** using that generated dataset. -1. **Curation**: You extract your engine's core headers and architectural documentation. -2. **Instruction Synthesis (Cloud)**: You provide these files to a Cloud model and task it with generating technical 'Instruction/Response' pairs. Because the Cloud model has massive reasoning capabilities, it can "understand" your API and create complex questions that test that understanding. -3. **Local Retraining**: You use this "Cloud-generated Textbook" to fine-tune your local model. +Since the cloud model is working from your actual headers, the resulting fine-tuned local model tends to learn the logic of your engine's abstractions — not just isolated snippets — and can end up handling your specific synchronization primitives or allocators better than a much larger model that only has generic Vulkan knowledge. -**The "Specialized SLM" Advantage** -By using a Cloud model as a "Teacher," your local 17B model doesn't just memorize snippets; it learns the **logic** of your engine. It becomes a specialized tool that knows exactly how to use your custom synchronization primitives or memory allocators, often outperforming much larger models that only have general Vulkan knowledge. +== Why bother with LoRA specifically -== Why LoRA? (The "VMA" Problem) +Every sizable engine has its own conventions — one might use `vmaCreateBuffer`, another a custom suballocator built on Android's `AHardwareBuffer`, another the `vk::raii` C{pp} wrappers. Without a LoRA adapter, you end up repeatedly using RAG or manual prompting to remind the model of these preferences each session. -Every large-scale engine has its own "Vulkan Dialect." One engine might use `vmaCreateBuffer`, while another uses a custom sub-allocation scheme based on Android's `AHardwareBuffer`, and yet another relies on the `vk::raii` C{pp} wrappers. +== Training a LoRA adapter -Without LoRA, you have to constantly use **RAG** or manual prompts to remind the AI of these preferences. +A single high-end consumer GPU (RTX 3090/4090) is enough to train a LoRA adapter for a 17B model like Llama 4 in under an hour. -== Real-World Experience: Training the "Style Patch" +**Workflow:** -You do not need a supercomputer to train a LoRA. A single high-end GPU (like an RTX 3090 or 4090) can train a LoRA for a 17B model (like **Llama 4**) in less than an hour. +1. **Collect a dataset.** Extract header/implementation pairs from your repo — for instance, pairing `PipelineBuilder.hpp` with a `main.cpp` that uses it. +2. **Run training** with a tool like Unsloth. You'll see the loss drop as the model stops suggesting a generic `VkGraphicsPipelineCreateInfo` and starts suggesting your engine's own `GraphicsPipelineBuilder`. +3. **Use the result.** Load the resulting adapter (often under 50MB) into Ollama or LM Studio. When you start a new render pass, the model uses your engine's synchronization wrappers without extra prompting. -**The Workflow:** +== Step-by-step -1. **Dataset Collection:** You run a simple script that extracts "Header/Implementation" pairs from your engine's repository. For example, it pairs your `PipelineBuilder.hpp` with a complex `main.cpp` that uses it. -2. **The Training Loop:** You use a tool like **Unsloth** to "teach" the model. You will see the "Loss" curve drop as the model stops suggesting generic `VkGraphicsPipelineCreateInfo` and starts suggesting your engine's `GraphicsPipelineBuilder`. -3. **The Result:** You load this tiny (50MB) adapter into your IDE (via Ollama or LM Studio). Now, when you start typing a new render pass, the AI automatically uses your engine's specific synchronization wrappers without any extra prompting. +=== Step 1: curate the dataset -== Practical Tutorial: Training a LoRA Adapter +Use a strong cloud model to generate instruction/response pairs from your own headers. -To specialize a 17B SLM (Small Language Model) like **Llama 4** on your graphics engine, follow this pipeline to curate your dataset and execute the training loop. - -=== Step 1: Curate the Synthetic Dataset -You need a "Synthetic Textbook" of **Instruction-Response pairs**. Use a high-reasoning Cloud model (Claude 4.6/GPT-5.3) to generate this from your engine's source code. - -**The "Teacher" Prompt:** +**Example prompt:** [source,text] ---- I am uploading our core engine headers (Renderer.hpp, Buffer.hpp) @@ -59,16 +53,18 @@ pairs in JSONL format that teach a model how to: 3. Implement our custom 'ImageMemoryBarrier' logic for layout transitions. ---- -=== Step 2: Format for Training (JSONL) -Ensure your data follows the standard training format. Each line in your `train.jsonl` should look like this: +=== Step 2: format as JSONL + +Each line of `train.jsonl` should look like: [source,json] ---- {"instruction": "Create a vertex buffer using our VMA wrapper.", "input": "Target: 1024 bytes, Usage: VERTEX_BUFFER_BIT", "output": "Buffer::Builder(device).setSize(1024).setUsage(VMA_MEMORY_USAGE_GPU_ONLY).build();"} ---- -=== Step 3: The Training Loop with Unsloth -Use the **Unsloth** library on a Linux machine with at least 24GB of VRAM (e.g., RTX 3090/4090). +=== Step 3: train with Unsloth + +Run on a Linux machine with at least 24GB VRAM (RTX 3090/4090): [source,python] ---- @@ -81,7 +77,7 @@ model, tokenizer = FastLanguageModel.from_pretrained("unsloth/llama-4-8b-bnb-4bi # 2. Add LoRA adapters model = FastLanguageModel.get_peft_model( model, - r = 16, # Rank (higher = more 'memory' but larger file) + r = 16, # Rank (higher = more capacity but larger file) target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"], lora_alpha = 16, lora_dropout = 0, @@ -92,15 +88,19 @@ model = FastLanguageModel.get_peft_model( model.save_pretrained_gguf("vulkan_expert_lora", tokenizer, quantization_method = "q4_k_m") ---- -When you execute the training loop, you will see the "Loss" value decrease—mathematical proof that the model is learning your engine's API. This process takes roughly 30 to 60 minutes on an RTX 4090. +Watching the loss value decrease over the run is a reasonable check that the model is actually picking up your engine's API. On an RTX 4090 this typically takes 30–60 minutes. -== Specializing the SLM: From Generalist to Expert +== Combining LoRA and RAG -One of the most powerful aspects of specialized development is that it allows **Small Language Models (SLMs)**—typically models in the 17B to 30B parameter range—to outperform 400B+ parameter "Giant" models in the specific domain of your game engine. By combining the "Style Patch" of a LoRA with the "Ground Truth" of a RAG index, the SLM no longer needs to waste its internal parameters trying to remember general C{pp} patterns; it focuses 100% of its reasoning on the logic you provide in your engine's specific dialect. +A small model (roughly 17–30B) with both a LoRA adapter and a RAG index can hold its own against much larger general-purpose models on tasks specific to your engine. The LoRA adapter handles general C{pp} conventions in your dialect; the RAG index handles facts that change (docs, current task list), so the model isn't relying on its frozen weights for either. -In a team environment, you often want a single, high-performance SLM that "knows" your engine deeply—including the LoRA you just trained—without forcing every developer to sacrifice significant VRAM. To set up a shared inference server, start by configuring the host machine (one with substantial VRAM) with **Ollama**. You must explicitly expose the service by setting the `OLLAMA_HOST` environment variable to `0.0.0.0` before running `ollama serve`. Finally, prepare the specialized model by pulling a capable base such as **Qwen 3-Coder (30B)**. +== Setting up a shared inference server -Every developer on the team then configures their local **Goose** instance to use the remote server. The beauty of this setup is the **Local-Agent/Remote-Model** split: the Goose agent remains on the developer's machine with direct access to local files and tools, but it offloads the heavy reasoning "thinking" to the shared server. Because the OpenAI API has become the industry standard for AI communication, we use the `openai` provider in Goose to talk to our remote Ollama server. Open your Goose configuration—located at `~/.config/goose/config.yaml` on Unix-like systems or `%APPDATA%\goose\config.yaml` on Windows—and add a provider entry that treats the remote server as an endpoint, using the host machine's IP address. +In a team setting, you typically want one capable model — including the LoRA you trained — running centrally, rather than every developer paying the VRAM cost individually. + +On a host machine with enough VRAM, configure Ollama to listen on all interfaces by setting `OLLAMA_HOST` to `0.0.0.0` before running `ollama serve`, then pull a capable base model such as Qwen 3-Coder (30B). + +Each developer then points their local Goose instance at the remote server. The agent itself stays local — it can read local files directly — but offloads inference to the shared machine. Since the OpenAI API has become a common interface for this, Goose can talk to the remote Ollama server using the `openai` provider. Edit your Goose config (`~/.config/goose/config.yaml` on Unix, `%APPDATA%\goose\config.yaml` on Windows) to add an entry pointing at the host's IP: [source,yaml] ---- @@ -112,16 +112,13 @@ models: api_key: "not-needed" ---- -**Why this works:** When you give Goose a task, it first uses its local tools to "Read" your local files. It then packages those file snippets together and sends them over the network to the shared server. The server generates the response and sends it back to your machine, where the local Goose agent then writes the code to your disk. You get the power of a 30B model without using a single byte of your local GPU's VRAM. - -To ensure consistency across the team, you must standardize your RAG sources by placing technical manuals and engine specifications in a shared directory tracked by Git. Instruct Goose to always index this directory first to maintain a synchronized "Knowledge Vault." +In this setup, Goose reads your local files, sends the relevant snippets to the shared server for inference, and writes the response back to disk locally — so you get a 30B model's output without spending your own GPU's VRAM on it. -**Real-World Experience: The Team Audit** -In this shared configuration, if a developer encounters a new synchronization bug, they can say: *"Goose, query our shared `docs/sync_patterns.md` and the remote 30B model to audit my current pipeline barrier."* +To keep the team consistent, put technical manuals and specs in a shared, git-tracked directory and have Goose index that directory first in each session. -Because the RAG retrieval happens locally, the agent sees the developer's exact line of code. Because the inference happens on the shared server, the agent has the "Reasoning Depth" of a massive model without causing the developer's Vulkan frame rate to drop to zero. +**Example.** A developer hitting a new synchronization bug can ask: "Goose, query our shared `docs/sync_patterns.md` and the remote 30B model to audit my current pipeline barrier." Retrieval happens locally against the developer's actual code, while the heavier reasoning runs on the shared server — so the developer's own frame rate isn't affected. -Once you have modified your `config.yaml`, verify that Goose is correctly talking to the shared server by running a simple diagnostic. First, list the available models to ensure the 'shared-vulkan-expert' is visible, then start a session using that specific model. +After updating `config.yaml`, verify the connection: [source,bash] ---- @@ -132,18 +129,18 @@ goose models goose session --model shared-vulkan-expert ---- -If the connection is successful, you can now task Goose with a complex graphics problem. Because the model is running on the shared server, your local system's VRAM usage will remain stable, even as the AI reasons through a complex 100-line shader. +If that works, your local VRAM usage should stay flat even while the model reasons through a large shader. -== RAG vs. LoRA: When to Use Which? +== RAG vs. LoRA -Choosing between these two techniques depends on your goals. Use **RAG** for facts that change often, such as third-party library manuals, project-specific requirements, or your engine's current task list. Use **LoRA** for deep, structural elements—teaching the AI your engine's "Soul," including its naming conventions, memory management philosophy, and specific C{pp} abstractions. +Use RAG for things that change often — third-party docs, project requirements, current task lists. Use LoRA for things that are structural and stable — naming conventions, memory management philosophy, and engine-specific abstractions. -== Summary: The Integrated Engineer +== Summary -Fine-tuning via LoRA is the final step in creating a truly personalized AI assistant. It bridges the gap between a "generic" programmer and a **Senior Engineer on your specific team**. +Fine-tuning with LoRA is the deepest level of specialization covered in this section — it closes the gap between a generically competent programmer and one who already knows your team's conventions. -With the completion of Section 3, you have established a comprehensive specialized environment. You have selected a base model, calculated a sustainable VRAM budget, integrated ground-truth knowledge via RAG and MCP, injected your engine's unique style using LoRA, and deployed the result to a shared inference host for your team. +With this section done, you've selected a base model, worked out a VRAM budget, added ground-truth knowledge via RAG and MCP, adjusted the model's style with LoRA, and optionally deployed it to a shared server for your team. -Your assistant is now fully specialized. In the next section, we will give your assistant visual reasoning capabilities as we explore **Multimodal AI** and its ability to diagnose visual graphics bugs like shadow acne and Z-fighting. +Next, we look at multimodal AI — using vision-capable models to diagnose visual bugs like shadow acne and z-fighting. -Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc[Multimodal AI: Visual Reasoning Capabilities] +Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc[Multimodal AI] diff --git a/en/AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc b/en/AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc index abac6886..966b8df1 100644 --- a/en/AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc @@ -2,32 +2,28 @@ = Multimodal AI: Visual Reasoning Capabilities -== Introduction: The Visual Nature of Graphics +== Introduction -Vulkan programming is inherently visual. When something goes wrong in your engine, the first evidence isn't usually a crash or a log message; it's a flicker on the screen, a missing shadow, or a strange moiré pattern on a texture. +Vulkan bugs are usually visual before they're anything else. When something goes wrong in your engine, the first evidence isn't a crash or a log line — it's a flicker on screen, a missing shadow, or a moiré pattern on a texture. -Until recently, AI assistants were limited to text-based inputs. You had to describe your visual bugs in text: *"The shadows have jagged edges"* or *"The textures are flickering when I move the camera."* This created a massive **Communication Gap**. You knew what the bug looked like, but the AI could only reason about it based on your description. This section introduces **Multimodal AI**, which gives your assistant the ability to analyze your screenshots and diagnose visual errors directly. +Text-only assistants make you describe that: "the shadows have jagged edges," "the textures flicker when the camera moves." A lot gets lost in that translation. This section covers multimodal models, which can take a screenshot directly and reason about what's in it, instead of relying entirely on your description. -== The Core Concept: Pixels to State +== What a multimodal model actually does -A **Multimodal Model** (like **Qwen 3-VL** or **GPT-5.3**) has been trained on both images and text simultaneously. It doesn't just "see" an image as a collection of pixels; it understands the relationship between those pixels and the underlying concepts of graphics engineering. +A multimodal model (Qwen 3-VL and GPT-5.3 are current examples) is trained on both images and text, so it can connect pixels in a screenshot to graphics concepts rather than just describing what's visible. -Consider the case where you are implementing shadow maps for the first time and your shadows are covered in strange, self-shadowing stripes. A generic AI might suggest a shader error or a texture sampling problem if you describe it as "stripes on the floor." However, with a multimodal solution, you can upload a screenshot and the AI immediately identifies the pattern as **Shadow Acne**. It doesn't just give you a generic fix; it explains that your `depthBiasConstantFactor` is too low for your current shadow map resolution and suggests a specific range of values based on the "stair-stepping" it sees in the image. +Take shadow acne as an example: stripes of self-shadowing across a surface. If you describe that as "stripes on the floor," a text-only model might guess at a shader bug or a texture sampling issue. A multimodal model looking at the actual screenshot has a better shot at naming the pattern correctly and pointing at `depthBiasConstantFactor` being too low for your shadow map resolution — though you should still treat that as a starting hypothesis to check, not a confirmed diagnosis. -== Why Multimodal Matters for Vulkan +== Why this is useful for Vulkan specifically -Vulkan's complexity means that a single visual bug can be caused by dozens of different state mismatches. Flickering could be a synchronization hazard, a depth-testing error, or a swapchain image acquisition race. Similarly, missing meshes could be a back-face culling issue, an incorrect vertex buffer offset, or a shader stage mismatch. +A single visual symptom in Vulkan can have several unrelated causes. Flickering might be a synchronization hazard, a depth-testing error, or a swapchain acquisition race. A missing mesh might be back-face culling, a wrong vertex buffer offset, or a shader stage mismatch. A multimodal assistant can narrow this down — for example, recognizing that triangles popping in and out looks like Z-fighting, and suggesting you check your near/far plane ratio against your depth buffer precision. It's a way to generate plausible candidates faster; you still verify against the validation layers and RenderDoc. -A multimodal assistant acts as a **Visual Analysis Tool**. It analyzes the artifact and identifies potential causes, such as suggesting that triangles popping in and out might indicate Z-fighting. It could then recommend checking your near/far plane ratios in your projection matrix if they look too aggressive for a 24-bit depth buffer. +== What's in this section -== The Multimodal Strategy +1. **xref:AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc[Vision Models for Graphics]:** The current vision-capable models (Qwen 3-VL, Gemma 4) and how they differ from the code-focused models covered earlier. +2. **xref:AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc[Visual Bug Diagnosis]:** Walking through screenshots of common Vulkan artifacts — shadow acne, texture bleeding, Z-fighting — and how to prompt for a useful diagnosis. +3. **xref:AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc[Expectations & Limits]:** Where this breaks down — resolution limits, hardware-specific behavior, and why the model still needs your code as context. -In the following chapters, we will explore how to integrate visual reasoning into your development loop: +By the end, you should be able to use a vision-capable model as one more diagnostic input, alongside the validation layers and RenderDoc, not as a replacement for either. -1. **xref:AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc[The Vision Specialist Models]:** We will introduce the models designed for visual reasoning, such as **Qwen 3-VL** and **Gemma 4**. We'll discuss how these models differ from the coding-only models we explored in the previous section. -2. **xref:AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc[Visual Bug Diagnosis (The Experience)]:** This is the heart of the section. We will show you how to provide screenshots of common Vulkan errors—like **Shadow Acne**, **Texture Bleeding**, and **Z-fighting**—and how to prompt the AI to identify the underlying Vulkan state mismatch. -3. **xref:AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc[Expectations & Limits]:** AI is not an automated "fix-it" tool for hardware. We will discuss the diagnostic role, the limits of sub-pixel resolution, and why the AI still needs your code context to verify its visual analysis. - -By the end of this section, you will be able to treat your AI assistant as a diagnostic tool, connecting what you see on the screen with the code that generated it. - -Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc[The Vision Specialist Models] +Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc[Vision Models for Graphics] diff --git a/en/AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc b/en/AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc index 02c00ebe..373a091f 100644 --- a/en/AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc +++ b/en/AI_Assisted_Vulkan/04_multimodal_ai/02_multimodal_models.adoc @@ -2,60 +2,57 @@ = Understanding Multimodal Models: Vision for Graphics -== Introduction: The "Vision-Language" Paradigm +== Introduction -In the previous sections, we focused on models like **Qwen 3-Coder**, which are "blind" but possess deep knowledge of C{pp} and the Vulkan spec. While they are masters of syntax, they are fundamentally limited when it comes to visual debugging. They can "read" your code, but they cannot "see" the result. +The models covered in earlier sections, like Qwen 3-Coder, are text-only: strong on C{pp} and the Vulkan spec, but they can't look at a screenshot. To debug visually, you need a vision-language model (VLM) — a model that pairs a vision encoder with a language model so it can reason about an image, not just recognize objects in it. -To bridge this gap, we use **Vision-Language Models (VLMs)**. These are not just "chatbots that can see"; they are specialized AI architectures that integrate a **Vision Encoder** with a standard Large Language Model to perform high-level **Geometric Reasoning**. +== How it works -== The Core Concept: The Vision Encoder +A VLM has two main parts. A vision encoder (often a Vision Transformer) processes the image and extracts features — shapes, depth cues, lighting patterns. That gets passed to the language model, which maps those features onto graphics concepts like "moiré pattern" or "Z-fighting." -A Multimodal model consists of two primary components working in tandem. The **Vision Encoder** is a separate neural network (often a Vision Transformer) that "scans" an image to identify high-level features such as shapes, depth cues, patterns, and lighting anomalies. This data is then passed to the **Language Model**, which acts as the reasoning engine to translate those visual features into technical graphics concepts like "Moiré patterns" or "Z-fighting." +This is a step beyond OCR, which can read text on screen but has no model of the relationships between pixels. A VLM can look at a flickering triangle and connect it to a likely cause, such as a depth-testing mismatch or precision loss in the projection matrix — again, a hypothesis worth checking, not a verdict. -**Why This Matters for Vulkan:** -Traditional OCR (Optical Character Recognition) can read the text on your screen, but it cannot understand the **Relationships** between pixels. A Multimodal model, however, can look at a flickering triangle and understand that the issue is likely a **Depth-Testing** mismatch or a **Precision Error** in the projection matrix. +== Evaluating a VLM for graphics work -== The Concept: How to Evaluate Vision-Language Models +A few things matter more than general image-recognition benchmarks: -To evaluate a Vision-Language Model (VLM) for graphics development, you must look beyond simple image recognition. You are looking for a model capable of **Spatial and Geometric Reasoning**. +**Resolution and detail retention.** Graphics bugs often live in fine detail — a single row of Z-fighting pixels, a subtle moiré pattern. If a model downsamples your 4K screenshot to something like 224x224 before processing, it will miss these. Look for models that support high-resolution or patch-based input. -The first criterion is **Resolution and Detail Retention**. Graphics bugs often hide in the fine details—a single pixel of "Z-fighting" or a subtle moiré pattern. If a model "downsamples" your 4K screenshot to 224x224 before processing it, it will be blind to these artifacts. Look for models that support **High-Resolution Input** or "patch-based" processing. +**Domain vocabulary.** A general-purpose VLM might describe "lines" on a model; a graphics-aware one distinguishes aliasing, shadow acne, and texture bleeding. A quick test is to show it a clear example of one of these and see whether it names it correctly. -The second criterion is **Domain-Specific Vocabulary**. A general VLM might see "lines" on a 3D model, but a graphics-specialized VLM will understand the difference between **Aliasing**, **Shadow Acne**, and **Texture Bleeding**. To test this, provide the model with a clear example of a common artifact and ask it to describe the "visual symptoms" using technical terms. +**Spatial reasoning.** Ask it something like "is the shadow correctly aligned with the light source and the occluder?" A model that can identify a break in the shadow's projection is more useful for debugging than one that just reports "a dark spot." -Finally, evaluate the model's **Spatial Reasoning**. Ask it to describe the relationship between two objects in a scene: *"Is the shadow correctly aligned with the light source and the occluder?"* A model that can correctly identify the "break" in a shadow's projection is significantly more valuable for debugging than one that simply sees a "dark spot." +== A few current vision models -== Leading 2026 Vision Models (Examples) +=== Qwen 3-VL -=== 1. Qwen 3-VL: High-Resolution Visual Analysis -Alibaba's **Qwen 3-VL** is currently one of the most capable models for technical image analysis, particularly for identifying complex graphics artifacts like shadow acne or reading text from UI overlays in tools like RenderDoc. For example, if you are debugging a subtle **Banding** artifact in your skybox and upload a 4K screenshot, Qwen 3-VL can identify the pattern and suggest that your `VkImageFormat` (such as B8G8R8A8_UNORM) lacks the precision for a smooth gradient, recommending a switch to a 10-bit or 16-bit float format instead. +Alibaba's Qwen 3-VL handles high-resolution technical images reasonably well — identifying artifacts like shadow acne, or reading UI text from a RenderDoc capture. For example, given a 4K screenshot showing banding in a skybox, it can flag that an 8-bit format like `B8G8R8A8_UNORM` doesn't have enough precision for a smooth gradient and suggest a 10-bit or 16-bit float format instead. -=== 2. Gemma 4: High-Speed Visual Feedback -Google's **Gemma 4** offers a smaller, faster alternative that is ideal for real-time visual feedback and rapid classification of bugs—such as determining if a black screen is due to a culling error or a missing swapchain image. It is highly optimized for performance on Apple Silicon and modern NVIDIA cards, providing fast categorization of visual errors. +=== Gemma 4 -== Choosing Your Vision Strategy +Google's Gemma 4 is smaller and faster, which makes it a reasonable choice for quick, local classification — telling apart a culling error from a missing swapchain image, for instance. It runs well on Apple Silicon and recent NVIDIA GPUs. + +== Picking a model [cols="1,1,2", options="header"] |=== -| Strategy | Recommended Model | Best Use Case +| Priority | Model | Use case -| **Local Inference (Privacy)** +| Local, private | Gemma 4 -| Quick bug classification and visual-focused coding. +| Quick bug classification, visual-adjacent coding. -| **High-Precision Analysis** +| Detail-sensitive analysis | Qwen 3-VL -| Deep diagnosis of subtle artifacts like aliasing or banding. +| Diagnosing subtle artifacts like aliasing or banding. -| **High-Reasoning Cloud Model** +| Complex planning | Claude 4.6 / GPT-5.3 -| Planning complex visual systems from sketches or whiteboards. +| Working from sketches or whiteboard photos. |=== -== Summary: The Visual Bridge - -By adding a Multimodal model to your toolbelt, you provide your assistant with the same visual context that you have: the pixels on the screen. This allows the AI to analyze the results of your Vulkan code and provide feedback based on what it actually sees. +== Summary -In the next chapter, we will put these models to work as we explore **Visual Bug Diagnosis** and the most common graphics artifacts in Vulkan. +Adding a multimodal model to your toolset gives your assistant access to the same pixels you're looking at, so its suggestions can be grounded in what actually rendered rather than only in your description of it. Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc[Visual Bug Diagnosis] diff --git a/en/AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc b/en/AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc index 6ae80991..a46b7e5d 100644 --- a/en/AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc +++ b/en/AI_Assisted_Vulkan/04_multimodal_ai/03_visual_bug_diagnosis.adoc @@ -2,49 +2,53 @@ = Visual Bug Diagnosis: Identifying Graphics Artifacts -== Introduction: The Artifact Hunt +== Introduction -In the life of a Vulkan developer, the "Black Screen" is only the first hurdle. Once you finally have images on the screen, the real challenge begins: **The Artifact Hunt.** Visual bugs like flickering, jagged shadows, and texture anomalies are the hallmarks of subtle state errors or synchronization hazards. +Getting past a black screen is only the first hurdle. Once you have images rendering, the next round of problems shows up as flickering, jagged shadows, or texture anomalies — usually the visible signature of a state error or synchronization hazard somewhere in the pipeline. -To truly master **Collaborative Engineering**, we use Multimodal AI to identify these artifacts and translate them back into the Vulkan code that caused them. +This section covers using a multimodal model to identify those artifacts from a screenshot and connect them back to the Vulkan state that's causing them. -== The Core Concept: Pixels as Symptoms +== Common patterns worth recognizing -In **Collaborative Engineering**, we treat pixels as symptoms. Just as a doctor uses an X-ray to diagnose a fracture, a graphics developer uses a screenshot to diagnose a state mismatch. Most Vulkan bugs follow predictable visual patterns that an AI can be trained to recognize. +Most Vulkan visual bugs follow recognizable patterns, whether you're doing the pattern-matching yourself or asking a model to do it. -=== 1. Z-Fighting (Depth Fighting) +=== Z-fighting (depth fighting) -**The Visual Symptom:** You see two overlapping surfaces "flickering" or showing a jagged, alternating pattern between them as the camera moves. -**The Real-World Experience:** You are building a large outdoor scene. Your floor tiles and your ground plane are perfectly aligned, but as you move away, they begin to flicker and "pop." -**The AI Diagnosis:** You upload a screenshot to your Multimodal AI. It identifies the pattern as **Z-fighting**. It doesn't just suggest "checking your code"; it explains that your **Near Plane** value (`0.0001f`) is likely too small for your **Far Plane** (`10,000f`), causing a loss of depth precision in a 24-bit depth buffer. It recommends a near plane of `0.1f` to regain stability. +**Symptom:** two overlapping surfaces flicker or alternate as the camera moves. -=== 2. Shadow Acne (Self-Shadowing) +**Typical scenario:** floor tiles and a ground plane are aligned correctly up close, but start flickering as the camera moves away. -**The Visual Symptom:** Strange, dark, striped or "moiré" patterns on the surface of an object that is supposed to be lit. -**The Real-World Experience:** You've just implemented your first **Directional Light**. The floor looks like it's covered in barcodes. -**The AI Diagnosis:** The AI, seeing the "stripes," identifies the issue as **Shadow Acne**. It suggests that your `depthBiasConstantFactor` or `slopeFactor` in your `VkPipelineRasterizationStateCreateInfo` is too low for your shadow map's current resolution. It explains the relationship between the light's angle and the bias required to prevent the shadow from "sampling itself." +**Likely cause:** a near plane set too small relative to the far plane (say, `0.0001f` near vs `10000.0f` far) loses depth precision in a 24-bit depth buffer. Moving the near plane to something like `0.1f` usually restores stability. A model shown the screenshot can suggest this, but you should confirm the actual near/far values in your code before changing them. -== The Diagnostic Prompting Template: Pixel-to-Code Correlation +=== Shadow acne (self-shadowing) -To get a high-quality diagnosis from a Multimodal AI, you must provide a combination of **Visual Evidence** and **Systemic Context**. +**Symptom:** dark, striped, or moiré-like patterns on a lit surface. -Start by capturing a high-resolution **Screenshot** of the artifact. This is the "Symptom" the AI will analyze. Next, provide the **System Context**—tell the assistant exactly what kind of renderer you are using (e.g., Clustered Forward, Deferred) and what GPU you are targeting. Finally, ask a **Specific Question** that focuses the AI's analysis on the problem area. +**Typical scenario:** the first directional light is in and the floor looks like it's covered in barcodes. -For example, you might say: *"Analyze the jagged, barcode-like shadows on the ground in this 4K screenshot. I am using a Cascaded Shadow Map system with Vulkan 1.3. Identify the likely artifact and suggest which `VkPipelineRasterizationStateCreateInfo` settings I should adjust to mitigate this 'Shadow Acne' without losing depth detail."* +**Likely cause:** `depthBiasConstantFactor` or `slopeFactor` in `VkPipelineRasterizationStateCreateInfo` is too low for the shadow map's resolution, given the light's angle relative to the surface. -By providing this triple-threat of information (Image, Context, and Goal), you ensure that the AI's response is grounded in the actual engineering requirements of your Vulkan engine, rather than just a generic guess based on a low-resolution thumbnail. +== Getting a useful diagnosis from a model -== Practical Tutorial: The AI-Assisted Debugging Workflow +A screenshot alone usually isn't enough context. Three things help: -To get a high-quality diagnosis from a Multimodal AI, you must provide a combination of **Visual Evidence** and **Systemic Context**. This turns the AI from a "guessing game" into a precise diagnostic tool. +1. A high-resolution screenshot of the artifact itself. +2. The relevant system context — renderer type (clustered forward, deferred, etc.), target GPU, depth buffer format. +3. A specific question, rather than "what's wrong with this." -Follow this step-by-step process for identifying any visual artifact: +For example: -* **Capture High-Resolution Evidence:** - Take a 4K screenshot of the artifact. If the bug is a flickering or motion-based issue, take a short screen recording (MP4) if your AI supports video, or a series of frames to show the "temporal" nature of the bug. +____ +Analyze the jagged, barcode-like shadows on the ground in this 4K screenshot. I'm using a cascaded shadow map system with Vulkan 1.3. Identify the likely artifact and suggest which `VkPipelineRasterizationStateCreateInfo` settings to adjust to reduce this without losing depth detail. +____ -* **Define the "Hardware Baseline":** - Provide the AI with your GPU's specs and the Vulkan features you are using. +Giving the model the image, the context, and a specific goal produces a more grounded answer than a vague description and a low-resolution thumbnail would. + +== A step-by-step workflow + +* **Capture the evidence.** Take a 4K screenshot of the artifact. If it's a flickering or motion-based bug, capture a short recording or a sequence of frames so the model can see how it changes over time. + +* **State your hardware baseline.** [source,text] ---- @@ -52,24 +56,23 @@ I am running on an NVIDIA RTX 4070 with Vulkan 1.3. I am using a 24-bit D24_UNORM_S8_UINT depth buffer format. ---- -* **The "Symptom to Spec" Prompt:** - Use a structured prompt that describes the visual symptom and asks for a spec-level diagnosis. +* **Describe the symptom and ask for a spec-level diagnosis.** + [source,text] ---- Reference the screenshot file: xxx.png -Analyze the 'jagged' alternating pixels between the two overlapping planes +Analyze the jagged, alternating pixels between the two overlapping planes in this image. I am currently using a 0.01 to 10000.0 depth range. Diagnose the artifact and suggest the specific Vulkan depth bias or projection matrix adjustments needed to resolve it. ---- -* **Execute the "Verify-Fix" Loop:** - The AI suggests a fix (e.g., *"Use a logarithmic depth buffer"*). You then ask it to generate the GLSL code for the vertex shader and the C{pp} code to enable the `VK_EXT_depth_clip_enable` feature if needed. +* **Verify and implement.** If the model suggests something like switching to a logarithmic depth buffer, ask it to generate the GLSL for the vertex shader and the C{pp} to enable `VK_EXT_depth_clip_enable` if needed — then test it against the actual bug before assuming it's fixed. -== Summary: From Pixels to Code +== Summary -By training your AI to recognize these patterns, you reduce the "Guesswork" phase of debugging. The AI points to the most likely cause, and you—the human engineer—verify the code. This is the heart of the **Collaborative Loop**: the AI identifies a potential problem, and you verify and fix the root cause. +A model that can read screenshots narrows down candidate causes faster than working from a text description alone, but it's still guessing from pixels. You confirm the actual cause and verify the fix. -However, as we will see in the next chapter, it is critical to understand the **Limits** of this visual reasoning. The AI can suggest the "Why," but it still lacks the "Ground Truth" of your actual GPU hardware unless you provide it through the **Context Bridge**. +The next chapter covers where this approach falls short — resolution limits, hardware differences, and cases where the model needs more than a screenshot to be useful. Next: xref:AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc[Expectations & Limits] diff --git a/en/AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc b/en/AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc index 3a6c05a3..4d35b383 100644 --- a/en/AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc +++ b/en/AI_Assisted_Vulkan/04_multimodal_ai/04_expectations_limits.adoc @@ -2,32 +2,28 @@ = Setting Realistic Expectations: The Limits of Vision -== Introduction: Visual Analysis Limits +== Introduction -By now, you have built an assistant that can "see." It can identify shadow acne, Z-fighting, and banding with a high level of speed. However, it is easy to fall into the trap of over-reliance—expecting the AI to automatically "fix" every pixel anomaly. +By this point you have an assistant that can identify shadow acne, Z-fighting, and banding reasonably quickly. It's worth being explicit about what it can't do: it's a diagnostic aid, not a compiler, and it doesn't see your actual GPU state — only the pixels you gave it. -In **Collaborative Engineering**, we must remember that the AI is a **Diagnostic Assistant, not a Compiler.** It sees the outcome of your code, but it doesn't see the underlying GPU state. Understanding these limits is what separates a novice user from an expert engineer. +== Pixels are not state -== The Core Concept: Pixels are not State +A multimodal model only sees a 2D array of pixels from a screenshot. It has no access to your actual Vulkan objects unless you give it that context directly. -The most fundamental limit of Multimodal AI is that it only sees a 2D array of pixels from your screenshot. It does not have access to your actual Vulkan objects unless you provide that context through the **Context Bridge**. +For example, a flickering screen might get diagnosed as Z-fighting when the real cause is a synchronization hazard — a race condition where you're reading from an image before a previous write finishes. From pixels alone, depth-fighting and a sync-related flicker can look nearly identical. Treat a visual diagnosis as a hypothesis to check against the validation layers and RenderDoc, not a confirmed answer. -For example, imagine you show the AI a flickering screen. The AI sees the pattern and suggests **Z-fighting**. You might spend an hour adjusting your near plane only to find the flickering remains. In reality, the cause was a **Synchronization Hazard**—a race condition where you were reading from an image before the previous write had finished. To an AI that only sees pixels, a depth-fighting artifact and a synchronization flicker can look nearly identical. This is why you should always treat an AI's visual diagnosis as a **Hypothesis** to be verified against the **Validation Layers** and **RenderDoc**. +== Resolution limits -== The "Resolution Gap" +Most multimodal models downscale images before processing — a 4K screenshot might get compressed well before the vision encoder ever sees it. That can erase exactly the high-frequency detail you're trying to diagnose: sub-pixel aliasing, micro-flickering, thin texture seams. If a model says an image looks fine despite a visible shimmering artifact, try cropping tightly to the affected area first — that gives the artifact more of the model's effective resolution to work with. -Most Multimodal models downscale images during processing, which can lead to a "Resolution Gap." A 4K screenshot of your scene might be compressed to a much smaller size before the Vision Encoder scans it. This compression can cause subtle, high-frequency artifacts like **Sub-pixel Aliasing**, **Micro-flickering**, or **Texture Seams** to be completely lost. If you send a full-screen screenshot to an AI and it claims the image is clear despite a visible shimmering issue, you should use a **Crop and Zoom** mitigation. By cropping the image specifically to the area of the artifact, you ensure the relevant pixels occupy the majority of the AI's limited vision buffer, preserving the detail needed for a correct diagnosis. +== Hardware differences -== The "Hardware Gap" +Vulkan is explicit, and the same bug can look different across vendors — an artifact on NVIDIA might present differently on AMD or on a mobile Adreno GPU. A model has no way to know which you're targeting unless you tell it. If you're working on mobile, say so directly: "I'm debugging on a tile-based deferred renderer (TBDR) architecture." That one line changes reasonable next steps from "check your barriers" to "check your subpass dependencies and load/store ops." -Vulkan is an explicit API that behaves differently on different hardware. A visual bug that appears on an NVIDIA card might look completely different on an AMD card or a mobile Adreno GPU. +== Summary -Unless you explicitly provide the **Hardware Context**, the AI will assume a "Generic Desktop GPU." If you are developing for mobile, you must include this in your prompt: *"I am debugging on a Tile-Based Deferred Renderer (TBDR) architecture."* This small piece of context changes the AI's reasoning from "Check your barriers" to "Check your subpass dependencies and load/store ops." +A multimodal model is useful for quickly narrowing down which of the usual suspects you're looking at. It's not a substitute for verifying synchronization, memory safety, or driver-specific behavior yourself. -== Summary: The Informed Graphics Engineer - -Multimodal AI is a powerful tool for the rapid identification of graphics artifacts, but it is not a replacement for your engineering judgment. Use it for **Rapid Pattern Recognition** and identifying "the usual suspects," but avoid relying on it for the final confirmation of synchronization, memory safety, or hardware-specific driver anomalies. - -With the completion of Section 4, you have integrated visual reasoning into your AI toolbelt. In the next section, we will bring everything together into a **Complete Workflow**, moving from high-level architectural planning to the line-by-line implementation of your Vulkan features. +That closes out this section on visual reasoning. Next, we bring this together with the rest of the toolset into a full workflow, from planning through implementation. Next: xref:AI_Assisted_Vulkan/05_workflow/01_introduction.adoc[Workflow: Architecture to Implementation] diff --git a/en/AI_Assisted_Vulkan/05_workflow/01_introduction.adoc b/en/AI_Assisted_Vulkan/05_workflow/01_introduction.adoc index e396c236..d138ded9 100644 --- a/en/AI_Assisted_Vulkan/05_workflow/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/05_workflow/01_introduction.adoc @@ -2,34 +2,34 @@ = Workflow: System Design to Implementation -== Introduction: The Collaborative Engineering Loop +== Introduction -Up to this point, we have built our toolbelt, selected our models, and established our visual reasoning capabilities. Now, it is time to put these components into motion. A truly efficient AI-assisted Vulkan workflow is not a single step; it is a **Three-Phase Cycle** that mirrors the traditional graphics development process but at a significantly accelerated pace. +So far we've set up a toolbelt, picked models, and covered multimodal debugging. Now it's time to put these pieces together into an actual workflow. A useful AI-assisted Vulkan workflow isn't a single step — it maps roughly onto three phases of traditional graphics development, just faster: design, implementation, and refactoring. -In **Collaborative Engineering**, we don't just "ask for code." We follow a structured loop that moves from high-level architectural intent to line-by-line implementation and finally to automated evolution. +Rather than treating each request to the model as a one-off, it helps to follow a loop that moves from high-level architectural intent down to line-by-line implementation, and finally to longer-running cleanup and migration work. -== The Core Concept: Separation of Concerns +== Matching the model to the phase -The secret to preventing AI from "going off the rails" is to match the model's reasoning depth to the specific phase of your development. +The main failure mode with AI-assisted development is using the wrong kind of model for the wrong kind of work — asking a fast local model to make an architectural call it doesn't have the context for, or burning a slow, expensive reasoning model on boilerplate it doesn't need. Matching the model to the phase avoids most of that. -1. **Phase 1: System Design (The Design Phase):** We use high-reasoning models (like **Claude 4.6** or **GPT-5.3**) to design the core architecture of the system. In this phase, we care about data flow, frame graphs, and memory safety, not specific bitmasks. -2. **Phase 2: Implementation & The Inner Loop (The Implementation Phase):** We use specialized coding models (like **Qwen 3-Coder**) to handle the "Syntactic Noise." This is where we generate the verbose `VkPipeline` structures and write the Slang/HLSL shaders. -3. **Phase 3: Automated Evolution (The Review Phase):** We use autonomous agents (like **Goose**) to migrate legacy code to modern patterns. The AI acts as a verification tool, ensuring that our new feature doesn't break existing synchronization or violate Vulkan 1.3/1.4 best practices. +1. **Phase 1: System design.** Use a high-reasoning model (such as Claude or GPT-class models) to work out the architecture: data flow, frame graph structure, memory ownership. At this stage the specific bitmasks and struct fields don't matter yet. +2. **Phase 2: Implementation.** Use a fast, code-specialized model (such as Qwen 3-Coder) to generate the verbose parts — `VkPipeline` structures, descriptor set boilerplate, and Slang/HLSL shader code. +3. **Phase 3: Refactoring and review.** Use an agent (such as Goose) for larger, multi-file migrations — moving legacy code to newer patterns, and checking that a new feature doesn't quietly break existing synchronization or violate Vulkan 1.3/1.4 best practices. -== Why This Workflow for Vulkan? +== Why this matters for Vulkan -Vulkan's **Design-to-Code** ratio is uniquely challenging. A simple architectural decision—like *"Let's move to a Bindless Rendering system"*—requires changes across your entire engine: descriptor management, shader inputs, resource allocation, and synchronization barriers. +Vulkan has an unusually high ratio of code to architectural decision. A single decision like "move to bindless rendering" touches descriptor management, shader inputs, resource allocation, and synchronization barriers across the whole engine. -By following this structured workflow, you ensure that the AI remains grounded in your architectural intent (Phase 1) while it handles the massive amount of boilerplate required for implementation (Phase 2) and refactoring (Phase 3). This prevents the "Code Rot" that often occurs when using AI to generate disconnected snippets of code. +Following a structured design → implementation → refactor loop keeps the AI's output grounded in what you actually decided in Phase 1, rather than producing disconnected snippets that drift out of sync with each other as the feature grows. -== The Workflow Roadmap: A Practical Example +== Walkthrough: a Bloom post-process pipeline -Throughout this section, we will walk through a concrete, real-world task: **Implementing a modern Bloom Post-Process Pipeline.** +The rest of this section works through one concrete example — a Bloom post-process pipeline — across all three phases: -1. **xref:AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc[Phase 1: System Design]:** You will learn how to prompt a model to design a technical roadmap. We'll discuss how to define the "Resource Lifecycle" and the specific layout transitions required for a downsample/upsample chain. -2. **xref:AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc[Phase 2: The Implementation Loop]:** We will move to your local specialist. You will learn how to use your technical roadmap as context to generate the necessary pipelines and compute shaders with near-zero manual typing. -3. **xref:AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc[Phase 3: Automated Evolution]:** Finally, we will use an AI agent to verify our implementation—checking for synchronization hazards and ensuring our memory usage matches the hardware limits we defined in the **Context Bridge**. +1. **xref:AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc[Phase 1: System Design]:** Prompting a model to produce a technical plan, including the resource lifecycle and layout transitions for a downsample/upsample chain. +2. **xref:AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc[Phase 2: The Implementation Loop]:** Using a local model, with the plan from Phase 1 as context, to generate the pipelines and compute shaders. +3. **xref:AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc[Phase 3: Refactoring]:** Using an agent to check the implementation for synchronization hazards and confirm memory usage matches the hardware limits established earlier via the Context Bridge. -By the end of this journey, you will have a repeatable "Blueprint" for building complex graphics features using AI as a deeply informed extension of your own engineering expertise. +By the end, you should have a repeatable process for building graphics features with AI involved at each stage, without losing track of the overall design. Next: xref:AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc[Phase 1: System Design] diff --git a/en/AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc b/en/AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc index 402a3fc8..c240e485 100644 --- a/en/AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc +++ b/en/AI_Assisted_Vulkan/05_workflow/02_phase_1_planning.adoc @@ -1,54 +1,53 @@ :pp: {plus}{plus} -= Phase 1: System Design (The Design Phase) += Phase 1: System Design -== Introduction: The skipped Step +== Introduction: the step people skip -In traditional Vulkan development, the "Planning" phase is often the most skipped step. Developers tend to jump straight into writing the `VkGraphicsPipelineCreateInfo` boilerplate, only to realize halfway through that their descriptor management won't support the new Bindless texture system they want to implement. +Planning is the phase developers most often skip. It's tempting to jump straight into `VkGraphicsPipelineCreateInfo` boilerplate, only to realize halfway through that the descriptor management doesn't support the bindless texture system you actually wanted. -In **Collaborative Engineering**, planning is not optional—it is the **Foundation**. By using a high-reasoning model to design your feature *before* you write any code, you ensure that the implementation phase is grounded in a consistent, spec-accurate roadmap. +Using a high-reasoning model to design the feature before writing any code helps here — not because planning is a novel idea, but because a model is good at surfacing the parts of a design you'd otherwise only discover through a rewrite. The goal is a roadmap that's specific enough to implement against. -== The Core Concept: The "Contract" +== What a good plan covers -In **Collaborative Engineering**, we treat the planning phase as a **Technical Contract**. This is a detailed roadmap that defines the "Laws" of the system before any code is written. A successful roadmap for a Vulkan feature must address several critical pillars. +A useful plan for a Vulkan feature needs to cover a few things clearly: -The first pillar is the **Resource Lifecycle**. You must define which buffers, images, and samplers are required, who "owns" them, and exactly when they are created and destroyed. Without this, your AI-assisted implementation will likely suffer from memory leaks or use-after-free errors. +**Resource lifecycle.** Which buffers, images, and samplers does the feature need, who owns them, and when are they created and destroyed? Skipping this is the most common source of memory leaks and use-after-free bugs in AI-generated code, because the model has no way to infer ownership you never wrote down. -The second pillar is the **Synchronization Strategy**. This is where Vulkan development often fails. Your roadmap should explicitly state the required `VkImageMemoryBarrier2` transitions for every stage of the feature. By defining these transitions in the planning phase, you provide the AI with the "Syntactic Truth" it needs to generate correct synchronization code later. +**Synchronization strategy.** State the required `VkImageMemoryBarrier2` transitions for each stage of the feature explicitly. This is the part of Vulkan development most likely to go wrong, and writing it down during planning gives the model something concrete to implement against later, instead of guessing. -Finally, you must define the **Interface Boundaries**. How will this new feature interact with your existing `VulkanDevice`, `Swapchain`, or `CommandPool` wrappers? By providing the AI with your existing C{pp} class signatures, you ensure that the new code integrates smoothly into your engine's architecture without requiring massive refactors. +**Interface boundaries.** How does the new feature interact with your existing `VulkanDevice`, `Swapchain`, or `CommandPool` wrappers? Giving the model your existing class signatures up front means the generated code is more likely to fit your engine instead of requiring a rewrite to integrate. -== Why Use a High-Reasoning Model? +== Why use a high-reasoning model for this phase -Architectural planning requires **Systemic Reasoning**—the ability to hold multiple, interacting systems in mind at once. While small local models are excellent at writing single functions, they can struggle with the high-level implications of a change across your entire engine. +Architectural planning means holding several interacting systems in mind at once. Small local models are good at writing an individual function, but tend to struggle when a change has implications across the whole engine. -Models like **Claude 4.6** or **GPT-5.3** excel at this phase. They can analyze 500 lines of your engine's existing architecture and suggest how to integrate a new feature without breaking existing synchronization or violating the memory alignment rules of your target hardware. +Larger reasoning models (Claude- or GPT-class) are better suited here — they can take a few hundred lines of existing architecture as context and reason about how a new feature fits in without breaking existing synchronization or violating the memory alignment rules of your target hardware. They're still not infallible, and it's worth checking their suggestions against the spec rather than taking them at face value. -== Real-World Experience: Designing a Bloom Pipeline +== Example: designing a Bloom pipeline -Let's design a modern **Bloom Post-Process Pipeline** using the Collaborative Loop. +=== Step 1: Give it real context -=== Step 1: Providing the "Deep Context" -Do not just ask: *"Design a bloom system."* Instead, feed the AI the results of your **Context Bridge**. Provide the `VulkanContext.hpp` file and your engine's memory management headers. -**The Prompt:** *"I am implementing a Bloom pass. Our engine uses Vulkan 1.3 with Dynamic Rendering and VMA for memory. Here is our `Renderer` class. Design a Bloom pipeline that uses a 13-tap downsample/upsample chain. Provide a roadmap that defines the class structure and the image layout transitions for each mip-level."* +Don't just ask "design a bloom system." Give the model the actual files it needs — your `VulkanContext.hpp`, your memory management headers, and the relevant parts of your renderer. -=== Step 2: Auditing the Roadmap -The AI generates a plan, and as the human lead, you perform a **Semantic Audit**. For example, if the AI suggests using `STORAGE_IMAGE_OPTIMAL` for your bloom pass, you might realize your target mobile hardware (such as a Mali GPU) performs better if you use `ATTACHMENT_OPTIMAL` with input attachments instead. You can then provide a correction by telling the AI to adjust the roadmap to use subpass-like input attachments for the upsample pass to save bandwidth on mobile. +**Example prompt:** "I am implementing a Bloom pass. Our engine uses Vulkan 1.3 with Dynamic Rendering and VMA for memory. Here is our `Renderer` class. Design a Bloom pipeline that uses a 13-tap downsample/upsample chain. Provide a roadmap that defines the class structure and the image layout transitions for each mip level." -=== Step 3: Finalizing the "Contract" -The result is a verified `bloom_design.md` file. This file contains the detailed technical specification that you will use in the next phase to generate the actual code. +=== Step 2: Check the plan against your hardware -== Practical Tutorial: Building a Graphics Feature Roadmap +The model will produce a plan, but you still need to review it against constraints it doesn't know about. For example, if it suggests `STORAGE_IMAGE_OPTIMAL` for the bloom pass, you might know your target mobile hardware (a Mali GPU, say) performs better with `ATTACHMENT_OPTIMAL` and input attachments instead. Push back and ask it to revise the plan to use input-attachment-style reads for the upsample pass to save bandwidth on mobile. -Planning a complex feature like a **Frame Graph** or a **Clustered Renderer** with an AI is a specific skill. It's the difference between a high-level conversation and a set of engineering blueprints. +=== Step 3: Write it down -Follow this workflow to create a high-quality "Contract" for your next feature: +The output is a `bloom_design.md` file — a concrete technical spec you'll use as the basis for the implementation phase. -1. **Context Loading:** - Use your **MCP Tools** (specifically the `mcp-Vulkan` server) to query the latest specification for the feature you want to build. This ensures the AI is using current Vulkan 1.3 or 1.4 extensions instead of legacy patterns. +== Practical steps for planning a feature -2. **Define the "System Boundaries":** - Clearly state your engine's constraints. +Planning something larger, like a frame graph or a clustered renderer, benefits from a bit more structure than a single prompt: + +1. **Load context first.** + Use your MCP tools (for example, an `mcp-vulkan` server) to pull the current spec for the feature you're building, so the model is working from Vulkan 1.3/1.4 semantics rather than outdated patterns it may have seen in training. + +2. **State your constraints explicitly.** + [source,text] ---- @@ -58,22 +57,21 @@ We use VMA for all allocations. We are using Dynamic Rendering (no manual RenderPasses). ---- -3. **Request a Multi-Layered Roadmap:** - Ask the assistant to break the plan into three specific layers. +3. **Ask for the plan in layers.** + [source,text] ---- Generate a 3-layer roadmap: -1. Architectural Design: (Class hierarchy, state management) -2. Data Flow: (Descriptor layout, push constants, VMA allocation flags) -3. Synchronization Map: (Pipeline barriers, layout transitions, and image memory dependencies) +1. Architectural design: class hierarchy, state management +2. Data flow: descriptor layout, push constants, VMA allocation flags +3. Synchronization map: pipeline barriers, layout transitions, image memory dependencies ---- -4. **Refine and Commit (The "Truth" File):** - As the human lead, audit the AI's suggestions. Once the plan is solid, have the AI write it to a `DESIGN_DOC.md` file in your repository. Use this file as the "Ground Truth" for the implementation phase. +4. **Review, then commit it to a file.** + Read through the plan critically before accepting it. Once you're satisfied, have the model write it to a `DESIGN_DOC.md` file in your repo — this becomes the reference for the implementation phase. -== Summary: System Design Mastery +== Summary -Phase 1 is about **Human-AI Collaboration at the Design Level.** By leveraging the reasoning power of high-end LLMs to create a solid "Contract," you turn the overwhelming complexity of Vulkan into a series of well-defined implementation tasks. You have effectively "pre-designed" your architecture before a single line of C{pp} has been written. +Phase 1 is about using a high-reasoning model to turn a feature idea into a concrete, reviewable plan before writing any C{pp}. The output — a design doc with resource lifecycle, synchronization, and interface boundaries spelled out — is what makes the implementation phase go smoothly instead of turning into a series of disconnected patches. -Next: xref:AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc[Phase 2: Implementation & The Implementation Loop] +Next: xref:AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc[Phase 2: Implementation] diff --git a/en/AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc b/en/AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc index 3b223d11..d7d0b250 100644 --- a/en/AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc +++ b/en/AI_Assisted_Vulkan/05_workflow/03_phase_2_implementation.adoc @@ -1,53 +1,51 @@ :pp: {plus}{plus} -= Phase 2: Implementation & The Implementation Loop (The Implementation Phase) += Phase 2: Implementation -== Introduction: Turning the Roadmap into Reality +== Introduction: turning the plan into code -Phase 1 provided the technical specification—a verified roadmap for our post-process system. Now, we enter **Phase 2: The Implementation Loop**. This is where we turn that design into the verbose, high-precision C{pp} and Slang/HLSL code that Vulkan requires. +Phase 1 gave us a design doc — a verified plan for the post-process system. Phase 2 is where that plan becomes actual C{pp} and Slang/HLSL code. -In **Collaborative Engineering**, this is the "Implementation" phase. We move from high-reasoning Cloud models to **Task-Specialized Models** like **Qwen 3-Coder (30B)** or **Mistral-Nemo (12B)**. These models excel at the "Syntactic Noise" of Vulkan—the descriptor set updates, the pipeline state objects, and the intricate shader logic—providing near-instant feedback while you type. +This is where it's worth switching from a high-reasoning cloud model to a faster, code-specialized local model — something like Qwen 3-Coder (30B) or Mistral-Nemo (12B). These models are well suited to the repetitive parts of Vulkan: descriptor set updates, pipeline state objects, and shader logic, with fast enough turnaround to stay in a tight edit-and-check loop. -== The Core Concept: Conversational Coding +== Working in small steps -The implementation of a Vulkan feature is not a single "long generation." Instead, it is a series of focused, incremental steps that we call **Conversational Coding**. +Implementing a feature works better as a series of small, focused requests than one large generation. -* **The Problem:** You need to write a `VkGraphicsPipelineCreateInfo` for your Bloom upsample pass. In a traditional workflow, you'd spend 20 minutes copying and pasting an existing function and manually changing the blending states and shader stages. -* **The AI Solution:** You use your local specialist. You provide the **Technical Roadmap** from Phase 1 as context and say: *"Generate the upsample pipeline state. Use additive blending for the color attachment and ensure the vertex input state matches our `ScreenQuad` class."* +* **The problem:** you need a `VkGraphicsPipelineCreateInfo` for the Bloom upsample pass. Writing it by hand usually means copying an existing pipeline and manually adjusting blend states and shader stages. +* **A faster path:** give your local model the technical roadmap from Phase 1 as context and ask directly: "Generate the upsample pipeline state. Use additive blending for the color attachment and ensure the vertex input state matches our `ScreenQuad` class." -The AI doesn't just "guess." It uses the roadmap's "Contract" to ensure the blending factors are correct and uses your engine's context to match your existing quad-rendering logic. +Because the model has the roadmap as context, it can get the blending factors and quad-rendering conventions right on the first pass more often than if you just described the pipeline from scratch. It's still worth checking the output against the plan rather than assuming it's correct. -== Why Use a Local Specialist? +== Why a local model for this phase -Implementing a graphics feature requires **Zero Latency**. You will find yourself asking hundreds of small, syntactic questions every hour: +Implementation work benefits from a fast feedback loop. You'll end up asking a lot of small, syntactic questions over the course of an hour: -* *"What was the enum for a 1D sampler again?"* -* *"Generate the `VkDescriptorImageInfo` array for these mips."* -* *"Refactor this shader to use a structured buffer for the bloom weights."* +* "What's the enum for a 1D sampler again?" +* "Generate the `VkDescriptorImageInfo` array for these mips." +* "Refactor this shader to use a structured buffer for the bloom weights." -A local model like **Qwen 3-Coder** or **Mistral-Nemo** (running via Ollama) provides the response in milliseconds. It handles the tedious details so you can stay in your "Creative Flow." +A local model running through Ollama answers these in milliseconds rather than the round-trip of a hosted API call, which matters more here than raw reasoning quality does. -== Real-World Experience: Implementing the Bloom Pass +== Example: implementing the Bloom pass -Continuing with our Bloom example, let's see how the implementation phase unfolds. +=== Step 1: Skeleton generation +Point the local model at `bloom_design.md`. +**Prompt:** "Generate the `BloomRenderer.hpp` header. Use the `VulkanDevice` wrapper and implement the `recordDownsampleCommand` and `recordUpsampleCommand` methods as defined in our design doc." +**Result:** a header that follows your engine's RAII conventions and naming, which you should still skim before accepting. -=== Step 1: Skeleton Generation -You point your local AI at the `bloom_design.md` roadmap. -**The Prompt:** *"Generate the `BloomRenderer.hpp` header. Use the `VulkanDevice` wrapper and implement the `recordDownsampleCommand` and `recordUpsampleCommand` methods as defined in our design doc."* -**The Result:** A perfectly formatted C{pp} header that matches your engine's RAII wrappers and naming conventions. +=== Step 2: Pipeline boilerplate +Vulkan pipeline setup is a common source of bugs, mostly from copy-paste errors. +**Prompt:** "In `BloomRenderer.cpp`, implement the pipeline creation logic. Use the roadmap's image layout requirements. For the downsample pass, disable depth testing and set the polygon mode to FILL." +**Result:** a working first draft of the pipeline initialization code, worth a careful read against the design doc before moving on. -=== Step 2: Automating the Verbosity -Vulkan pipelines are where most bugs are born. -**The Prompt:** *"In `BloomRenderer.cpp`, implement the pipeline creation logic. Use the roadmap's image layout requirements. For the downsample pass, disable depth testing and set the polygon mode to FILL."* -**The Result:** 150 lines of correct `VkPipeline` initialization code generated in 5 seconds. +=== Step 3: Shader iteration +Shader development is somewhere local models can speed up iteration, but the output needs closer review than the C{pp} side. +**Prompt:** "Generate a Slang shader for the 13-tap downsample filter. Use the Karis average to prevent fireflies in the bloom, as discussed in the roadmap's performance section." +**Result:** a shader implementing the requested technique — but be aware that most models have seen far more HLSL and GLSL than Slang in training, and will often produce HLSL-shaped output that doesn't compile as-is. Treat the generated shader as a starting point: check that Slang-specific constructs (`[shader("fragment")]`, `ParameterBlock`, `interface` types) are used correctly, and expect to fix several issues before it compiles cleanly. -=== Step 3: Shader Iteration -Shader development is where local models can accelerate iteration, though results require careful review. -**The Prompt:** *"Generate a Slang shader for the 13-tap downsample filter. Use the Karis average to prevent fireflies in the bloom, as discussed in the roadmap's performance section."* -**The Result:** A shader implementing the requested technique. However, be aware that most LLMs have been trained primarily on HLSL and GLSL, and will often produce HLSL-style output that needs manual adjustment to compile as valid Slang. Treat the generated shader as a starting point: verify the syntax, check that Slang-specific constructs (e.g., `[shader("fragment")]`, `ParameterBlock`, or `interface` types) are used correctly, and expect to fix several issues before the shader compiles cleanly. +== Summary -== Summary: The Accelerated Implementation Loop +Using a local model for implementation turns a lot of Vulkan's inherent verbosity into short iteration cycles rather than long manual typing sessions. The tradeoff is that you're reviewing more generated code, more often — the roadmap from Phase 1 is what keeps that review manageable instead of open-ended. -By using a local specialist to handle the "Heavy Lifting" of Vulkan's verbosity, you turn a multi-day implementation task into a series of 15-minute iteration cycles. You remain the **Director**, ensuring every line of code matches the high-level architecture, while the AI manages the low-level precision. - -Next: xref:AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc[Phase 3: Automated Evolution] +Next: xref:AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc[Phase 3: Refactoring] diff --git a/en/AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc b/en/AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc index 499539a5..1b07d45f 100644 --- a/en/AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc +++ b/en/AI_Assisted_Vulkan/05_workflow/04_phase_3_refactoring.adoc @@ -1,59 +1,55 @@ :pp: {plus}{plus} -= Phase 3: Automated Evolution (The Review Phase) += Phase 3: Refactoring -== Introduction: The Evolution of an Engine +== Introduction: engines keep changing -Vulkan code is not static. What was a "Best Practice" in Vulkan 1.0 (like using `VkRenderPass` and `VkFramebuffer` for every tiny subpass) is now considered "Legacy" in Vulkan 1.3, which prioritizes **Dynamic Rendering**. +Vulkan best practices shift over time. What was standard in Vulkan 1.0 — a `VkRenderPass` and `VkFramebuffer` for every small subpass — is now considered legacy next to Vulkan 1.3's Dynamic Rendering. -This final phase of the workflow is where we move from "Co-Coding" to **Supervision**. We use **Autonomous Agents** (like **Goose**) to perform the "Deep Refactor." Unlike an IDE assistant that focuses on single functions, an agent can "look" across your entire engine to ensure that a major architectural shift is implemented consistently from the initialization code to the final draw call. +This phase is less about writing new code with the model and more about supervising an agent that does a broader refactor on its own. Unlike an IDE assistant working on one function at a time, an agent (such as Goose) can work across the whole engine, which matters for changes that need to land consistently from initialization through to the final draw call. -== The Core Concept: The "Agentic Shift" +== Handing off the work -In **Phase 3**, the roles are reversed. You are no longer writing code with the AI's help; you are giving the AI an engineering goal and acting as the **Technical Reviewer**. +In Phase 3, you're not co-writing code — you're giving the agent a goal and reviewing what it produces. -* **The Problem:** You want to migrate your engine from GLSL to **Slang**. This involves updating every shader, changing your descriptor binding logic, and modifying your shader compilation build scripts. For a human, this is a week of tedious, error-prone work. -* **The Agentic Solution:** You give an agent access to your terminal and your project. You say: *"Migrate our `post_process/` folder to Slang. Update the `CMakeLists.txt` to use the Slang compiler, refactor the C{pp} to use the new reflection data, and fix any compilation errors."* +* **The problem:** migrating your engine from GLSL to Slang. This touches every shader, your descriptor binding logic, and your shader compilation scripts — a tedious, error-prone week of work if done by hand. +* **The agentic approach:** give an agent access to your terminal and project, and describe the goal: "Migrate our `post_process/` folder to Slang. Update the `CMakeLists.txt` to use the Slang compiler, refactor the C{pp} to use the new reflection data, and fix any compilation errors." -The agent doesn't just "suggest" changes. It reads the files, applies the refactor, runs the compiler, sees the errors, and **iterates** until the task is complete. +A capable agent doesn't just propose a diff — it edits the files, runs the build, reads the resulting errors, and keeps iterating until it compiles. That doesn't mean the result is correct; it means the build passes, which is a lower bar than "correct." Review is still your job. -== Why Use an Autonomous Agent? +== Why an agent works for this kind of change -Refactoring Vulkan code is often a "House of Cards" task. If you remove a `VkRenderPass`, you must also update: +Refactoring Vulkan code tends to have cascading dependencies. Removing a `VkRenderPass` also means updating: 1. The `VkGraphicsPipelineCreateInfo` to use `VkPipelineRenderingCreateInfo`. 2. The `vkCmdBeginRenderPass` call to `vkCmdBeginRendering`. 3. The layout transitions in every affected memory barrier. -An autonomous agent is uniquely suited for this because it can follow the **"Error Chain"** across multiple files, fixing the compilation errors it creates until the system reaches a new stable state. +An agent that can run the build and read the compiler's errors is well suited to this specific kind of task — it can follow the chain of resulting errors across files and keep fixing them until the build succeeds again. -== Real-World Experience: The "Dynamic Rendering" Migration +== Example: migrating Bloom to Dynamic Rendering -Let's see how an agent handles the migration of our Bloom system to Dynamic Rendering. +=== Step 1: Set boundaries first +Start from a clean git branch. Be explicit about scope: "Only modify the `BloomRenderer` class and its associated shaders. Do not touch the core `VulkanDevice` initialization yet." -=== Step 1: Setting the Guardrails -Before starting, ensure you are on a clean Git branch. -**Your Role:** You define the scope. *"Only modify the `BloomRenderer` class and its associated shaders. Do not touch the core `VulkanDevice` initialization yet."* +=== Step 2: Let it iterate +Give the instruction: "Refactor `BloomRenderer.cpp` to use Dynamic Rendering. Replace all `VkRenderPass` members with `VkPipelineRenderingCreateInfo`. Ensure the image barriers are updated to use `VkImageMemoryBarrier2`." -=== Step 2: The Agentic Loop -You give the command: *"Refactor `BloomRenderer.cpp` to use Dynamic Rendering. Replace all `VkRenderPass` members with `VkPipelineRenderingCreateInfo`. Ensure the image barriers are updated to use `VkImageMemoryBarrier2`."* +Watch what the agent actually does. In a typical run it will: -You watch the terminal. The agent: +* Read the code and work out the dependency chain. +* Modify the pipelines. +* Run the build (`ninja`, or whatever your project uses). +* Hit a "variable not found" error because it missed a `vulkan_core.h` update for synchronization2. +* Fix the include and rebuild. -* Reads the code and identifies the dependency chain. -* Modifies the pipelines. -* Runs `ninja`. -* Sees a "variable not found" error because it forgot to include the `vulkan_core.h` update for synchronization2. -* Fixes the include and rebuilds. +=== Step 3: Review before merging +A passing build isn't the same as good code. Read the diff. It's common to find the agent used `IMAGE_LAYOUT_GENERAL` everywhere just to get things working — correct it explicitly: "Optimize the layouts in the Bloom pass to use `ATTACHMENT_OPTIMAL` for the render targets and `READ_ONLY_OPTIMAL` for the source textures." -=== Step 3: Technical Review -The build is passing, but is the code *good*? -**Your Role:** You review the diff. You notice the agent used `IMAGE_LAYOUT_GENERAL` for everything to "make it work." You correct it: *"Optimize the layouts in the Bloom pass to use `ATTACHMENT_OPTIMAL` for the render targets and `READ_ONLY_OPTIMAL` for the source textures."* +== Summary -== Summary: The Evolving Engine +Phase 3 uses an agent for the kind of multi-file refactor that's mechanical but error-prone by hand, while you stay responsible for reviewing what actually lands. That review step isn't optional — a passing build only tells you the code compiles, not that it's correct or idiomatic for your engine. -Phase 3 is the culmination of the **Collaborative Loop**. By using autonomous agents for the "Heavy Refactor," you ensure that your engine never becomes trapped in legacy patterns. You focus on the **Innovation** of your graphics technology, while the AI manages the intricate details of the evolution. +That closes out the three-phase loop covered in this section: design with a high-reasoning model, implementation with a fast local model, and refactoring with an agent. Next, we look at AI-assisted debugging with RenderDoc. -With the completion of Section 5, you have mastered the **Feature Lifecycle**: from Cloud-based Design, through Local-based Implementation, to Autonomous Evolution. In the next section, we will explore the "Dark Arts" of **Pro-Level Debugging with AI and RenderDoc.** - -Next: xref:AI_Assisted_Vulkan/06_debugging/01_introduction.adoc[Pro-Level Debugging with AI & RenderDoc] +Next: xref:AI_Assisted_Vulkan/06_debugging/01_introduction.adoc[Pro-Level Debugging] diff --git a/en/AI_Assisted_Vulkan/06_debugging/01_introduction.adoc b/en/AI_Assisted_Vulkan/06_debugging/01_introduction.adoc index f1532a34..cd1b1da8 100644 --- a/en/AI_Assisted_Vulkan/06_debugging/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/06_debugging/01_introduction.adoc @@ -2,37 +2,37 @@ = Pro-Level Debugging with AI & RenderDoc -== Introduction: The Diagnostic Paradigm Shift +== Introduction -In traditional graphics development, debugging is a game of "Find the Needle in the Haystack." You are presented with a cryptic **Validation Layer Error (VUID)** or, worse, a "Black Screen," and you must manually trace through 10,000 lines of pipeline state, synchronization barriers, and shader logic to find the single bit that is set incorrectly. +Vulkan debugging is usually a search problem: you get a cryptic **Validation Layer Error (VUID)**, or worse, a black screen, and you have to trace through pipeline state, synchronization barriers, and shader logic to find the one bit that's wrong. -In **Collaborative Engineering**, we move from **Manual Tracing** to **Automated Diagnosis.** We transform our AI assistant from a "Code Writer" into a **diagnostic tool.** This section explores how to use AI to analyze your running application and find the root cause of its failures using the **Evidence Chain**. +An AI assistant can help here by acting as a diagnostic aid rather than just a code generator: given the right context, it can correlate error strings, GPU state, and your source code faster than you can do it by hand. This section covers how to set that up. -== The Core Concept: The "Evidence Chain" +== Three sources of evidence -A professional debugging workflow uses three distinct sources of truth to solve a bug: +A debugging session usually draws on three sources of truth: -1. **The Syntactic Evidence (VUIDs):** The specific error strings from the Vulkan Validation Layers. These tell you *what* rule you broke. -2. **The State Evidence (RenderDoc):** The raw state of the GPU—the pipelines, descriptors, and buffers—at the exact moment of the draw call. This tells you *what* the GPU actually saw. -3. **The Logical Evidence (The AI Reasoning Engine):** The reasoning power of the AI, which correlates the VUID with the RenderDoc state and your C{pp} source code. This tells you **why** the error happened and how to fix it. +1. **VUIDs.** The specific error strings from the Vulkan Validation Layers. These tell you *what* rule you broke. +2. **RenderDoc state.** The raw state of the GPU — pipelines, descriptors, buffers — at the moment of the draw call. This tells you what the GPU actually saw. +3. **The AI's reasoning over both.** When you give an assistant the VUID, the RenderDoc state, and your C{pp} source together, it can often connect them faster than manually cross-referencing all three yourself. -== Why Use AI for Vulkan Debugging? +== Why use AI for Vulkan debugging -Vulkan's validation errors are notoriously dense. A VUID like `VUID-VkGraphicsPipelineCreateInfo-pStages-01565` is technically accurate but emotionally exhausting. By feeding these strings and GPU captures into an AI assistant, you give it the ability to: +Vulkan's validation errors are accurate but dense. A VUID like `VUID-VkGraphicsPipelineCreateInfo-pStages-01565` tells you exactly what's wrong, but not always where to look. Feeding these strings and GPU captures to an assistant can help with: -* **Translate Cryptic VUIDs:** Instantly explain a complex "Image Layout Mismatch" in plain English, including the likely locations in your code where you forgot the transition. -* **Audit GPU State:** "Scan" a RenderDoc capture to identify why a specific descriptor set is "Incompatible" with a pipeline, without you having to manually compare member-by-member. -* **Shader Reasoning:** Use variable values from a specific "failing" pixel to find logical errors in your HLSL or Slang code that the compiler couldn't catch. +* **Translating VUIDs.** Turning a dense "image layout mismatch" message into a plain explanation, plus likely locations in your code where the transition was missed. +* **Reading GPU state.** Scanning a RenderDoc capture to find why a descriptor set is incompatible with a pipeline, instead of comparing fields by hand. +* **Reasoning about shader values.** Taking the input values for a specific failing pixel and checking them against your HLSL or Slang logic to catch mistakes the compiler wouldn't flag. -== The Debugging Roadmap +It won't replace understanding what the validation layers and RenderDoc are telling you — treat it as a way to cut down the manual cross-referencing, not as a black box that produces answers you don't need to check. -In the following chapters, we will build your "Forensic Lab": +== What's in this section -1. **xref:AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc[The VUID Auto-Fix]:** How to use your assistant to get instant code corrections from raw error strings. -2. **xref:AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc[AI + RenderDoc Integration]:** Using autonomous agents to analyze your GPU captures to identify redundant state changes and sync hazards. -3. **xref:AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc[Shader Logs & API Dumps]:** Analyzing the underlying behavior of your app by feeding high-volume API logs to an AI to find the source of a state mismatch. -4. **xref:AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc[AI + GFXReconstruct]:** Capturing and auditing multi-frame Vulkan call streams for driver regressions and resource lifecycle hazards. +1. **xref:AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc[VUID Auto-Fix]:** Turning a raw validation error string into a targeted code fix. +2. **xref:AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc[AI + RenderDoc Integration]:** Using an agent to read GPU captures and flag redundant state changes or sync hazards. +3. **xref:AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc[Shader Logs & API Dumps]:** Feeding large API logs to an assistant to find the source of a state mismatch. +4. **xref:AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc[AI + GFXReconstruct]:** Capturing and auditing multi-frame Vulkan call streams for driver regressions and resource lifecycle issues. -By the end of this journey, you will no longer fear the "Black Screen." You will have a systematic, AI-driven process for diagnosing even the most elusive Vulkan bugs. +By the end, you'll have a repeatable process for working through Vulkan validation and rendering bugs with an assistant in the loop. -Next: xref:AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc[The VUID Auto-Fix] +Next: xref:AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc[VUID Auto-Fix] diff --git a/en/AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc b/en/AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc index f0cd8711..e1344582 100644 --- a/en/AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc +++ b/en/AI_Assisted_Vulkan/06_debugging/02_vuid_autofix.adoc @@ -1,24 +1,22 @@ :pp: {plus}{plus} -= The VUID "Auto-Fix": Mastering Validation Layers += VUID Auto-Fix: Working Through Validation Layers -== Introduction: Translation Capabilities +== Introduction -Vulkan **Validation Layers** are the most powerful tool in your debugging arsenal. They catch thousands of common errors, from incorrect structure sizes to sophisticated synchronization hazards. However, for a novice or even an intermediate developer, the sheer density of a single Validation Error can be paralyzing. +Vulkan **Validation Layers** are one of the most useful tools you have — they catch a huge range of errors, from bad struct sizes to subtle synchronization hazards. But a single validation error can be dense enough to be hard to parse, especially if you're newer to Vulkan. -In **Collaborative Engineering**, we use our AI assistant to translate validation errors. It doesn't just "read" the error; it correlates the cryptic VUID (Valid Usage ID) with your actual C{pp} source code to provide a definitive fix. +An AI assistant can help by correlating the VUID (Valid Usage ID) with your actual C{pp} source to suggest a specific fix, rather than leaving you to decode the error string on your own. -== The Core Concept: The "VUID to Source" Bridge +== VUID, location, and spec context -A **VUID Auto-Fix** is not a simple search-and-replace. It is a three-way correlation between the **Symptom**, the **Location**, and the **Spec Context**. +A useful auto-fix workflow combines three pieces of information: the **VUID** itself (a stable identifier for a specific rule in the Vulkan Specification), the **location** in your C{pp} source where the offending call happens, and the **spec context** — the actual rule text, which an MCP server can pull live from `vk.xml` so the assistant isn't relying on outdated training data. -The **Symptom** is the full VUID string provided by the Validation Layers. These are unique identifiers that point to a single, immutable rule in the Vulkan Specification. The **Location** is your actual C{pp} source code—the specific function or class where the offending Vulkan call is being made. Finally, the **Spec Context** is the ground truth provided by your **Model Context Protocol (MCP)** server, which allows the AI to "read the manual" for that specific VUID in the latest version of the SDK. +With all three, the assistant can tell you not just that there's a type mismatch, but that your shader expects a `storage_buffer` because you're writing to it, while your C{pp} code declares it as a `uniform_buffer` — and propose a fix consistent with the rest of your code. -By bridging these three elements, the AI can perform a **diagnostic analysis**. It doesn't just see a "type mismatch"; it sees that your shader is expecting a `storage_buffer` because you are writing to it, while your C{pp} code is initializing it as a `uniform_buffer`. The resulting fix isn't just a "best guess"—it is a spec-accurate correction that is consistent with your engine's architecture. +== Example: a descriptor mismatch -== Real-World Experience: Fixing a Descriptor Mismatch - -Imagine you are implementing a complex **Bindless Texture** system. You run your app, and the console explodes with this error: +Say you're implementing a bindless texture system, and you run the app to see this: [source,text] ---- @@ -27,27 +25,31 @@ Object: 0x5555559868f0 | Message: pBindings[0].descriptorType is VK_DESCRIPTOR_T but shader expects VK_DESCRIPTOR_TYPE_STORAGE_BUFFER... ---- -=== Step 1: The Diagnostic Query -You don't just "search" for this error. You provide the AI with the **Context Bridge**. -**The Prompt:** *"I am getting this VUID: [PASTE ERROR]. Here is my `DescriptorSetLayout` creation code: [PASTE CODE]. Based on our engine's `Buffer` class, why is this type mismatch occurring?"* +=== Step 1: Ask with context + +Give the assistant the error and the relevant code, not just the error alone. + +**Prompt:** *"I am getting this VUID: [PASTE ERROR]. Here is my `DescriptorSetLayout` creation code: [PASTE CODE]. Based on our engine's `Buffer` class, why is this type mismatch occurring?"* + +=== Step 2: Let it correlate shader and C{pp} code -=== Step 2: The Logical Correlation -The AI doesn't just tell you to change an enum. It analyzes your **Shader Code** (using RAG) and your **C{pp} Initialization**. -**The AI Diagnosis:** *"Your shader is expecting a `storage_buffer` because you are writing to it in the fragment stage. However, your C{pp} code is initializing it as a `uniform_buffer`. You must update the descriptor type to `STORAGE_BUFFER` and ensure your `VkBufferUsageFlags` include the `STORAGE_BUFFER_BIT`."* +With RAG access to your shader code and your C{pp} initialization, a reasonable response looks like: -=== Step 3: The Contextual Fix -The AI provides the exact code change, tailored to your engine's RAII wrappers. It even warns you that this change might require an increase in your `maxDescriptorSetStorageBuffers` limit, a detail often missed by human developers. +**"Your shader is expecting a `storage_buffer` because you are writing to it in the fragment stage. However, your C{pp} code is initializing it as a `uniform_buffer`. You must update the descriptor type to `STORAGE_BUFFER` and ensure your `VkBufferUsageFlags` include the `STORAGE_BUFFER_BIT`."** -== Advanced: The "Always-Current" VUID Dictionary +=== Step 3: Apply the fix -One of the most powerful aspects of this workflow is using **MCP** to query the **live Vulkan Spec**. As discussed in xref:AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc[The Context Bridge], your assistant can query the `vk.xml` file on your disk. This means that if a new VUID is introduced in Vulkan 1.4, your AI knows the rule the same day you update your SDK, even if the model itself was trained years ago. +A good assistant will give you the change in the style of your existing RAII wrappers, and will flag secondary effects — for example, that this change might push you over your `maxDescriptorSetStorageBuffers` limit, which is easy to miss even for an experienced developer. -== Practical Tutorial: Performing a VUID Auto-Fix +== Keeping the VUID list current -When you encounter a validation error, do not panic. Follow these steps to transform a cryptic warning into a solid fix. +One genuinely useful part of this workflow is having MCP query the live Vulkan spec, as covered in xref:AI_Assisted_Vulkan/02_environment_setup/06_mcp_context_bridge.adoc[The Context Bridge]. Your assistant can read the `vk.xml` file on disk directly, so a VUID introduced in a newer Vulkan version is available to it the same day you update your SDK, regardless of when the underlying model was trained. -=== Step 1: Isolate and Load Context -Copy the VUID string from your terminal. Then, in your AI assistant (Goose or IDE Chat), ensure the relevant code is loaded. +== Practical steps: performing a VUID auto-fix + +=== Step 1: Isolate and load context + +Copy the VUID string from your terminal, then make sure the relevant code is loaded into your assistant (Goose or IDE chat). [source,bash] ---- @@ -55,8 +57,9 @@ Copy the VUID string from your terminal. Then, in your AI assistant (Goose or ID goose session --instruction "Analyze Renderer.cpp and main.frag. We have a VUID error to solve." ---- -=== Step 2: The Diagnostic Prompt -Provide the exact error and ask for a correlation with your engine's logic. +=== Step 2: Ask for the correlation + +Provide the exact error and ask for it to be checked against your engine's logic. [source,text] ---- @@ -68,8 +71,9 @@ but shader expects VK_DESCRIPTOR_TYPE_STORAGE_BUFFER. Based on our 'DescriptorSetManager' class, why is this mismatch occurring? ---- -=== Step 3: Verify and Apply -When the AI suggests a solution, ask for a "Pre-Flight Check" to ensure it doesn't break other parts of the pipeline. +=== Step 3: Verify before applying + +Before applying a suggested fix, ask it to check for side effects elsewhere in the pipeline. [source,text] ---- @@ -78,12 +82,12 @@ Will this require any changes to our memory alignment logic or buffer usage flags in 'Buffer.cpp'? ---- -Finally, apply the fix and re-run your application. If the Validation Layer remains silent, the fix is verified. +Apply the fix and re-run your application. If the validation layer stays quiet, the fix worked. -In most use cases, in an IDE with the chat interface, one can simply copy the VUID and paste it into the AI assistant prompt. The AI will infer that the error is happening and it will go and fix it. This process is seamless and efficient, leveraging AI to bridge the gap between validation errors and code fixes. However, the above is an example of an explicit fix request pipeline that local LLMs do well with and is what one should select when dealing with a local LLM for best results. +In an IDE chat interface, this often reduces to pasting the VUID and letting the assistant find and fix the issue directly — it's a fast path that works well for common cases. The explicit prompt structure above is worth using with local models, which tend to do better with a clearly scoped request than with an open-ended "fix this." -== Summary: Specification-Aware Debugging +== Summary -By using AI as a bridge between the Validation Layers and your source code, you move directly from **Error** to **Understanding** to **Solution.** You are no longer "guessing" why a barrier failed; you are receiving a spec-accurate, codebase-consistent diagnosis. +Using AI to bridge validation errors and source code cuts out a lot of the manual cross-referencing between the spec, the error string, and your code. It's still worth checking the reasoning, especially for anything touching memory layout or synchronization — but for routine VUID triage, it saves real time. Next: xref:AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc[AI + RenderDoc Integration] diff --git a/en/AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc b/en/AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc index c6400c47..6145cc12 100644 --- a/en/AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc +++ b/en/AI_Assisted_Vulkan/06_debugging/03_renderdoc_ai_integration.adoc @@ -2,44 +2,46 @@ = AI + RenderDoc Integration: Automated GPU Analysis -== Introduction: Freezing Time +== Introduction -**RenderDoc** is the definitive tool for Vulkan debugging. It allows you to freeze time and inspect every draw call, descriptor set, and pipeline state object on your GPU. However, for a complex scene, a single frame can contain thousands of events. Finding the one draw call that is causing your "Shadow Acne" or "Missing Texture" is often a daunting, manual task. +**RenderDoc** is the standard tool for Vulkan debugging — it lets you freeze a frame and inspect every draw call, descriptor set, and pipeline state object on the GPU. But a complex scene can have thousands of events in a single frame, and finding the one draw call responsible for shadow acne or a missing texture is often slow, manual work. -In **Collaborative Engineering**, we combine the raw state-tracking of RenderDoc with the reasoning power of an AI Agent. We transform the "Artifact Hunt" from a manual search into an **automated state audit**. +Combining RenderDoc's state-tracking with an AI agent turns some of that manual search into something closer to a scripted audit. -== The Core Concept: Programmatic Inspection +== Programmatic inspection -To integrate AI with RenderDoc, we move beyond the GUI and utilize the **RenderDoc command-line tool (`renderdoccmd`)**. This allows an autonomous agent (like **Goose**) to "browse" your GPU capture programmatically. +To connect AI with RenderDoc, you go through the **RenderDoc command-line tool (`renderdoccmd`)** instead of the GUI. This lets an agent (such as Goose) browse a GPU capture programmatically. -Consider a typical scenario where you encounter a "Black Screen." You know the draw call is happening, but you don't know which of the 50 pipeline states is misconfigured. In a traditional workflow, you'd spend an hour clicking through the RenderDoc UI. +Say you're facing a black screen: the draw call is happening, but you don't know which of the pipeline states is misconfigured. Working through the RenderDoc UI by hand can easily take an hour for a case like this. -Using an agentic solution, you give your agent a tool that can run `rdc-cli` and simply ask it to analyze the capture. You might say: *"Analyze the latest RenderDoc capture. Find the draw call for the 'MainPass' and audit the descriptor sets. Why is the fragment shader receiving a zero-value for the diffuse texture?"* +With an agent that can run `rdc-cli`, you can instead ask it directly: *"Analyze the latest RenderDoc capture. Find the draw call for the 'MainPass' and audit the descriptor sets. Why is the fragment shader receiving a zero-value for the diffuse texture?"* -The agent doesn't just "look" at the screen. It exports the **Structured Pipeline State** (in JSON or Markdown), compares it against your C{pp} source code (via the **Context Bridge**), and identifies the mismatch. +The agent exports the pipeline state (as JSON or Markdown), compares it against your C{pp} source via the Context Bridge, and reports back what it finds. -== Real-World Experience: The "Ghost" Texture +== Example: a missing texture -Imagine you are debugging a missing texture. In RenderDoc, you see that the texture is bound, but the shader output is black. +Say you're debugging a missing texture. In RenderDoc, the texture looks bound, but the shader output is black. -=== Step 1: Automated State Search -Instead of clicking through tabs, you ask your assistant: *"Goose, use the RenderDoc tool to check the `VkSampler` state for the third draw call in the 'G-Buffer' pass."* +=== Step 1: Ask for a targeted state check -=== Step 2: The Discovery -The agent exports the sampler state and finds that `minLod` is set to `10.0f` while your texture only has 5 mip-levels. -**The AI Diagnosis:** *"You are binding a sampler that is forcing the GPU to sample from a non-existent mip-level. This is why the texture appears black. Check your `SamplerBuilder` in `VulkanContext.cpp`."* +Instead of clicking through tabs, ask directly: *"Goose, use the RenderDoc tool to check the `VkSampler` state for the third draw call in the 'G-Buffer' pass."* -=== Step 3: The Automated Fix -Because the agent also has access to your source code, it doesn't just tell you the problem—it offers to fix it. It refactors your sampler creation logic to automatically clamp `minLod` to the actual mip-count of the bound image. +=== Step 2: The finding -== Practical Tutorial: Connecting Goose to RenderDoc +The agent exports the sampler state and finds `minLod` set to `10.0f`, while the texture only has 5 mip levels. -To truly automate your GPU analysis, you must give your agent the ability to use the **RenderDoc command-line tool (`renderdoccmd`)**. This allows the AI to "inspect" a `.rdc` capture file as if it were a text document. +**"You are binding a sampler that is forcing the GPU to sample from a non-existent mip-level. This is why the texture appears black. Check your `SamplerBuilder` in `VulkanContext.cpp`."** -Follow these steps to build the "RenderDoc Bridge": +=== Step 3: The fix -1. **Ensure RenderDoc is in your PATH:** - The `renderdoccmd` binary (or `renderdoccmd.exe` on Windows) must be accessible from your terminal. +Since the agent also has your source code, it can propose a fix directly — for example, refactoring the sampler creation logic to clamp `minLod` to the actual mip count of the bound image. + +== Connecting Goose to RenderDoc + +To automate this kind of analysis, give your agent access to the **RenderDoc command-line tool (`renderdoccmd`)** so it can treat a `.rdc` capture like a document it can query. + +1. **Make sure RenderDoc is on your PATH.** + The `renderdoccmd` binary (or `renderdoccmd.exe` on Windows) needs to be reachable from your terminal. + [source,bash] ---- @@ -49,11 +51,11 @@ $env:Path += ";C:\Program Files\RenderDoc" export PATH=$PATH:/usr/bin/renderdoc ---- -2. **Capture a Frame:** - Use the RenderDoc GUI to capture a frame from your Vulkan application. Save it as `debug_capture.rdc`. +2. **Capture a frame.** + Use the RenderDoc GUI to capture a frame from your Vulkan application and save it as `debug_capture.rdc`. -3. **Commanding the Inspection:** - In your Goose session, ask the assistant to analyze the capture. The following prompt is designed to list all draw calls and find the one that uses a specific shader. +3. **Ask the agent to inspect it.** + In your Goose session, ask it to list draw calls and find the one using a specific shader. + [source,text] ---- @@ -62,8 +64,8 @@ Find the one using 'MainPass' and export its full pipeline state to a structured file named 'state.txt'. ---- -4. **Performing the Logical Audit:** - Once the state is exported, task the AI to correlate the GPU's current configuration with your source code. +4. **Run the comparison against source.** + Once the state is exported, ask the assistant to compare it with your source. + [source,text] ---- @@ -73,17 +75,17 @@ depth test settings or cull mode that could explain why our geometry is missing? ---- -5. **The "Fix-to-Source" Step:** - The AI identifies that `depthTestEnable` is `VK_FALSE` in the capture, while your code *should* be setting it to `VK_TRUE`. It then searches your project to find the line of code that is incorrectly disabling the depth test and offers to fix it. +5. **Fix at the source.** + The agent identifies that `depthTestEnable` is `VK_FALSE` in the capture when it should be `VK_TRUE` in your code, finds the line responsible, and offers a fix. -== Why Structured State Matters +== Why structured state helps -An AI cannot "see" the RenderDoc GUI effectively, but it is a master of **Structured Data**. By exporting your GPU state to a format the AI can parse, you give it the ability to identify redundant state changes. For example, it can notice if you are rebinding the same pipeline five times in a command buffer without any intervening draw calls. +An AI model can't usefully "look at" the RenderDoc GUI, but it can work well with structured data. Once you export GPU state to a format it can parse, it can catch things like redundant pipeline rebinds — for instance, the same pipeline bound five times in a command buffer with no draw calls in between. -It can also audit memory alignment by checking if your vertex buffer offset (e.g., 12 bytes) meets your hardware's requirement (e.g., 16-byte alignment). Furthermore, it can cross-reference VUIDs by taking a Validation Layer error and finding the exact moment in the RenderDoc timeline where that error originated. +It can also check memory alignment (does a 12-byte vertex buffer offset satisfy a 16-byte hardware requirement?), and cross-reference a validation layer error against the exact point in the RenderDoc timeline where it originated. -== Summary: Automated GPU Analysis +== Summary -By integrating AI with RenderDoc, you turn graphics debugging into a high-level conversation. You no longer spend your afternoon clicking through nested menus in a GUI; you simply ask the AI to **Find and Explain** the state mismatch. This is a significant efficiency gain for a graphics engineer. +Feeding RenderDoc's structured state to an AI agent turns some GPU debugging from clicking through nested menus into asking a direct question and checking the answer. It won't replace understanding what the pipeline state means, but it does cut down on the manual comparison work. Next: xref:AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc[Shader Debugging & Log Parsing] diff --git a/en/AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc b/en/AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc index 25481446..2117f4eb 100644 --- a/en/AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc +++ b/en/AI_Assisted_Vulkan/06_debugging/04_shader_debugging_logs.adoc @@ -1,19 +1,18 @@ :pp: {plus}{plus} -= Shader Debugging & Log Parsing: Finding the Needle += Shader Debugging & Log Parsing -== Introduction: The Noise of Graphics +== Introduction -Vulkan is a noisy API. Between the 2,000-page specification, the 10,000 lines of API logs, and the thousands of variables in a single fragment shader, finding the "Needle" of a bug is often a matter of filtering out the "Noise." +Vulkan generates a lot of text: a 2,000-page specification, API dumps that can run to tens of thousands of lines, and shaders with hundreds of intermediate variables. Finding the specific bug in all of that is largely a filtering problem, and it's one an AI assistant can help with — reasoning about shader logic, or scanning a large API dump for a specific pattern. -In **Collaborative Engineering**, we use AI as a high-precision filter. Whether it's reasoning about shader logic or parsing massive API dumps, the AI acts as a digital sieve, catching the anomalies that are invisible to the naked eye. +== Building a log diagnostic script -== Tutorial: Building the Log Diagnostic Script +You can automate a lot of the log search with a small script that captures an API dump and feeds it to **Goose** for review. -To automate the "Needle-in-a-Stack" search, you can use a simple Bash script that captures your API dump and feeds it to **Goose** for an architectural audit. +=== Step 1: Capture the API dump -=== Step 1: Capturing the API Dump -Enable the `VK_LAYER_LUNARG_api_dump` layer and redirect the output to a file. On Linux or Windows, you can do this via environment variables. +Enable the `VK_LAYER_LUNARG_api_dump` layer and redirect the output to a file. [source,bash] ---- @@ -21,8 +20,9 @@ Enable the `VK_LAYER_LUNARG_api_dump` layer and redirect the output to a file. O VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_api_dump ./VulkanApp > api_dump.txt ---- -=== Step 2: The Agentic Diagnostic Script -Create a script `diagnose_logs.sh` that provides the agent with the log context and a specific "Pattern Match" instruction. +=== Step 2: Script the review + +Create `diagnose_logs.sh` to hand the agent the relevant log context and a specific instruction. [source,bash] ---- @@ -45,18 +45,21 @@ barrier parameters for 'VkImageMemoryBarrier2'. " ---- -=== Step 3: Resolving the "Ghost" Artifacts -In its audit, the agent identifies that at line 8,452, you recorded a dispatch to your bloom shader but forgot the barrier before the final composition draw. The bug only appeared when the GPU was under heavy load, which is why it was so hard to find manually. By providing the agent with the log, it transforms the search from a "Manual Scrape" into a "Pattern-Matched Diagnosis." +=== Step 3: Reviewing the result + +In one real case, the agent flagged that at line 8,452, a dispatch to a bloom shader was recorded but the barrier before the final composition draw was missing. The bug only showed up under heavy GPU load, which is part of why it was hard to catch by scanning the log manually. Automating the search this way turns a slow manual scrape into a targeted pattern match — though it's worth spot-checking the flagged lines yourself before assuming the diagnosis is complete. + +== Simulating shader logic + +Shaders are hard to debug because you can't set a breakpoint the way you would in C{pp}. One workaround is to give an assistant the actual input values for a failing fragment and ask it to reason through the shader logic manually. -== Tutorial: Shader Logical Simulation +=== Step 1: Extract fragment data -Debugging a shader is notoriously difficult. Unlike C{pp}, you cannot set a breakpoint. However, you can use **Goose** to "Simulate" the logic of a single fragment. +Use RenderDoc to find a failing pixel and pull its input variables (for example, `prevCoord`, `currentDepth`, `motionVector`). -=== Step 1: Extract Fragment Data -Use RenderDoc to find a failing pixel and extract its input variables (e.g., `prevCoord`, `currentDepth`, `motionVector`). +=== Step 2: Ask it to work through the logic -=== Step 2: The Simulation Prompt -Provide the values and your HLSL/Slang source code to the assistant. +Provide the values along with your HLSL or Slang source. [source,text] ---- @@ -71,18 +74,14 @@ why is this pixel flickering? Is our motion vector scaling correctly for 4K resolution? ---- -=== Step 3: The Discovery -The AI calculates the depth difference, checks the rejection threshold in your code, and identifies that your motion vector scaling is using the wrong resolution—a tiny mathematical error that you would have spent hours manually calculating. +=== Step 3: The result -== Summary: Precision Debugging and Log Analysis +In practice, this kind of prompt can catch a mistake like motion vector scaling using the wrong resolution constant — a small arithmetic error that's tedious to trace by hand but quick for a model to check once it has the actual values in front of it. -Shader reasoning and log parsing are the "Last Mile" of pro-level debugging. By using AI to process the massive amounts of data generated by Vulkan's logging layers and debuggers, you turn **Information Overload** into **Actionable Intelligence**. +== Summary -With the completion of Section 6, you have built a complete, AI-enhanced forensic lab: -1. **The Syntactic Evidence (VUIDs):** Translated and auto-fixed. -2. **The State Evidence (RenderDoc):** Programmatically audited. -3. **The Logical Evidence (Shaders & Logs):** Filtered and reasoned about. +Shader reasoning and log parsing round out the debugging toolkit from this section. Between VUID translation, RenderDoc state audits, and log/shader analysis, you now have three complementary ways to bring AI into a debugging session — matched to VUIDs, GPU state, and shader/log data respectively. -In the next section, we move into **Advanced MCP Tooling**, where you will learn how to build your own custom tools to give your AI direct access to the Vulkan Registry and hardware limits. +Next, we move into MCP tooling, covering how to build custom tools that give your assistant direct access to the Vulkan Registry and hardware limits. Next: xref:AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc[AI-Assisted Trace Analysis: GFXReconstruct] diff --git a/en/AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc b/en/AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc index e9b78dd2..246be527 100644 --- a/en/AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc +++ b/en/AI_Assisted_Vulkan/06_debugging/05_gfxreconstruct_ai.adoc @@ -2,27 +2,25 @@ = AI-Assisted Trace Analysis: GFXReconstruct -== Introduction: The Stream of Truth +== Introduction -While **RenderDoc** is the master of the "Frozen Frame," it can sometimes miss the forest for the trees. Many Vulkan bugs—especially synchronization hazards, resource lifecycle issues, and driver-specific regressions—happen across multiple frames or during the initialization phase. +RenderDoc is built around a single frozen frame, which means it can miss bugs that only show up across multiple frames — synchronization hazards, resource lifecycle issues, and driver-specific regressions during initialization, for example. -**GFXReconstruct** is the definitive tool for capturing and replaying a stream of Vulkan API calls. It allows you to record every command sent to the GPU, from the first `vkCreateInstance` to the final `vkDestroyDevice`. By combining this "Stream of Truth" with the reasoning power of an AI assistant, you transform a multi-gigabyte trace file into a **searchable, audited knowledge base**. +**GFXReconstruct** captures and replays a full stream of Vulkan API calls, from the first `vkCreateInstance` to the final `vkDestroyDevice`. Feeding that stream to an AI assistant turns a multi-gigabyte trace file into something you can actually query, instead of scrolling through by hand. -== Why AI + GFXReconstruct? +== Why pair AI with GFXReconstruct -In **Collaborative Engineering**, GFXReconstruct provides the raw data, and the AI provides the **Architectural Audit**. A trace can contain millions of calls; an AI can "watch" that stream to identify: +GFXReconstruct gives you the raw call data; the AI helps you search it. A trace can run to millions of calls, and an assistant can scan that stream for things like: -* **Initialization Hazards:** Did you enable an extension that isn't supported on your target hardware? The AI can cross-reference `gfxrecon-info` output with your `vk.xml` registry context. -* **Resource Lifecycle Audits:** Are you destroying a buffer before its last use in a command buffer? The AI can parse the "JSON conversion" of a trace to find the exact call sequence where the object became invalid. -* **Driver Regression Analysis:** If a trace works on one driver but fails on another, the AI can audit the trace for "Undefined Behavior" that one driver might be more forgiving of than another. +* **Initialization hazards.** Did you enable an extension unsupported on the target hardware? The assistant can cross-reference `gfxrecon-info` output against your `vk.xml` registry context. +* **Resource lifecycle issues.** Are you destroying a buffer before its last use in a command buffer? The assistant can parse a JSON-converted trace to find the exact call sequence where the object became invalid. +* **Driver regression comparisons.** If a trace works on one driver but not another, the assistant can look for undefined behavior that one driver happens to tolerate and another doesn't. -== Tutorial: The Trace Audit Workflow +== The trace audit workflow -To truly leverage GFXReconstruct with an AI agent like **Goose**, you must convert the binary trace into a format the agent can reason about: **JSON**. +To let an agent like Goose reason over a GFXReconstruct trace, convert the binary capture to JSON first. -Follow these steps to perform an automated trace audit: - -1. **Capture the Trace:** +1. **Capture the trace.** Run your application with the GFXReconstruct layer enabled. + [source,bash] @@ -33,10 +31,10 @@ vkconfig # Use the GUI to enable GFXReconstruct VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct ./VulkanApp ---- + -This will generate a file like `vulkan_capture_20260313_174700.gfxr`. +This generates a file like `vulkan_capture_20260313_174700.gfxr`. -2. **Convert to Structured JSON:** - Use the `gfxrecon-convert` tool to transform the binary trace into a structured text format. For long traces, you should filter for a specific frame range to avoid "Context Window" limits. +2. **Convert to JSON.** + Use `gfxrecon-convert` to turn the binary trace into structured text. For long traces, filter to a specific frame range so you don't blow past the model's context window. + [source,bash] ---- @@ -44,8 +42,8 @@ This will generate a file like `vulkan_capture_20260313_174700.gfxr`. gfxrecon-convert --to-json --frames 100-110 capture.gfxr capture_frames.json ---- -3. **The Agentic Audit:** - In your Goose session, task the assistant to audit the JSON for specific logical hazards. +3. **Ask the agent to audit it.** + In your Goose session, ask it to check the JSON for specific hazards. + [source,text] ---- @@ -56,11 +54,11 @@ Is there any point where we are transitioning FROM 'UNDEFINED' after the first frame? If so, this is a redundant clear hazard. ---- -== Advanced: Building the GFXReconstruct MCP Server +== Building a GFXReconstruct MCP server -To make your assistant truly "Hardware-Aware" and "Trace-Aware," you can build a custom **MCP Server** that wraps the GFXReconstruct CLI. This allows the AI to "query" your trace files directly without you having to manually run shell commands. +If you want your assistant to query trace files without you running the CLI by hand each time, you can wrap the GFXReconstruct CLI in a small MCP server. -Below is a Python implementation of a **GFXReconstruct-Connector** using the MCP SDK: +Here's a minimal Python implementation using the MCP SDK: [source,python] ---- @@ -96,24 +94,26 @@ if __name__ == "__main__": server.run() ---- -By registering this server in your `goose/config.yaml`, the agent gains the ability to **self-diagnose** traces. You can simply say: *"Goose, analyze the hardware requirements of 'crash.gfxr' and tell me if it uses any extensions that are missing on the M4 Max GPU."* +Once this server is registered in your `goose/config.yaml`, you can ask things like: *"Goose, analyze the hardware requirements of 'crash.gfxr' and tell me if it uses any extensions that are missing on the M4 Max GPU."* + +== Example: a layout hazard that only shows up on one driver + +Say you're porting your engine from Windows (NVIDIA) to Android (Qualcomm). It works fine on Windows, but you see flickering in the shadow maps on Android. -== Real-World Experience: The "Undefined" Layout Hazard +=== Checking manually -Imagine you are porting your engine from Windows (NVIDIA) to Android (Qualcomm). On Windows, your engine works perfectly. On Android, you see random "flickering" in your shadow maps. +You spend a day going over your barriers by hand. Everything looks correct, and you suspect a driver difference, but you don't have proof yet. -=== The Manual Attempt -You spend a day checking your barriers. Everything *looks* correct. You suspect a driver bug, but you have no proof. +=== Using the trace -=== The AI + GFXReconstruct Solution -You capture a trace and task your agent: *"Analyze the trace for any 'VkImageMemoryBarrier' that uses 'VK_IMAGE_LAYOUT_UNDEFINED' as the 'oldLayout' on a resource that already contains valid data."* +You capture a trace and ask the agent: *"Analyze the trace for any 'VkImageMemoryBarrier' that uses 'VK_IMAGE_LAYOUT_UNDEFINED' as the 'oldLayout' on a resource that already contains valid data."* -The AI uses the **GFXReconstruct MCP tool** to scan the calls. It finds that in your "Cascaded Shadow Map" pass, you are incorrectly resetting the layout to `UNDEFINED` for the second cascade, effectively discarding the data from the first cascade. On NVIDIA, the driver was "saving" you by ignoring the undefined transition; on Qualcomm, the tile-based renderer was (correctly) discarding the tile memory. +Scanning the calls, it finds that your cascaded shadow map pass resets the layout to `UNDEFINED` for the second cascade — discarding the first cascade's data in the process. NVIDIA's driver happened to tolerate the undefined transition; Qualcomm's tile-based renderer correctly discarded the tile memory instead, which is why the bug only showed up there. -**The Result:** The AI doesn't just find the bug; it identifies the specific line in your `ShadowRenderer.cpp` where the logic was flawed and provides the fix: `oldLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL`. +The agent points to the exact line in `ShadowRenderer.cpp` and proposes the fix: `oldLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL`. -== Summary: Capturing the Narrative +== Summary -GFXReconstruct gives your AI assistant **Timeline Context**. By moving beyond the single-frame snapshot of RenderDoc, you allow the AI to reason about the **Narrative of your Engine**—how it starts, how it allocates, and how it fails over time. This is the hallmark of a professional **Collaborative Engineering** workflow. +GFXReconstruct gives you the full timeline instead of a single frame, and an AI assistant makes that timeline searchable rather than something you scroll through by hand. It's a useful tool for cross-driver bugs and multi-frame hazards that RenderDoc alone won't catch. Next: xref:AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc[Advanced MCP Tooling & Automation] diff --git a/en/AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc b/en/AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc index 4ed08428..422750db 100644 --- a/en/AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc @@ -4,37 +4,31 @@ == Introduction: Beyond Generic Assistance -Up to this point, we have used the **Model Context Protocol (MCP)** as a passive bridge to read our documentation and source code. However, the true power of MCP lies in **Custom Tooling**—the ability to give your AI "Arms and Legs" to interact with the Vulkan ecosystem directly. +So far, we've used the **Model Context Protocol (MCP)** mainly as a read-only bridge, letting the AI look at our documentation and source code. MCP can do more than that: you can give the AI tools that let it act directly on the Vulkan ecosystem, not just read about it. -In **Collaborative Engineering**, we transform our AI from a "Chatbot" into an **Active Maintainer**. This section explores how to move beyond basic file reading to building specialized MCP servers that allow your AI to query the Vulkan Registry, audit hardware limits, and autonomously manage your engine's performance. +This section covers building specialized MCP servers so your AI assistant can query the Vulkan Registry, check hardware limits, and help manage your engine's performance, instead of relying only on what it can read in open files. -== The Core Concept: The AI "Plugin" +== The core concept: MCP servers as plugins -Think of an MCP server as a "Plugin" for your AI assistant's reasoning engine. By default, an AI doesn't know how to query a database or run a performance profiler. When you provide it with an MCP server, you are giving it a **Toolbox**. +An MCP server works like a plugin for your AI assistant. On its own, an AI can't query a database or run a performance profiler. An MCP server gives it a defined, callable interface to do that. -1. **The Registry Tool:** Allows the AI to query the live `vk.xml` file. -2. **The GPUInfo Tool:** Allows the AI to query hardware limits (e.g., `maxDescriptorSetSampledImages`) from databases like `gpuinfo.org`. -3. **The Profiler Tool:** Allows the AI to read your engine's timing logs and identify bottlenecks. +1. **A registry tool:** Lets the AI query the live `vk.xml` file. +2. **A GPU info tool:** Lets the AI query hardware limits (e.g., `maxDescriptorSetSampledImages`) from databases like `gpuinfo.org`. +3. **A profiler tool:** Lets the AI read your engine's timing logs and flag likely bottlenecks. -By giving an autonomous agent (like **Goose**) these tools, you enable it to perform **Agentic Automation**—tasks that previously required hours of manual cross-referencing between the spec and your code. +Giving an autonomous agent (like **Goose**) access to these tools lets it do multi-step lookups on its own — cross-referencing the spec against your code without you manually copying data back and forth. It's still worth checking its output; these tools reduce manual lookup, they don't remove the need to verify the result. -== Why Advanced MCP for Vulkan? +== Why this is useful for Vulkan -Vulkan's development cycle is dominated by **Data Lookup**. +A lot of Vulkan development involves looking things up: the spec (XML), hardware limits (JSON databases), and performance data (timing logs). AI models can work with structured data like this reasonably well once they have direct access to it, instead of needing you to paste it in manually. -* **The Spec:** Defined by XML. -* **The Hardware:** Defined by JSON limits. -* **The Performance:** Defined by microsecond-level timing logs. +Without MCP, you're the one copying data into the chat window. With a registry or hardware-limits tool wired up, you can ask a more specific question — "what's the max descriptor count on an Adreno 750, and does our budget fit?" — and let the agent go look it up itself. -AI models are highly efficient at reasoning about this structured data when they have direct access to it. Without MCP, you are the middleman, manually copy-pasting data into a chat window. With Advanced MCP, you tell the AI the goal ("Optimize our shadow map resolution for an Adreno 750"), and the AI **goes and gets the data itself.** +== What's in this section -== The Automation Roadmap +1. **xref:AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc[The Vulkan Registry & GPUInfo MCP]:** Giving your assistant real-time access to the spec and hardware limits. +2. **xref:AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc[Agentic Automation & Audits]:** Using an agent to help analyze CI/CD failures and look for inefficiencies in your command buffer recording logic. -In this section, we will move from "Consumer" to "Creator" of AI tools: - -1. **xref:AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc[The Vulkan Registry & GPUInfo MCP]:** How to provide your assistant with real-time access to the official spec and hardware limits. -2. **xref:AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc[Agentic Automation & Audits]:** Using AI to autonomously analyze CI/CD failures and perform "Performance Scans" on your command buffer recording logic. - -By the end of this journey, you will have built a suite of custom tools that make your AI assistant an expert not just in Vulkan, but in your specific hardware ecosystem and engineering pipeline. +By the end, you'll have a small set of custom tools that make your AI assistant more useful for your specific hardware targets and build pipeline, not just generic Vulkan questions. Next: xref:AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc[The Vulkan Registry & GPUInfo MCP] diff --git a/en/AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc b/en/AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc index f4956677..79739ae9 100644 --- a/en/AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc +++ b/en/AI_Assisted_Vulkan/07_advanced_mcp/02_vulkan_registry_mcp.adoc @@ -2,27 +2,25 @@ = The Vulkan Registry & GPUInfo MCP -== Introduction: The Ground Truth +== Introduction: Working from the source data -Vulkan is a data-driven API. Its structure, enums, and extensions are all defined in a central XML file: the **Vulkan Registry (`vk.xml`)**. Similarly, the capabilities of thousands of different GPUs are tracked in the **Vulkan Hardware Database (`gpuinfo.org`)**. +Vulkan is a data-driven API. Its structs, enums, and extensions are all defined in a central XML file, the **Vulkan Registry (`vk.xml`)**. Similarly, the capabilities of thousands of GPUs are tracked in the **Vulkan Hardware Database (`gpuinfo.org`)**. -In **Collaborative Engineering**, we give our AI direct access to these datasets. We move from "Probabilistic AI" (which guesses based on training data) to **"Deterministic AI"** (which queries the ground truth). This chapter explores how to use existing tools for the registry and how to **build your own** specialized MCP server to bridge your assistant to the `gpuinfo.org` database. +Giving your AI direct access to these sources means it can look up an answer instead of guessing from training data, which matters most for anything added or changed recently. This chapter covers using an existing MCP server for the registry, and building your own small connector for `gpuinfo.org`. -== The Registry Interrogation with mcp-Vulkan +== Querying the registry with mcp-Vulkan -The **mcp-Vulkan** project is a specialized MCP server designed to "interrogate" the `vk.xml` file. It provides your AI agent with tools like `search-vulkan-docs` and `get-vulkan-topic`. +The **mcp-Vulkan** project is an MCP server built around `vk.xml`. It exposes tools like `search-vulkan-docs` and `get-vulkan-topic` to your AI agent. -Consider a case where you are implementing a brand-new extension, such as `VK_KHR_maintenance7`. If your AI model was trained before this extension was released, it might hallucinate structure names or flags. By providing the AI with the `search-vulkan-docs` tool via mcp-Vulkan, you can ask it to query the latest spec. The AI calls the server, scans the registry, and returns the exact structure names—such as `VkPhysicalDeviceMaintenance7FeaturesKHR`—providing a 100% spec-accurate response. +This matters most for extensions released after a model's training cutoff. If you're implementing something like `VK_KHR_maintenance7` and the model wasn't trained on it, it may guess at structure or flag names. With `search-vulkan-docs` wired up via mcp-Vulkan, the AI can query the current spec directly and return the actual structure name — e.g., `VkPhysicalDeviceMaintenance7FeaturesKHR` — instead of inventing something plausible-looking. -== Tutorial: Building the GPUInfo MCP Server +== Building a GPUInfo MCP server -While `mcp-Vulkan` handles the official specification, connecting your AI to the **Vulkan Hardware Database (`gpuinfo.org`)** requires a custom "Connector" MCP server. Since `gpuinfo.org` is a community-driven web resource, we can build a simple Python-based MCP server that allows our assistant to "search" the database for specific hardware limits. +`mcp-Vulkan` covers the official spec, but connecting to the **Vulkan Hardware Database (`gpuinfo.org`)** needs a small custom server, since it's a community-run site rather than a structured API. Below is a minimal Python MCP server that gives your assistant a `query_gpu_limit` tool for checking hardware constraints during design. -In this tutorial, we will create a **GPUInfo Connector** using the Python MCP SDK. This server will provide a tool called `query_gpu_limit` that our AI can use to check hardware constraints during the architectural design phase. +=== Step 1: Setting up the environment -=== Step 1: Setting up the Environment - -First, create a new directory for your MCP server and install the necessary dependencies. You will need the `mcp` library for the protocol and `requests` for fetching data from the web. +Create a new directory for your MCP server and install the dependencies: `mcp` for the protocol, and `requests` for fetching data. [source,bash] ---- @@ -32,9 +30,9 @@ source venv/bin/activate pip install mcp requests ---- -=== Step 2: Defining the Server Logic +=== Step 2: Defining the server logic -Create a file named `server.py`. We will initialize an `McpServer` and define a tool that takes a GPU name and a specific limit (like `maxDescriptorSetSampledImages`) and returns the value from the database. +Create a file named `server.py`. It initializes an `McpServer` and defines a tool that takes a GPU name and a limit name (like `maxDescriptorSetSampledImages`) and returns the corresponding value. [source,python] ---- @@ -62,9 +60,9 @@ if __name__ == "__main__": server.run() ---- -=== Step 3: Registering with Your AI Agent +=== Step 3: Registering with your AI agent -To use this tool, add the server to your agent's configuration. If you are using **Goose**, add the following entry to your `~/.config/goose/config.yaml` file: +To use this tool, add the server to your agent's configuration. If you're using **Goose**, add this to `~/.config/goose/config.yaml`: [source,yaml] ---- @@ -74,14 +72,14 @@ extensions: args: ["/path/to/gpuinfo-mcp/server.py"] ---- -== Real-World Experience: Hardware-Aware System Design +== Example: checking a bindless design against hardware limits -Imagine you are architecting a new **Bindless Rendering** system. You ask your assistant: *"What is the `maxDescriptorSetSampledImages` limit for the Adreno 600 series, and will my 1024-texture strategy work?"* +Say you're designing a bindless rendering setup and ask your assistant: "What's the `maxDescriptorSetSampledImages` limit on the Adreno 600 series, and does a 1024-texture bindless array fit?" -With the GPUInfo MCP server connected, your assistant doesn't guess. It uses the `query_gpu_limit` tool, identifies the limit (which is 128 on many Adreno 600 devices), and immediately reports that your plan will fail. It then suggests a refactored architecture—such as a 'Tiled' descriptor strategy or a mobile-specific fallback—that is compatible with your target hardware. +With the GPUInfo MCP server connected, the assistant can call `query_gpu_limit` and report the actual figure (128 on many Adreno 600 devices) instead of guessing — which in this case would mean your 1024-texture plan doesn't fit that hardware. From there it can suggest an alternative, like a tiled descriptor strategy or a mobile-specific fallback path, though it's worth checking any such suggestion against your actual constraints before committing to it. -== Summary: Data-Driven Engineering +== Summary -By connecting your AI to the **Vulkan Registry** and building a custom **GPUInfo Connector**, you transform it into a **Hardware-Aware Assistant**. It no longer relies on static training data; it uses the same "Live Source of Truth" that professional graphics engineers use to design high-performance, cross-platform engines. +Connecting your AI to the Vulkan Registry and a GPUInfo lookup tool means it's working from the same live data professional graphics engineers check by hand, rather than from whatever it happened to see in training. It still won't catch everything, and it's still worth verifying anything load-bearing, but it cuts out a lot of the manual copy-paste lookup work. Next: xref:AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc[Agentic Automation & Audits] diff --git a/en/AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc b/en/AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc index 7b2b8789..21f0d2ee 100644 --- a/en/AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc +++ b/en/AI_Assisted_Vulkan/07_advanced_mcp/03_agentic_automation_qa.adoc @@ -2,33 +2,36 @@ = Agentic Automation & Performance Audits -== Introduction: Automated System Verification +== Introduction: Automated system verification -Vulkan development is a marathon of correctness and performance. In the final stage of our AI-assisted journey, we move from "Coding" and "Debugging" to **Autonomous Maintenance**. This is where we use **AI Agents** (like **Goose**) to perform high-level tasks that span your entire engine, ensuring that your system remains robust as it grows. +Vulkan development involves a lot of ongoing correctness and performance work, not just the initial implementation. This section covers using AI agents (like **Goose**) for higher-level, cross-file tasks — the kind that involve tracing something across your whole engine rather than a single file open in the IDE. -In **Collaborative Engineering**, the AI isn't just a helper; it is a **verification tool** that identifies the subtle inefficiencies and "Headless" failures that are often invisible to a human developer working on a single file. +An agent working this way can help surface inefficiencies and hard-to-reproduce failures that are easy to miss when you're focused on one file at a time. It's not infallible, and its suggestions still need review, but it can do the tedious cross-referencing work faster than doing it by hand. -== The Core Concept: Systemic Analysis +== What's different about an autonomous agent here -Unlike an IDE assistant, an autonomous agent can perform **Systemic Analysis**. It can hold the state of your build system, your test logs, and your source code in its context simultaneously. This enables it to perform **CI/CD Log Diagnosis**, where the model reads the cryptic logs from nightly "Headless" builds and identifies the root cause of failures on specific hardware. +Unlike an IDE assistant answering questions about the file you have open, an autonomous agent can hold your build system state, test logs, and source code together in context at once. That makes it useful for two kinds of tasks: -Simultaneously, it can perform **Performance Inefficiency Scans**, looking at your entire command buffer recording logic to find redundant state changes or synchronization barriers that are too broad for optimal GPU overlap. +- **Reading CI/CD logs.** Nightly "headless" build logs can be long and hard to parse by eye. An agent can go through them and try to identify the actual cause of a failure on specific hardware. +- **Scanning for inefficiencies.** It can look across your command buffer recording logic for redundant state changes or synchronization barriers that are broader than they need to be. -== Real-World Experience: The "Nightly" Fix +Treat its output as a starting point for investigation, not a verified diagnosis — check the specific claim against the log or the code before acting on it. -Vulkan engines often have automated test suites that run on remote servers. When these tests fail on a specific hardware farm (e.g., a "Mali GPU Farm"), the logs can be tens of thousands of lines long, making manual debugging difficult. +== Example: tracking down a nightly test failure -The agentic workflow begins with a notification that a test has failed. You provide the agent with a high-level command to analyze the logs and suggest a fix. In this scenario, the agent uses the **Context Bridge** to check the hardware limits of the specific device. +Vulkan engines often run automated test suites on remote hardware farms. When a test fails on specific hardware (say, a Mali GPU runner), the logs can run tens of thousands of lines, which makes manual debugging slow. -**The Forensic Discovery:** During its analysis, the agent identifies that the Adreno 750 runner is reporting a `maxComputeWorkGroupInvocations` limit exceeded. It realizes your Bloom shader is hardcoded to 512 threads, but this device only supports 256. It then automatically refactors the shader to use a specialization constant for the workgroup size, updates the C{pp} to set this constant at runtime, and triggers a new CI build to verify the fix. +A typical workflow: a nightly run fails, you point the agent at the log and ask it to investigate, and it uses your Context Bridge setup to check the hardware limits of the specific device that failed. -== Tutorial: Integrating the Autonomous Build Guard +In one case, the agent found that the failing runner reported a `maxComputeWorkGroupInvocations` limit that a bloom shader exceeded — the shader was hardcoded to 512 threads per group, but the device only supports 256. It proposed refactoring the shader to use a specialization constant for the workgroup size, updated the C{pp} side to set that constant at runtime, and triggered a new CI build to check the fix. This kind of result is worth double-checking rather than merging blind, but it saved the time of manually diffing hardware limit tables against shader constants. -Moving beyond the developer's local workstation, agentic automation can be integrated directly into your CI/CD pipeline. By using **Goose** in a "Headless" environment (like a GitHub Actions runner), you can automate the diagnosis of build failures and the resolution of complex merge conflicts. +== Running an agent in CI + +Agentic automation doesn't have to stay on your local machine — you can run it in your CI/CD pipeline too. Using **Goose** in a headless environment (like a GitHub Actions runner), you can automate a first pass at diagnosing build failures. === Configuring the GitHub Action -To run Goose in CI, we use a standard Linux runner and provide it with the necessary API keys and environment context. The following configuration demonstrates how to trigger an agentic diagnosis upon a build failure. +This workflow triggers an agent-based diagnosis when your build job fails: [source,yaml] ---- @@ -59,9 +62,9 @@ jobs: ./scripts/agentic_ci_diagnosis.sh ---- -=== The Diagnosis Script +=== The diagnosis script -The script `agentic_ci_diagnosis.sh` provides the agent with the necessary context and a clear instruction to resolve the issue. In this scenario, we use the build log and the pull request diff to identify the cause of the failure. +`agentic_ci_diagnosis.sh` gives the agent the build log and the PR diff, and asks it to identify the cause of the failure: [source,bash] ---- @@ -87,14 +90,13 @@ Vulkan validation error (VUID), propose a fix in a new branch " ---- -=== Native Platform Intelligence: GitHub Copilot Autofix +Treat the resulting PR comment or branch as a proposed fix to review, not something to auto-merge — that's true of any agent-generated change, but especially one produced from a truncated log excerpt. -=== Tutorial: Enabling the Guardian +=== GitHub Copilot Autofix -To enable **GitHub Copilot Autofix** for your Vulkan project, follow these steps: +Separately from a Goose-based CI step, **GitHub Copilot Autofix** can catch a different category of issue: static-analysis findings from CodeQL, such as an unchecked `vkMapMemory` result or a potential buffer overflow in a descriptor update. -1. **Configure Code Scanning:** - Create a new GitHub Actions workflow file at `.github/workflows/codeql.yml`. This will run the **CodeQL** static analysis engine, which provides the "Alerts" that Copilot uses to generate fixes. +1. **Configure code scanning.** Add a GitHub Actions workflow at `.github/workflows/codeql.yml` to run CodeQL, which produces the alerts Copilot Autofix works from. + [source,yaml] ---- @@ -124,17 +126,15 @@ jobs: uses: github/codeql-action/analyze@v3 ---- -2. **Enable Autofix in Repository Settings:** - Navigate to your repository on GitHub. Go to **Settings > Code security and analysis**. Ensure that **GitHub Copilot Autofix** is toggled to **Enabled**. +2. **Enable Autofix in repository settings.** Go to **Settings > Code security and analysis** and toggle **GitHub Copilot Autofix** on. -3. **Reviewing a Fix:** - When a Pull Request is created, CodeQL will scan your code. If it finds a vulnerability—such as an unchecked `vkMapMemory` result or a potential buffer overflow in a descriptor update—Copilot will append a "Propose a fix" section to the security alert. Click the **"Fix with Copilot"** button to review the suggested code change and commit it directly to your branch. +3. **Review a suggested fix.** When CodeQL flags something on a pull request, Copilot Autofix appends a "Propose a fix" section to the alert. Click **"Fix with Copilot"** to review the suggested change before committing it to your branch — review it the same way you would any other suggested patch. -This represents a different level of the **Collaborative Engineering** paradigm. While **Goose** is your proactive "Agent" that can handle complex, multi-file architectural refactors, **Copilot Autofix** is your reactive "Guardian" that ensures the basic safety and correctness of your code before it ever reaches the review phase. +Goose in CI and Copilot Autofix cover different ground: Goose is better suited to larger, multi-file changes you explicitly ask it to investigate, while Autofix reacts to specific static-analysis alerts as they come up. Neither replaces normal code review. -=== Semantic Merge Conflict Resolution +=== Semantic merge conflict resolution -Merge conflicts in complex graphics engines often involve changes to synchronization logic or resource management that simple text-based merge tools cannot resolve safely. An agent can analyze the semantic intent of both branches and integrate them without introducing race conditions. This can be used to resolve existing conflicts or to audit a pull request for "Hidden" semantic conflicts that git might miss. +Merge conflicts in a graphics engine often touch synchronization logic or resource management in ways a text-based merge tool can't reason about safely. An agent can look at what each branch is actually trying to do and propose a merge that keeps both changes intact — though as with any merge, it's worth checking the result for race conditions or ordering issues the agent may have missed. [source,bash] ---- @@ -148,14 +148,15 @@ Descriptor Buffer paths correctly use the updated Command Pool lifecycle. " ---- -You can also task the agent to proactively check if a merge will cause systemic issues, such as pipeline incompatibilities or synchronization hazards, before the merge is even attempted. +You can also ask an agent to check, before merging, whether two branches are likely to conflict at a semantic level — pipeline incompatibilities or synchronization hazards that wouldn't show up as a textual conflict. + +== Running a GPU state audit -== Tutorial: Running a GPU State Audit +A performance audit here means looking for redundant work, not correctness bugs. You can pair **Goose** with the **RenderDoc CLI (`rdc-cli`)** to automate a first pass over a captured frame. -A "Performance Audit" is not about finding bugs; it is about finding **Waste**. To automate this, we can use **Goose** in conjunction with the **RenderDoc CLI (`rdc-cli`)**. +=== Step 1: Capture and export -=== Step 1: Capture and Export -First, generate a RenderDoc capture (`.rdc`) of your frame. Then, use a script to export the command list to a text format that the agent can "read". +Generate a RenderDoc capture (`.rdc`) of your frame, then export the command list to a text format the agent can read: [source,bash] ---- @@ -163,8 +164,9 @@ First, generate a RenderDoc capture (`.rdc`) of your frame. Then, use a script t rdc-cli replay --cmds --pipeline-state output_state.txt capture_frame.rdc ---- -=== Step 2: Commanding the Audit -Point **Goose** at the exported `output_state.txt` and provide a specific architectural goal. +=== Step 2: Running the audit + +Point Goose at `output_state.txt` with a specific question: [source,text] ---- @@ -175,15 +177,16 @@ Also, identify any 'vkCmdPipelineBarrier' calls that use alternatives for a depth-only pass. ---- -=== Step 3: The AI Report and Fix -In its audit, the agent might identify that you are binding the 'Shadow' pipeline three times per frame because your draw call sorting is grouped by material rather than pipeline state. It recommends refactoring your `DrawBucket` to prioritize pipeline sorting. Additionally, it notices that your barrier for the Depth Buffer uses `BOTTOM_OF_PIPE_BIT`, which is causing a full GPU stall. It suggests narrowing the scope to `LATE_FRAGMENT_TESTS_BIT` to allow the next pass to begin its vertex work early. +=== Step 3: Reviewing the findings + +In one run, the agent found the shadow pipeline was being bound three times per frame because draw calls were sorted by material rather than pipeline state, and recommended reordering the `DrawBucket` sort to group by pipeline first. It also flagged a depth-buffer barrier using `BOTTOM_OF_PIPE_BIT` that was stalling the GPU, and suggested narrowing it to `LATE_FRAGMENT_TESTS_BIT` so the next pass could start its vertex work earlier. Both are worth confirming against a profiler before assuming they're correct, but they're reasonable starting points for a specific investigation you'd otherwise do by scanning the state dump manually. -Once the report is generated, you can ask Goose to **apply the refactor** directly to your `Renderer.cpp` file. +Once you've reviewed the findings, you can ask Goose to apply the refactor directly to `Renderer.cpp`. -== Summary: The Pro-Level Partner +== Summary -Agentic automation is the ultimate goal of the AI-assisted workflow. By using AI as an autonomous verification tool, you ensure that your engine remains both **Correct** and **High-Performance**. You focus on the creative "Vision" of your engine, while the AI manages the data-heavy tasks required to keep it running smoothly across thousands of hardware configurations. +Agent-driven automation is useful for the data-heavy, cross-file verification work that's tedious to do manually: reading long CI logs, scanning command buffer state for waste, or checking a design against hardware limits. It doesn't replace review, and its output should be checked rather than merged on trust, but it can meaningfully cut down the time spent on that kind of lookup and cross-referencing across a large engine. -In the final section of our tutorial, we move to **Deployment**, where we explore how to bring these AI-assisted workflows to Android, iOS, and Embedded systems. +In the next section, we move to deployment, covering how these AI-assisted workflows apply to Android, iOS, and embedded targets. Next: xref:AI_Assisted_Vulkan/08_deployment/01_introduction.adoc[Desktop, Mobile, and Embedded Deployment] diff --git a/en/AI_Assisted_Vulkan/08_deployment/01_introduction.adoc b/en/AI_Assisted_Vulkan/08_deployment/01_introduction.adoc index 2cd4e67d..b26ab5c8 100644 --- a/en/AI_Assisted_Vulkan/08_deployment/01_introduction.adoc +++ b/en/AI_Assisted_Vulkan/08_deployment/01_introduction.adoc @@ -4,31 +4,27 @@ == Introduction: The Multi-Platform Chain -Vulkan's greatest strength is its portability. A well-written application can run on a high-end Windows desktop, a Samsung smartphone, and a Raspberry Pi. However, each of these platforms has its own "Laws" when it comes to memory management, asset loading, and performance optimization. +Vulkan's greatest strength is its portability. A well-written application can run on a high-end Windows desktop, a Samsung smartphone, and a Raspberry Pi. Each of these platforms comes with its own constraints around memory management, asset loading, and performance optimization. -In **Collaborative Engineering**, we use our assistant as a **Platform-Aware Specialist**. We use AI to bridge the "Context Gap" between platforms, automating the tedious JNI boilerplate for mobile, suggesting optimizations for tiled GPUs, and managing the strict resource constraints of embedded systems. +An AI assistant is useful here mainly because it can be given those platform constraints up front and hold onto them: it can help write the tedious JNI boilerplate for mobile, suggest optimizations suited to tiled GPUs, and flag when you're about to exceed the resource limits of an embedded target. -== The Core Concept: Platform Context +== Giving the assistant platform context -A "Platform-Aware" assistant is an AI that has been provided with the specific hardware constraints of your target. By using the **Context Bridge**, we inform the AI of the "Target Reality": +The pattern in this chapter is to hand your assistant the specific hardware constraints of your target before asking it for code, using the same MCP/RAG setup covered earlier in this series: -1. **Mobile Context:** The AI understands **Tile-Based Deferred Rendering (TBDR)**. It prioritizes subpass dependencies and memory-less attachments to maximize on-chip tile memory performance. -2. **Desktop Context:** The AI understands the performance trade-offs of different vendor extensions. It knows when to use `VK_NV_ray_tracing` versus the cross-vendor `VK_KHR_ray_tracing_pipeline`. -3. **Embedded Context:** The AI understands strict VRAM and CPU budgets. It suggests quantization strategies for shaders and textures to fit within architectures that lack traditional paging. +1. **Mobile:** Point it at Tile-Based Deferred Rendering (TBDR). Once it has that context, it's better at prioritizing subpass dependencies and memory-less attachments to keep data in on-chip tile memory. +2. **Desktop:** Give it the relevant vendor extension tradeoffs so it can tell you, for example, when `VK_NV_ray_tracing` is worth using over the cross-vendor `VK_KHR_ray_tracing_pipeline`. +3. **Embedded:** Give it your VRAM and CPU budget so it can suggest quantization strategies for shaders and textures that actually fit your target, instead of generic advice. -== Why Use AI for Deployment? +== Why bother with AI here specifically -Deployment is where "Syntactic Noise" becomes a blocker to portability. -* **Android/iOS:** You must write hundreds of lines of "Glue Code" (JNI or Objective-C{pp}) just to get a window surface and an asset loader running. -* **The AI Solution:** You describe the asset structure of your engine, and the AI generates the complete multi-language bridge, ensuring that your C{pp} Vulkan core remains clean and platform-agnostic. +Deployment code is mostly glue: hundreds of lines of JNI or Objective-C{pp} just to get a window surface and asset loader running on a given platform. That's tedious to write by hand and easy for an AI assistant to draft correctly once you've described your engine's asset structure and target platform — you still need to review it, since it's exactly the kind of platform-specific code where a subtle mistake (wrong lifetime, wrong threading assumption) won't show up until you're on-device. -== The Deployment Roadmap +== What's next -In this final section, we move from the "Lab" to the "World": +1. **xref:AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc[Android & iOS]:** Automating JNI boilerplate and optimizing for tiled GPU architectures. +2. **xref:AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc[Embedded & Safety-Critical Systems]:** Working within strict hardware limits and memory-constrained environments. -1. **xref:AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc[Android & iOS: The Mobile Frontier]:** Automating JNI boilerplate and optimizing for Tiled GPU architectures. -2. **xref:AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc[Embedded & Safety Critical Systems]:** Managing strict hardware limits and memory-constrained environments. - -By the end of this journey, you will be able to take your Vulkan application from a single-platform prototype to a globally portable, high-performance engine that runs optimally on every device it touches. +By the end of this section, you should be able to take a Vulkan application from a single-platform prototype to something that runs reasonably well across desktop, mobile, and embedded targets. Next: xref:AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc[Android & iOS: The Mobile Frontier] diff --git a/en/AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc b/en/AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc index ff222b6f..56b9570a 100644 --- a/en/AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc +++ b/en/AI_Assisted_Vulkan/08_deployment/02_android_ios_mobile.adoc @@ -1,27 +1,24 @@ :pp: {plus}{plus} -= Android & iOS: The Mobile Frontier += Android & iOS Deployment -== Introduction: The Bandwidth Battle +== Introduction -Mobile development is the ultimate test of a Vulkan engine. You must balance high-performance rendering with strict battery limits, thermally constrained GPUs, and the complexities of multi-language "Glue Code." +Mobile Vulkan development means balancing high-performance rendering against battery limits, thermally throttled GPUs, and a fair amount of platform glue code. This chapter covers using AI to draft that glue code and to help optimize your rendering pipeline for Tile-Based Deferred Renderers (TBDR). -In **Collaborative Engineering**, we use AI to handle the "Syntactic Noise" of platform integration and to act as a **Bandwidth Analysis Assistant** for mobile GPU architectures. This chapter explores how to automate JNI boilerplate and optimize your rendering pipeline for **Tile-Based Deferred Renderers (TBDR)**. +== The glue code problem -== The Core Concept: The "Glue" Engineer - -Mobile development requires hundreds of lines of code before you even reach your first `vkCreateInstance` call. Whether you are on Android or iOS, you must manage the lifecycle of the application and the "Glue Code" that bridges the platform's native language to your C{pp} core. +Before you reach your first `vkCreateInstance` call on mobile, you need platform-specific setup: lifecycle handling and the bridge code between the platform's native language and your C{pp} core. It's mechanical, well-documented, and exactly the kind of code an AI assistant can draft quickly — you still need to read through what it generates, since a wrong assumption about surface lifetime or threading here can be hard to debug later. == Tutorial: Building the Mobile Entry Point -To make your Vulkan engine cross-platform, you must implement the platform-specific "Glue Code" that provides a native surface to your C{pp} core. Follow these steps to automate this process. +=== Step 1: The JNI Bridge (Android) -=== Step 1: Automating the JNI Bridge (Android) -Create a new file `VulkanEntry.cpp` and use your AI assistant (Goose or IDE Chat) to generate the JNI boilerplate. +Create a new file `VulkanEntry.cpp` and ask your assistant (Goose or your IDE's chat) to generate the JNI boilerplate. [source,cpp] ---- -// Example JNI bridge generated by AI for surface management +// Example JNI bridge for surface management #include #include #include "VulkanRenderer.hpp" @@ -33,10 +30,11 @@ Java_com_example_vulkan_MainActivity_onSurfaceCreated(JNIEnv* env, jobject thiz, } ---- -**Commanding the AI:** *"Goose, generate the full `VulkanEntry.cpp` JNI bridge. We need to handle `onSurfaceCreated`, `onSurfaceChanged`, and `onSurfaceDestroyed`. Ensure we use `ANativeWindow_fromSurface` and pass it to our renderer's `initSurface` method."* +**Example prompt:** *"Generate the full `VulkanEntry.cpp` JNI bridge. We need to handle `onSurfaceCreated`, `onSurfaceChanged`, and `onSurfaceDestroyed`. Use `ANativeWindow_fromSurface` and pass it to our renderer's `initSurface` method."* === Step 2: The iOS SwiftUI Bridge -On iOS, you must wrap your Vulkan surface in a SwiftUI-compatible view. Task the AI to generate the `UIViewRepresentable` bridge. + +On iOS, you need to wrap your Vulkan surface in a SwiftUI-compatible view. Ask the assistant to generate the `UIViewRepresentable` bridge. [source,swift] ---- @@ -49,7 +47,7 @@ struct VulkanView: UIViewRepresentable { let view = UIView() let metalLayer = CAMetalLayer() view.layer.addSublayer(metalLayer) - // AI will add the logic to pass this layer to C++ here + // Logic to pass this layer to C++ goes here return view } @@ -57,18 +55,19 @@ struct VulkanView: UIViewRepresentable { } ---- -**Commanding the AI:** *"Generate a SwiftUI `UIViewRepresentable` that provides a `CAMetalLayer` for our Vulkan engine. We need to support both MoltenVK and KosmicKrisp, so ensure the bridge is flexible enough to handle different native handle types."* +**Example prompt:** *"Generate a SwiftUI `UIViewRepresentable` that provides a `CAMetalLayer` for our Vulkan engine. We need to support both MoltenVK and KosmicKrisp, so the bridge should handle different native handle types."* == Tutorial: Optimizing for Tile Memory (TBDR) -Unlike desktop GPUs, mobile GPUs (Adreno, Mali, Apple) are **Tiled**. To maximize performance, you must use **Subpasses** to keep your G-Buffer data on-chip. +Unlike desktop GPUs, mobile GPUs (Adreno, Mali, Apple) are tile-based. To get good performance, you generally want to use subpasses to keep G-Buffer data on-chip rather than round-tripping it through VRAM. === Step 1: The Subpass Refactor -If your engine currently uses separate render passes for G-Buffer and Lighting, ask the AI to refactor them into a single pass with subpasses. + +If your engine currently uses separate render passes for G-Buffer and lighting, ask the assistant to combine them into a single pass with subpasses. [source,text] ---- -Goose, refactor our 'DeferredRenderer::createRenderPass' method. +Refactor our 'DeferredRenderer::createRenderPass' method. Combine the G-Buffer and Lighting passes into a single 'VkRenderPass' with two subpasses. Use 'subpassLoad' in the fragment shader to read the G-Buffer data directly @@ -76,11 +75,12 @@ from tile memory. ---- === Step 2: The Dependency Logic -The AI will generate the `VkSubpassDependency` structures needed to ensure the GPU doesn't write the tile data to VRAM prematurely. + +Have it generate the `VkSubpassDependency` structures needed so the GPU doesn't write tile data out to VRAM prematurely. [source,cpp] ---- -// AI-generated subpass dependency for on-chip data flow +// Subpass dependency for on-chip data flow VkSubpassDependency dependency = {}; dependency.srcSubpass = 0; // G-Buffer dependency.dstSubpass = 1; // Lighting @@ -91,10 +91,10 @@ dependency.dstAccessMask = VK_ACCESS_INPUT_ATTACHMENT_READ_BIT; dependency.dependencyFlags = VK_DEPENDENCY_BY_REGION_BIT; ---- -By using this "On-Chip" data flow, you reduce VRAM bandwidth by up to 70%, significantly increasing battery life and frame rate on mobile devices. +Keeping this data on-chip instead of round-tripping through VRAM can meaningfully cut memory bandwidth, which matters for both frame rate and battery life on mobile. -== Summary: Battery-Efficient Graphics +== Summary -Mobile Vulkan is about **Bandwidth Management**. By using AI to automate the JNI "Glue" and optimize your subpass dependencies for Tiled GPUs, you ensure that your engine remains fast and battery-efficient. You move from being a "C{pp} Coder" to a **Mobile Graphics Engineer**, managing the complex interplay between the platform's native code and the GPU's hardware architecture. +Mobile Vulkan work is largely about managing bandwidth and platform glue. Using AI to draft the JNI/Objective-C{pp} bridge code and to help restructure render passes around tile memory can save real time — but the review step matters more here than in most other parts of the engine, since platform-specific bugs are often invisible until you're running on the actual device. Next: xref:AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc[Embedded & Safety Critical Systems] diff --git a/en/AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc b/en/AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc index 16e58ae4..2a543de7 100644 --- a/en/AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc +++ b/en/AI_Assisted_Vulkan/08_deployment/03_embedded_safety_critical.adoc @@ -4,16 +4,17 @@ == Introduction: Managing the Minimums -Embedded systems—like the Raspberry Pi, NVIDIA Jetson, or custom SoC (System on a Chip) hardware—represent the most constrained environments in the Vulkan ecosystem. In these systems, you are often limited by a single gigabyte of shared memory, and your CPU is a fraction of the speed of a desktop processor. In this environment, "Close Enough" is not an option. +Embedded systems — the Raspberry Pi, NVIDIA Jetson boards, or custom SoC hardware — are the most constrained environments in the Vulkan ecosystem. You're often working with a single gigabyte of shared memory and a CPU that's a fraction of the speed of a desktop processor. There's little room for approximate answers here. -In **Collaborative Engineering**, we use our assistant as a **Safety Verification Tool**. We use AI to continuously check our resource allocation against hard hardware limits and to suggest the aggressive quantization needed to fit modern graphics into a "Small" silicon footprint. +An AI assistant can help by continuously checking your resource allocation against hard hardware limits and by suggesting the aggressive quantization needed to fit modern graphics techniques into a small silicon footprint. Treat its output as a starting point to verify against your actual device profile, not as a guarantee of correctness. == Tutorial: Managing the Embedded Memory Map -Unlike desktop development, embedded development is about **hard limits**. Use your AI assistant to audit your resource allocation against the specific constraints of your SoC. +Embedded development is about hard limits rather than reasonable defaults. Use your assistant to audit your resource allocation against the specific constraints of your SoC. === Step 1: Loading the Hardware Profile -Feed the AI your device's memory map and the requirements for your target resolution. + +Feed the assistant your device's memory map and the requirements for your target resolution. [source,text] ---- @@ -27,39 +28,42 @@ for the swapchain and our G-Buffer? ---- === Step 2: Optimizing the Allocation -When the AI reports the audit results, ask for a refactored allocation strategy that minimizes "Padding Waste." + +Once you have the audit results, ask for a refactored allocation strategy that reduces wasted padding. [source,cpp] ---- -// Example of an AI-suggested memory allocation for embedded +// Example memory allocation for embedded targets VkMemoryAllocateInfo allocInfo = {}; allocInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; allocInfo.allocationSize = reclaimedSize; // Optimized size -// AI will suggest specific memory types for the Jetson's unified memory +// The assistant can suggest specific memory types for the Jetson's unified memory allocInfo.memoryTypeIndex = unifiedMemoryIndex; ---- == Tutorial: Shader Quantization for Mali/Adreno -For embedded systems, the precision of your shaders has a direct impact on power consumption. Downgrading variables to `mediump` (FP16) can increase performance significantly. +On embedded systems, shader precision has a direct effect on power consumption. Downgrading variables to `mediump` (FP16) can noticeably improve performance, though it comes with a real risk of visible artifacts if applied carelessly. === Step 1: The Precision Audit -Provide the AI with your fragment shader and ask it to identify safe candidates for quantization. + +Give the assistant your fragment shader and ask it to identify candidates for quantization, then check its suggestions against your own visual testing rather than taking them on faith. [source,text] ---- -Goose, audit 'PBR.frag' for a Mali-G710 GPU. +Audit 'PBR.frag' for a Mali-G710 GPU. Which variables can we safely downgrade to 'mediump' (FP16) without causing visible artifacts in our specular reflections or normal mapping? ---- === Step 2: Applying the Quantization -The AI will provide the refactored shader. By switching to `mediump` for normalization and intermediate lighting steps, you reduce register pressure, allowing more threads to run in parallel. + +Review the refactored shader it proposes. Switching to `mediump` for normalization and intermediate lighting steps reduces register pressure, which allows more threads to run in parallel — but verify the visual result before committing to it. [source,glsl] ---- -// AI-optimized fragment shader for embedded +// Fragment shader using FP16 intermediates for embedded targets layout(location = 0) out vec4 outColor; void main() { mediump vec3 normal = normalize(inNormal); @@ -70,26 +74,28 @@ void main() { == Tutorial: Vulkan SC Compliance Check -In automotive and industrial environments, we use **Vulkan SC**. This requires every resource to be pre-allocated. +In automotive and industrial environments, Vulkan SC requires every resource to be pre-allocated ahead of time. This is a case where you should treat AI output as a first pass, not a certification — actual SC compliance needs to be verified against the spec and your certification process, not just an assistant's opinion. === Step 1: The Compliance Audit -Feed your initialization sequence to the AI and ask it to check for SC 1.0 violations. + +Feed your initialization sequence to the assistant and ask it to check for SC 1.0 violations. [source,text] ---- -Goose, audit 'VulkanContext::initializeSC'. +Audit 'VulkanContext::initializeSC'. Check if our 'PipelineCache' warming logic is compliant with the Vulkan SC 1.0 specification for offline compilation. Are we using any illegal dynamic state like 'VK_DYNAMIC_STATE_LINE_WIDTH'? ---- === Step 2: The SC Refactor -If the AI finds a violation (like dynamic line width), ask it to generate the "Pre-Baked" pipeline array needed to maintain compliance. -== Summary: The Precision Partner +If it finds a violation, such as dynamic line width, ask it to generate the pre-baked pipeline array needed to stay compliant, then confirm the result yourself against the spec. + +== Summary -Embedded deployment is about **Engineering for the Edge**. By using AI to audit your memory budgets and suggest quantization strategies, you can build a high-performance engine that runs on hardware that others would consider "Too Small." +Embedded deployment comes down to working within hard memory and CPU budgets. AI can help audit allocations and suggest quantization strategies, but this is one of the areas in this series where the assistant's output deserves the most scrutiny — the cost of an error is a device that doesn't work, not just a slower frame. -With the completion of Section 8, you have mastered the art of bringing your AI-assisted Vulkan engine to every corner of the computing world. In the final section of our tutorial, we will provide a comprehensive **FAQ** and a **Final Project** to solidify your skills as a Collaborative Engineer. +That covers desktop, mobile, and embedded deployment. The next section is a FAQ and a capstone project that ties the tools from this series together. Next: xref:AI_Assisted_Vulkan/09_faq.adoc[Common Questions & Troubleshooting (FAQ)] diff --git a/en/AI_Assisted_Vulkan/09_faq.adoc b/en/AI_Assisted_Vulkan/09_faq.adoc index 4c57f47b..277f8300 100644 --- a/en/AI_Assisted_Vulkan/09_faq.adoc +++ b/en/AI_Assisted_Vulkan/09_faq.adoc @@ -2,57 +2,48 @@ = Common Questions & Troubleshooting (FAQ) -== Introduction: The Collaborative Obstacle Course +== Introduction -As you begin your journey into AI-assisted Vulkan development, you will inevitably encounter obstacles. These aren't just technical bugs; they are the "Cultural" and "Privacy" hurdles of the **Collaborative Engineering** paradigm. This chapter addresses the most common concerns and provides troubleshooting tips for building a stable, secure, and highly effective AI-enhanced graphics lab. +As you work through AI-assisted Vulkan development, you'll run into more than just technical bugs — privacy concerns, hallucinated code, and hardware contention all show up in practice. This chapter covers the most common issues and how to work around them. == Privacy & IP Security **Q: "I am working on a proprietary rendering engine. How do I ensure my source code doesn't leak or get used to train other models?"** -**A: The 'Local Gating' Strategy.** -The only way to maximize privacy is to use **Local Inference Engines** (Ollama, LM Studio) as your primary daily drivers. By running models like **Qwen 3-Coder** on your own hardware, your shaders and C{pp} files never leave your machine. A common "Hybrid Rule" is to use local models for **Implementation** while reserving cloud models like Claude or GPT for high-level **System Design**, ensuring you use **API-based access** to prevent your data from being used for future training. +**A:** The most reliable way to keep code private is to run local inference (Ollama, LM Studio) as your daily driver. Running a model like Qwen 3-Coder on your own hardware means your shaders and C{pp} files never leave your machine. A common approach is to use local models for day-to-day implementation and reserve cloud models like Claude or GPT for higher-level system design, using API-based access (rather than a consumer chat product) so your data isn't used for training. == Accuracy & Hallucinations -**Q: "The AI keeps suggesting 'invented' extensions or legacy Vulkan 1.0 patterns. I'm spending more time fixing the AI's code than writing my own."** +**Q: "The AI keeps suggesting invented extensions or legacy Vulkan 1.0 patterns. I'm spending more time fixing the AI's code than writing my own."** -**A: Context is King.** -Hallucinations are almost always a symptom of a **Context Gap**. To resolve this, ensure you are using the **Model Context Protocol (MCP)** to link your assistant directly to the `vk.xml` registry; without this ground truth, the AI is effectively guessing. You should also explicitly start your session with a **Baseline Prompt** such as: *"We are using Vulkan 1.3 with Dynamic Rendering. Do not suggest `VkRenderPass` or `VkFramebuffer` unless I explicitly ask for legacy support."* For consistent results in complex projects, consider a tiny **LoRA adapter** to "default" the AI to your specific engine style and memory allocation patterns. +**A:** This is almost always a context problem — the model doesn't have the information it needs and is filling the gap with something plausible-sounding from its training data. Connect your assistant to the `vk.xml` registry via the Model Context Protocol (MCP) so it can check current struct and extension definitions instead of guessing. It also helps to start the session with an explicit baseline, e.g.: *"We are using Vulkan 1.3 with Dynamic Rendering. Do not suggest `VkRenderPass` or `VkFramebuffer` unless I explicitly ask for legacy support."* For projects with a strong house style, a small LoRA adapter can help the model default to your engine's conventions and allocation patterns — see xref:AI_Assisted_Vulkan/03_model_selection_specialization/05_fine_tuning_lora.adoc[Chapter 3.5]. == Multimodal & Vision Limits **Q: "I sent a screenshot of a subtle banding issue in my skybox, but the AI says the image is clear. Why can't it see the bug?"** -**A: The Resolution Compression.** -Most Multimodal models downscale images to a small "Vision Buffer" (often 1024x1024 or less). A subtle artifact that spans only a few pixels in a 4K frame can be completely smoothed out by this compression. The solution is to **Crop and Zoom**—before sending an image to your AI, crop the screenshot to focus entirely on the artifact. If the issue takes up 80% of the AI's vision buffer instead of 2%, the model will identify it much more reliably. +**A:** Most multimodal models downscale input images, often to 1024x1024 or less, before processing them. An artifact that only spans a few pixels in a 4K frame can simply get smoothed away by that downscaling. Crop the screenshot so the artifact fills most of the frame before sending it — if the issue takes up 80% of the image instead of 2%, the model is much more likely to catch it. == Hardware & VRAM Management **Q: "My Vulkan app keeps crashing with a 'Device Lost' (TDR) error when the AI starts generating code. Do I need a new GPU?"** -**A: The Resource Contention Trap.** -You are likely hitting a **Timeout Detection and Recovery (TDR)** event because both the AI and the renderer are competing for 100% of the GPU at once. To mitigate this, use a 4-bit (Q4_K_M) quantization to reduce the AI's VRAM footprint, and set a limit in your inference engine (like Ollama or LM Studio) on the number of GPU layers used by the AI to leave some compute headroom for your desktop and renderer. For professional use, the ultimate solution is a **Dedicated AI GPU**; even an older card in a secondary slot can host a 17B model like **Llama 4**, leaving your primary high-end card entirely free for Vulkan. +**A:** You're likely hitting a Timeout Detection and Recovery (TDR) event because the AI and your renderer are both competing for the full GPU at once. Try a 4-bit (Q4_K_M) quantization to shrink the model's VRAM footprint, and cap the number of GPU layers your inference engine (Ollama, LM Studio) offloads, leaving some headroom for the renderer. If you do a lot of this work, a second, older GPU dedicated to running the model (even something that can only host a 7-8B model) frees your primary card entirely for Vulkan. **Q: "I downloaded a model in Ollama, but it fails to run or crashes my system. What happened?"** -**A: The VRAM/GPU RAM Limit.** -Just because a model is downloaded doesn't mean it can run on your current hardware. If you attempt to load a model (like a 30B Qwen 3-Coder) that exceeds your available VRAM, Ollama may attempt to "offload" layers to your system RAM, which is significantly slower and can lead to a system-wide freeze or a "Device Lost" error in your Vulkan application. Always check the model size against your **VRAM Budgeting Formula** (from xref:AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc[Chapter 3.3]) before attempting to run it. +**A:** Downloading a model doesn't guarantee it fits in your VRAM. If you load something like a 30B model that exceeds available VRAM, Ollama may offload layers to system RAM, which is much slower and can cause a system-wide freeze or a Device Lost error in your Vulkan app. Check the model's size against your VRAM budget (see xref:AI_Assisted_Vulkan/03_model_selection_specialization/03_hardware_vram.adoc[Chapter 3.3]) before loading it. -**Q: "Should I use the Ollama WebUI (localhost) or the Windows Native App?"** +**Q: "Should I use the Ollama WebUI (localhost) or the native app?"** -**A: Native for Performance, WebUI for Management.** -While a WebUI (running on `localhost`) provides a familiar browser-based interface for managing models, the **Windows Native App** (or the CLI) is generally preferred for graphics developers. The native app has direct access to Windows' hardware scheduling and often provides lower-latency communication with IDE extensions and agents like Goose. If you find your AI responses are lagging while your Vulkan renderer is active, ensure you are using the native background service rather than a browser-based abstraction. +**A:** The native app (or CLI) is generally the better choice for graphics development. It has direct access to the OS's hardware scheduling and tends to have lower latency when talking to IDE extensions and agents like Goose. A browser-based WebUI is fine for managing and browsing models, but if you notice AI responses lagging while your renderer is active, switch to the native background service. **Q: "I added an extension (like Filesystem Access) to Goose, but it's not working. How do I enable it?"** -**A: The 'Recipe' Activation.** -In some versions of Goose, simply adding an extension to the configuration is not enough; it must be explicitly enabled in your active **Recipe**. If you find that your "Filesystem Access" (HPI) extension is not responding, open your Goose settings and ensure the extension is checked or "allowed" within the specific task profile you are using. Without this activation, the agent may "know" the tool exists but lack the permission to use it in your current session. +**A:** In some versions of Goose, adding an extension to the configuration isn't enough — it also needs to be enabled in the active recipe. Check your Goose settings and confirm the extension is checked or allowed for the specific task profile you're using. Without that, the agent may know the tool exists but not have permission to call it in your current session. -== Summary: Informed Graphics Engineering +== Summary -The transition to **Collaborative Engineering** requires a shift in mindset. You are no longer just a "Coder"; you are an **Infrastructure Manager** and an **Audit Lead**. By mastering these troubleshooting techniques, you ensure that your AI assistant remains a significant asset rather than a source of frustration. +Most of the problems above come down to the same few things: giving the model accurate context, giving it enough visual detail to work with, and not starving it (or your renderer) of GPU resources. None of it requires you to stop thinking about the Vulkan spec yourself — if anything, understanding these failure modes is what lets you use the tooling without getting burned by it. -In our final chapter, we will put every tool and concept into practice with the **AI-Driven Graphics Lab**. - -Next: xref:AI_Assisted_Vulkan/10_final_project.adoc[Final Project: The AI-Driven Graphics Lab] +Next: xref:AI_Assisted_Vulkan/10_final_project.adoc[Final Project: AI-Assisted Graphics Lab] diff --git a/en/AI_Assisted_Vulkan/10_final_project.adoc b/en/AI_Assisted_Vulkan/10_final_project.adoc index 0cd9462a..8d15e028 100644 --- a/en/AI_Assisted_Vulkan/10_final_project.adoc +++ b/en/AI_Assisted_Vulkan/10_final_project.adoc @@ -1,12 +1,12 @@ :pp: {plus}{plus} -= Final Project: The "AI-Driven" Graphics Lab += Final Project: AI-Assisted Graphics Lab -== Introduction: The Collaborative Capstone +== Introduction -Congratulations! You have built your environment, specialized your models, mastered the "Design-Implementation-Review" cycle, and explored the forensic arts of debugging and deployment. Now, it is time to bring every tool in your new toolbelt together for a final, comprehensive challenge: **The AI-Driven Graphics Lab.** +By this point you've set up your environment, worked with different models, gone through a design-implement-review cycle, and used AI to help with debugging and deployment. This final project brings those pieces together on one feature, end to end. -In this capstone project, you will work within the **Simple Engine** (found in the `attachments/simple_engine/` directory from the Building a Simple Engine tutorial). You will not just "use" AI; you will **Direct** a team of digital specialists as they help you architect, implement, debug, and optimize a complex graphics feature from scratch. This is your final exam in **Collaborative Engineering**. +You'll work within the **Simple Engine** (found in `attachments/simple_engine/` from the Building a Simple Engine tutorial), using a cloud model for design, a local model for implementation, and RenderDoc/GFXReconstruct-driven review to verify the result. == The Project: A Dual-Filtering Blur System @@ -52,7 +52,7 @@ for an implementation model and save it to LLM_NOTES.md. ---- === Step 3: Implementation and Refactoring -Once the design is reviewed and your `LLM_NOTES.md` is ready, task your local specialist (like **Qwen 3-Coder**) to generate the `BlurRenderer` header and implementation. By providing the notes as direct context, you ensure the local model follows the established architectural contract without needing high-reasoning cloud access for the bulk implementation phase. +Once the design is reviewed and your `LLM_NOTES.md` is ready, have a local model (like **Qwen 3-Coder**) generate the `BlurRenderer` header and implementation. Passing the notes as direct context lets the local model follow the agreed design without needing a cloud model for the bulk of the implementation work. [source,bash] ---- @@ -94,10 +94,10 @@ GitHub Actions workflow in '.github/workflows/main.yml' to include a new step that compiles and runs this test. ---- -By tasking the AI with the creation of the test and the update of the pipeline, you ensure that the "Technical Debt" of testing is handled simultaneously with implementation, maintaining a high standard of engineering without manual overhead. +Having the AI write the test and update the CI config alongside the implementation means test coverage doesn't lag behind the feature itself. === Step 6: Multi-Frame Analysis with GFXReconstruct -To truly verify your synchronization logic across the entire pass-chain of the blur system, you must look beyond a single frame. Use **GFXReconstruct** to capture a stream of API calls and task your autonomous agent with a "Timeline Audit." This is particularly effective for catching race conditions or incorrect image layout transitions that might only manifest intermittently or during the first few frames of initialization. +Verifying synchronization across the whole blur pass-chain usually means looking across several frames, not just one. Use **GFXReconstruct** to capture a stream of API calls and have your agent check layout transitions across that range. This catches race conditions or incorrect image layout transitions that only show up intermittently or during the first few frames of initialization. [source,bash] ---- @@ -127,10 +127,10 @@ If you find a hazard, suggest the specific line in 'BlurRenderer.cpp' that needs adjustment. ---- -By auditing the "Stream of Truth," you ensure that your engine isn't just working by "accident" on your current developer driver, but is strictly compliant with the Vulkan specification across time. +Auditing the full call stream this way is what tells you whether the engine is actually spec-compliant, rather than just happening to work on your current driver. === Step 7: Visual Frame Inspection with RenderDoc -While GFXReconstruct provides the multi-frame "Timeline of Truth," **RenderDoc** allows you to perform a deep-dive into the visual state of a single frame. Use the **RenderDoc command-line tool (`renderdoccmd`)** in conjunction with your autonomous agent to verify that each downsample and upsample pass in your blur system is producing the expected visual output. This is especially useful for identifying "Black Textures" or "Sampling Artifacts" that arise from incorrect sampler settings. +Where GFXReconstruct covers the multi-frame timeline, **RenderDoc** lets you inspect the visual state of a single frame in detail. Use the **RenderDoc command-line tool (`renderdoccmd`)** together with your agent to verify that each downsample and upsample pass produces the expected output. This is especially useful for tracking down black textures or sampling artifacts caused by incorrect sampler settings. [source,bash] ---- @@ -159,11 +159,11 @@ If you find a mismatch, explain how to refactor the beyond the available mip-levels. ---- -By integrating RenderDoc with your AI agent, you move beyond "guessing" why a texture is black. The AI identifies the specific state mismatch (like a misconfigured LOD clamp) and correlates it directly to the line of code in your engine that created the faulty resource. +Feeding RenderDoc captures to the agent turns "why is this texture black" from a guessing game into a direct trace: the AI identifies the specific state mismatch, like a misconfigured LOD clamp, and points at the line of code that produced it. -== Summary: Mastering the Collaborative Engineering Workflow +== Summary -By completing this lab, you have practiced managing the complexity of a modern Vulkan feature using a **Collaborative Engineering** approach — combining high-reasoning cloud models for architecture, local specialists for implementation, and established tools like RenderDoc and GFXReconstruct for verification. +Completing this lab means you've practiced the full loop on a real feature: a cloud model for architecture, a local model for implementation, and RenderDoc/GFXReconstruct for verification. AI assistance does not replace the need to understand Vulkan; it reduces the time spent on boilerplate and repetitive lookups so you can focus on the decisions that matter. The Vulkan specification remains your authoritative reference, and your judgment as an engineer remains the critical ingredient. What changes is how efficiently you can navigate that specification and iterate on your implementation. diff --git a/en/AI_Assisted_Vulkan/introduction.adoc b/en/AI_Assisted_Vulkan/introduction.adoc index 34c0ee84..97ca3aa2 100644 --- a/en/AI_Assisted_Vulkan/introduction.adoc +++ b/en/AI_Assisted_Vulkan/introduction.adoc @@ -1,50 +1,46 @@ :pp: {plus}{plus} -= AI-Assisted Vulkan Development: The Collaborative Engineering Paradigm += AI-Assisted Vulkan Development == Introduction -Imagine you are deep in the weeds of a complex Vulkan project. You’ve just finished a custom frame graph, but your validation layers are screaming with a `VUID-VkImageMemoryBarrier-oldLayout-01197` error. In the "Traditional Engineering" era, you’d spend the next 45 minutes digging through the Vulkan Specification, cross-referencing your image layout transitions, and manually tracing the lifecycle of your textures. +Say your validation layers throw a `VUID-VkImageMemoryBarrier-oldLayout-01197` error somewhere in a custom frame graph. Without help, you're looking at the Vulkan spec, tracing image layout transitions by hand, and following the texture's lifecycle back through however many files touch it. -Now, imagine a different scenario. You highlight the error in your IDE. An AI assistant, which has been quietly indexing your entire engine and the latest Vulkan 1.4 registry, doesn't just "fix" the code. It explains the *why* behind the mismatch, identifies a missing barrier in a completely different file, and suggests a more efficient synchronization pattern you hadn't even considered. +An AI assistant that has indexed your engine and the current Vulkan registry can shortcut a lot of that: it can point at the mismatch, find the missing barrier even if it's in an unrelated file, and sometimes suggest a cleaner synchronization pattern than the one you were about to write. It isn't magic and it isn't always right, but used well it removes a lot of the tedious lookup work so you can spend more time on the actual design problem. -This is **Collaborative Engineering**, and it is the future of high-performance graphics development. +This series is about using that capability deliberately, with a clear sense of where it helps and where it doesn't. -== The Paradigm Shift: Beyond "Copilot" +== Why this is worth doing for Vulkan specifically -Vulkan is a powerful but notoriously verbose API. Historically, the "barrier to entry" was not just mathematical or architectural—it was the sheer volume of boilerplate and the cognitive load of managing explicit GPU state. +Vulkan is verbose by design. A lot of the initial difficulty in learning it comes from boilerplate and explicit state tracking rather than from the underlying graphics concepts, and that's exactly the kind of work LLMs tend to handle reasonably well, for a few reasons: -With the rise of Large Language Models (LLMs) like **Qwen 3-Coder** and autonomous agents like **Goose**, the role of the graphics programmer is evolving. We are shifting from "Code Writers" to "System Designers." The AI doesn't replace your expertise; it amplifies it by managing the "Syntactic Noise" so you can focus on "Semantic Innovation." +1. **Vulkan's strictness helps the model.** Structs, bitmasks, and enums give an LLM a much narrower space of valid answers than a dynamically typed language would. That doesn't eliminate hallucination, but it makes it easier to catch. +2. **The spec is machine-readable.** Using the Model Context Protocol (MCP), you can give your assistant direct access to `vk.xml` so it can look up an extension or struct instead of guessing from training data that may be a version or two out of date. +3. **Vision-capable models can read screenshots.** You can hand a multimodal model a frame capture and ask about shadow acne, z-fighting, or texture bleeding, which is a reasonable starting point for tracking down the cause. -== Who is the Audience? +None of this replaces understanding what the API is doing. It's a way to spend less time on lookup and boilerplate, and more time on the parts of the problem that actually need judgment. -This series is designed for **Intermediate Vulkan developers** who have completed the core xref:00_Introduction.adoc[Vulkan Tutorial] and are ready to supercharge their workflow. You should already understand the fundamentals, such as pipelines, descriptor sets, and the common pitfalls of queue presentation. If you're looking for more efficiency and are curious about how to safely and effectively integrate AI into a high-performance C{pp} codebase without getting bogged down in boilerplate, this series is for you. +== Who is the audience? -== Why AI for Vulkan? +This series is for **intermediate Vulkan developers** who have completed the core xref:00_Introduction.adoc[Vulkan Tutorial]. You should already be comfortable with pipelines, descriptor sets, and the usual queue-presentation pitfalls. If you want a practical look at where AI tooling fits into a C{pp} graphics workflow, and where it doesn't, this series is for you. -Vulkan's unique properties make it the perfect candidate for AI assistance: +== The toolset -1. **Strictness as a Feature:** Unlike loosely typed languages, Vulkan's strict structures and bitmasks provide a "Syntactic Guardrail" for AI. A well-configured AI is much less likely to hallucinate in Vulkan than in Python because the "valid" state space is so well-defined. -2. **The Context Bridge:** Using the **Model Context Protocol (MCP)**, we can connect our AI directly to the `vk.xml` registry. This means your assistant is never "out of date"—it can query the exact spec for a new extension the day it’s released. -3. **Multimodal Debugging:** Modern AI has "eyes." We can now use vision-language models to diagnose "Shadow Acne," "Z-Fighting," or "Texture Bleeding" from simple screenshots, bridging the gap between what the user sees and what the code does. +The series uses two kinds of models: cloud models with strong reasoning, for structural planning and math-heavy problems, and local models running through Ollama, for fast, private, everyday completion and chat. Which one is worth using depends on the task, your hardware, and your tolerance for sending code off-machine; later chapters go into the tradeoffs rather than treating either category as a default choice. The other piece is the **Context Bridge** (MCP and RAG), which gives your assistant access to your engine's actual coding conventions and the current Vulkan API surface, instead of relying only on whatever the model happened to see during training. -== The 2026 AI Strategy +== The roadmap: what we will build -In this series, we move beyond generic chat-bots. We will build a specialized **AI Toolbelt** that leverages high-reasoning **Cloud Models** (like **Claude 4.6** or **GPT-5.3**) for structural planning and complex math, alongside privacy-first, ultra-fast **Local Models** (like **Qwen 3-Coder**, **Mistral-Nemo**, or **Llama 4**) running on your local GPU via **Ollama**. This strategy is anchored by the **Context Bridge** (MCP & RAG), which connects your IDE to the live Vulkan ecosystem, ensuring your assistant knows your engine's specific coding standards and the latest API specifications. +Throughout this tutorial, we will build up an AI-assisted development setup piece by piece: -== The Roadmap: What We Will Build - -Throughout this tutorial, we will systematically build an AI-enhanced graphics lab: - -1. **xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[The Modern AI Toolbelt]:** Transforming your IDE (CLion, Android Studio, Visual Studio, or Xcode) into an AI-enhanced hub and setting up your first autonomous agent. -2. **xref:AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc[Model Selection & Specialization]:** Deep dives into VRAM budgeting, quantization, and the "Intelligence-to-Performance" ratio. -3. **xref:AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc[Multimodal AI]:** Diagnosing visual bugs and analyzing your frames directly with AI. -4. **xref:AI_Assisted_Vulkan/05_workflow/01_introduction.adoc[Workflow: System Design to Implementation]:** Mastering the "Collaborative Loop" from high-level planning to shader implementation. -5. **xref:AI_Assisted_Vulkan/06_debugging/01_introduction.adoc[Pro-Level Debugging]:** Using AI to solve Validation Layer errors and analyze RenderDoc captures. -6. **xref:AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc[Advanced MCP Tooling]:** Building custom tools that let your AI query hardware limits and the Vulkan Registry. +1. **xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[The AI Toolbelt]:** Setting up your IDE (CLion, Android Studio, Visual Studio, or Xcode) with AI assistance and configuring your first agent. +2. **xref:AI_Assisted_Vulkan/03_model_selection_specialization/01_introduction.adoc[Model Selection & Specialization]:** VRAM budgeting, quantization, and picking a model that's actually a good fit for the task. +3. **xref:AI_Assisted_Vulkan/04_multimodal_ai/01_introduction.adoc[Multimodal AI]:** Diagnosing visual bugs by handing frame captures to a vision-capable model. +4. **xref:AI_Assisted_Vulkan/05_workflow/01_introduction.adoc[Workflow: System Design to Implementation]:** Moving from high-level planning to shader implementation with AI in the loop. +5. **xref:AI_Assisted_Vulkan/06_debugging/01_introduction.adoc[Pro-Level Debugging]:** Using AI to work through validation layer errors and RenderDoc captures. +6. **xref:AI_Assisted_Vulkan/07_advanced_mcp/01_introduction.adoc[Advanced MCP Tooling]:** Building custom tools so your AI can query hardware limits and the Vulkan Registry directly. 7. **xref:AI_Assisted_Vulkan/08_deployment/01_introduction.adoc[Deployment Automation]:** Streamlining the path to Android, iOS, and Desktop. -8. **xref:AI_Assisted_Vulkan/10_final_project.adoc[The AI-Driven Graphics Lab]:** A comprehensive capstone project implementing a Dual-Filtering Blur system using every tool in the AI toolbelt. +8. **xref:AI_Assisted_Vulkan/10_final_project.adoc[Capstone Project]:** A Dual-Filtering Blur implementation that uses the tools covered throughout the series. -By the end of this journey, you will be capable of leveraging current AI models for building complex graphics systems with a level of speed and precision that would have been significantly more difficult to achieve just a few years ago. +By the end, you should be able to use current AI models to build complex graphics systems faster, without losing track of what the code is actually doing. -Let's begin: xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Next: The Modern AI Toolbelt] +Let's begin: xref:AI_Assisted_Vulkan/02_environment_setup/01_introduction.adoc[Next: The AI Toolbelt] diff --git a/en/conclusion.adoc b/en/conclusion.adoc index dea67aca..82322095 100644 --- a/en/conclusion.adoc +++ b/en/conclusion.adoc @@ -47,11 +47,11 @@ You'll learn engine architecture and design patterns, scene management with hier The series also covers essential math and rendering concepts needed for advanced techniques like Forward+ rendering with tiled lighting, shadow mapping techniques, HRTF spatial audio integration, GPU-accelerated physics simulation using compute shaders, and Ray Query for hybrid rendering effects. This series builds upon the fundamentals you've learned here to help you create more structured and reusable rendering solutions, and assumes you've completed all the chapters in this Core Vulkan series. -2. link:AI_Assisted_Vulkan/introduction.adoc[*AI-Assisted Vulkan Development*] - Want to accelerate your development with AI? -This tutorial series explores how to leverage Cloud and Local LLMs, specialized AI agents (MCP), and multimodal vision models to navigate the complexities of Vulkan. -You'll learn to set up an AI-enhanced environment with tools like Ollama and MCP servers, select and specialize models for graphics programming, automate the generation of verbose Vulkan boilerplate, and use AI as a multimodal consultant to diagnose visual bugs from screenshots. -The series culminates in an AI-driven graphics lab where you'll use your new toolbelt to architect, implement, and debug a custom post-process effect. -This series focuses on "Collaborative Engineering" and is ideal for developers looking to optimize their workflow and master the tools of the modern graphics architect. +2. link:AI_Assisted_Vulkan/introduction.adoc[*AI-Assisted Vulkan Development*] - A practical look at using AI tools in a Vulkan workflow. +This series covers cloud and local LLMs, MCP-based agent tooling, and multimodal vision models, and where each one is and isn't a good fit for Vulkan work. +You'll set up an AI-assisted environment with Ollama and MCP servers, pick models suited to graphics programming, cut down on boilerplate, and use vision models to help diagnose visual bugs from screenshots. +The series ends with a capstone project where you use these tools to build, implement, and debug a custom post-process effect. +It's aimed at developers who want a clearer, less hyped picture of what AI tooling can realistically do for a Vulkan codebase. Future, planned tutorials will guide you through implementing more sophisticated rendering techniques and architectures.