Skip to content

Metal backend: generated kernels index a packed copy of a non-packed view out of bounds (wrong results, crash in MPSGraph)#22983

Description

@abdelaziz-mahdy

馃悰 Describe the bug

With inductor's default layout optimization, models built on chunk + cat along C (the YOLO C2f block) return wrong values on the Metal backend, and yolov8n crashes inside MPSGraph.

aoti_torch__reinterpret_tensor replaces a view that is not densely packed with a packed copy (materialize_packed). Kernels generated by inductor never look at the tensor they are handed: they index with the strides they were compiled with, and they write through such views as well as read them. In channels-last a chunk along C is non-packed, and a cat is filled by writing through views of it:

buf52 = reinterpret_tensor_wrapper(buf55, 4, {1, 32, 80, 80}, {819200, 1, 10240, 128}, 32)  // alias

buf55 has 819200 elements and the copy that stands in for buf52 has 204800. The kernel writing "into buf52" indexes up to 819200 into that copy, which is an out-of-bounds GPU access, and nothing reaches buf55. #22957 lists the lost writes under "Not fixed here" (linear_chunk_cat_last_dim is skipped for it); the out-of-bounds part is new, and it is what takes yolov8n down: 10 out of 10 runs crash with EXC_BAD_ACCESS in -[MPSGraphExecutable runInternalWithDevice:...] -> getFuncOp -> mlir::SymbolTable::lookupSymbolIn, called from aoti_torch_mps_convolution, on a pointer that reads as two floats. With torch._inductor.config.layout_optimization = False there are no such views and the same model runs correctly.

Repro. The C2f block reduced to 1x1 convs, so that it only needs matmuls. For MODULE_REGISTRY in backends/apple/metal/tests/test_modules.py:

class PointwiseC2f(nn.Module):
    class Inner(nn.Module):
        def __init__(self, channels: int):
            super().__init__()
            self.conv1 = nn.Conv2d(channels, channels, kernel_size=1)
            self.conv2 = nn.Conv2d(channels, channels, kernel_size=1)

        def forward(self, x):
            return x + self.conv2(torch.relu(self.conv1(x)))

    def __init__(self):
        super().__init__()
        self.conv_in = nn.Conv2d(16, 16, kernel_size=1)
        self.inner = nn.ModuleList(PointwiseC2f.Inner(8) for _ in range(2))
        self.conv_out = nn.Conv2d(32, 16, kernel_size=1)

    def forward(self, x):
        parts = list(self.conv_in(x).chunk(2, dim=1))
        for block in self.inner:
            parts.append(block(parts[-1]))
        return self.conv_out(torch.cat(parts, dim=1))


MODULE_REGISTRY["pointwise_c2f"] = {
    "model_class": PointwiseC2f,
    "input_shapes": [(2, 16, 8, 8)],
    "description": "C2f block whose cat is filled through non-packed channels-last views",
}

Output mismatch - max_atol=0.586, max_rtol=1.64 in float32, 0.755 / 1.71 in bfloat16.

Versions

ExecuTorch: #22957 @ 89dbe34077 (main @ 9b91b43098 plus that PR)
PyTorch version: 2.14.0
OS: macOS 27.0 (26A428), arm64, Apple M2 Pro
Clang version: 21.0.0 (clang-2100.3.34.2)
CMake version: 4.4.3
Python version: 3.10.11

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugmodule: metalIssues related to the AOTI Metal backendtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions