RFC: run an ExecuTorch model from Python without PyTorch

[shoumikhin]

The problem

The executorch package does not list torch as one of its dependencies. That is on purpose. You
export a model with whichever torch version you like, and the runtime that runs it afterwards is meant
to be independent of that choice.

The Python bindings do not keep that promise. In an environment with no torch, importing them fails,
and the message blames the package:

import executorch.runtime
# ModuleNotFoundError: the prebuilt extension module ...pybindings._C is not found.
# Please reinstall ExecuTorch from pip.

Nothing is missing from the package. There are two real reasons:

1. The Python wrapper runs import torch when it loads, so that libtorch is in the process before the
   extension needs it.
2. The extension links libtorch, because it turns a torch.Tensor into a runtime tensor in C++.

So anyone who wants to run a .pte file from Python has to install torch, even though none of torch
runs during inference. It is there to hold numbers on the way in and on the way out. On a small machine
it is often the largest thing on the disk.

This proposal makes the bindings able to take and return plain memory, and adds a build that links no
torch at all, while keeping every existing caller working.

What this would change for a caller

This section is the shortest way to understand the proposal. Everything after it explains how to get
here.

Installing and importing

# today, in an environment with numpy and no torch
import executorch.runtime
# ModuleNotFoundError, even though the package never asked for torch

# proposed
import numpy
from executorch.runtime import Runtime

method = Runtime.get().load_program("model.pte").load_method("forward")
out = method.execute([numpy.ones((1, 3, 224, 224), numpy.float32)])[0]

Where torch is present, nothing changes

import torch

out = method.execute([torch.ones(1, 3, 224, 224)])[0]
out.softmax(dim=1)          # a torch tensor, with torch methods, as today

Inputs that are not torch tensors

# today
method.execute([numpy_array])
# RuntimeError: Unsupported python type <class 'numpy.ndarray'>

# proposed
method.execute([numpy_array])              # works
method.execute([memoryview_of_bytes])      # works
method.execute([cupy_array])               # works, through DLPack
method.execute([torch_tensor, numpy_array])  # mixing is fine

One input, without a list

# today
method.execute(tensor)     # works, through a separate overload
method.execute(array)      # not possible

# proposed
method.execute(array)            # counted as one input, not split by row
method.execute(previous_output)  # a result passed straight back in

Reading a result, the same line either way

values = numpy.asarray(method.execute([x])[0])

This is the line to recommend, because numpy.asarray reads a torch tensor and the new result type the
same way, with no copy in either case.

Mistakes that should be loud

method.execute([x.to(memory_format=torch.channels_last)])
# today: runs, and returns wrong numbers, for a method exported with the default layout
# proposed: raises, and names the layout the method expects

method.execute([x.astype(numpy.float64)])
# today: RuntimeError: Failed to execute method forward, error: 0x12
# proposed: raises, and names both dtypes

What comes back from execute

This is the part most likely to surprise someone, so it is worth its own section.

Today there is one answer: a torch.Tensor, always, because there is nothing else to return. With the
proposed build there are four cases, and only one of them is new.

build              torch in the process    execute returns
links torch        yes                     torch.Tensor          today, unchanged
links torch        no                      nothing, the bindings cannot be imported
links no torch     yes                     torch.Tensor          existing code keeps working
links no torch     no                      TensorBuffer          the new case

The new case only happens in an environment where a model could not run at all before, so no existing
caller lands in it by accident.

What each result supports. A torch tensor is unchanged, so this is really the contract of the new
type:

out = method.execute([x])[0]

# the same on both
numpy.asarray(out)      # the values, reading the same memory, no copy
out.shape               # (2, 2)

# different
out.dtype               # a torch dtype on one, a readable runtime dtype on the other

# a torch tensor only
out + 1                 # the new type holds memory and does no arithmetic
out.numpy()
len(out)

So code that reads results through numpy needs no change. Code that does arithmetic on the result keeps
working wherever torch is present, which is every environment where it works today.

The rule to implement, and it needs care. The question is whether the process has torch, and it is
asked once, when a method is loaded, not on every call. Asking per call lets the answer change under
a caller in the middle of a run, because importing the public runtime package is enough to pull torch
in, so a caller who imports nothing themselves is exactly the one who would be surprised. Never import
torch in order to answer the question, since importing it is what loads libtorch.

Goals and non-goals

Goals. Let a caller run a model with nothing but the runtime and something that holds memory. Keep
every existing caller working, including one that passes and receives torch.Tensor. Take and return
memory that lives on an accelerator, without copying it through the host. Refuse the inputs that currently
produce wrong numbers.

Non-goals. Export still uses torch; this is about running a model after export. Inference does not
get faster.

How it would work

The bindings sit between Python and the runtime. Today a torch.Tensor becomes a runtime tensor in
C++, and that is what pulls libtorch in. The idea is to keep torch types on the Python side of that
line, so nothing torch shaped ever crosses into C++.

python                          c++, the extension              runtime
---------------------------     --------------------------      -------------------
numpy array, memoryview,   -->  the buffer protocol gives
bytes, array.array              a pointer, a shape, strides
                                and a format code
                                          \
cupy array, any producer   -->  a capsule gives a pointer,       one description of
that speaks DLPack              device, dtype, shape, strides    some memory
                                          |                      ------> Method
torch.Tensor               -->  torch's own PYTHON api gives       copies it, or keeps
                                data_ptr, shape, stride, dtype     the pointer and reads
                                          /                        it when the method runs
                                TensorBuffer, which owns its    <------ results
result        <-------------    memory, or a torch tensor
                                built over it

Every kind of input becomes one description. Whatever a caller passes, the extension ends up with a
pointer, a shape, a layout, a dtype and a device, and hands that to the runtime. The runtime sees one
path, not three.

Nothing torch shaped gets linked. Reading a tensor through its Python API needs no torch headers and
no torch symbols. To the compiler, the tensor is an ordinary Python object. That is what lets one
package link no torch and still serve callers who have torch.

The bindings own what they hand out. Today they borrow from torch and let torch own things. That has
to be replaced with explicit ownership, which is the part most worth designing carefully.

Components and changes:

where	what changes
the runtime	refuse an input whose layout is not the one the method expects
the extension	accept plain memory and DLPack, and read a torch tensor through Python
the build	an option that links no torch
the Python wrapper	import torch only when the extension needs it

The control path, before and after

An input, on the way down

Today, for a torch.Tensor:

python      method([tensor])
bindings    dispatch on the name of the type
            cast the object to at::Tensor                <- this is what links libtorch
            copy sizes and strides into a runtime tensor
            work out a dimension order from the strides
            hand both to the bridge, which checks they agree and sets the pointer
runtime     Method::set_input, per input
              copies the bytes into planned memory, or keeps the pointer
            Method::execute

The cast forces the link. And nothing is held: the tensor and the
runtime tensor are locals, so once the call returns, whatever the runtime kept a pointer to belongs to
nobody.

Proposed, with several ways in that meet in the middle:

python      method(inputs)                one input, or a sequence of them
bindings    normalise that, so a single buffer counts as one input

            a torch.Tensor          something with memory       a DLPack producer
            --------------------    ---------------------      --------------------
            check it is one         ask for the buffer          ask for the capsule,
            refuse a lazy view      dtype from the format       and claim it so the
            read dtype, shape,      code, layout from the       producer stops owning
            stride, data_ptr and    strides, raw bytes only     it. dtype, device,
            device through          for a dtype no code         shape and strides all
            torch's PYTHON api      can name                    come from the capsule
            --------------------    ---------------------      --------------------
                       one description of some memory
                                     |
            point at it, or copy it, by one rule: does the runtime keep this
            pointer, and can this source promise the memory stays put and may
            be written
                                     |
            keep alive what must be: the view, the capsule, any copy, the object
            itself, and for anything pointed at, a note of where its memory was
runtime     Method::set_input checks the layout against the method's metadata,
              then copies the bytes or keeps the pointer
            Method::execute

A result, on the way back

Today the bridge views each output as a torch tensor, clones it, and casts it to Python, which needs
libtorch again. Proposed:

runtime     before running, every output the method has no planned memory for gets
              a buffer of its own, fresh for this call, skipping an output that is
              also an input
            execute writes into those buffers
bindings    for each tensor:
              already handed back in this execution? return the same object
              is its data exactly the buffer we allocated?
                 yes -> move that buffer into the result object, no copy
                 no  -> copy the bytes into a new result object
            does this method return torch tensors, decided when it was loaded?
                 no  -> return that object, which lends its memory out
                 yes -> build a torch tensor over it, so the bytes stay owned by
                        the object the tensor holds
python      a TensorBuffer, or a torch.Tensor

The shape of the change is the same on both sides. Before, the bindings borrow from torch and lean on
it to own things. After, they own what they hand out and know exactly what they borrowed.

What to build

1. Check the input layout, in the runtime

A method is exported with a memory layout for each input, and its kernels read that layout. Today a
caller can pass the same values in a different layout and nothing notices, because only byte counts are
compared. The kernels then read the values in the wrong order and the method returns wrong numbers with
no error at all.

Do this first, or every new way of passing an input inherits the problem. Land it on its own, because it
is in the shared runtime: about twenty callers outside Python reach that function, and some of them wrap
it in a macro that ends the process on failure.

Three traps:

• Read the expected layout from the method's metadata, not from the destination tensor. The
  destination is resized first, and in ATen mode that resize rewrites its strides, so the declared
  layout is gone before there is anything to compare.
• Compare strides on both sides, never a dimension order against strides. A dimension of size one
  carries no layout information and must be exempt, and a dimension order cannot express that
  exemption, because such a dimension may legally sit anywhere in an order.
• A tensor may carry strides, or a dimension order, or neither. Reading the one it does not have builds
  an array view over a null pointer, which asserts where assertions are on and reads a null pointer
  where they are not. Ask before reading, and accept a tensor that states no layout at all, because
  there is nothing to compare it against.

const auto meta = this->method_meta().input_tensor_meta(input_idx);
if (meta.ok() && meta->dim_order().size() == static_cast<size_t>(source.dim())) {
  StridesType wanted[kTensorDimensionLimit];
  ET_CHECK_OK_OR_RETURN_ERROR(dim_order_to_stride(
      source.sizes().data(), meta->dim_order().data(), source.dim(), wanted));

  StridesType actual[kTensorDimensionLimit];
  bool states_layout = true;
#ifdef USE_ATEN_LIB
  for (ssize_t i = 0; i < source.dim(); ++i) {
    actual[i] = static_cast<StridesType>(source.strides()[i]);
  }
#else
  const auto* impl = source.unsafeGetTensorImpl();
  if (impl->has_strides()) {
    for (ssize_t i = 0; i < source.dim(); ++i) {
      actual[i] = static_cast<StridesType>(source.strides()[i]);
    }
  } else if (impl->has_dim_order()) {
    ET_CHECK_OK_OR_RETURN_ERROR(dim_order_to_stride(
        source.sizes().data(), source.dim_order().data(), source.dim(), actual));
  } else {
    states_layout = false;  // nothing to compare against
  }
#endif
  for (ssize_t i = 0; states_layout && i < source.dim(); ++i) {
    // A dimension of size one says nothing about layout.
    if (source.size(i) > 1 && actual[i] != wanted[i]) {
      return Error::InvalidArgument;  // log the dimension, and what was expected
    }
  }
}

The two predicates that appear there do not exist yet and are worth adding beside the accessors they
guard, because every caller of those accessors has the same problem:

/// Returns true if this tensor was built with a dim order. Callers must ask before
/// calling dim_order(), which builds a view over a null pointer when it was not.
bool has_dim_order() const {
  return dim_order_ != nullptr;
}

One more thing belongs in this change. The input helper that the runners share always builds contiguous
tensors, so a channels-last method will start refusing them until that helper follows the method's
layout instead.

2. Accept plain memory

Take any object that exposes its memory through CPython's buffer protocol. numpy arrays, memoryview,
bytes and array.array all do. The dtype comes from the format code and the layout from the strides.

The format code names the kind and the item size decides the width, because some codes vary by platform.
Only the host's own byte order can be read:

std::optional<ScalarType> buffer_scalar_type(const py::buffer_info& info) {
  // strip a leading '@' or '=', accept '<' on a little endian host, refuse the rest
  switch (code) {
    case 'e': case 'f': case 'd':
      switch (info.itemsize) {
        case 2: return ScalarType::Half;
        case 4: return ScalarType::Float;
        case 8: return ScalarType::Double;
      }
      return std::nullopt;
    // ... the integer and bool codes, also by item size
  }
}

Some dtypes have no format code at all. bfloat16 is the one that matters, because exported models use
it. For those, accept a flat run of raw bytes that fills the input exactly, and take the dtype, shape and
layout from the method. Keep it narrow. Every condition below is load bearing: without the first, a
byte-typed image of the right size is silently read as float data, and without the stride check, a
strided byte view has its gaps read as data.

const bool raw_bytes = expected.ok()
    && !buffer_format(expected->scalar_type()).has_value()  // no format code names it
    && info.ndim == 1 && info.itemsize == 1                 // a flat run of bytes
    && info.strides[0] == 1                                 // packed, not strided
    && nbytes == expected->nbytes();                        // fills the input exactly

One thing belongs in a single place: a numpy array also behaves like a sequence, unlike a torch tensor,
so a bare array passed where a list is expected must count as one input rather than being taken apart
into one input per row. The entry points therefore take any object, not a sequence:

py::sequence as_input_sequence(const py::object& inputs) {
  if (!py::isinstance<py::buffer>(inputs) && py::isinstance<py::sequence>(inputs)) {
    return py::cast<py::sequence>(inputs);
  }
  py::list one;
  one.append(inputs);
  return one;
}

3. Read a torch tensor without linking torch

Today the conversion needs torch at compile time:

auto at_tensor = python_input.cast<at::Tensor>();   // needs torch headers and symbols

Everything the runtime wants is reachable through torch's Python API, which costs neither. Two details
matter: torch counts strides in elements while a layout check counts bytes, and shape and strides arrive
in separate calls, so nothing guarantees they agree.

const auto itemsize = py::cast<py::ssize_t>(t.attr("element_size")());
const auto shape = py::cast<std::vector<py::ssize_t>>(t.attr("shape"));
auto strides = py::cast<std::vector<py::ssize_t>>(t.attr("stride")());
if (strides.size() != shape.size()) {
  throw std::runtime_error("shape and strides disagree, so the layout cannot be read");
}
for (auto& stride : strides) {
  stride *= itemsize;
}
auto* data = reinterpret_cast<uint8_t*>(py::cast<uintptr_t>(t.attr("data_ptr")()));

Use one table of dtype names for both directions, so an input and an output of the same dtype cannot
disagree about what it is. Names rather than dtype objects, because this build has no torch headers to
compare against.

Never import torch to do any of this. Use torch only when the caller has already imported it.

4. Take and give memory through DLPack

DLPack is how array libraries hand each other memory. numpy, torch, CuPy and JAX all produce and consume
it. It has two things the buffer protocol does not: it can name every dtype, and it can describe memory
that is not the host's.

Taking one in, the ownership step is the one that gets missed. The protocol says a consumer claims a
capsule by renaming it, which tells the producer not to free what is inside, and then calls the deleter
when it is done. Holding both halves in one object means the promise is kept even when a conversion
throws:

class DlpackInput final {
 public:
  explicit DlpackInput(const py::object& source) {
    py::object capsule = source.attr("__dlpack__")();
    auto* raw = PyCapsule_GetPointer(capsule.ptr(), "dltensor");
    if (raw == nullptr) {
      PyErr_Clear();
      throw std::runtime_error("this object offers __dlpack__ but handed over nothing");
    }
    PyCapsule_SetName(capsule.ptr(), "used_dltensor");  // claimed: ours to free now
    managed_ = static_cast<DLManagedTensor*>(raw);
  }
  ~DlpackInput() {
    if (managed_ != nullptr && managed_->deleter != nullptr) {
      managed_->deleter(managed_);
    }
  }
  const DLTensor& tensor() const { return managed_->dl_tensor; }
  // movable, not copyable
 private:
  DLManagedTensor* managed_ = nullptr;
};

Two details in the description itself: DLPack counts strides in elements, and leaves them out entirely
when the elements are packed in the order the shape implies, so a null strides pointer is normal and
means contiguous. There is also a byte offset to add to the pointer.

Giving one out, the result object keeps the memory and the capsule holds a reference to the result
object, so a consumer can outlive the result it read from:

py::capsule dlpack(py::object self) {
  auto managed = std::make_unique<DLManagedTensor>();
  managed->dl_tensor = describe(buffer_);          // pointer, device, dtype, shape, strides
  managed->manager_ctx = self.inc_ref().ptr();     // keep this object alive
  managed->deleter = [](DLManagedTensor* tensor) {
    py::gil_scoped_acquire acquired;               // the consumer may free from anywhere
    py::handle(static_cast<PyObject*>(tensor->manager_ctx)).dec_ref();
    delete tensor;
  };
  // Named "dltensor" so a consumer can claim it. If nobody does, this destructor
  // runs instead and releases what was reserved above.
  return py::capsule(managed.release(), "dltensor", [](PyObject* capsule) {
    if (PyCapsule_IsValid(capsule, "dltensor")) {
      auto* tensor = static_cast<DLManagedTensor*>(
          PyCapsule_GetPointer(capsule, "dltensor"));
      if (tensor != nullptr && tensor->deleter != nullptr) {
        tensor->deleter(tensor);
      }
    }
  });
}

Two checks stay the bindings' own whatever the transport. A torch view that carries a pending sign flip
loses that sign when it crosses DLPack, so the values underneath are not the values the tensor shows.
And bfloat16 does not survive the trip into numpy, which limits what can be handed back rather than what
can be taken in.

5. A result type for when torch is absent

Without torch there is nothing to return a tensor as, so the bindings need a small type of their own: an
object that owns its memory and lends it out, with a shape and a dtype a reader can inspect.

Four details decide whether this is pleasant or a trap.

• Publish strides in bytes. The buffer protocol counts bytes while a runtime tensor counts elements.
  Publishing the runtime's numbers unconverted hands a reader zeros and denormals with nothing raised.
• Publish the real strides, so a result in an unusual layout is not presented as if it were
  contiguous. torch.frombuffer cannot consume those, so the torch side needs as_strided over the
  same memory, or a copy, rather than frombuffer alone.
• Never present a shape the result does not have. For a dtype with no format code it is tempting to
  hand back a flat run of bytes, but that is a different shape, silently. Keep the shape and present an
  unsigned integer of the same width, so a reader sees the right elements in the right places and only
  reinterprets what each one means. Offer DLPack too, which names the dtype properly.
• Hand over only what the result occupies. With dynamic shapes the buffer is sized for the largest
  shape the method can return.

py::buffer_info buffer_info() {
  const auto format =
      buffer_format(scalar_type_).value_or(unsigned_format_of_width(itemsize_));
  return py::buffer_info(
      data(), itemsize_, format, shape_.size(), shape_, strides_in_bytes_);
}

6. A build that links no torch

Add an option that drops the C++ tensor conversion, the bridge it uses, the torch libraries, their
include directories, and the search path that finds them at run time.

The traps, all of which make the option look like it works when it does not:

• Every place that looks for torch has to be guarded, not only the one in the bindings. A lookup at
  the top of the build runs whenever bindings are enabled, and the runner extension has its own Python
  module that looks for torch too. Either one leaves this build unable to configure without torch.
• The preset that builds the bindings turns on things that need torch, including training bindings
  that read torch tensors in C++. If the option refuses that combination, the preset has to stop asking
  for it, or the supported way of building the bindings cannot use the option at all.
• ATen mode is a contradiction with this option, because that runtime is built on torch's own tensor
  type. Refuse it when the build is configured rather than at link time.
• The optimized kernels compile against torch's headers. A build that wants them still needs torch
  present, even though nothing links it. Say so, rather than letting somebody discover it.

The Python wrapper needs care too. It must not import torch when the extension does not link it, and it
must import torch when the extension does, because those conversions go through torch's own Python types.
Let the extension say which build it is, and read that defensively, because an extension built before the
flag existed does not report it and a missing attribute should not become a failed import:

_needs_torch = getattr(_C, "_links_torch", True)

Watch for anything else on the import path that reaches for torch indirectly, such as a warning class
that lives in the export half of the package.

7. Memory that lives on an accelerator

A model exported for an accelerator can keep its inputs and outputs there, with no copy at the boundary.
That is an export time choice, and it comes with a second requirement: the runtime must also stop
reserving its own buffers, or it would fill them from the caller's memory and the copy would be back.

ExecutorchBackendConfig(
    propagate_device_config=PropagateDeviceConfig(
        skip_h2d_for_method_inputs=True,
        skip_d2h_for_method_outputs=True,
    ),
    enable_non_cpu_memory_planning=True,
    memory_planning_pass=MemoryPlanningPass(
        alloc_graph_input=False, alloc_graph_output=False
    ),
)

So a device resident method is the unplanned case from the ownership rules, with the memory somewhere the
host cannot read. Most of what this needs is already there: a tensor carries the device it lives on, the
tensor maker can set one, device memory can be allocated the same way planned device buffers are, and
handing the runtime a buffer for an output does not care where that buffer is.

The metadata has to say which device an input or an output is on. Before a method runs, the bindings
choose a host or a device allocation for each output that has no planned memory, and that choice needs the
output's device. The exported file carries it, and the metadata does not report it. A planned buffer can be
asked where it belongs, but a device resident output is by definition not planned, so that query does not
help. Add this first, because nothing else on the device path can be decided without it.

An output on a device gets a device allocation. Same shape as the host path, a different allocator, and
the same rule that each call gets its own so a result already handed back is not written through.

A device result goes back through DLPack. The buffer protocol cannot describe memory the host cannot
read, so it cannot be the way out. The capsule carries the device kind and index, and the consumer, torch
or CuPy or anything else, wraps it without a copy:

import cupy
x = cupy.ones((1, 3, 224, 224), cupy.float32)   # already on the device
out = method.execute([x])[0]                    # nothing copied on the way in
y = cupy.from_dlpack(out)                       # nothing copied on the way out

A device result cannot be handed back as a host
buffer, so a caller who wants one copies it across itself. And borrowing a device allocation is only safe
for producers whose behaviour is known, because holding a tensor does not stop a caching allocator
replacing its storage; everything else is copied into a device buffer these bindings own. The ownership
rules below say which is which.

Ownership: the part worth designing carefully

Most of the difficulty is not conversion. It is deciding who owns which memory and for how long. Each
rule below is a silent memory bug when it is wrong, and each one deserves a test.

An input belongs to the caller. The bindings hold a reference to it for exactly the window in which
the runtime may read it: from when the inputs are set until they are replaced, or until the method is
gone.

Whether to copy has one rule, and it depends on what the export asked for and on what the source can
promise:

bool input_needs_owned_copy(const Result<TensorInfo>& expected, bool pins_memory) {
  if (!expected.ok()) {
    return true;           // nothing known, so do not point at anything
  }
  if (expected->is_memory_planned()) {
    return false;          // the runtime copies this itself, inside the call
  }
  return !pins_memory;     // otherwise it depends on the source
}

A source pins its memory when it both keeps it in place and allows it to be written. A writable buffer
view does. A claimed DLPack capsule does, because claiming it is a promise. A tensor object does not,
because it can be resized or freed while the object stays alive. A read only buffer does not either, and
that one is easy to miss: the runtime may hand that memory to a kernel that writes, and Python shares
immutable objects, so writing into one reaches whatever else holds it.

Do not trust the pin, verify it. The protocol says an exporter keeps memory in place while a view is
held, and Python enforces that for a well behaved exporter: it refuses to release a memoryview or resize
a bytearray that is lent out. numpy offers an explicit way out with resize(refcheck=False), which moves
the allocation anyway. So for anything the runtime was given a pointer to, record where that memory was
and how much of it there was, and check both before running. An address alone is not enough, because
a smaller allocation can keep the same address.

Setting inputs is not all or nothing. The runtime installs them one at a time and stops at the first
it refuses, so a refused call can leave it holding some new inputs and some old. Keep both sets alive and
refuse to run until the whole set is given again. Two things follow: nothing may read those inputs while
the set is incomplete, including the outputs, and a caller that retries in a loop must not accumulate
them, so drop an incomplete set when the next call arrives.

Every way into the outputs refuses while the inputs are incomplete. The refusal above deliberately
drops what was held, so anything that then reads an output tied to that memory reads a hole. Reading
outputs is the one that gets forgotten.

Never hand out a pointer into memory that will be freed. An output written into a buffer the bindings
own gets a fresh buffer per call, so a result already handed over is not written through. That means a
caller asking for no copy cannot be given a reference into it. A planned output is different: it lives in
the arena, which lasts as long as the method, so no copy there still means no copy.

Hand a result over once. Return the same objects if the outputs are read twice in one execution.

A method is not safe to share. It holds the inputs it was given and the outputs it handed back, and
converting an output calls into Python, which lets another thread in. Refuse to run a method already
running on another thread, and say to load one per thread. Within one thread allow it, because a call
that does several steps holds the same claim across them.

Hold the bytes a program is read out of. A program is read in place, not copied, so whoever reads it
must keep it. Put that reference beside the loader, not on the object the caller happens to hold, because
the shortest way to write the call leaves the program object as a temporary:

method = load_program(exported.buffer).load_method("forward")

The bundled program loaders need the same treatment.

An output can share an input's bytes. A method that returns an input unchanged has one value serving
as both, so giving that output a buffer of its own moves the input into that buffer too. Detect it and
leave such an output alone.

The host cannot copy what it cannot read. Device memory is passed by pointer in every case, and it is
never copied at the boundary.

Failure should be loud and specific

Each case below has to raise an error that names the input and says what to do. None may return numbers,
and none may end the process.

• An input whose layout is not the one the method expects.
• A dtype the method did not ask for, naming both dtypes rather than returning a code.
• A layout no runtime layout describes, such as a transposed or strided view, saying to make it
  contiguous first.
• A rank beyond what the runtime can describe. The stride computation fails an assertion past its limit
  and that ends the process, so check the limit before asking, on every path, including the one that
  takes its shape from the method rather than from the caller.
• A size the runtime cannot hold. Sizes narrow to a smaller signed type on the way in, and a larger one
  wraps to a negative number, which fails an assertion deep inside. A tensor with no elements can carry
  such a size without using any memory, so this needs nothing unusual to reach.
• A tensor on a device the runtime cannot name, and a tensor with no memory at all.
• A negated or conjugated view, because torch applies the sign when something reads the tensor, so the
  memory holds different values than the tensor does.
• An object that only claims to be a tensor. Inputs are told apart by the name of their type, and any
  object can carry that name, so check it against the real class, and check that what it reports about
  itself is self consistent.
• A result on a device, where the caller asked for it as host memory. Hand it back through DLPack so it
  stays where it is, and say plainly that reading it on the host means copying it there first.

One thing cannot be defended against, and the design should say so rather than imply a check exists. A
view built with numpy's as_strided can describe memory that does not exist, and the buffer protocol
then advertises the inflated length, so the number a consumer would check against is part of the lie.
numpy crashes reading its own view, with no runtime involved. It belongs with resize(refcheck=False):
a caller who has stepped outside the protocol's promises.

How to know it works

The tests are the deliverable as much as the code is, because most of what goes wrong here is invisible
in a passing suite.

The ways in agree with the model. Run the same model through every input path and compare against
what the model computes in eager mode, not against another call of the same binding. A test that compares
one call against another agrees with itself whether or not the layout was read correctly. Use inputs
whose values all differ, so a wrong layout cannot pass by luck.

Every test can fail. Remove the fix, watch the test fail, put it back, and say how that was
established. Two ways a test here can look fine and prove nothing: reading its result the copying way,
which hides whether a buffer was reused, and using a model that never writes its input, which hides
whether read only memory is protected.

Ownership needs two calls. Set the inputs, then run, as separate calls. The window in which the
runtime holds a pointer does not open inside a single call, so a test written that way cannot see any of
the ownership rules.

An end to end proof that torch is absent. This is the one the whole change exists for, and it cannot
be written in the ordinary way, because a test file that imports torch cannot observe what happens
without it. Split it: the test process exports the models, which needs torch, and a child process
imports the bindings with torch blocked and does the asserting. Have the child list the shared libraries
the process actually mapped, which an import hook cannot fake, so a build that linked torch is caught
rather than skipped.

Cover these, because each one is a way this goes wrong quietly:

an input resized with refcheck=False after being set    must raise, not read freed memory
a read only buffer into a method that writes its input  the bytes must be unchanged afterwards
a dimension too large for the runtime, on an empty
  tensor                                                must raise, not end the process
a program loaded from bytes nobody else holds           must work
a method whose output is one of its inputs              must return the input's values
outputs read twice in one execution                     the same objects both times
outputs read after a refused call                       must raise, not crash
an earlier result after a later call                    unchanged, including with no copy asked for
a channels-last method, and a size-one dimension        accepted, and compared against eager
a dtype no format code can name                         round trips, and keeps its shape
two threads on one method                               must raise, and say one per thread

Fuzz the input space. Random dtypes, ranks, transposes, strided slices, scalars, None, strings,
dicts, generators, a self-referential list, and objects that lie about themselves. Every case must either
run or raise, and none may end the process.

Watch for leaks and for references. Loops over execute, over setting inputs, and over load and drop
should not grow. The bindings should hold exactly one reference to an input while it may be read, and none
once the inputs are replaced, and the same for the bytes a program was loaded from.

Both builds, and a job that builds the new one. Run the suite against the ordinary build and the new
one. Without a job that builds the new configuration, the tests that only run there skip everywhere
forever, and the first change that adds a torch dependency to the extension breaks it quietly. A test
that finds the wrong build where a job asked for the new one must fail rather than skip, or a miswired
build passes.

More info

A mismatched layout is refused from the first release, with no warning period. The behaviour being
replaced is not a contract: a mismatched layout returns wrong numbers today, silently. A warning release
would leave those callers computing wrong answers for another release while telling them so in a log
nobody reads. The cost is that a caller outside the tree relying on the old behaviour gets an error, which
is the intent.

One package is published, and it is the build that links no torch. The package already declares no
dependency on torch, so this is the build that matches what it says about itself, and a second package
would double the release matrix on every platform forever. The cost is carried by the next decision.

A result is a torch tensor when the process has torch, and the new type otherwise, decided once when a
method is loaded. Deciding per call lets the type change under a caller who imported nothing themselves.
Deciding at load removes that, and keeping the torch tensor for a caller who has torch is what makes the
package change invisible. No public argument selects it; one can be added later without breaking anyone.

Raw bytes stay narrow, and DLPack carries what they cannot. A dynamically shaped input of a dtype no
format code can name comes in through DLPack, where the shape comes from the capsule rather than from the
method. No descriptor is added, because that would be public API for a case already covered.

A backend answers one query about the padding and alignment it needs, and a backend that has not
answered counts as unknown rather than as needing nothing. That default is the whole point: assuming a
backend needs nothing is what lets a delegate read past the end of a caller's buffer. Unknown means copy
into owned storage with room at the end on the host, and refuse on a device.

Device memory is borrowed from a named list of producers, and copied from everything else. Holding a
producer's tensor does not stop its storage being replaced, so the tensor is the wrong anchor, and even the
storage is only safe while nothing resizes it. So borrowing is allowed only for producer and allocator
combinations whose behaviour is known, anchored on the storage rather than on the tensor, with the caller
promising not to resize it while the method holds it. Anything not on that list still works: its memory is
copied into a device buffer these bindings own. Nothing is refused for being on a device, and nothing about
device memory waits for a later version.

There are three ways in, not one. The buffer protocol for plain host memory, DLPack for tensor-like
objects, and reading a torch tensor through its Python API. DLPack cannot be the only path, because
bytes, bytearray, memoryview and array.array do not speak it. The three converge on one
description immediately, so the runtime still sees one.

A host input is copied unless both its owner and its size are proved. A copy costs nothing at small
sizes and about a millisecond for the weights of a small model, so copying everything would be paid mostly
by the callers who least need it, while copying nothing is what keeps going wrong.

Host memory and device memory are both in scope from the start. A model exported for an accelerator
can keep its inputs and outputs there, and that is the case this work exists to serve as much as the host
one, so the two are built together rather than one behind the other. Four pieces make it work and all four
belong in the same change: the metadata says which device an input or an output is on, the bindings
allocate on that device for an output the method has no planned memory for, DLPack carries the memory in
both directions, and the borrowing rule above decides when a pointer may be kept. The cost is that the
first version is larger, and the reason to pay it is that splitting it would make every device caller
migrate twice.

[Gasoonjia]

Sry but im not super clear about why we need to support running model from PyThon without PyTorch:

1. In what scenerio user may want to run model from python in the real world production? Running from Python may sacrifice model performance and not every backend supports pybinding now.
2. When user using python ET lib, for majority of cases they want to do AOT stuff, like model quantization, exportation and serialization, all of which depend on PyTorch. What's the best interface we should provide our user "this is a full packaged python ET" vs "this is a inference-only python ET"?

[shoumikhin]

@Gasoonjia fair question, and the answer is not production serving. I would not propose Python for that either.

The package already makes this promise. pip install executorch brings 30 dependencies and none of them is torch. Then import executorch.runtime fails and tells you to reinstall ExecuTorch, which is misleading, since nothing is missing. So this is an existing promise that does not hold rather than a new feature.

pip download executorch --no-deps -d /tmp/et && grep -c "^Requires-Dist: torch$" /tmp/et/*.whl  # 0

The person running a model is often not the person who exported it. Export needs torch and always will. Running does not, and it happens in places where torch is pure cost: checking that an exported file gives the right numbers before it ships, CI jobs that run .pte files and pay a multi gigabyte install per job, and devices where the code around inference is Python, such as a camera loop or a robot node.

LeRobot is one of those. lerobot-rollout needs torch for one reason, that the policy is a torch model, so running an exported .pte there would need none. I have two open proposals to keep torch out of the paths that never run a model (4690, 4693), where I measured that on a Jetson Orin Nano with 7 GB of memory, importing torch alone costs 2.5 s and 527 MB before any robot code runs.

On the interface, one package rather than two. The split already exists inside it: the AOT half imports torch itself, so quantization and export are unaffected, and the runtime half stops linking it. Someone who exports installs torch as they do now. Someone who only runs does not, and nobody has to pick a wheel.

Two things I am not claiming: this does not change which backends have bindings, and it does not make Python inference fast enough for serving.

[JacobSzwejbka]

Theres a fair amount of claudism in the post I skipped over, but iiuc the general ask is to allow the pybindings to be torchless and then the rest is sort of diving into what is desirable in the torchless tensor? Ill put up a change to allow it to be torchless and the torchless tensor will be a real barebone wrapper around a data blob. We can then iterate from there. Does that sound good @shoumikhin