Release 4.11.0

Changes from 4.10.1 to 4.11.0

Nullability in CTable is rebuilt on Arrow's own model. A nullable column now
keeps its nulls in a per-column validity sidecar instead of reserving a value
from its own range, which is what makes it lossless — an int8 column can hold
-128, a utf8 one can hold "", and a float64 one can tell NaN from
missing. That is the new default for columns created from now on; nothing on
disk changes, and sentinel storage remains supported indefinitely, one keyword
away. Built on top of it: predicates follow three-valued (Kleene) logic, so
~(t.price > 10) no longer returns the null rows, and column indexes are
null-aware, which makes min/max over a nullable column answer from the
index instead of scanning.

On the packaging side, wheels are now a single Stable ABI (abi3) build per
platform covering CPython 3.11+, with free-threaded 3.14 and 3.15 shipping
alongside.

New features

Mask-based nullable columns for CTable, and they are now the default

• A nullable column keeps its nulls in a per-column validity sidecar —
  Arrow's own model — instead of reserving a value from its own range. This is
  what a bare nullable=True resolves to, and what every nullable column
  inferred from Arrow, Parquet or CSV gets: blosc2.bool(nullable=True) no
  longer reserves 255 and keeps np.bool_, blosc2.int8(nullable=True) has
  all 256 values usable, blosc2.utf8(nullable=True) accepts any string
  including "" and "\x00", and blosc2.complex128(nullable=True) is
  nullable at all for the first time.
• This is what makes nullability lossless. A sentinel steals a value from
  the dtype, so a nullable int8 could not hold -128, a free-text utf8
  column had no safe sentinel at all, and Arrow columns whose type had no value
  to spare could not be imported. to_arrow(from_arrow(x)) now returns x for
  nullable bool, full-range int8/uint8, float64 containing
  nan/±inf/-0.0 as values, utf8 containing "" and
  "__BLOSC2_NULL__", and timestamp with int64.min as a value — none of
  which round-trip through a sentinel.
• None is how you write a null under mask storage (t.append((None,)),
  t["price"][3] = None), which a fixed-width sentinel column cannot accept at
  all. is_null() is unchanged and remains the uniform API across every kind.
• Nothing on disk changes. The new default governs creation only: opening
  a stored table never re-resolves anything, so every existing table keeps the
  storage, dtype and sentinel it was written with, and every rewrite rule for
  the reserved 255 stays permanently in place.
• Sentinel storage is supported indefinitely and is one keyword away, per
  column (null_storage="sentinel", or any explicit null_value=) or globally
  through NullPolicy. Setting a type-wide NullPolicy sentinel field still
  implies sentinel storage for the kinds it covers, so existing
  NullPolicy(float_value=...) code is unaffected — with one unavoidable
  exception: 255 is the only value a nullable bool may reserve, so it is also
  bool_value's default, and NullPolicy(bool_value=255) carries no
  information to act on. A bool column that wants a sentinel has to say so with
  null_storage or column_null_values.
• A table containing a mask column records schema version 3. Only such
  tables do: a table with no nullable column still records version 1, exactly
  as before, and a sentinel one does too. Readers older than 4.11.0 refuse a
  version-3 table rather than misreading it, but their message is a bare
  ValueError: Unsupported schema version 3 — the hint naming
  convert_nulls(to='sentinel') ships in 4.11.0, so only readers that can
  already open the file will print it.
• If some of your data has to stay readable by an earlier release, pin the
  storage rather than discovering this downstream. Per column as above, or
  process-wide — including for schemas inferred from Arrow, Parquet and CSV —
  with blosc2.null_policy(blosc2.NullPolicy(null_storage="sentinel")). That
  reinstates the sentinel's lossiness (a float column's nulls become NaN
  again, and a type with no value to spare still cannot be imported), which is
  the trade being made.
• Column.null_storage reports where a column keeps its nulls and info
  tags each column (int64 nullable[mask]), so CTable.convert_nulls() can
  move columns between the two in either direction — never implicitly, and
  refusing rather than silently relabelling data when a sentinel is
  unavailable.
• One deliberate semantic difference: in a mask column NaN is a value,
  following Arrow, and only the sidecar marks a null. Sentinel float columns
  keep NaN-as-null. See "Where nulls are stored" in the CTable reference.

Column indexes are null-aware

• Per-segment min/max are taken over the rows that carry a value. A
  column's nulls are read from its validity channel and left out, and a segment
  with no value at all is flagged rather than summarised. This applies to both
  storages — an INT64_MIN sentinel is exactly as invisible to a summary as a
  mask column's fill.
• Column.min/Column.max answer from the index for a nullable column
  (236x on a 20M-row int64, measured), where before every nullable column
  but a NaN-sentinel float had to scan.
• where() with an OR over a nullable indexed column uses the index
  instead of falling back to a full scan (1.6x on a 20M-row two-column
  probe). The fallback existed because the only null filtering available was
  global, and a global filter drops a row that is null in one branch but
  matches the other; the segment path never needed it, because it evaluates
  the predicate, which has been null-aware per leaf since the string-predicate
  fix below.
• Indexes written by an earlier release are read as not null-aware and keep
  the old fallback, so nothing silently changes meaning; rebuild_index()
  promotes them.
• Building an index over a nullable column that holds nulls costs one
  decompression pass (33 ms for a 20M-row int64 column), because the
  incremental per-block summaries folded during writes carry no validity; a
  nullable column with no nulls keeps that fast path untouched.

Predicates over nulls follow three-valued (Kleene) logic

• A comparison against a null is now unknown, the third value SQL and Arrow
  both use, and &, |, ^ and ~ combine it by Kleene's rules instead of
  collapsing it to False at the leaf. t.where(t.price > 10) gives the rows
  definitely above 10, t.where(~(t.price > 10)) the rows definitely not
  above 10 — nulls in neither.
• where() keeps what a predicate is true for, so the rows it returns for
  a plain comparison are unchanged. What this fixes is everything built on top
  of one: ~(t.price > 10) used to invert a null that had already been
  collapsed to False and so returned every null row — the exact opposite of
  the intent — and ~((a > 10) & (b == 999)) dropped rows that qualify,
  because unknown & false is false, not unknown, and only a real third
  value can express that. Both query forms are covered and both now agree with
  SQL: the string form carries the second channel through an AST rewrite under
  negation.
• A predicate can be asked about its unknown rows rather than only filtered
  with: p.is_null() gives the rows it cannot answer for, p.null_count()
  counts them, and t.where(p.fillna(True)) keeps what cannot be ruled out.
  fillna(False) is the other reading, and is what where() applies
  implicitly.
• Predicates over non-nullable columns are untouched and cost nothing
  extra; the result of a nullable comparison is still a blosc2.LazyExpr, so
  it computes, indexes and plans exactly as before. Measured cost of the exact
  answer: a negated two-column conjunction over a 20M-row nullable table runs
  1.15x slower than the wrong answer it replaces; every other predicate shape
  is unchanged.
• Two consequences worth knowing. t.where(dict_col != "x") no longer
  returns the rows where dict_col is null (its reserved code differs from
  every value's, so it used to match); dict_col == None remains how to ask
  for them. And Column.isin() stays deliberately two-valued — it returns a
  materialized array and has its own spelling for nulls (None among the
  values).

Packaging and other changes

• Wheels are a single Stable ABI (abi3) build per platform. Instead of one
  wheel per CPython minor version, each platform ships one cp311-abi3 wheel
  that serves CPython 3.11 and every later version, including ones released
  after this one. Nothing changes for pip install blosc2; what changes is
  that a new CPython no longer has to wait for a blosc2 release to be
  installable from a wheel. CI installs that one wheel on 3.11 through 3.15 and
  runs a slice of the suite on each, since cibuildwheel only tests a wheel on
  the interpreter that built it.
• Free-threaded builds are shipped too, as version-specific cp314t and
  cp315t wheels — the free-threaded stable ABI (abi3t, PEP 803) starts at
  3.15 and Cython cannot emit it yet. The limited API costs nothing measurable
  here: across the Linux and Windows cells of a throwaway benchmark matrix the
  worst ratio over every benchmark was 1.075x and 1.054x respectively,
  including the call-heavy ones where an ABI cost would show up first.
• CSV reads and writes take an encoding=. from_csv() defaults to
  utf-8-sig (plain UTF-8, absorbing a byte-order mark if present) and
  to_csv() to utf-8, but either can be given another codec, which is what
  an existing file written in a platform codec needs.
• **Three new [optimization tips](https://www.blosc....

Release 4.10.1

Changes from 4.10.0 to 4.10.1

A correctness release: lazy indexing and reductions now follow NumPy in a
batch of cases where they quietly did not, the stores close several
cross-process read races, and wheels finally ship usable C-Blosc2 development
files. Bundled C-Blosc2 moves to 3.3.2.

Bug fixes

Lazy expressions and indexing

• Indexing a lazy expression with an integer squeezed too much.
  expr[0] dropped every length-1 axis of the result, including ones the
  index kept, so a (1, 4) expression indexed at [0] came back as (4,)
  where NumPy gives (4,) only for the consumed axis and keeps the rest.
  Only the dimensions the integer indices actually consumed are dropped now.
  where() results, whose length is data-dependent, are left alone.
  Closes #319.
• LazyUDF and broadcast operands mis-indexed on None. A None in the
  key inserts an axis in the result but consumes none in the operand;
  aligning it as if it did shifted every axis to its left by one, so
  expr[None, 2] read the wrong operand region. Closes #403, #688.
• Indexing a full reduction materialized the whole operand. (a + b).sum() [key] evaluated the reduction over everything and then indexed; the
  operands are sliced first now. Closes #457.
• Datetime comparisons in lazy expressions raised. numexpr has no datetime
  type, so t1 < t2 on datetime64/timedelta64 died with unknown type datetime64[s] and the error was re-raised rather than letting the NumPy
  fallback try. They are compared as their underlying int64 counts, which is
  exact, keeping the fast path. Mixed units and NaT decline that route and
  fall back to NumPy, since raw counts would silently lie about both.
  Closes #409.
• NDArray.nbytes reported the padded size. It now returns the logical
  size * itemsize, matching NumPy, whenever the shape does not fill the
  chunk grid exactly. cratio still measures the stored (padded) data, so
  nbytes / cbytes need not equal cratio; use .schunk.nbytes for the
  padded figure. Closes #544.

Tables and stores

• CTable.where() applied a short boolean mask to the wrong rows. A mask
  shorter than the live-row count was padded out to the physical length,
  which aligned it with the underlying column and selected rows outside the
  view. A mask no longer than the live-row count is now treated as logical —
  entry i selects the i-th live row — with a short one simply leaving the
  trailing rows unselected. Closes #607.
• Cross-process read races in EmbedStore and DictStore. Both resolved
  a key under the store lock but read the data after releasing it, so a
  concurrent writer could be caught mid-mutation: EmbedStore.__getitem__
  returned bytes from a stale offset, and DictStore.__getitem__ opened an
  external leaf that a concurrent overwrite had just removed or half-rewritten
  (KeyError, or RuntimeError: Error while getting the buffer). The
  resolve, the existence check and the open now share one lock; non-shared
  stores skip it entirely. Fixes #691, #692.
• Overwriting an external DictStore leaf is now atomic. __setitem__
  removed the old leaf and rebuilt it at the same path, and the handle
  __getitem__ returns holds no file descriptor — the C layer re-opens the
  leaf by path for every chunk it decompresses — so a read already in flight
  could open a truncated file. The new leaf is built beside its final name and
  moved in with a single os.replace(), so every such re-open sees one
  complete cframe or the other. A crash mid-write now leaves a stray .tmp
  staging file rather than a partial leaf.
• TreeStore.close() swallowed inline handle failures. A CTable that
  failed to flush left an archive whose row count disagreed with a varlen
  column, reported only on read, long after close() said it succeeded. Every
  handle still gets a chance to close and the store is still packed; the
  failure is then re-raised.

Other

• unpack_tensor() turned padding into a phantom column. np.dtype()
  renames the empty-named padding fields of a structured descr ('' -> f2),
  so a packed tensor with padding came back with an extra field. Closes #287.

Packaging

• Wheels ship usable C-Blosc2 development files. Install paths are now
  relative to CMAKE_INSTALL_PREFIX; the absolute ones made C-Blosc2 generate
  a blosc2.pc and exported targets pointing into the build tempdir, so
  pkg-config and find_package(Blosc2) both failed against an installed
  wheel. Verified by building and linking a C program against a wheel three
  ways: pkg-config, Blosc2::blosc2_shared and Blosc2::blosc2_static.
  miniexpr's license texts are mirrored into .dist-info/licenses, where PEP
  639 tooling looks. Closes #627.
• Wheels carry two copies of libblosc2 instead of three, ~1.5 MB
  smaller. C-Blosc2 set both VERSION and SOVERSION, and scikit-build-core
  follows symlinks, so the fully versioned file — referenced by nothing but
  the symlinks pointing at it — was shipped as a third full copy.

Development

• The test suite runs in parallel by default (-n auto --dist loadfile in
  pytest.ini), 130s -> 35s locally, falling back to a serial run when
  pytest-xdist is absent. The heavy tests, 58% of everything collected and
  excluded from every push-time job, now run in a nightly workflow. The
  network tests run once per push on a single Linux job instead of five times.
• The ruff rule set is spelled out with select rather than extend-select,
  so a ruff release widening its defaults no longer redefines what CI
  enforces.

Releare 4.10.0

Changes from 4.9.1 to 4.10.0

This release is the string-support milestone: string expressions and DSL
kernels now run on miniexpr, utf8() and dictionary() columns gain full
indexing, comparisons, and a documented conversion pair, and NumPy's
StringDType is understood by the array constructors. Alongside, slicing
with plain keys is up to 1.7x faster, a new blosc2.random module provides
chunk-parallel NumPy-quality random constructors, and the optimization-tips
guide gained two new tips and refreshed figures.

New features

• String-valued expressions and DSL kernels over fixed-width <Un
  arrays now run on miniexpr instead of falling back to NumPy.
  Concatenation (arr + "suffix", "prefix=" + arr) plus lower,
  upper, strip/lstrip/rstrip, removeprefix, removesuffix,
  replace, substr and split_part all produce string results, and
  @blosc2.dsl_kernel accepts method syntax (name.lower()) and tuple
  unpacking (before, after = desc.split(sep, 1)), which are rewritten to
  the DSL grammar. The output width is inferred by miniexpr and the
  container is allocated from it, so nothing truncates — .dtype may be
  wider than NumPy's exact answer, never narrower.

• Bytes (S) arrays go through the same engine, with NumPy's S
  semantics rather than <U's: ASCII-only case mapping (so upper/lower
  keep the width instead of growing) and ASCII-only stripping. S and <U
  operands do not mix in one expression, which is what NumPy does too.
  Variable-width utf8() columns still use the NumPy path.

• String expressions now work on utf8() columns.
  t.where("name == 'x'"), startswith/endswith/contains and mixed predicates such as
  t.where("(name == 'b') | (x > 2)") used to raise NotImplementedError;
  only the operator form t[t.name == "x"] was available. A variable-length
  column cannot be an expression operand (its offsets and data have
  independent chunk grids, so the prefilter contract does not apply), so
  these are evaluated span by span, each span materialized to a fixed-width
  array whose width is rounded up to a power of two and handed to miniexpr.
  Nulls are materialized to "" before any kernel sees them and re-masked
  afterwards, so a null never satisfies a predicate — the same answer the
  operator form gives.

• Scalar comparisons on utf8() columns are 5-6x faster in expression
  form. t.where("name == 'x'") (and !=, <, <=, >, >=, either
  operand order) is now answered by the same raw-byte scan the operator form
  t[t.name == "x"] uses, instead of decoding the column to fixed-width
  first: 156 -> 28 ms over 1M short values, 268 -> 56 ms over 1M ~31-byte
  values. Mixed expressions get whatever they can -- in
  startswith(name, 'x') | (name == 'zz') the comparison takes the fast
  path and startswith still decodes.

• New blosc2.utf8_array(seq, spec=None) builds a UTF8Array from an
  iterable of strings; UTF8Array is exported too. Previously the only
  construction path was UTF8Array(spec) + .extend() + .flush(), which
  was not exported at all.

• df.apply(f, axis=1, engine=blosc2.jit) now runs row["colname"]
  kernels that contain an if. Neither dispatch route could before:
  tracing evaluated the branch over a whole column (truth value ... is ambiguous)
  and the DSL parser rejected the subscript. Such references are
  now rewritten into named parameters, so the function is compiled and every
  branch runs. This is not string-specific — numeric row kernels with a
  branch were equally blocked. String columns reach this route too, which
  makes the pandas-3 "format room info" kernel run unmodified. Nulls in a
  string column are rejected rather than substituted, since a row-wise kernel
  over a null raises in pandas as well.

• New blosc2.random module: seedable, NumPy-quality random NDArray
  constructors. Each chunk gets its own independent SeedSequence-spawned
  stream and is generated concurrently in a thread pool, giving full PCG64
  quality with genuinely parallel generation (measured ~3x faster than
  asarray(np.random.default_rng(...).random(...)) on a 100M-element array).
  Covers 42 of numpy.random.Generator's 43 public methods (full
  compatibility table in doc/reference/random.rst):

  • Core: random, integers, normal, uniform, choice
    (replace=True only).
  • 30 scalar distributions: beta, binomial, chisquare,
    exponential, f, gamma, geometric, gumbel, hypergeometric,
    laplace, logistic, lognormal, logseries, negative_binomial,
    noncentral_chisquare, noncentral_f, pareto, poisson, power,
    rayleigh, standard_cauchy, standard_exponential,
    standard_gamma, standard_normal, standard_t, triangular,
    vonmises, wald, weibull, zipf.
  • 4 vector-valued distributions (output shape shape + (k,), one draw
    per trailing vector): dirichlet, multinomial,
    multivariate_hypergeometric, multivariate_normal.
  • permutation, permuted, shuffle: unlike the rest of the module,
    these are not chunk-parallel — whole-array shuffling is inherently
    sequential, so they materialize the full array and shuffle it
    single-threaded. shuffle additionally requires its argument to
    already be an NDArray, since it mutates in place and returns None,
    matching numpy.
  • Not implemented: bytes (returns raw bytes, not an NDArray).

• create_index() now works on utf8() columns, the last string flavour
  without one. Both utf8 and dictionary are indexed by the alphabetical
  rank of each value: sorting by rank is sorting by the decoded string, so an
  int32 rank column drives the same machinery a numeric column uses. At 1M
  rows / cardinality 20k: sort_by 424 ms -> 7.2 ms, sorted_slice 458 ms ->
  43 ms, and the index is the cheapest of the three flavours to build (277 ms
  against 867 ms for <U). Scalar comparisons are served from it too — utf8
  == 29.0 ms -> 5.5 ms, < 34.6 ms -> 5.5 ms, and the dictionary operator
  form t[t.c == v] 329.6 ms -> 8.4 ms. startswith/substring searches are
  not accelerated (no index covers them), and ranks are frozen at build time,
  so a value inserted ahead of existing ones sends the index stale until it is
  rebuilt.

• CTable.add_column() accepts values=, a sequence with one entry per
  live row, as an alternative to backfilling from a declared default. This is
  the supported way to land a result computed outside the table back into it,
  which matters most for utf8() columns: string-returning expressions are
  evaluated on fixed-width arrays, and the result previously had to be written
  through the private t._cols[name].set_all(...). A declared default is still
  honoured for rows appended later, so the two can be combined. values= is
  checked against the constraints declared on the spec, like the constructor
  and extend() are: without that, coercion to a fixed-width dtype would
  truncate an over-long string to max_length instead of complaining.

• blosc2.from_utf8() / blosc2.to_utf8() and UTF8Array.astype() make
  the conversion between variable-length and fixed-width text an explicit,
  documented pair. utf8 columns store and filter text compactly, but
  string-returning expressions need miniexpr's compile-time output width, so
  they run on fixed-width arrays; the rule is now written down (see "Computing
  strings on a utf8 column" in the CTable reference) rather than left for
  callers to discover. from_utf8() sizes the result to the longest value in
  codepoints, counted from the raw bytes without decoding a row, so nothing
  truncates and non-ASCII text does not over-allocate the 3-4x a byte-length
  bound would.

• The array constructors dispatch on NumPy's StringDType.
  blosc2.asarray(np.array([...], dtype=StringDType())) used to raise
  TypeError: data type 'StringDType()' not understood, and
  blosc2.zeros(n, dtype=StringDType()) a malformed node ValueError; both
  now return a UTF8Array, as do empty, ones and full, with the same
  fill values NumPy uses ('', '', '1', str(fill_value)). The dispatch
  is on the target dtype, so asarray(utf8_source, dtype="<U8") still gives
  a fixed-width NDArray. StringDType still cannot back an NDArray — it keeps
  each row's payload outside the array buffer and offers no buffer protocol,
  so compressing that buffer would persist pointers — which is why the
  variable-length container is what comes back.

• UTF8Array gained .shape, .ndim, .size and __array__, so it now
  satisfies the blosc2.Array protocol (.shape was the only member it
  lacked) and np.asarray(arr) returns StringDType instead of silently
  widening to a fixed-width <Un — for 200-character values that was 1600
  bytes where the payload is 203, and a different dtype than arr[:] reported
  for the same object.

• Column.assign() works on utf8, vlstring, vlbytes, struct and object
  columns. It previously raised TypeError: UTF8Array assignment index must be int, leaving no public way to overwrite a variable-length column's
  values. These are now rewritten whole (one write per backing batch) rather
  than row by row, which for the batched varlen columns would have rewritten a
  whole batch per row.

• @blosc2.jit dispatches control flow to the DSL, and the DSL engine
  widened its operand and function coverage. miniexpr's prefilter now
  gathers blocks directly from raw NumPy buffers instead of converting
  operands with asarray(), and Series are accepted as DSL operands.
  np.foo(...) calls inside a jitted function are rewritten to the bare
  names miniexp...

Release 4.9.1

Changes from 4.9.0 to 4.9.1

A small hot-fix release for the Arrow interop work in 4.9.0: a real
performance regression in dictionary-column export, and a clearer error
message when opening a nonexistent CTable in append mode.

Improvements

• CTable.iter_arrow_batches() (and therefore to_arrow() and the Arrow
  PyCapsule interchange, __arrow_c_stream__) no longer recomputes the
  full live-row-position array from scratch on every batch, for every
  dictionary column — an O(n_rows) scan that was repeated
  O(n_rows / batch_size) times. The position array is now computed once
  per export call instead. Measured 6-14x faster export for
  dictionary-encoded string columns (e.g. company) on a 1M-row table.
• Reminder for anyone consuming a CTable through the Arrow PyCapsule
  protocol (DuckDB, pyarrow, Polars, pandas): the raw Arrow C Stream
  interface has no column-projection pushdown, so a consumer that only
  needs a few columns still triggers export of every column in the table.
  Use CTable.select([...]) to project down to the columns you actually
  need before handing the table to the consumer, particularly if any
  column is an expensive nested/list type.

Bug fixes

• Opening a CTable with mode="a" at a path that doesn't exist yet now
  raises a clear FileNotFoundError ("mode='a' opens an existing table;
  use mode='w' to create a new one") instead of silently falling through
  and creating a new, empty table.

Release 4.9.0

Changes from 4.8.1 to 4.9.0

This release is about cooperation: CTable now speaks the tabular
ecosystem's own protocols instead of asking it to speak blosc2's. Arrow
tools (pyarrow, DuckDB, Polars, and pandas >= 3.0 via DataFrame.from_arrow())
can consume or produce a CTable directly through the Arrow PyCapsule
interface; a new utf8() string column stores text in Arrow's own
offsets+bytes layout and reads back as NumPy StringDType; and
engine=blosc2.jit now runs correctly inside pandas 3 itself. Alongside
that, CTable gained a proper missing-data story (fillna/dropna,
null-safe arithmetic and comparisons) and a pandas-3-style chaining API
(assign()/col(), UDF aggregations, CTable.apply()).

New features

Ecosystem interop

• Arrow PyCapsule interchange: CTable.__arrow_c_stream__ lets
  pyarrow, DuckDB, and Polars consume a CTable directly as a stream of
  record batches, with bounded memory — no to_arrow()/copy step
  required. pandas >= 3.0 can do the same via the new
  pandas.DataFrame.from_arrow() classmethod (the plain pd.DataFrame(t)
  constructor does not use this protocol). CTable.from_arrow() now
  accepts any object implementing the same protocol on ingest
  (single-argument form), in addition to the existing (schema, batches)
  form — including Arrow's string_view layout (Polars' default string
  export type), which raised TypeError before this release.
• blosc2.utf8(): a new column type for high-cardinality/free-text
  strings, storing each column as two companion NDArrays — int64 row
  offsets plus a UTF-8 byte blob — the same layout Arrow uses for
  large_string. A row costs exactly its encoded byte length (7-13x
  smaller uncompressed than fixed-width string() on high-cardinality
  text), and reads materialize as NumPy StringDType arrays (NumPy >=
  2.0 required; older NumPy falls back to vlstring on Arrow/Parquet
  import with a clear message). Full query surface: comparisons,
  where(), sort_by, group_by keys, fillna, and Arrow export
  (large_string, sentinel-null mask) / import (Arrow/Parquet string
  columns now default to utf8 instead of vlstring). Measured on the
  1e7-row NYC-taxi company column: ingest 3622 ms -> 598.6 ms
  (6.05x faster, only 1.22x slower than string() and 2.63x faster
  than vlstring()), full-column read 2472.6 ms -> 165.3 ms (14.96x
  faster), equality filter ~1900 ms -> 162 ms (~11.7x faster), and
  groupby-key factorization 3304 ms -> 558 ms (2.83x, down from a
  16.9x-slower initial fallback, now faster than fixed-width string()
  keys). See the new "Choosing a string column type" guide in the
  CTable reference and examples/ctable/utf8_strings.py. Known gaps:
  string-expression filters (t.where("name == 'x'")) and
  create_index on utf8 columns still raise a clear
  NotImplementedError; use fixed-width string() if you need those.
• pandas engine, pandas 3 compatible: engine=blosc2.jit for
  DataFrame.apply now returns a properly indexed DataFrame/Series
  under pandas 3.0.3's default raw=False (previously a raw NumPy
  array, so results only matched by value, never by type).
  Series.map(func, engine=blosc2.jit) is now implemented (it
  previously always raised NotImplementedError). Non-numeric columns
  now raise a clear ValueError instead of a deep numexpr error. See
  the guide "Using Blosc2 as a pandas engine" and
  bench/bench_pandas_engine.py.

pandas-style CTable API

• CTable.assign(**named_exprs): return a view with additional computed
  columns, without mutating the table or copying column data. Pairs with
  the new blosc2.col(name) — an unbound column expression that defers
  operator replay until it's bound to a table (assign(), t[...],
  where()) — to write pandas-3-style chains:
  t.assign(profit=col("revenue") - col("cost"))[col("profit") > 0].sort_by("profit", ascending=False).head(10).
• Missing data: Column.fillna(value) replaces sentinel/None
  values for scalar, dictionary, and varlen-scalar columns;
  CTable.dropna(subset=None) returns a view excluding rows where any
  nullable column (or a chosen subset) is null. Column arithmetic
  (+ - * / // % **) and comparisons (< <= > >= == !=) on nullable
  int/timestamp/bool columns now propagate nulls instead of operating on
  the raw sentinel: arithmetic promotes to float64/NaN, comparisons
  follow SQL WHERE semantics (a null operand never satisfies any
  comparison) — fixing filters like t[t.x < 0] wrongly matching null
  rows. Also fixes null detection for timestamp columns
  (is_null()/null_count()/dropna() previously missed every
  NaT).
• UDF aggregations: group_by().agg() accepts a custom callable as
  the op via the named form (output_name=(column, callable[, dtype]));
  it receives each group's live, non-null values as a 1-D NumPy array.
  CTable.apply(func, columns=None, dtype=None, engine="auto") applies
  a UDF across the table's live rows, sugar over blosc2.lazyudf().
  group_by(engine=...) now accepts "auto"/"numpy" explicitly
  alongside the existing default.
• CTable views are now read-only for value writes: Column.__setitem__
  and Column.assign() raise ValueError on a view, pointing at
  take()/copy() as the escape hatch (structural mutations already
  raised; this closes the one unguarded cell-write path).
• NDArray.iter_sorted()/argsort() on a FULL-indexed array now reads
  the sidecar range directly instead of building the full permutation —
  ~52x faster and ~193x less memory on a 20M-element array for
  iter_sorted(start=-k)-style tail queries. See the new optimization
  tip.

Improvements

• String-key group_by() (fixed-width string() keys) now factorizes
  via an exact hash of each row's raw bytes instead of a NumPy
  UTF-32 argsort, with a vectorized collision-checked verify pass
  keeping the result bit-identical: 1157 ms -> 737 ms on a 1e7-row
  benchmark. Every caller benefits automatically; no engine= switch
  involved.
• An example showing ctables and ndarrays bundled together in the same
  TreeStore.
• C-Blosc2 bumped to 3.2.3.

Bug fixes

• Fixed a @blosc2.dsl_kernel-decorated function crashing unconditionally
  when passed as a groupby UDF aggregation (g.agg(name=(col, dsl_kernel_fn))): it now runs like the equivalent undecorated callable.
• Fixed CTable.head()/tail() silently discarding row order when called
  on a lazily-sorted view (e.g. t.sort_by("col", ascending=False) on a
  view, or any .sort_by() result chained off a prior filter): they
  ignored _cached_live_positions and built a plain physical-order mask
  instead, so t.where(...).sort_by("x", ascending=False).head(10) came
  back in the wrong order.
• Fixed a NameError when nan/inf scalars appeared in lazy
  expressions; ShapeInferencer no longer ignores user-provided shapes
  for nan/inf.
• Fixed CTable.from_arrow() raising TypeError: No blosc2 spec for Arrow type DataType(string_view) on any Arrow string_view/binary_view
  column — the layout Polars exports by default through the PyCapsule
  protocol. string_view/binary_view now import exactly like
  string/large_string/binary/large_binary everywhere a column's
  Arrow type is inspected (schema inference, null-sentinel selection,
  list/struct/dictionary value types).

Release 4.8.1

Changes from 4.8.0 to 4.8.1

Improvements

• Read-only memory mapping for CTable stores: CTable.open() (and
  FileTableStorage) gain an mmap_mode="r" parameter, mirroring
  blosc2.open(). All members of a read-only store — scalar, list, varlen
  and dictionary columns alike — are then read from mapped pages; for .b2z
  archives, in place at their offsets inside the single mapped container
  file. With several concurrent readers on one file this pays off quickly:
  2.5x/4.4x/4.5x faster wall time for 1/4/8 readers in our benchmark
  (bench/optim_tips/tip_10_mmap_many_readers.py).
• Reduced memory consumption in CTable.extend() when passed an NDArray.

Bug fixes

• Fixed lifetime/use-after-free hazards around zero-copy cframes and
  vlmeta: schunk_from_cframe()/ndarray_from_cframe() with copy=False
  (the default) returned objects pointing into the caller's bytes buffer
  without keeping it alive, so a temporary cframe (e.g.
  ndarray_from_cframe(response.content)) could be reclaimed under the live
  object, corrupting reads. The buffer is now pinned on the returned object.
  Also, vlmeta read paths (__getitem__/__len__/__iter__) now raise
  ReferenceError on an orphaned owner instead of segfaulting, matching the
  write paths.
• BatchArray.delete() / ObjectArray.delete(): negative-step slices
  (e.g. del arr[3:0:-1]) deleted chunks in ascending order, shifting the
  indices of chunks still to be deleted and removing the wrong ones (or
  raising RuntimeError).
• ListArray.extend_arrow(): Arrow chunks were appended to the backend
  without flushing pending cells first, reordering unflushed rows after the
  new ones.
• Shape inference: stack() with a negative axis inserted the new dimension
  one position too early, so a lazyexpr's reported .shape disagreed with
  its computed result; vecdot() also normalized positive axes as if they
  were negative.
• Chunked matmul(): broadcast (size-1) operand batch dims were sliced with
  the result-chunk coordinates, producing empty slices when the broadcast
  dim spans several result chunks.
• DictStore.__setitem__(): overwrite semantics depended on value size —
  embedded keys refused overwrite ("already exists"), while an
  embedded-to-external overwrite double-stored the key and resurrected the
  stale embedded value after a delete. Assignments now behave uniformly
  dict-like, dropping any previous value.
• Explicitly passing cparams=None or dparams=None to NDArray
  constructors crashed; both now mean "defaults".
• Fixed a builtin-shadowing bug that made store detection in
  blosc2.open() recurse ~250 times (silently swallowed) on every
  .b2z/.b2d open; opening a .b2z is now ~10x faster under allocation
  tracing.

Documentation

• Restructured docs (#674), with a new
  Optimization tips
  section, including new tips on grouping related data into a single
  memory-mapped .b2z file and on using mmap_mode="r" with many
  concurrent readers.

Release 4.8.0

Changes from 4.7.0 to 4.8.0

Sharing containers across processes

• New locking storage parameter (and the BLOSC_LOCKING environment
  variable to enable it fleet-wide) serializes accesses to an on-disk
  SChunk/NDArray/EmbedStore/DictStore against other handles and other
  processes, via a small sidecar lock file (.b2lock). Advisory: every
  handle touching the container must opt in.
• SChunk.holding_lock() / NDArray.holding_lock(): a context manager to
  hold the exclusive lock across several operations, making a multi-step
  mutation atomic to other locked handles.
• New SChunk.refresh(), mirroring the existing NDArray.refresh().
• Fixed a data-loss bug in NDArray.append(): it read the cached,
  unrefreshed shape before computing the resize target, so under
  concurrent growth/shrink — even inside holding_lock() — another writer's
  just-appended data could be silently deleted.
• EmbedStore and DictStore (.b2d) now support cross-process writers
  under locking: transactional writes plus key-map re-sync, so readers
  follow keys added or removed by another process.
• DictStore.to_b2z() (and TreeStore, which inherits from it) now replaces
  the target file atomically, so concurrent readers always see either the old
  or the new archive, never a torn one.
• Growth-SWMR (single writer, multiple readers): a reader NDArray handle
  opened before a resize() made through another handle follows the new
  shape on its next data access, or via the new explicit NDArray.refresh().
• New user guide page,
  Sharing containers across processes,
  covering all of the above plus the caveats (NFS, mmap_mode, Windows
  in-use-file rename).

Bug fixes

• Fixed detect_aligned_chunks() (used internally to fast-path aligned
  slice reads/writes): a floor-division undercounted the chunk grid for
  arrays whose shape isn't a multiple of the chunk shape, which could
  silently return the wrong chunk's data for an otherwise-aligned slice
  with a nonzero start in an earlier dimension.

Others

• Raised the manylinux wheel baseline from manylinux2014 (CentOS 7, glibc
  2.17, GCC 10.2) to manylinux_2_28 (AlmaLinux 8, glibc 2.28, GCC 12),
  fixing a build failure with NumPy >=2.5 which requires GCC >=10.3.

Release 4.6.0

Changes from 4.5.1 to 4.6.0

CTable.sort_by(view=True): zero-copy sorted views

• CTable.sort_by() now accepts view=True, returning a lightweight
  sorted view that shares the parent's column data and gathers rows on
  demand in sorted order — no whole-table copy. This is ideal for reading a
  sorted slice of a large (possibly on-disk) table::

  t.sort_by("col", view=True)[:10]      # top-10 without materialising
  

  Sorting on a fully indexed column streams directly from the index, so the
  table is never materialised. Multi-column sorts and dotted (nested) leaf
  names are supported (e.g. t.sort_by(["trip.begin.lon", "payment.fare"], ascending=[True, False])).

where on dictionary (string) columns

• where expressions now work over dictionary-encoded (string) columns,
  including membership tests such as '"Acme" in company', so categorical
  text columns can be filtered without decoding the whole column.

b2view is now an opt-in extra

• The b2view terminal browser and its TUI stack (textual,
  textual-plotext) are no longer core dependencies: a plain
  pip install blosc2 no longer pulls them, keeping the compression library
  lean (and dropping deps that are unusable under wasm32, which has no TTY).
  Install the viewer with pip install "blosc2[tui]", or
  pip install "blosc2[hires]" to also get the high-res h view. The
  b2view command prints this hint if the dependencies are missing.

group_by: flexible aggregation naming

• CTable.group_by(...).agg() now accepts a list of (column, ops) pairs
  and explicit output names (pandas-style keyword arguments), alongside the
  existing auto-suffixed mapping; the forms can be combined::

  g.agg({"sales": ["sum", "mean"]})              # auto: sales_sum, sales_mean
  g.agg([(t.sales, ["sum", "mean"])])            # auto, but accepts Column objects
  g.agg(revenue=("sales", "sum"))                # explicit: revenue
  g.agg({"sales": "sum"}, n=("*", "size"))       # combined, with a named row count
  

  The list-of-pairs and named forms accept Column objects (t.sales), which
  the mapping form cannot because Column is unhashable and so cannot be a dict
  key.

• Aggregation ops may also be given as the matching blosc2 reduction functions
  (blosc2.sum, mean, min, max, argmin, argmax), matched by
  identity -- e.g. g.agg([(t.sales, [blosc2.sum, "mean"])]). This is a
  naming shorthand only; arbitrary/UDF callables (and look-alikes such as
  np.sum or a user function named sum) are rejected rather than silently
  misinterpreted.

group_by / group_reduce: tri-state sort=

• Vectorized dictionary group ordering: group_by() result building now
  batch-decodes dictionary (string) keys in one pass (decode_batch) instead of
  one decode() per group, making high-cardinality string group-bys dramatically
  faster (end-to-end group_by().size() dropped from seconds to milliseconds on
  ~100k-group workloads).
• sort= is now a tri-state (None / True / False) on both
  CTable.group_by() and blosc2.group_reduce():
  • True — always return groups sorted by key.
  • False — never sort; deterministic but unspecified order.
  • None (the new default) — auto: sort only when cheap. Integer and
    dictionary keys are sorted (free / vectorized); float and multi-key results,
    whose only ordering is an O(G log G) Python sort over every distinct group,
    are left unsorted to avoid a cost that can rival the grouping itself on
    high-cardinality data.
• Behavior changes (the two APIs had different prior defaults, so they move
  in opposite directions):
  • CTable.group_by() previously returned results always sorted. Under the
    new None default, float-key and multi-key group-bys are no longer
    key-sorted by default — pass sort=True to restore sorted output. This is
    a deliberate divergence from pandas (which defaults to sort=True), suited
    to blosc2's large / on-disk datasets.
  • blosc2.group_reduce() previously defaulted to sort=False (unsorted).
    Under the new None default its cheap kernels now sort by default —
    most visibly float keys, which previously came out in hash order. Integer
    keys were already ascending; the generic Python fallback stays unsorted.
    Pass sort=False to opt out.

Accelerated reductions from index summaries

• min/max on indexed Columns, and argmin/argmax inside group_by, are
  now accelerated using the index's per-block min/max summaries: when an
  index is available these reductions run from the precomputed summaries instead
  of decompressing the underlying data, which is dramatically faster on large
  columns. A fast path also builds min/max envelope plots from any index.
• The last group_by operation is memoized and reused when the same
  grouping is requested again, avoiding recomputation in interactive / repeated
  workflows (e.g. b2view).

b2view: group-by, sort, and richer plots

• Interactive group-by (G): group a CTable by a column (integer, string,
  or now float keys) directly in the viewer, with a three-list / two-column
  menu; while grouped, S/R operate on the grouped result and the data
  panel's subtitle shows a G(roup) chip. The last grouping is memoized for
  instant reuse.
• Sort by column (S): sort a CTable by a fully indexed column via a
  dropdown (R toggles reverse) as a zero-copy sort_by(view=True) that streams
  from the index — the table is never materialised, Esc restores the original
  order, and a SORTED chip shows in the status bar. Non-indexed columns can
  now be sorted too. Sort and filter are mutually exclusive; a row window
  composes over a sort, and an filter is preserved across Sort / Group.
• Better plots of grouped/sorted views: a grouped view plots bars for a
  categorical key and lines for a numeric key; numeric-key group plots
  render as stem/impulse charts rather than misleading connected lines. Bar
  plots gain an hi-res counterpart mirroring the line/scatter plots, and +/-
  zoom about the view's left edge.
• --max maximizes the current panel, and escape is now the single,
  consistent way to back out of every modal.

Other / bug fixes

• C-Blosc2 upgraded to 3.1.5.
• Open-file cache correctness: cached open handles are now validated against
  the file's fingerprint (st_mtime_ns, st_size) and cached index handles are
  released when a table closes, so a file changed underneath an open handle is no
  longer served stale.
• NumPy 2.5 compatibility: adjusted for deprecations in NumPy 2.5.
• Substantially reduced test-suite runtime, and emscripten builds no longer
  attempt to spawn subprocesses (unsupported there).

Release 4.5.1

Changes from 4.5.0 to 4.5.1

This follow-up release builds the b2view terminal viewer into a richer
data-exploration tool — a scatter plot, a searchable column picker, a
one-shot demo download, refreshed chrome, and several interaction fixes — and
upgrades the bundled C-Blosc2 to 3.1.4. WASM/Pyodide is now a fully
supported platform, and CTable.info reports per-column compressed sizes.

b2view: richer exploration

• Scatter plots: from a column plot, press s to scatter the current column
  (X) against another column (Y) chosen from a list, over the current (zoomed)
  row range; h then opens a high-resolution matplotlib scatter.
• High-res for 1-D series is now an envelope plot (matching the in-terminal
  view), and a new r key toggles between the min/max envelope and the raw
  values (strided-sampled when the range is wide).
• Searchable column picker: the c go-to-column key now opens a searchable,
  selectable list (type to filter, ↑/↓, Enter) for CTables, instead of a text
  field; N-D arrays still go by numeric index.
• Show/hide columns: / opens a searchable multi-select to pick which CTable
  columns are displayed.
• Demo download: b2view --download fetches a demo bundle
  (chicago-taxi-flat.b2z by default) into the current directory if it is not
  already there, then opens it.
• Refreshed chrome: a branded header, a left-docked filename label in the
  title, and clearer status chips.

b2view: interaction fixes

• Go-to-row/column pre-fill is now pre-selected, so the first keystroke
  replaces the current index instead of appending to it (typing a column name no
  longer produced e.g. 0payment.fare).
• Escape keeps its layered exit while a panel is maximized: with the data
  panel maximized, escape now unlocks a plot's locked row window (and clears
  filters) as documented, instead of being hijacked into restoring the panel —
  use r to restore (ESCAPE_TO_MINIMIZE = False).
• Test-suite robustness fixes (a timing flake and a Windows rendering glitch).

Other

• C-Blosc2 upgraded to 3.1.4.
• WASM/Pyodide is now a fully supported platform, with more frequent CI runs.
• CTable.info shows per-column compressed sizes (cbytes and cratio),
  and print_versions() uses clearer Python-Blosc2 / C-Blosc2 labels.

Release 4.5.0

Changes from 4.4.5 to 4.5.0

This release teaches the b2view terminal viewer to plot — peak-preserving
envelope line plots of any series, with zoom, a row-window lock, and an optional
high-resolution matplotlib view — and gives CTable a pandas-like display and
CSV experience. It also publishes WASM/Pyodide wheels to PyPI and adds
faster strided reads for NDArray and Column.

b2view: plotting and data inspection

• In-terminal plots: press p on a numeric series (a CTable column or an
  array row) to draw a braille line plot. Plots are peak-preserving min/max
  envelopes by default, so no spike or trough is hidden however large the
  series is; large local series stream their envelope exactly in bounded
  spans (only remote c2arrays fall back to a labeled strided sample).
• Zoom and row-window lock: zoom the plot into a row range and pan it; press
  v to lock the data grid to the plotted range so paging stays inside it
  (escape unlocks). The plot and high-res views honor the locked window.
• High-resolution view: h opens a high-res matplotlib image of the
  plotted range (new optional hires extra: matplotlib + textual-image).
• On-demand cell decode: enter decodes a single skipped/expensive CTable
  cell, and SChunk nodes now preview as a paged hex dump.
• Fixes and polish: row paging re-aligns to the page grid after dim-mode
  single-row scrolls; the data panel now focuses correctly with
  --path ... --panel data; status chips are branded yellow.

CTable display

• CTable.to_string() now renders the whole table by default (every row and
  every column), like pandas' DataFrame.to_string(). New max_rows and
  max_width parameters truncate on demand. Behaviour change: previously
  to_string() returned the truncated view; code that relied on that should
  pass max_rows=/max_width= (or use str()).
• The [N rows x M columns] dimensions footer now follows pandas: omitted by
  to_string() (pass show_dimensions=True to force it), and shown by
  str/repr/print only when the view is actually truncated. Previously it
  was always appended.
• repr(ctable) now shows the same truncated table as str(ctable)
  (pandas/polars convention), instead of the one-line CTable<…> summary. The
  compact summary remains available via ctable.info.
• New display options in set_printoptions: display_width controls the
  column-fitting width budget (None = auto-detect terminal, -1 = show all
  columns, positive int = fixed budget), and display_rows now accepts -1 to
  show all rows (0 still shows none).
• New blosc2.printoptions(...) context manager temporarily sets the display
  options and restores them on exit, e.g.
  with blosc2.printoptions(display_rows=-1, display_width=-1): print(t).

CTable I/O

• CTable.to_csv() now accepts no path, returning the CSV as a string like
  pandas' DataFrame.to_csv(). Passing a path still writes the file (and
  returns None); the returned string is byte-for-byte the same as the file.

Performance

• Faster strided reads: NDArray.__getitem__ gains a sparse-gather fast
  path for large strides, and Column.__getitem__ short-circuits when the
  logical positions equal the physical ones.
• Fix: a negative step in Column getitem could return []; it now
  returns the reversed selection.

Indexing

• Fix: a sidecar-handle cache collision could return the wrong SUMMARY
  index for a compact-store column.
• Cross-column index pruning is now enabled for compact CTable queries, so
  more predicates prune blocks before any data is materialized. The docs also
  note when summary indexes are not created automatically.

Packaging

• WASM/Pyodide wheels on PyPI: the main wheel build now also produces
  pyemscripten wheels for CPython 3.13 (2025 ABI) and 3.14 (2026 ABI) and
  uploads them to PyPI, so blosc2 is micropip-installable in Pyodide, and
  b2view prints a clear message instead of crashing when run under WASM.
  Known limitation: slicing an in-memory SChunk loaded from a frame fails on
  the Pyodide 0.29.x Emscripten toolchain (cp313); it works on Pyodide 314
  (cp314) and natively. See issue #664.
• cibuildwheel updated to 4.1.