Skip to content

feat(index): Support vector index rerank#493

Merged
JingsongLi merged 6 commits into
apache:mainfrom
JingsongLi:rerank
Jul 9, 2026
Merged

feat(index): Support vector index rerank#493
JingsongLi merged 6 commits into
apache:mainfrom
JingsongLi:rerank

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

This pull request adds rerank support for vector index search, aligned with the Java implementation. When a positive refine factor is configured, the index search first retrieves a larger candidate set and then reranks those candidates by reading raw vectors and computing exact scores.

Brief change log

  • Support vector index rerank with refine_factor / refine-factor / rerank_factor / rerank-factor option aliases.
  • Resolve rerank options from search options first, then table options, with field/index-specific prefixes taking precedence over global options.
  • Expand index search limit to limit * refine_factor and rerank candidates back to the original query limit.
  • Read raw vectors for the union of candidate row IDs while scoring only the candidates relevant to each query.
  • Keep Java-compatible ranking semantics: higher score first, and smaller row id first when scores tie.
  • Optimize rerank performance with bounded top-K heaps, row-id-to-query routing, zero-copy Arrow vector scoring, and streaming per-query top-K collection.

Tests

  • cargo fmt --all
  • cargo test -p paimon vector_search --lib
  • cargo check -p paimon --all-targets
  • git diff --check

API and Format

This change adds vector search option aliases for rerank/refine factor. It does not change the storage format.

Documentation

No user-facing documentation is updated in this pull request. The supported options are covered by unit tests.

@leaves12138 leaves12138 changed the title [test] Support vector index rerank Jul 9, 2026
@JingsongLi JingsongLi changed the title Support vector index rerank feat(index): Support vector index rerank Jul 9, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The rerank implementation is aligned with the Java behavior, keeps the expected score ordering semantics, and the follow-up optimization avoids unnecessary allocations and full sorts. The accidental README change has been removed, and the check job is now passing.

@JingsongLi JingsongLi merged commit 1352504 into apache:main Jul 9, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants