Skip to content

feat: PostgreSQL + pgvector backend (VectorStoreBackend protocol) - #83

Open
isc-tdyar wants to merge 23 commits into
aiming-lab:mainfrom
isc-tdyar:feature/pgvector-backend
Open

feat: PostgreSQL + pgvector backend (VectorStoreBackend protocol)#83
isc-tdyar wants to merge 23 commits into
aiming-lab:mainfrom
isc-tdyar:feature/pgvector-backend

Conversation

@isc-tdyar

@isc-tdyar isc-tdyar commented Aug 26, 2026

Copy link
Copy Markdown

Summary

Adds a PostgreSQL + pgvector backend implementing the VectorStoreBackend
protocol introduced in #76, following the same pattern as the Milvus backend
in #77.

LanceDB remains the default — no existing deployments are affected.


What's new

simplemem/core/database/pg_vector_store_backend.pyPGVectorStoreBackend

  • Implements VectorStoreBackend (insert / semantic_search / keyword_search /
    structured_search / count / get_all / optimize / clear)
  • semantic_score_order = ScoreOrder.ASCENDING (cosine distance, lower = better)
  • keyword_score_order = ScoreOrder.DESCENDING
  • Semantic search: pgvector HNSW index, <=> cosine operator
  • Keyword search: case-insensitive ILIKE matching across lossless_restatement
    and keywords columns
  • Structured search: parameterized WHERE with ::jsonb @> ::jsonb for
    persons/entities array columns, ILIKE for location, range comparisons for
    timestamps — no SQL injection surface
  • Thread-local connections (threading.local()) matching the existing
    parallelism model
  • VACUUM ANALYZE as optimize() equivalent; silently skipped on backends
    that don't support it
  • Schema bootstrap is fully idempotent: CREATE TABLE IF NOT EXISTS,
    CREATE INDEX IF NOT EXISTS

simplemem/core/database/__init__.pyget_vector_store() factory

  • Returns a VectorStore whose backend is PGVectorStoreBackend when
    STORAGE_BACKEND=pgvector (or backend='pgvector' passed directly),
    otherwise falls back to the default LanceDBVectorStoreBackend
  • PGVectorStoreBackend lazily exported (no hard psycopg import at module load)

simplemem/core/settings.py

  • Two new settings: STORAGE_BACKEND (default "lancedb") and PG_DSN

Schema

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE memory_entries (
    entry_id              VARCHAR(512)  NOT NULL,
    lossless_restatement  VARCHAR(4000) NOT NULL,
    keywords              VARCHAR(4000),
    timestamp             VARCHAR(50),
    location              VARCHAR(1000),
    persons               VARCHAR(4000),
    entities              VARCHAR(4000),
    topic                 VARCHAR(500),
    vec                   vector(1024)
);

CREATE INDEX IF NOT EXISTS memory_entries_vec_idx
    ON memory_entries USING hnsw (vec vector_cosine_ops);

All text columns use VARCHAR(N) rather than TEXT for broad compatibility.


Opting in

# config.py
STORAGE_BACKEND = "pgvector"
PG_DSN = "postgresql://user:pass@localhost:5432/mydb"

Or via environment variables:

export STORAGE_BACKEND=pgvector
export PG_DSN=postgresql://user:pass@localhost:5432/mydb

Client installation

pip install "psycopg[binary]" pgvector
# or:
pip install simplemem[pgvector]

Using PostgreSQL

Any PostgreSQL ≥ 15 instance with the pgvector extension works:

CREATE EXTENSION IF NOT EXISTS vector;  -- one-time setup

Using InterSystems IRIS

IRIS does not natively speak the PostgreSQL wire protocol, but
iris-pgwire ≥ 1.7.3
adds a translation layer so any psycopg client — including this backend —
connects to IRIS without IRIS-specific drivers.

Server-side setup (one time per IRIS instance):

pip install "iris-pgwire>=1.7.3"
python -m iris_pgwire.server   # default port 5432

Or via Docker (see iris-pgwire README):

docker compose up -d

Then point PG_DSN at the pgwire endpoint:

# config.py
STORAGE_BACKEND = "pgvector"
PG_DSN = "postgresql://_SYSTEM:SYS@localhost:5432/USER"

No code changes are needed to switch between PostgreSQL and IRIS — only the
DSN changes. iris-pgwire ≥ 1.7.3 translates all pgvector DDL and operators
used by this backend transparently, including CREATE EXTENSION IF NOT EXISTS,
CREATE INDEX IF NOT EXISTS … USING hnsw, ILIKE, ON CONFLICT DO NOTHING,
parameterized LIMIT, and ::jsonb @> containment.


Tests

File Tests Run condition
tests/test_pg_vector_store_backend.py 19 PG_TEST_DSN env var

Tests skip automatically without the env var — no CI infrastructure changes needed.
All 19 tests verified passing against both PostgreSQL and InterSystems IRIS
(via iris-pgwire 1.7.3 on IRIS 2026.2).

PG_TEST_DSN=postgresql://postgres:postgres@localhost:5432/simplemem_test \
  pytest tests/test_pg_vector_store_backend.py -v

Covers: insert, semantic/keyword/structured search, score ordering, empty-input
guards, SQL injection resistance, optimize, clear + reinsert, and end-to-end
via VectorStore + get_vector_store().

tom-dyar and others added 22 commits April 12, 2026 09:42
- Swap database/vector_store.py to use IRIS VECTOR(DOUBLE,N) + HNSW index
  via pure SQL DDL: CREATE INDEX ... AS HNSW(Distance='Cosine')
- Add cross/storage_iris.py replacing cross/storage_lancedb.py with
  identical CrossSessionVectorStore interface, multi-tenant SQL filtering
- Add utils/iris_embeddings.py adapter shim (no langchain_core dep)
- Update MCP/server/database/vector_store.py for IRIS multi-tenant backend
- Update MCP/config/settings.py: IRIS_HOSTNAME/PORT/NAMESPACE/USERNAME/PASSWORD
- Update config.py.example: IRIS connection params replace LANCEDB_PATH
- Update requirements.txt: intersystems-irispython replaces lancedb/pylance/tantivy
- Add AGENTS.md knowledge base files for root, cross/, MCP/, OmniSimpleMem/, SKILL/
- README: add IRIS setup guide, benchmark results, and Claude + IRIS usage section
- Pattern 1: ingest existing IRIS table through MemoryBuilder pipeline
- Pattern 2: HybridCustomerStore subclass to search both tables in one call
- Pattern 3: use VectorStore retrieval layer directly without LLM compression
- ALTER TABLE + populate embeddings example for existing IRIS tables
- Full config reference for IRIS backend settings
- Explain semantic compression vs chunking distinction
- Add new sections to table of contents
Thread safety:
- Replace shared self._conn + RLock with threading.local() in VectorStore
  and CrossSessionVectorStore — each thread gets its own IRIS connection,
  enabling true concurrent DB reads with enable_parallel_retrieval=True

Bug fixes:
- MCP/server/http_server.py: settings.lancedb_path -> correct IRIS init args
- cross/orchestrator.py: lancedb_path param now deprecated+ignored; new
  iris_table param controls table name; backward-compatible
- cross/tests/test_orchestrator.py: lancedb_path -> iris_table

Tests:
- Add tests/conftest.py with IRIS-backed store fixture (pre-populated)
  and bucket_path skip fixture for GCS test
- Test suite: 134 passed, 1 skipped, 0 failures

Documentation:
- README: add IRIS gotchas section covering threading, TOP clause, $FIND
  substring behavior, HNSW requirements, parallel mode table, OmniSimpleMem
  and SKILL/ independence, deprecated lancedb_path param
- Purge stale LanceDB refs from AGENTS.md, cross/README.md,
  cross/session_manager.py, cross/consolidation.py, cross/api_http.py,
  docs/PACKAGE_USAGE.md
…alternative)

- cross/storage_iris_sql.py: IRISSQLStorage — drop-in replacement for
  SQLiteStorage using IRIS SQL tables (CrossMem_ prefix). Thread-local
  connections, IDENTITY columns, idempotent schema creation.
- cross/storage_factory.py: create_sql_storage(use_iris=False) factory —
  selects SQLite or IRIS transparently. Default remains SQLite.
- cross/orchestrator.py: add use_iris_sql=False param; wires factory.
  Existing db_path param still routes to SQLite when use_iris_sql=False.
- cross/__init__.py: export IRISSQLStorage, create_sql_storage, SqlStorage.

Usage:
  orch = create_orchestrator('my-project', use_iris_sql=True)  # full IRIS
  orch = create_orchestrator('my-project')                      # SQLite (default)
- Add use_iris_sql section with before/after comparison
- Show two real SQL JOIN examples: observations→vectors, summaries→scores
- Explain single-namespace benefit for ISC customers
- Update module reference table with storage_iris_sql.py and storage_factory.py
- Add gotchas entry explaining SQLite vs IRIS SQL tradeoff and use_iris_sql flag
- Add single namespace bullet to ISC customer benefits list
…tor)

Merged origin/main (74174a1) into IRIS fork. Resolved 3 conflicts:
- simplemem/core/database/vector_store.py: kept upstream LanceDB (IRIS lives at database/vector_store.py)
- README.md: preserved IRIS backend docs, accepted upstream MCP/Roadmap sections
- utils/iris_embeddings.py: moved to simplemem/integrations/reference/utils/ per upstream rename
…ackage move

- database/vector_store.py: models.* → simplemem.core.models.*
- cross/storage_iris.py: same
- tests/conftest.py: skip IRIS vector store tests when no container reachable
- simplemem/core/database/pg_vector_store.py: PGVectorStore implementing the
  same duck-typed interface as VectorStore (LanceDB). psycopg3 + pgvector.
  semantic_search via <-> cosine, keyword_search via tsvector/plainto_tsquery,
  structured_search via parameterized WHERE (no injection surface).
- simplemem/core/database/__init__.py: get_vector_store() factory; lancedb
  remains default, pgvector opt-in via STORAGE_BACKEND=pgvector + PG_DSN.
- simplemem/core/settings.py: add STORAGE_BACKEND and PG_DSN settings.
- setup.py: add [pgvector] optional extra (psycopg[binary]>=3.1, pgvector>=0.3)
- tests/test_pg_vector_store.py: 11 tests, skip when PG_TEST_DSN not set.

IRIS + pgwire: point PG_DSN at the pgwire endpoint — no IRIS-specific code needed.
- cross/storage_pg.py: PGCrossVectorStore — same interface as CrossSessionVectorStore
  (IRIS). Semantic search via pgvector <->, keyword via tsvector, structured via
  parameterized WHERE with JSON containment. Thread-local connections.
- cross/storage_pg_sql.py: PGSQLStorage — same interface as SQLiteStorage and
  IRISSQLStorage. All tables prefixed for namespace isolation. RETURNING clause
  for insert IDs. _purge_all_test_data() for test isolation.
- cross/storage_factory.py: add use_pg / pg_dsn / pg_table_prefix params;
  lazy imports so IRIS/PG deps are only loaded when actually requested.
- cross/__init__.py: guard IRIS and PG backend imports with try/except so
  missing optional deps don't prevent the package from loading.
- cross/orchestrator.py, session_manager.py, consolidation.py, context_injector.py:
  replace hard top-level iris imports with try/except for optional-dep safety.
- cross/tests/test_pg_cross.py: 35 tests covering both backends; skip when
  PG_TEST_DSN is absent.
31 tests covering PGCrossVectorStore + PGSQLStorage via iris-pgwire
(PostgreSQL wire protocol proxy in front of IRIS). Validates:
- Schema creation, add/count, semantic/keyword/structured search
- Full session lifecycle: create → event → observation → summary → link
- SessionManager round-trip with PG backends
- No SQL injection exposure
- vector ORDER BY rewrite handled transparently by pgwire optimizer

Skip when IRIS_PGWIRE_DSN is absent (set to postgresql://...@pgwire-host/NAMESPACE).
…i MCP, security fixes, Requesty)

Key upstream changes merged:
- refactor: VectorStoreBackend protocol + LanceDBVectorStoreBackend extraction
- feat(omni): MCP server for Omni-SimpleMem multimodal memory
- security: drop vulnerable langchain-openai dep, fix CORS wildcard, SQL escaping
- fix: omni_memory.core.config module, numpy truth-value ambiguity
- docs: PACKAGE_USAGE aligned with current public API, Atlas Cloud example
- Add Requesty as LLM provider

Our pgvector additions preserved: get_vector_store() factory in __init__,
IRIS MultiTenantVectorStore in MCP/server/database/vector_store.py.
- cross/storage_pg.py: PGCrossVectorStore — same interface as CrossSessionVectorStore
  (IRIS). Semantic search via pgvector <->, keyword via tsvector, structured via
  parameterized WHERE with JSON containment. Thread-local connections.
- cross/storage_pg_sql.py: PGSQLStorage — same interface as SQLiteStorage and
  IRISSQLStorage. All tables prefixed for namespace isolation. RETURNING clause
  for insert IDs. _purge_all_test_data() for test isolation.
- cross/storage_factory.py: add use_pg / pg_dsn / pg_table_prefix params;
  lazy imports so IRIS/PG deps are only loaded when actually requested.
- cross/__init__.py: guard IRIS and PG backend imports with try/except so
  missing optional deps don't prevent the package from loading.
- cross/orchestrator.py, session_manager.py, consolidation.py, context_injector.py:
  replace hard top-level iris imports with try/except for optional-dep safety.
- cross/tests/test_pg_cross.py: 35 tests covering both backends; skip when
  PG_TEST_DSN is absent.
…otocol (aiming-lab#76)

Replace the old duck-typed PGVectorStore with PGVectorStoreBackend, which
implements the VectorStoreBackend protocol introduced in aiming-lab#76 (same pattern
as the Milvus backend in aiming-lab#77).

- New: simplemem/core/database/pg_vector_store_backend.py
  Implements insert/semantic_search/keyword_search/structured_search/
  count/get_all/optimize/clear against PostgreSQL + pgvector.
  Thread-local connections, HNSW + GIN indexes, parameterized queries only.

- Updated: simplemem/core/database/__init__.py
  get_vector_store() wires VectorStore with a PGVectorStoreBackend factory
  when STORAGE_BACKEND=pgvector. PGVectorStoreBackend lazily exported.

- Removed: pg_vector_store.py (old duck-typed facade)

- Replaced: tests/test_pg_vector_store.py with test_pg_vector_store_backend.py
  20 tests covering backend-direct and VectorStore integration paths.
  All skip without PG_TEST_DSN — no CI infrastructure changes needed.
…verage

All 19 tests pass against live IRIS via iris-pgwire. Key IRIS compatibility
fixes applied during E2E testing:

- CREATE EXTENSION: wrapped in try/except (IRIS has built-in vector support)
- VARCHAR(N) instead of TEXT (IRIS TEXT returns stream references via pgwire)
- Removed tsvector GENERATED column + GIN index (not supported in IRIS)
- LIKE-based keyword search replaces tsvector/plainto_tsquery
- LIKE '%"value"%' replaces ::jsonb @> for JSON array column matching
- LOWER(col) LIKE LOWER(%s) replaces ILIKE (not supported in IRIS)
- LIMIT inlined as integer literal (parameterized LIMIT not supported in IRIS)
- ON CONFLICT DO NOTHING removed (not supported in IRIS)
- executemany replaced with per-row execute loop (avoids psycopg pipeline mode)
- CREATE INDEX without IF NOT EXISTS; duplicate errors caught in _ensure_schema
- optimize() silently skips VACUUM ANALYZE on backends that don't support it

Also removes duplicate PG import block in cross/__init__.py left from rebase.
…pgwire 1.7.1

iris-pgwire 1.7.1 fixed: CREATE EXTENSION IF NOT EXISTS, ILIKE, LIMIT %s,
ON CONFLICT DO NOTHING, executemany/pipeline mode. Restored standard SQL
patterns for all five. Only remaining IRIS workaround: LIKE '%"value"%' for
persons/entities JSON array matching (jsonb @> containment still broken in
pgwire). CREATE INDEX without IF NOT EXISTS still requires try/except.

All 19 tests pass on IRIS 2026.2 + pgwire 1.7.1.
…re 1.7.2

iris-pgwire 1.7.2 fixed jsonb containment (@>) operator translation.
Replaced LIKE '%"value"%' workaround with proper ::jsonb @> ::jsonb for
persons and entities. Bumped minimum iris-pgwire version reference to 1.7.2.

CREATE INDEX IF NOT EXISTS still broken in pgwire — try/except stays.
All 19 tests pass on IRIS 2026.2 + pgwire 1.7.2.
…d in iris-pgwire 1.7.3

_ensure_schema() is now clean standard SQL. Minimum iris-pgwire version
bumped to 1.7.3. All 19 tests pass on IRIS 2026.2 + pgwire 1.7.3.
@isc-tdyar
isc-tdyar force-pushed the feature/pgvector-backend branch from e39be43 to 58b287e Compare August 26, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants