feat: PostgreSQL + pgvector backend (VectorStoreBackend protocol) - #83
Open
isc-tdyar wants to merge 23 commits into
Open
feat: PostgreSQL + pgvector backend (VectorStoreBackend protocol)#83isc-tdyar wants to merge 23 commits into
isc-tdyar wants to merge 23 commits into
Conversation
- Swap database/vector_store.py to use IRIS VECTOR(DOUBLE,N) + HNSW index via pure SQL DDL: CREATE INDEX ... AS HNSW(Distance='Cosine') - Add cross/storage_iris.py replacing cross/storage_lancedb.py with identical CrossSessionVectorStore interface, multi-tenant SQL filtering - Add utils/iris_embeddings.py adapter shim (no langchain_core dep) - Update MCP/server/database/vector_store.py for IRIS multi-tenant backend - Update MCP/config/settings.py: IRIS_HOSTNAME/PORT/NAMESPACE/USERNAME/PASSWORD - Update config.py.example: IRIS connection params replace LANCEDB_PATH - Update requirements.txt: intersystems-irispython replaces lancedb/pylance/tantivy - Add AGENTS.md knowledge base files for root, cross/, MCP/, OmniSimpleMem/, SKILL/ - README: add IRIS setup guide, benchmark results, and Claude + IRIS usage section
- Pattern 1: ingest existing IRIS table through MemoryBuilder pipeline - Pattern 2: HybridCustomerStore subclass to search both tables in one call - Pattern 3: use VectorStore retrieval layer directly without LLM compression - ALTER TABLE + populate embeddings example for existing IRIS tables - Full config reference for IRIS backend settings - Explain semantic compression vs chunking distinction - Add new sections to table of contents
Thread safety: - Replace shared self._conn + RLock with threading.local() in VectorStore and CrossSessionVectorStore — each thread gets its own IRIS connection, enabling true concurrent DB reads with enable_parallel_retrieval=True Bug fixes: - MCP/server/http_server.py: settings.lancedb_path -> correct IRIS init args - cross/orchestrator.py: lancedb_path param now deprecated+ignored; new iris_table param controls table name; backward-compatible - cross/tests/test_orchestrator.py: lancedb_path -> iris_table Tests: - Add tests/conftest.py with IRIS-backed store fixture (pre-populated) and bucket_path skip fixture for GCS test - Test suite: 134 passed, 1 skipped, 0 failures Documentation: - README: add IRIS gotchas section covering threading, TOP clause, $FIND substring behavior, HNSW requirements, parallel mode table, OmniSimpleMem and SKILL/ independence, deprecated lancedb_path param - Purge stale LanceDB refs from AGENTS.md, cross/README.md, cross/session_manager.py, cross/consolidation.py, cross/api_http.py, docs/PACKAGE_USAGE.md
…alternative)
- cross/storage_iris_sql.py: IRISSQLStorage — drop-in replacement for
SQLiteStorage using IRIS SQL tables (CrossMem_ prefix). Thread-local
connections, IDENTITY columns, idempotent schema creation.
- cross/storage_factory.py: create_sql_storage(use_iris=False) factory —
selects SQLite or IRIS transparently. Default remains SQLite.
- cross/orchestrator.py: add use_iris_sql=False param; wires factory.
Existing db_path param still routes to SQLite when use_iris_sql=False.
- cross/__init__.py: export IRISSQLStorage, create_sql_storage, SqlStorage.
Usage:
orch = create_orchestrator('my-project', use_iris_sql=True) # full IRIS
orch = create_orchestrator('my-project') # SQLite (default)
- Add use_iris_sql section with before/after comparison - Show two real SQL JOIN examples: observations→vectors, summaries→scores - Explain single-namespace benefit for ISC customers - Update module reference table with storage_iris_sql.py and storage_factory.py - Add gotchas entry explaining SQLite vs IRIS SQL tradeoff and use_iris_sql flag - Add single namespace bullet to ISC customer benefits list
…tor) Merged origin/main (74174a1) into IRIS fork. Resolved 3 conflicts: - simplemem/core/database/vector_store.py: kept upstream LanceDB (IRIS lives at database/vector_store.py) - README.md: preserved IRIS backend docs, accepted upstream MCP/Roadmap sections - utils/iris_embeddings.py: moved to simplemem/integrations/reference/utils/ per upstream rename
…ackage move - database/vector_store.py: models.* → simplemem.core.models.* - cross/storage_iris.py: same - tests/conftest.py: skip IRIS vector store tests when no container reachable
- simplemem/core/database/pg_vector_store.py: PGVectorStore implementing the same duck-typed interface as VectorStore (LanceDB). psycopg3 + pgvector. semantic_search via <-> cosine, keyword_search via tsvector/plainto_tsquery, structured_search via parameterized WHERE (no injection surface). - simplemem/core/database/__init__.py: get_vector_store() factory; lancedb remains default, pgvector opt-in via STORAGE_BACKEND=pgvector + PG_DSN. - simplemem/core/settings.py: add STORAGE_BACKEND and PG_DSN settings. - setup.py: add [pgvector] optional extra (psycopg[binary]>=3.1, pgvector>=0.3) - tests/test_pg_vector_store.py: 11 tests, skip when PG_TEST_DSN not set. IRIS + pgwire: point PG_DSN at the pgwire endpoint — no IRIS-specific code needed.
- cross/storage_pg.py: PGCrossVectorStore — same interface as CrossSessionVectorStore (IRIS). Semantic search via pgvector <->, keyword via tsvector, structured via parameterized WHERE with JSON containment. Thread-local connections. - cross/storage_pg_sql.py: PGSQLStorage — same interface as SQLiteStorage and IRISSQLStorage. All tables prefixed for namespace isolation. RETURNING clause for insert IDs. _purge_all_test_data() for test isolation. - cross/storage_factory.py: add use_pg / pg_dsn / pg_table_prefix params; lazy imports so IRIS/PG deps are only loaded when actually requested. - cross/__init__.py: guard IRIS and PG backend imports with try/except so missing optional deps don't prevent the package from loading. - cross/orchestrator.py, session_manager.py, consolidation.py, context_injector.py: replace hard top-level iris imports with try/except for optional-dep safety. - cross/tests/test_pg_cross.py: 35 tests covering both backends; skip when PG_TEST_DSN is absent.
31 tests covering PGCrossVectorStore + PGSQLStorage via iris-pgwire (PostgreSQL wire protocol proxy in front of IRIS). Validates: - Schema creation, add/count, semantic/keyword/structured search - Full session lifecycle: create → event → observation → summary → link - SessionManager round-trip with PG backends - No SQL injection exposure - vector ORDER BY rewrite handled transparently by pgwire optimizer Skip when IRIS_PGWIRE_DSN is absent (set to postgresql://...@pgwire-host/NAMESPACE).
…i MCP, security fixes, Requesty) Key upstream changes merged: - refactor: VectorStoreBackend protocol + LanceDBVectorStoreBackend extraction - feat(omni): MCP server for Omni-SimpleMem multimodal memory - security: drop vulnerable langchain-openai dep, fix CORS wildcard, SQL escaping - fix: omni_memory.core.config module, numpy truth-value ambiguity - docs: PACKAGE_USAGE aligned with current public API, Atlas Cloud example - Add Requesty as LLM provider Our pgvector additions preserved: get_vector_store() factory in __init__, IRIS MultiTenantVectorStore in MCP/server/database/vector_store.py.
- cross/storage_pg.py: PGCrossVectorStore — same interface as CrossSessionVectorStore (IRIS). Semantic search via pgvector <->, keyword via tsvector, structured via parameterized WHERE with JSON containment. Thread-local connections. - cross/storage_pg_sql.py: PGSQLStorage — same interface as SQLiteStorage and IRISSQLStorage. All tables prefixed for namespace isolation. RETURNING clause for insert IDs. _purge_all_test_data() for test isolation. - cross/storage_factory.py: add use_pg / pg_dsn / pg_table_prefix params; lazy imports so IRIS/PG deps are only loaded when actually requested. - cross/__init__.py: guard IRIS and PG backend imports with try/except so missing optional deps don't prevent the package from loading. - cross/orchestrator.py, session_manager.py, consolidation.py, context_injector.py: replace hard top-level iris imports with try/except for optional-dep safety. - cross/tests/test_pg_cross.py: 35 tests covering both backends; skip when PG_TEST_DSN is absent.
…otocol (aiming-lab#76) Replace the old duck-typed PGVectorStore with PGVectorStoreBackend, which implements the VectorStoreBackend protocol introduced in aiming-lab#76 (same pattern as the Milvus backend in aiming-lab#77). - New: simplemem/core/database/pg_vector_store_backend.py Implements insert/semantic_search/keyword_search/structured_search/ count/get_all/optimize/clear against PostgreSQL + pgvector. Thread-local connections, HNSW + GIN indexes, parameterized queries only. - Updated: simplemem/core/database/__init__.py get_vector_store() wires VectorStore with a PGVectorStoreBackend factory when STORAGE_BACKEND=pgvector. PGVectorStoreBackend lazily exported. - Removed: pg_vector_store.py (old duck-typed facade) - Replaced: tests/test_pg_vector_store.py with test_pg_vector_store_backend.py 20 tests covering backend-direct and VectorStore integration paths. All skip without PG_TEST_DSN — no CI infrastructure changes needed.
…verage All 19 tests pass against live IRIS via iris-pgwire. Key IRIS compatibility fixes applied during E2E testing: - CREATE EXTENSION: wrapped in try/except (IRIS has built-in vector support) - VARCHAR(N) instead of TEXT (IRIS TEXT returns stream references via pgwire) - Removed tsvector GENERATED column + GIN index (not supported in IRIS) - LIKE-based keyword search replaces tsvector/plainto_tsquery - LIKE '%"value"%' replaces ::jsonb @> for JSON array column matching - LOWER(col) LIKE LOWER(%s) replaces ILIKE (not supported in IRIS) - LIMIT inlined as integer literal (parameterized LIMIT not supported in IRIS) - ON CONFLICT DO NOTHING removed (not supported in IRIS) - executemany replaced with per-row execute loop (avoids psycopg pipeline mode) - CREATE INDEX without IF NOT EXISTS; duplicate errors caught in _ensure_schema - optimize() silently skips VACUUM ANALYZE on backends that don't support it Also removes duplicate PG import block in cross/__init__.py left from rebase.
…pgwire 1.7.1 iris-pgwire 1.7.1 fixed: CREATE EXTENSION IF NOT EXISTS, ILIKE, LIMIT %s, ON CONFLICT DO NOTHING, executemany/pipeline mode. Restored standard SQL patterns for all five. Only remaining IRIS workaround: LIKE '%"value"%' for persons/entities JSON array matching (jsonb @> containment still broken in pgwire). CREATE INDEX without IF NOT EXISTS still requires try/except. All 19 tests pass on IRIS 2026.2 + pgwire 1.7.1.
…re 1.7.2 iris-pgwire 1.7.2 fixed jsonb containment (@>) operator translation. Replaced LIKE '%"value"%' workaround with proper ::jsonb @> ::jsonb for persons and entities. Bumped minimum iris-pgwire version reference to 1.7.2. CREATE INDEX IF NOT EXISTS still broken in pgwire — try/except stays. All 19 tests pass on IRIS 2026.2 + pgwire 1.7.2.
…d in iris-pgwire 1.7.3 _ensure_schema() is now clean standard SQL. Minimum iris-pgwire version bumped to 1.7.3. All 19 tests pass on IRIS 2026.2 + pgwire 1.7.3.
isc-tdyar
force-pushed
the
feature/pgvector-backend
branch
from
August 26, 2026 14:08
e39be43 to
58b287e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a PostgreSQL + pgvector backend implementing the
VectorStoreBackendprotocol introduced in #76, following the same pattern as the Milvus backend
in #77.
LanceDB remains the default — no existing deployments are affected.
What's new
simplemem/core/database/pg_vector_store_backend.py—PGVectorStoreBackendVectorStoreBackend(insert / semantic_search / keyword_search /structured_search / count / get_all / optimize / clear)
semantic_score_order = ScoreOrder.ASCENDING(cosine distance, lower = better)keyword_score_order = ScoreOrder.DESCENDING<=>cosine operatorILIKEmatching acrosslossless_restatementand
keywordscolumnsWHEREwith::jsonb @> ::jsonbforpersons/entities array columns,
ILIKEfor location, range comparisons fortimestamps — no SQL injection surface
threading.local()) matching the existingparallelism model
VACUUM ANALYZEasoptimize()equivalent; silently skipped on backendsthat don't support it
CREATE TABLE IF NOT EXISTS,CREATE INDEX IF NOT EXISTSsimplemem/core/database/__init__.py—get_vector_store()factoryVectorStorewhose backend isPGVectorStoreBackendwhenSTORAGE_BACKEND=pgvector(orbackend='pgvector'passed directly),otherwise falls back to the default
LanceDBVectorStoreBackendPGVectorStoreBackendlazily exported (no hard psycopg import at module load)simplemem/core/settings.pySTORAGE_BACKEND(default"lancedb") andPG_DSNSchema
All text columns use
VARCHAR(N)rather thanTEXTfor broad compatibility.Opting in
Or via environment variables:
Client installation
Using PostgreSQL
Any PostgreSQL ≥ 15 instance with the pgvector extension works:
CREATE EXTENSION IF NOT EXISTS vector; -- one-time setupUsing InterSystems IRIS
IRIS does not natively speak the PostgreSQL wire protocol, but
iris-pgwire ≥ 1.7.3
adds a translation layer so any psycopg client — including this backend —
connects to IRIS without IRIS-specific drivers.
Server-side setup (one time per IRIS instance):
Or via Docker (see iris-pgwire README):
Then point PG_DSN at the pgwire endpoint:
No code changes are needed to switch between PostgreSQL and IRIS — only the
DSN changes. iris-pgwire ≥ 1.7.3 translates all pgvector DDL and operators
used by this backend transparently, including
CREATE EXTENSION IF NOT EXISTS,CREATE INDEX IF NOT EXISTS … USING hnsw,ILIKE,ON CONFLICT DO NOTHING,parameterized
LIMIT, and::jsonb @>containment.Tests
tests/test_pg_vector_store_backend.pyPG_TEST_DSNenv varTests skip automatically without the env var — no CI infrastructure changes needed.
All 19 tests verified passing against both PostgreSQL and InterSystems IRIS
(via iris-pgwire 1.7.3 on IRIS 2026.2).
Covers: insert, semantic/keyword/structured search, score ordering, empty-input
guards, SQL injection resistance, optimize, clear + reinsert, and end-to-end
via
VectorStore+get_vector_store().