Add pg_deltax#933
Conversation
pg_deltax is Xata's time-series PostgreSQL extension that adds columnar storage and compression to a partitioned hits table. Implements the per-system script interface (install/start/check/stop/load/query/ data-size + benchmark.sh shim) so the shared driver in lib/ runs it end-to-end. Includes a c6a.4xlarge result run from the new methodology (true-cold cycles + concurrent QPS).
|
|
The xataio/pg_deltax URL still works via GitHub's 301 redirect, but the canonical repo name is xataio/deltax (matches the link in the PR description). Use the canonical URL directly so the clone does not silently depend on the redirect.
| @@ -0,0 +1,8 @@ | |||
| #!/bin/bash | |||
| # Report the test database's on-disk size: tables + indexes + TOAST. Excludes | |||
There was a problem hiding this comment.
I'm no Postgres expert but in my thinking, WAL data is conceptually part of the persistent table data. In this case, the dataset is static, so I assume that the WAL can be shrunk to zero size using some Postgres administration statement (something like "merge", "consolidate", or whatever the "compact-the-wal" thingy in Postgres is called).
There was a problem hiding this comment.
With Postgres, WALs are kept on disk even after they are consumed and transactions stored in the table data. There is no merge/compact/etc that will reduce WALs to zero, however Postgres will reuse the disk space when new transactions are coming in. So in practice once the WAL files reach max_wal_size, even after consuming them completely, the disk usage will stay around max_wal_size from then on. Default max_wal_size is 1GB, but I see some entries in this benchmark use 32GB.
Looking over some of the other entries, there seem to be three possible approaches:
- timescale entry does
hypertable_size()which is strictly the table (no WAL, no database overhead) - greenplaum does
pg_database_size()(no WAL, but database overhead included) - pgpro_tam does
du -bcs .../data/base(no WAL, but database overhead included) - oriole does something similar but
du -bcs /var/lib/postgresql/data/orioledb_data(no WAL, database overhead not included) - plain postgres does
du .../main/(WAL included, database overhead included)
I pushed a commit to do du -bcs .../data/base (like pgpro_tam), IMHO this is the most correct (includes DB overhead, no WALs) and has the advantage that it can be used as such across most of these entries.
It slightly disadvantages deltax compared with say timescale, but not by much. If we include the WALs, because default max_wal_size is 1GB in postgres, I think that will add ~1GB to the deltax result. If that is the decision, it is fine with me as well, happy to update the PR.
Side note: the plain postgres entry uses max_wal_size: 32GB and data-size includes the WALs. IMHO that's unfair to Postgres data size, it adds up to 32GB to the measured data (OTOH, it might speed up loading via what I'd consider tuning).
There was a problem hiding this comment.
Thanks, that was a good summary and I learned something about Postgres as well.
I'm good with the method you proposed (du -bcs .../data/base). If you think that other Postgres-based databases are treated unfairly or inconsistently, feel free to send a separate PR which rectifies this.
| # boundary calculation to the dataset's epoch (the hits data is from 2013). | ||
| sudo -u postgres psql -v ON_ERROR_STOP=1 -t test < create.sql | ||
| sudo -u postgres psql -v ON_ERROR_STOP=1 -t test -c "SET pg_deltax.mock_now = '2013-07-01 12:00:00'; SELECT deltax.deltax_create_table('hits', 'eventtime', '3 days'::interval, 15)" | ||
| sudo -u postgres psql -v ON_ERROR_STOP=1 -t test -c "SELECT deltax.deltax_enable_compression('hits', order_by => ARRAY['counterid', 'userid', 'eventtime'], segment_size => 30000)" |
There was a problem hiding this comment.
Re deltax_enable_compression: I will not oppose this but it slightly violates the spirit of the benchmark. The rules say:
It's better to use the default settings and avoid fine-tuning. Configuration changes can be applied if it is considered strictly necessary and documented.
Fine-tuning and optimization for the benchmark are not recommended but allowed. In this case, add results for the vanilla configuration and tunes results separately (e.g. 'MyDatabase' and 'MyDatabase-tuned')
Running without deltax_enable_compression or making compression deltax's default will be the preferred option.
There was a problem hiding this comment.
So I consider deltax_enable_compression() part of enabling the extension, because compression is really the main point of it. That selects the table on which you want columnar storage. Perhaps I should consider another name for the function, but it's the equivalent of this for Timescale: https://github.com/ClickHouse/ClickBench/blob/main/timescaledb/load#L13-L14
The segment_size would indeed be a form of tuning, but 30K is already the default, so I pushed a commit to remove it from the config.
Regarding the order_by setting, I'd say that's the equivalent of ClickHouse setting the PK here https://github.com/ClickHouse/ClickBench/blob/main/clickhouse/create.sql#L108
Let me know if you see it otherwise, or anything I could clarify here.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
@rschu1ze Many thanks for reviewing! Will go through the comments today. Quick question about the link you posted, that is internal I assume, right? |
Correct. However, the content wasn't secret at all, so let me copypaste it in a public Pastila as well: https://pastila.nl/?00d8628b/de15e62d9757400aad269c52d058e962#xRrW24qhTWOxPGBXk/WSEw==GCM |
|
Thanks again for the thoughtful review, replied to the comments and pushed some changes. I didn't update the results file, assuming that you will do it. But your run and mines are consistent within noise level. Let me know if you'd like me to update the result file. |
3 participants
Hi,
I'd like to add an entry for pg_deltax (https://github.com/xataio/deltax)
It is a time-series extension for PostgreSQL, which we just made open source.
I did read the rules for contribution, hopefully I didn't miss anything important. Please let me know if you have any feedback.