From: | Jian Guo <gjian(at)vmware(dot)com> |
---|---|
To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
Cc: | Georgios Kokolatos <gkokolatos(at)protonmail(dot)com> |
Subject: | Re: Teach pg_receivewal to use lz4 compression |
Date: | 2021-09-11 05:02:42 |
Message-ID: | 163133656252.1167.2818066389662879524.pgcf@coridan.postgresql.org |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
/*
* Check that the segment has the right size, if it's supposed to be
* completed. For non-compressed segments just check the on-disk size
- * and see if it matches a completed segment. For compressed segments,
- * look at the last 4 bytes of the compressed file, which is where the
- * uncompressed size is located for gz files with a size lower than
- * 4GB, and then compare it to the size of a completed segment. The 4
- * last bytes correspond to the ISIZE member according to
+ * and see if it matches a completed segment. For zlib compressed
+ * segments, look at the last 4 bytes of the compressed file, which is
+ * where the uncompressed size is located for gz files with a size lower
+ * than 4GB, and then compare it to the size of a completed segment.
+ * The 4 last bytes correspond to the ISIZE member according to
* http://www.zlib.org/rfc-gzip.html.
+ *
+ * For lz4 compressed segments read the header using the exposed API and
+ * compare the uncompressed file size, stored in
+ * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
*/
- if (!ispartial && !iscompress)
+ if (!ispartial && wal_compression_method == COMPRESSION_NONE)
{
struct stat statbuf;
char fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
- else if (!ispartial && iscompress)
+ else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
{
int fd;
char buf[4];
@@ -322,6 +378,70 @@ FindStreamingStart(uint32 *tli)
continue;
}
}
+ else if (!ispartial && compression_method == COMPRESSION_LZ4)
+ {
+#ifdef HAVE_LIBLZ4
+ int fd;
+ int r;
+ size_t consumed_len = LZ4F_HEADER_SIZE_MAX;
+ char buf[LZ4F_HEADER_SIZE_MAX];
+ char fullpath[MAXPGPATH * 2];
+ LZ4F_frameInfo_t frame_info = { 0 };
+ LZ4F_decompressionContext_t ctx = NULL;
+
+ snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+ fd = open(fullpath, O_RDONLY | PG_BINARY, 0);
Should close the fd before exit or abort.
From | Date | Subject | |
---|---|---|---|
Next Message | Michael Paquier | 2021-09-11 05:44:11 | Re: strange case of "if ((a & b))" |
Previous Message | Dilip Kumar | 2021-09-11 04:17:00 | Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints |