CSVQL? CSV SQL? tab-separated table I/O? RENAME COLUMN

From: Jim Michaels <jmichae35(at)gmail(dot)com>
To: pgsql general <pgsql-general(at)postgresql(dot)org>
Subject: CSVQL? CSV SQL? tab-separated table I/O? RENAME COLUMN
Date: 2018-05-02 21:29:08
Message-ID: CAFWCvVhwRJdc7YEkhrWqb3XGd0jSyOfbBt_XUfzebCseqm_yWQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

what do you think about foreign data wrappers getting CSV file table I/O?
- I had thought that CSVQL db could be implemented completely with
small amount of memory and file I/O, line at a time. EOL detection
would be needed. can be: CR, LF, CR+LF. sometimes beginners get it
backwards (LF+CR), but it's stll detectable because it's an immediate
sequence detectable by state machine or while loop. it should be
written as CR+LF because of standards compliance X2J4579 - that means
go look for it (I have not found it yet).
while (!feof(...)&&(ch=='\r'||ch=='\n')) {
do {
if (1!=fread(&ch, 1, 1, ...)) {
ch=0;
break;
}
if (ch!='\r'&&ch!='\n') {
break;//process non-EOL character
}
//at this point, it's an EOL character
} while (true);
or
while (!feof(...)&&ch=='\t') {
if (1==fread(&ch, 1, 1, ...)) {//read 1 character
}
}
this code could be read in chunks using a buffer. the last chunk would
need to be handled as a special case, since if it exists, it exists as
<full buffer size. try %

- the microsoft patented CSV would be required for implementation. it
handles special data with commas and double-quotes in them
- tab-separated I/O would be nice as well.
- you could see >2,000,000 rows, so don't limit it. try GCC's
fopen64/fsetpos,fgetpos/fpos_t/fclose, other vendors use plain fopen
and that works with >=64-bit file sizes. and it's fast. avoid 32-bit
fseek/ftell.
- indexing could be file/RAM array of uint64_t-like file pointers
convertable to fpos_t if needed.
- biggest needed feature is an easier-to-use ALTER TABLE RENAME. a
memorable alternative/alias would be simply RENAME COLUMN columnName
TO newColumnName.

--
======
Jim Michaels <jmichae35(at)gmail(dot)com>

Responses

Browse pgsql-general by date

  From Date Subject
Next Message David G. Johnston 2018-05-02 21:49:18 Re: CSVQL? CSV SQL? tab-separated table I/O? RENAME COLUMN
Previous Message Alvaro Herrera 2018-05-02 20:32:36 Re: Index/trigger implementation for accessing latest records