Re: Character Encoding Question

On Thu, Mar 28, 2013 at 1:30 PM, Daniele Varrazzo <
daniele(dot)varrazzo(at)gmail(dot)com> wrote:

> Please keep the mailing list in copy.
>
> Yes... My apologies - I have gotten accustomed to munging lists.... Will
try to remember (or maybe stop using gmail for this list).

> On Thu, Mar 28, 2013 at 5:17 PM, Don Parris <parrisdc(at)gmail(dot)com> wrote:
> > Here is the result of the query you suggested:
>
<SNIP>

> >Try (without setting the encoding to utf8):
>
> cur.execute("select [ your stuff, including non-ascii records")
> while 1:
> record = cur.fetchone()
> if not record:
> break
>
> I'm pretty sure you will get an encoding error too.
>
> You are right about this. It stopped at record #7 (entity_id = 7) and
gave me the traceback. That is, record #7 was the last record printed.
Does that have anything to do with the position mentioned in the
traceback? Would #8 have been the 'guilty' record?

This record was loaded (as were all of the first 80-100 in this table) via
\copy command from within psql. The remaining records have been inserted
manually via PGAdmin, simply opening the table and adding the rows. I also
played with my pattern matching search query, which has turned up great
results, until I searched on record #8 - then it broke.

Given that most of the early records were loaded via \copy, why & how would
the character encoding change in the middle of a \copy command? Or is it
possible they were all different when initially saved as a CSV file? I
could understand better if the encoding broke the program, say at the first
record I loaded via \copy, but maybe worked on the first record I inserted
via PGAdmin - or vice versa.

>
> > I created the DB in postgresql using the following command:
> > CREATE DATABASE mydb
> > WITH TEMPLATE template0 ENCODING 'UTF8';
> >
> > Although I cannot now recall my reason for doing so, there *is* a reason
> why
> > I'm using this approach. :-/ I should have said "why" in my comments!
>
> I don't see anything wrong here. The only question is what is setting
> the connection to SQL_ASCII. Maybe something in postgresql.conf or a
> setting of the database user. It's not a psycopg issue anyway: the
> driver is following the instructions.
>
> I can't say there's anything wrong with psycopg - I just want to better
understand what is happening with my data and front-end code.

--
D.C. Parris, FMP, Linux+, ESL Certificate
Minister, Security/FM Coordinator, Free Software Advocate
http://dcparris.net/
<https://www.xing.com/profile/Don_Parris><http://www.linkedin.com/in/dcparris>
GPG Key ID: F5E179BE