#578797 DAViCal: UTF-8 capable PostgreSQL setup required for database creation

#578797#5
Date:
2010-04-22 19:16:57 UTC
From:
To:
Hi,

I have some problems while trying to install a calendar.
I tried to follow the documentation at:

http://davical.org/installation.php

and the problem is:

# su postgres -c /usr/share/davical/dba/create-database.sh
createdb: database creation failed: FEHLER:  Kodierung UTF8 stimmt nicht mit Locale de_DE überein
DETAIL:  Die gewählte LC_CTYPE-Einstellung verlangt die Kodierung LATIN1.
Unable to create database

I am not really sure how the system determines, this de_DE thing. It
is not really tranparent to me. I thought it is related to variables
in the shell:


# env | grep -i LANG
LANG=C
LANGUAGE=us

# env | grep -i LC_
LC_MESSAGES=

but it looks that there are other ways of doing it. Anyway this is where I am stuck.
Could you please give me a hint, what is wrong here?

kind regards
  Frank

#578797#10
Date:
2010-08-26 06:43:53 UTC
From:
To:
Hi,

I found a workaround/solution:

postgres stores some information at install time about the locale:

postgresql.conf:lc_messages = 'en_US.UTF-8'                     # locale for sys
postgresql.conf:lc_monetary = 'en_US.UTF-8'                     # locale for mon
postgresql.conf:lc_numeric = 'en_US.UTF-8'                      # locale for num
postgresql.conf:lc_time = 'en_US.UTF-8'                         # locale for tim

previously that was set to de_DE which is apparently not working in
combination with davical.

regards
  Frank

#578797#15
Date:
2010-09-02 02:05:54 UTC
From:
To:
severity 578797 wishlist
tag 579797 +wontfix
retitle 578797 DAViCal: UTF-8 capable PostgreSQL setup required for database creation
thanks

DAViCal creates the database with a UTF-8 encoding and PostgreSQL does
not allow UTF-8 encodings to be created if the database was originally
initialised with an incompatible encoding such as ISO-8859-15 etc.

There's not really any way around this.  I don't believe that the
PostgreSQL folk will fix the issue - there are excellent performance
reasons behind their decision.  CalDAV specifies UTF-8 as the One True
Encoding and so naturally DAViCal wants to store in a database that is
capable of that.

Possibly I could add a check during the installation, but it would say
little more than the PostgreSQL error message did, and it would most
likely say it in English, which would be a poor option indeed.

Regards,
					Andrew.
------------------------------------------------------------------------
#578797#24
Date:
2010-09-02 08:14:20 UTC
From:
To:
Andrew McMillan <andrew@morphoss.com> writes:
Hi Andrew,

thank you very much for your detailed explanation!

From a normal user prespective I would very much welcome a check and an
explanation or hint for a workaround from the davical side. And english
is very much acceptable. However the same error message in english would
not help at all and not because of the english part.

I would need a level of detail like:

grep UTF-8 postgresql.conf or die "ERROR: davical needs UTF-8, but your
postgresql.conf does not allow this. Please consider changing the lc_*
settings in postgresql.conf"

to get me any further.


Let me explain from my no-database-guru  point of view.

   * I have never used postgresql and I installed it especially for
     davical. I do not maintain SQL servers for many users, instead
     I want to sync my telephone calendar with my desktop.

   * I did not configure postgresql for the locale de_DE as far as I
     remember. This is against my intention. I have explicitly set my
     root account and /etc/default/locale to LANG=C in order to get
     english error messages for system things, which helps for
     retrieving solutions in google a lot.

   * Most gnome session are using de_DE as my children like their
     openoffice menu in german

   * the de_DE 'leaked' somehow to postgresql, probably because of the
     way synaptic or the "gnome root terminal" or sudo behave when
     getting root. As you have explained there is an additional
     performance impact connected to this setting. So quite strange that
     this is propagated automatically here. On the other hand reportbug
     thinks my gnome-session is using

Kernel: Linux 2.6.30-2-686 (SMP w/1 CPU core)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)

     This snippet comes from my original bug report, so I cannot give
     any real source for the de_DE value in postgresql.conf

This:

     Did never happen in my opinion. If I understand it correctly, I had
     zero databases while de_DE was in the postgresql.conf.


   * The postgresql error message of postgresql confused me, because the
     strings LC_CTYPE and LATIN1 are nowhere in any of the involved
     commands or configuration files. You need a lot of know-how to
     parse this error message and come to the explanation you gave
     above. Or employ try and error like I did later.


I am somehow aware that this is not related to davical at all, but more
to me being confronted to a complex SQL thing. :)

What do you think of adding a sentence/instruction to the
installation document, which allows the user to check his
postgresql.conf for UTF-8 support? Like the "grep UTF-8 postgresql.conf"
pseudo code above?



kind regards
  Frank

#578797#29
Date:
2010-09-02 09:23:59 UTC
From:
To:
Hi Frank,

There should not be a problem with de_DE.UTF-8, but with
de_DE.ISO-8859-15.  There are two issues here: Language vs. Character
encoding.  DAViCal does not have a problem with language, but with
character encoding.

I guess the problem may actually be that your LC_* settings did not
specify an encoding and sadly this means that much software does not
default to UTF-8 encoding, specifically PostgreSQL.

If you really did not have to reinstall PostgreSQL, but only to change
your LC_* setting then the problem is not with the *installation* of
PostgreSQL, which clearly *did* initialise the database area to support
UTF-8.  The problem I thought it was in my earlier e-mail...

Indeed, but *there* is where the bug lies.  The installation for
PostgreSQL should not have written that on it's own, without an encoding
specification.

No, in fact the installation of PostgreSQL creates an initial tablespace
containing several databases which are templates for the creation of
further databases.  PostgreSQL writes the does/doesn't support UTF-8 in
at the tablespace level and so if the tablespace is created without
UTF-8 support DAViCal will indeed be unable to create databases, without
any real blame being able to be assigned anywhere...

However it seems the tablespace created by your installation *does*
support UTF-8 encodings, and so perhaps there really *is* a bug here
about not allowing the creation of a database with UTF-8 encoding when
you have some strange setting in your config file.

It's obscure, however, and I really am even *less* inclined to consider
it a DAViCal bug.

At this point my intentions are to migrate away from the current
database creation script *entirely*, and I expect this will occur at
some point in the next few releases.  This is because I want to better
support people in other installation situations, such as 'in a hosted
environment on FreeBSD' and strange things like that.

This does increase the chance of the situation being caught with an
(eventually localisable) error message about what might not be working,
but it won't change in the short term.

Regards,
					Andrew McMillan.

#578797#34
Date:
2010-09-02 10:35:24 UTC
From:
To:
Andrew McMillan <andrew@morphoss.com> writes:

I have to educate myself on that topic. Currently it is like endianess
to me.

I did both:

/var/log/dpkg.log.1:2010-08-09 20:54:37 install postgresql-8.4 8.4.3-1 8.4.4-2
/var/log/dpkg.log.1:2010-08-09 21:14:41 remove postgresql-8.4 8.4.4-2 8.4.4-2
/var/log/dpkg.log.1:2010-08-09 21:14:41 purge postgresql-8.4 8.4.4-2 8.4.4-2
/var/log/dpkg.log.1:2010-08-09 21:15:26 install postgresql-8.4 <none> 8.4.4-2
/var/log/dpkg.log.4.gz:2010-05-21 21:44:37 remove postgresql-8.4 8.4.3-1 8.4.3-1
/var/log/dpkg.log.5.gz:2010-04-22 20:40:14 install postgresql-8.4 <none> 8.4.3-1


It looks like I manually edited that file at 2010-08-09 21:14:35, which
did not work, purged the installation and was then happy with the new
config file at 2010-08-09 21:15:55 (based on etckeeper autocommits)

I vaguely remember trying stuff in a CTRL-ALT-F1-Terminal as root (to
avoid 'leakage' of these environment settings ) like

LANG=C apt-get install postgresql....

and observing what went into the config files. I had no idea on
the existance of these initial sql tables....

Yes I agree! It is more me vs postgresql installation. Thank you for
your patience and explanations!

kind regards
  Frank