1

Im using UB 16.04 LTS Server and yesterday a new version of the locales-package has been released and installed:

Start-Date: 2019-02-21  09:44:05
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade
Upgrade: [...], locales:amd64 (2.23-0ubuntu10, 2.23-0ubuntu11), [...]
End-Date: 2019-02-21  09:44:45

I'm additionally using PostgreSQL 11 with databases relying on the character set windows-1252 for historical reasons:

Name  |  Owner   | Encoding |   Collate    |    Ctype     |   Access[...]
------+----------+----------+--------------+--------------+--------------
[...] | postgres | WIN1252  | de_DE.CP1252 | de_DE.CP1252 |

To make that available, the following command is used:

localedef -f CP1252 -i /usr/share/i18n/locales/de_DE /usr/lib/locale/de_DE.CP1252

Using locale-gen before/after localedef gives the following error:

locale-gen de_DE.CP1252
Error: 'de_DE.CP1252' is not a supported language or locale

That error message is correct, the file /usr/share/i18n/SUPPORTED doesn't contain my locale. From my understanding that's why I need to use localedef.

The problem now is that most likely after the new locales-package has been installed, my manually added one was removed automatically and access to the databases relying on that locale failed:

2019-02-21 09:42:45.109 CET [27039] FATAL:  Datenbank-Locale ist inkompatibel mit Betriebssystem
2019-02-21 09:42:45.109 CET [27039] DETAIL:  Die Datenbank wurde mit LC_COLLATE »de_DE.CP1252« initialisiert, was von setlocale() nicht erkannt wird.

From my understanding, all locales which were recognized and enabled by dpkg-reconfigure locales have been kept and my custom locale was not listed there. So I followed instructions from /etc/locale.gen and created the following file with the following line:

/usr/local/share/i18n/SUPPORTED
de_DE.CP1252 CP1252

That made my locale available in the end:

enter image description here

The problem is that this doesn't seem to be enough to solve my goal: Whenever I run dpkg-reconfigure locales, the created locale using localedef gets deleted from the folder /usr/lib/locale/de_DE.CP1252 and Postgres fails again. Even though locale -a prints my locale and such. So that's most likely what was happening with/after installing the new locales-package. If I run localedef like documented above manually again, Postgres instantly allows access to the legacy databases again.

One interesting thing I recognized is the locale C.UTF-8 being available in /usr/lib/locale like my custom one, BUT that never gets deleted automatically for some reason. Searching for that locale in the web, it seems it is provided as part of some package by the distribution instead of configured and generated locally as needed:

We now have an "uninstallable" C.UTF-8 locale that is available even if you delete locale-archive, or change the installed language set for locale-archive.

https://bugzilla.redhat.com/show_bug.cgi?id=902094#c20

/usr/lib/locale/C.UTF-8/LC_ADDRESS
/usr/lib/locale/C.UTF-8/LC_COLLATE

https://packages.debian.org/de/sid/sh4/libc-bin/filelist

So, what do I need to do to make my localedef-results survive dpkg-reconfigure locales or whatever has been done during installing the new locales-packages`

Thanks!

Thorsten Schöning
  • 643
  • 5
  • 16

1 Answers1

0

There are different things to note here:

Deleted results of localedef

Manually created results of localedef are really always deleted, simply because locale-gen is a script doing so:

if [ -z "$1" ] && [ -z "$KEEP" ]; then
        # Remove all old locale dir and locale-archive before generating new
        # locale data.
        rm -rf /usr/lib/locale/locale-archive || true
        for dir in /usr/lib/locale/*; do
                [ -e "$dir" ] || continue
                if [ "${dir#/usr/lib/locale/}" = C.UTF-8 ]; then
                        # owned by libc-bin
                        continue
                fi
                rm -rf "$dir" 2>/dev/null || true
        done
fi

To circumvent this, one really needs to make the locale available differently, so that it gets generated always when locales are generated at all.

/usr/local/share/i18n/SUPPORTED

That file IS in fact the correct way to make my locale available and how I added it IS correct. The line de_DE.CP1252 CP1252 results in the same locale being created using dpkg-reconfigure locales like using localedef manually. The only difference is that in the first case the locale is added to the file /usr/lib/locale/locale-archive, while the latter creates individual directories. That difference leads to the following problem...

Postgres not recognizing changes

I simply never restarted Postgres during my tests after executing dpkg-reconfigure locales and that is necessary!

When the individual directories created using localedef where removed or added, Postgres recognized that instantly whenever a session to a database using that locale has been created. That's why I thought locales are always read as needed, which doesn't seem to be the case. Instead I guess the archive file is only opened once per process and only if some locale is not found in there, the additional directories are recognized and that on-demand. So when the archive file has been deleted and created newly, Postgres simply didn't recognized and still worked with the old contents, which didn't ever contain my locale.

After restarting Postgres that used the new archive file with my locale embedded and things worked again. This can easily be reproduced by running dpkg-reconfigure locales to remove my locale again and without a restart of Postgres it simply continues to work. After a restart it fails again until the locale gets added to the archive file again.

Do not use localedef?

So it seems the correct approach to make results of localedef survive dpkg-reconfigure locales in the end is to NOT use localedef manually, but instead define the needed locale in the file /usr/local/share/i18n/SUPPORTED and in the worst case to restart the system.

Thorsten Schöning
  • 643
  • 5
  • 16