#417411 readpst: Need to address encoding differences

#417411#5
Date:
2007-04-02 16:47:28 UTC
From:
To:
Hi,

Thanks for interesting tool.  This is helping me to move my wife's old
Windows mail to a diifferent platform.

I realized that old windows (95,98,Me) versions used non-utf-8 encodings
for different countries.  So when one extracts the content of pst file,
filenames are encoded in traditional non-utf-8 format (Names like Inbox
are translated into each language in pst).

readpst just writes its content to the filename as is.  In case of Japan,
it was Shift-JIS.   But my sid/etch box is utf-8.

You need to convert filenames to UTF-8 encoding on modrn Linux system
and all other new platforms (Mac, NT, ...).

So having capability to pipe filenames created by readpst with iconv
comannd will be good thing to do if options are given.

  -f shift_jis
  -t utf-8
(Just use iconv or equivalent libray calls, please)

Also, the file contents seems to be in the traditional non-utf-8 format,
i.e., shift-jis for Japan.  This issue needs a bit more investigation.
Simple run of iconv will error out.  -c option helps processig them but
makes some mail with "mojibake", i.e., decoding error. Funny thing is
even if original mail is 7 bit encoded iso-2022-jp file, stored content
seems to be in shift-jis thus readable on utf-8 console running vim.
(Unix mbox usually keeps original 7 bit encoding while storing it.  MS
may be doing some shortcut here since pst is their proprietary format.)

So far just running iconv to convert entire generated file does the
decent job.

For multipart plain/text contents, I needed to change the stratig part:
to be
for mutt to read them OK. (Non-extensive test.  vim was always able to
read it since it does not care these mailbox specific encoding
directives.)

For mutt on Linux, I may have other ways to read it but this was problem
moving file to Mac.  This kind of rewrite is best done in readpst.

Once you adress the first issue, please downgrade this to the wish list.
"Lacks capability to adress encoding issue for the mail content".

I will post more detailed report soon.

I think addressing issues here properly will address non-ASCII people
(eastern europe, Asia, ... even ISO-8859-1).

#417411#10
Date:
2007-04-05 15:02:32 UTC
From:
To:
Hi,

Followup to this.

Filename can be fixed by attached script.  Can you include this in
example.  Also, I have few scripts I used to fix encoding isues.

Osamu

#417411#15
Date:
2021-04-03 02:21:01 UTC
From:
To:
Hi Osamu,

I recently adopted readpst and I am now triaging the bug reports.

Would it be possible for you to attach an example PST file with
non-UTF-8 content, without any private information?

The two scripts that you attached to the bug report have some coding
issues. I have fixed the issues reported by shellcheck, fixed some
other issues and attached the new versions to this email.

Would it be possible for you to discuss the scripts with the upstream
maintainer? They don't have time to maintain the project but are
willing to accept patches to the Mercurial repository.

Carl Byington <carl@five-ten-sg.com>
https://www.five-ten-sg.com/libpst/
http://hg.five-ten-sg.com/libpst/

#417411#20
Date:
2021-04-06 23:57:43 UTC
From:
To:
Paul,

Sorry.  It's been almost 10 years since I did it.

I recall that was from WindowsMe (Japanese ed.) to White old MacBook
(Japanese) data transfer.  I think it worked then.  If I recall
correctly, since Mac mail system used ISO2022-JP
https://en.wikipedia.org/wiki/ISO/IEC_2022
I may have done further conversion from UTF-8 to ISO2022-JP.

Since then, I scrapped my private old Windows machine, old white
Macbook died (data was migrated to Silver Macbook which didn't involve
pst), I don't think I can help.

Please note recent Windows (After Windows 10?) application may use
different encoding (UTF-16?) internally.  The application in the old
English system used to use Windows-1252 or CP-1252.  That's when
application in the localized Windows used encoding I mentioned such as
Shift-JIS.

*
https://docs.microsoft.com/en-us/windows/win32/intl/international-support
* https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows
* https://en.wikipedia.org/wiki/Windows-1252

Regards,

Osamu