Unicode encoding and decoding¶
Introduction: Why unicode is difficult?¶
Python 2.x does not make a clear distinction between:
- 8-bit strings (byte data)
- 16-bit unicode strings (character data)
Developers use these two formats interchangeably, because it is so easy and Python does not warn you about this.
However, it will only work as long as the input does not encounter any international, non-ASCII, characters. When 8-bit encoded string data and 16-bit raw Unicode string data gets mixed up, by being run through encoding first, really nasty things start to happen.
Read more:
safe_unicode()¶
Plone's core contains a helper function which allows you to safely decode strings to unicode without fear of UnicodeDecodeException. Use this in your own code to decode unicode in the cases you are not sure if the input is 8-bit bytestrings or real unicode strings.
https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/utils.py#L434
Example:
# -*- coding: utf-8 -*-
from Products.CMFPlone.utils import safe_unicode
foobar = safe_unicode("Ärrinmurrin taas on Plonea joku jättänyt dokumentoimatta")
sys.setdefaultencoding()¶
Python has a system-wide setting to enforce encoding of all unicode input automatically to utf-8 when used as 8-bit string.
Warning
This is a wrong way to fix things and it will break other things. You have been warned.
There is also
sitecustomization.py
trick to set
sys.setdefaultencoding("utf-8")
on per-script basis
UnicodeEncodeError¶
UnicodeEncodeError: 'ascii' codec can't encode character u'xe4' in position
4: ordinal not in range(128)
This is usually because you are trying to output/store unicode data using outdated methods, e.g.
- printing,
- logging,
- using 7-bit ids ...
Instead of:
print foo
do:
print foo.encode("utf-8") # You are sure this is a unicode string
Filtering example:
def safe_print(x):
""" Do not die on bad input when doing debug prints """
if type(x) == str:
print x
else:
print x.decode("utf-8")
Infamous non-breaking Unicode space \xa0¶
Press CTRL+space / AltGr space on Linux to accidentally create it.
You can't see it. But it breaks everything.
How to fix¶
Example to how to fix non-breaking space characters
which have ended up in reStructuredText
.txt
files. This is Unicode character code A0.
Example fix_wtf_space.py:
# -*- coding: utf-8 -*-
""" Fix non-breaking space characters which have ended up to reST
.txt files. This is Unicode character code A0.
Press CTRL+space / AltGr space on Linux to accidentally create it.
E.g. as a sympton the following exception is raised if you try
to upload Python egg::
File "/Library/Python/2.6/site-packages/docutils-0.6-py2.6.egg/docutils/parsers/rst/states.py", line 2621, in blank
self.parent += self.literal_block()
File "/Library/Python/2.6/site-packages/docutils-0.6-py2.6.egg/docutils/parsers/rst/states.py", line 2712, in literal_block
literal_block = nodes.literal_block(data, data)
File "/Library/Python/2.6/site-packages/docutils-0.6-py2.6.egg/docutils/nodes.py", line 810, in __init__
TextElement.__init__(self, rawsource, text, *children, **attributes)
File "/Library/Python/2.6/site-packages/docutils-0.6-py2.6.egg/docutils/nodes.py", line 798, in __init__
textnode = Text(text)
File "/Library/Python/2.6/site-packages/docutils-0.6-py2.6.egg/docutils/nodes.py", line 331, in __new__
return reprunicode.__new__(cls, data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 715: ordinal not in range(128)
"""
import os
def fix(name):
""" Fix a single .txt file
"""
input = open(name, "rt")
text = input.read()
input.close()
text = text.decode("utf-8")
# Show if we get bad hits
for c in text:
if c == u"\xa0":
print "Ufff"
text = text.replace(u"\xa0", u" ")
text = text.encode("utf-8")
output = open(name, "wt")
output.write(text)
output.close()
# Process all .txt files in the
# current folder
for f in os.listdir(os.getcwd()):
if f.endswith(".txt"):
fix(f)