5
APR2009
Ruby 1.9's Three Default Encodings
I suspect early contact with the new m17n (multilingualization) engine is going to come to Rubyists in the form of this error message:
invalid multibyte char (US-ASCII)
Ruby 1.8 didn't care what you stuck in a random String literal, but 1.9 is a touch pickier. I think you'll see that the change is for the better, but we do need to spend some time learning to play by Ruby's new rules.
That takes us to the first of Ruby's three default Encodings.
The Source Encoding
In Ruby's new grown up world of all encoded data, each and every String needs an Encoding. That means an Encoding must be selected for a String as soon as it is created. One way that a String can be created is for Ruby to execute some code with a String literal in it, like this:
str = "A new String"
That's a pretty simple String, but what if I use a literal like the following instead?
str = "Résumé"
What Encoding is that in? That fundamental question is probably the main reason we all struggle a bit with character encodings. You can't tell just from looking at that data what Encoding it is in. Now, if I showed you the bytes you may be able to make an educated guess, but the data just isn't wearing an Encoding name tag.
That's true of a frightening lot of data we deal with every day. A plain text file doesn't generally say what Encoding the data inside is in. When you think about that, it's a miracle we can successfully read a lot of things.
When we're talking about program code, the problem gets worse. I may want to write my code in UTF-8, but some Japanese programmer may want to write his code in Shift JIS. Ruby should support that and, in fact, 1.9 does. Let's complicate things a bit more though: imagine that I bundle up that UTF-8 code I wrote in a gem and the Japanese programmer later uses it to help with his Shift JIS code. How do we make that work seamlessly?
The Ruby 1.8 strategy of one global variable won't survive a test like this, so it was time to switch strategies. Ruby 1.9's answer to this problem is the source Encoding.
All Ruby source code now has some Encoding. When you create a String literal in your code, it is assigned the Encoding of your source. That simple rule solves all the problems I just described pretty nicely. As long my source Encoding is UTF-8 and the Japanese programmer's source Encoding is Shift JIS, my literals will work as I expect and his will work as he expects. Obviously if we share any data, we will need to establish some rules about our shared formats using documentation or code that can adapt to different Encodings, but we should have been doing that all along anyway.
Thus the only question becomes, what's my source Encoding and how do I change it?
There are a few different ways Ruby can select a source Encoding. Here are the options:
$ cat no_encoding.rb
p __ENCODING__
$ ruby no_encoding.rb
#<Encoding:US-ASCII>
$ cat magic_comment.rb
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment.rb
#<Encoding:UTF-8>
$ cat magic_comment2.rb
#!/usr/bin/env ruby -w
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment2.rb
#<Encoding:UTF-8>
$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'p __ENCODING__'
#<Encoding:UTF-8>
$ ruby -KU no_encoding.rb
#<Encoding:UTF-8>
The first example shows us two important things. The first is the main rule of source Encodings: source files receive a US-ASCII Encoding, unless you say otherwise. [Update: this was changed to UTF-8 in Ruby 2.0 and up.] This is where I expect programmers to run into the error I mentioned earlier. If you place any non-ASCII content in a String literal without changing the source Encoding, Ruby will die with that error. Thus you need to change the source Encoding to work with any non-ASCII data. The second thing we see here is the new __ENCODING__ keyword that can be used to get the source Encoding that's active where it is executed.
The second example shows the preferred way to set your source Encoding and it's called a magic comment. If the first line of your code is a comment that includes the word coding, followed by a colon and space, and then an Encoding name, the source Encoding for that file is changed to the indicated Encoding. If your code has a shebang line, the magic comment must come on the second line, with no spacing between them. Once set, all String literals you create in that file will have that Encoding attached to them.
The third example shows an exception to the rule for your convenience. When you feed Ruby some code on the command-line using the -e switch, it gets a source Encoding from your environment. I have UTF-8 set in the LC_CTYPE environment variable, but some people also use the LANG variable for this. This makes scripting easier since Ruby will (hopefully) match the Encoding of any other commands you chain together.
The fourth example is another interesting exception to the rule. Ruby 1.9 still supports the -K* style switches from Ruby 1.8 including the -KU switch I've recommended so heavily in this series. These switches have a couple of effects, but of particular note they are the only non-magic comment way to modify the source Encoding. This is good news for backwards compatibility, because some Ruby 1.8 code may be able to run on Ruby 1.9 without Encoding problems thanks to this. I must stress that this is just for backwards compatibility though, and magic comments are the future.
With magic comments the code will include its Encoding data. It will probably seem a little tedious to add them to all your source files at first, but it's really not that big of a change. In the past, I've recommended we stick the following shebang line at the top of our files:
#!/usr/bin/env ruby -wKU
Now, for Ruby 1.9, I'm recommending we switch to something like this:
#!/usr/bin/env ruby -w
# encoding: UTF-8
Note that the magic comment format rules are pretty loose and all of following examples would work the same:
# encoding: UTF-8
# coding: UTF-8
# -*- coding: UTF-8 -*-
This is nice for support in some text editors that also read such comments.
If we all get into that habit of adding magic comments, our code can work together regardless of the various Encodings we personally favor. Ruby will know how to handle each separate file. As an added bonus, we programmers also get to see these comments and know more about the code we are working with. That makes it a good habit to get into, I think.
The Default External and Internal Encodings
There's another way Strings are commonly created and that's by reading from some IO object. It doesn't make sense to give those Strings the source Encoding because the external data doesn't have to be related to your source code. Also, you really need to know how data is encoded to read it correctly. Even a simple concept like reading the next line of data changes if you are talking about UTF-8 or UTF-16LE (the LE stands for a Little Endian byte order) data. Thus, it makes sense for IO objects to have at least one Encoding attached to them. Ruby 1.9 is generous and gives them two: the external Encoding and the internal Encoding.
The external Encoding is the Encoding the data is in inside the IO object. That affects how data will be read and this is the Encoding data will be returned in as long as the internal Encoding isn't set (more on that in a bit). Let's look at an example of how this plays out in practice:
$ cat show_external.rb
open(__FILE__, "r:UTF-8") do |file|
puts file.external_encoding.name
p file.internal_encoding
file.each do |line|
p [line.encoding.name, line]
end
end
$ ruby show_external.rb
UTF-8
nil
["UTF-8", "open(__FILE__, \"r:UTF-8\") do |file|\n"]
["UTF-8", " puts file.external_encoding.name\n"]
["UTF-8", " p file.internal_encoding\n"]
["UTF-8", " file.each do |line|\n"]
["UTF-8", " p [line.encoding.name, line]\n"]
["UTF-8", " end\n"]
["UTF-8", "end\n"]
There are four things to notice in this example:
- I set the external
Encodingby tacking:UTF-8onto the end of my modeStringwhen I opened theFile - You can use
external_encoding()to check the externalEncodingas I have here -
internal_encoding()works the same for the internalEncoding, which will benilunless you explicitly set it - Note how each
Stringcreated as I read the data is given theexternal_encoding()
The internal Encoding just adds one more twist. When set, data will still be read in the external Encoding, but transcoded to the internal Encoding as the String is created. It's a convenience for you as the programmer. Watch how that changes things:
$ cat show_internal.rb
open(__FILE__, "r:UTF-8:UTF-16LE") do |file|
puts file.external_encoding.name
puts file.internal_encoding.name
file.each do |line|
p [line.encoding.name, line[0..3]]
end
end
$ ruby show_internal.rb
UTF-8
UTF-16LE
["UTF-16LE", "o\x00p\x00e\x00n\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00f\x00i\x00"]
["UTF-16LE", " \x00 \x00 \x00 \x00"]
["UTF-16LE", " \x00 \x00e\x00n\x00"]
["UTF-16LE", "e\x00n\x00d\x00\n\x00"]
There are a couple differences here:
- A second added
Encodingon the modeString(my:UTF-16LEin this example) sets theinternal_encoding()as I show with the secondputs() - This little change gets Ruby to translate all of the data for me (I just shortened the output because UTF-16LE is noisy)
The external Encoding works the same when writing. It still represents the Encoding in the IO object, or the Encoding data is going to. However, you don't need to specify an internal Encoding when writing. Ruby will automatically use the Encoding of a String you output as the internal Encoding and transcode as needed to reach the external Encoding. For example:
$ cat write_internal.rb
# encoding: UTF-8
open("data.txt", "w:UTF-16LE") do |file|
puts file.external_encoding.name
p file.internal_encoding
data = "My data…"
p [data.encoding.name, data]
file << data
end
p File.read("data.txt")
$ ruby write_internal.rb
UTF-16LE
nil
["UTF-8", "My data…"]
"M\x00y\x00 \x00d\x00a\x00t\x00a\x00& "
Note how my data was transcoded before it was written even though the internal_encoding() was nil. Ruby used the String's Encoding to decide what was needed.
Both of those IO Encodings should be pretty straight forward. The only question left about them is: what happens if you don't set them? The answer is that the IO inherits the default external Encoding and/or the default internal Encoding whenever one isn't set. Now we need to know how Ruby chooses those defaults.
The default external Encoding is pulled from your environment, much like the source Encoding is for code given on the command-line. Have a look:
$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'puts Encoding.default_external.name'
UTF-8
$ LC_CTYPE=ja_JP.sjis ruby -e 'puts Encoding.default_external.name'
Shift_JIS
The default internal Encoding is simply nil. You must actively change it to get anything else.
Both default IO Encodings have a global setter: Encoding.default_external=() and Encoding.default_internal=(). You can set them to an Encoding object or just the String name of an Encoding.
You can also change these default Encodings using some command-line switches. The new -E switch can be used to set one or both of the IO Encodings:
$ ruby -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, nil]
$ ruby -E Shift_JIS \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, nil]
$ ruby -E :UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, #<Encoding:UTF-16LE>]
$ ruby -E Shift_JIS:UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, #<Encoding:UTF-16LE>]
As you can see, the argument for this switch is just like what you would append to a mode String in a call to File.open().
There's one more command-line switch shortcut for those of us who prefer to just use UTF-8 everywhere. The new -U switch sets Encoding.default_internal() to UTF-8. Using that, you can just set the external Encoding for your IO objects, or let it default from your environment, and all Strings you read will be transcoded to the preferred UTF-8.
Probably the most important thing to note about Encoding.default_external() and Encoding.default_internal() is that you should really just treat them as shortcuts for your own scripting. Pulling Encodings from the environment or command-line switches can be handy when you're in control of where the code runs, but you're going to need to be more explicit for code you intend for others to run. When in doubt, set the external and internal Encodings the way you want them for each IO object. It's a bit more tedious, but also safer in that it won't mysteriously be changed by some outside force. Also remember that the defaults are global settings affecting all loaded code, including any libraries you require(). That can be a boon or bane, so just remember to factor it into your thinking when you're wondering, "Where does this String get its Encoding from?"
Comments (41)
-
James Edward Gray II April 6th, 2009 Reply LinkIt's probably worth noting that using the default
Encodingsetters will trigger warnings:$ ruby -we 'Encoding.default_internal = Encoding.default_external = "UTF-8"' -e:1: warning: setting Encoding.default_external -e:1: warning: setting Encoding.default_internalThat makes sense, as it's really too late to set these in code after
IOobjects may have already been created. -
James,
A possible typo:
"If the first line of your code is a comment that includes the word 'coding' <== (shouldn't this be 'encoding'), followed by a colon and space"
-
I see that both 'coding' and 'encoding' are valid but since the example shown immediately before that paragraph used 'encoding' and I had my finger on the trigger....
My apologies..
-
-
When using
File#open, is it still possible to specify the file mode using the integer values available through the constants of the File class? How would you represent encoding intentions while specifying a mode ofFile::WRONLY | File::CREAT | File::TRUNC, for example?-
It's easy to use an
Integermode with anEncoding. Mostopen()-like methods now take an optionalHashof arguments at the end where you can set things like:mode,:external_encoding, or:internal_encoding. Thus your example could be written as:$ cat modes_and_encoding.rb open( "utf16.txt", File::WRONLY | File::CREAT | File::TRUNC, external_encoding: "UTF-16BE" ) do |f| f.puts "Some data." end $ ruby modes_and_encoding.rb $ ruby -e 'p File.binread("utf16.txt")' "\x00S\x00o\x00m\x00e\x00 \x00d\x00a\x00t\x00a\x00.\x00\n"I do talk about this later in the series. I just had to spread some of these topics out a bit because there's a lot to cover and the articles where already very long.
-
Thanks for answering such an obvious question, James. I should have read the RDoc for
IO#new, which clearly describes the API changes.-
No worries. My hope is that we are making things better for all by talking this stuff out.
-
-
-
-
It's worth noting, Ruby currently requires that a source
Encodingbe ASCII compatible. -
What with class
Net::HTTP?
It take always#<Encoding:US-ASCII>-
I believe
Net::HTTPleaves it to the programmer to manage the conversions. Thus, it will likely always return content in something like ASCII-8BIT and leave it to you to callforce_encoding()using information you pull out of the headers, documentation for the service, or whatever.-
return US-ASCII, no
force_encoding.-
Well, it obviously can't return US-ASCII for all cases. What does it do if the data contains extended characters, like UTF-8?
-
OK
It returns US-ASCII if all the bytes in the content is < 128
but return ASCII-8BIT if I put any char >128 in the content.-
Right. At that point, you will need to call
force_encoding(), as I was referring to earlier, to set the proper encoding for your data.
-
-
-
-
-
-
Sorry.
StringIO.newnot seem to have an option to setexternal_encodingat its constructor; It saysStringIO#string.encodingis CP850 in mi pc.I had to set
Encoding.default_external="UTF-8"and nowStringIO#string.encodingis UTF-8.-
I think
StringIOwas a m17n enhanced a little later in the 1.9 conversion game:>> require "stringio" => true >> sio = StringIO.open("", "w:UTF-8") => #<StringIO:0x00000100848ae0> >> sio << "abc" => #<StringIO:0x00000100848ae0> >> sio.string.encoding => #<Encoding:UTF-8> >> RUBY_VERSION => "1.9.2"
It doesn't seem to support the
Hash-style arguments in my version though.-
Thanks.
Not running in 1.9.1 version.
irb(main):001:0> require 'stringio' => true irb(main):002:0> sio = StringIO.open("", "w:UTF-8") => #<StringIO:0x29d4a00> irb(main):003:0> sio << "abc" => #<StringIO:0x29d4a00> irb(main):004:0> sio.string.encoding => #<Encoding:CP850> irb(main):005:0> RUBY_VERSION => "1.9.1" irb(main):006:0>
-
-
-
Hey James,
thanks so much for the 'magic comment' hint.
-
James, thanks for this memorable ride across the enigmatic world of Unicode.
I have a couple of observations on the script
write_internal.rb:(1) The sentence note how my data was transcoded before it was written is not clear (it cannot mean "in the block, before it is written to the file", as the printout shows that the data is of course still in Utf-8); we only see it transcoded when we read from the file (or running an
od -cx data.txtfrom the shell), so I lost what that meant.(2) But the real problem was presented by the format of the string read from the file and printed at the end; I could not make sense of it. Finally I realized that while the string content is encoded as UTF-16LE, Ruby assigned to the string encoding UTF-8 (as per the script coding line); thus, the apparent oddity of the string derives from the fact that ruby is representing in UTF-8 an UTF-16LE string.
Only applying
force_encode("UTF-16LE")to the string read, the string made sense (the unicode triple dot is shown via its unicode codepoint, and all those zero bytes disappeared in the printout). And then when encoding the previous result to UTF-8, we find the exact string we had at the beginning.It all makes sense (although I confess that I had thought, even without realizing it, that Ruby would do at least the first step above, i.e. read the encoding from the file and present a string whose encoding matched the content).
If you have a chance, let me know if I am correct in interpreting this, or if miss something. In any case, thanks again for this extremely useful series.
Raul
-
I did mean, "before it was written to the file." Just think of in terms of
encode()instead ofencode!(), with the returned result being written into the file.As for your other comment, it's not always possible for Ruby to know the encoding from just the data, so it leaves it to us to specify what is intended.
But yeah, it sounds like you have it pretty figured out to me.
-
-
A note on the script
show_internal.rb; for each line, it prints the correct encoding (UTF-16LE), but then instead of the expected text it prints the internal byte structure (like it did not recognize the encoding).I got the same result running 1.9.1, but with 1.9.2 the problem does not occur and the text is the expected one. Just in case you can update it (perhaps printing the text and an
unpackof the line to show the structure).Thanks again for this tutorial
-
I have a collection of input files in several different formats - usascii, iso-8859-1, and utf-8
I need to read them all into a utf-8 encoding for further processing (regex,
split, etc).My solution, which seems to be working at the moment, is:
[user1@hoho6 ~]$ cat james.rb # coding: utf-8 puts `ruby -v` files = `ls -1 /home/user1/Accounts/Kto*.sta` files << `ls -1 /home/user1/Accounts/Kto*.scn` files << `ls -1 /home/user1/Accounts/umsMT940*.txt` pflag = true files.each_line {|fname| fname.chomp! enc = `file -bi "#{fname}"`.chomp.split('=')[1] # puts enc mode = "r" mode << ":#{enc}:utf-8" if enc != 'utf-8' File.open(fname,mode) {|f| f.each_line {|line| line.chomp! begin fields = line.split(':') rescue ArgumentError => e puts e.message puts fname puts line exit(1) end # <further processing of fields> if /UEBERZ/.match(line) != nil puts line if pflag pflag = false end } } } [user1@hoho6 ~]$ [user1@hoho6 ~]$ ruby james.rb ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-linux] :86:809?00UEBERZ.-ZINS?10999116?20ÜBERZIEHUNGSZINSEN?21ZURZEIT [user1@hoho6 ~]$I determine the encoding of each input file prior to opening by using the Unix command
file -bimodeis used to avoid spurious messages when trying to open using"r:utf-8:utf-8"The
splitcommand was my original problem when porting from Ruby 1.8.6 Rails 2.3.5 to Ruby 1.9.2 Rails 3.0.2. Testing each line of each file with arescuearound thesplitgives me some confidence that I can process these files.Any suggestions for code simplification would be appreciated. It would be nice if it could be done without the external Unix command.
-
You could definitely replace shelling out to
lswith a call toDir.glob(), but guessing encodings is a lot trickier. Therchardetgem can help figure it out, I believe.
-
-
Hello,
for now, in Ruby 1.9.2p136, the setting
LC_CTYPEorLANGmethod isn't working. Do you know why?-
Probably not. If you are having trouble it's best to discuss on the Ruby Core mailing list where you can get help from several people smarter than me.
-
-
Thank you for this great article.
Regarding magic comments:
my version of GNU Emacs 22.1.1 complained about an "Invalid coding system" when using this version of the magic comment:# -*- coding: UTF-8 -*-However, Emacs was happy with this:
# -*- coding: utf-8 -*-Just thought I'd pass along the info.
Thanks again for your post.
-
I guess somebody never heard of convention over configuration.
I mean would it really kill those Japanese gem developers (or the rest of us) to set text editors to UTF8 instead of SJIS (or US-ASCII)? The whole point of Unicode is that it's universal - a unique ID for every symbol of every language on earth. Instead of that, now every Ruby file on earth needs to have an extra line of garbage at the top.
-
Even if all the Japanese switch the UTF-8 today, they will have plenty of legacy data to contend with. They also have reasons for the slow adoption.
-
-
Sorry to get in so late on this.
I tried putting
# encoding: UTF-8to my file but I still get the error "invalid multibyte char (US-ASCII) (SyntaxError)". syntax error, unexpected '|'It doesn't seem to like "|"(pipe) character. My Ruby version is ruby 1.9.2p180 (2011-02-18) [i386-mingw32]
-
The encoding line must be the very first line of the file, or the second if you have a shebang line.
-
Thanks Edward, it worked. I had few comments in there.
-
-
-
Great article, since long I am trying to understand the encoding of
IO, gone through multiple articles but not convinced with anyone of them. Finally got my hand on this article. Now things are pretty clear. -
Thanks for a great series! Very helpful.
I just noticed a typo:
That makes sense, as it's really to late
should be
That makes sense, as it's really too late
-
Fixed. Thanks.
-
-
Thanks much for this!
using ruby 1.9.3, it does not appear that default_external applies to requiring files:
$ cat test.rb puts 'internal:' puts Encoding.default_internal puts 'external:' puts Encoding.default_external require './test2' $ cat test2.rb def test_str return "a string" end $ ruby -E UTF-8:UTF-8 test.rb internal: UTF-8 external: UTF-8 a string US-ASCIIHowever, the
-Kuoption does seem to work:$ ruby -E UTF-8:UTF-8 -Ku test.rb internal: UTF-8 external: UTF-8 a string UTF-8Any idea if this is by design or a bug?
-
Sorry for the formatting of the previous comment. The form ate my linefeeds pasted from terminal.
-
The line endings where still there, but these comments are int Markdown. I reformatted your comment to indent the code.
-
-
The default internal and external encodings apply to
IObased communication, not to the encoding of the source code. That's the third encoding type, separate from the other two.The source encoding is controlled via the "(en)coding" comments. Also, as you've noted,
-Kcan change it as well. That's really just for backwards compatibility though and you should be using the comments.Hope that helps.
-
-
Great series of articles, really clarifies the whole encoding mess for me.
The only contribution I can offer at this time is an insignificant typo to note. In point 2. of four things to note in the example, you start with "Use can use", but I think you meant to write "You can use".
Thanks for putting this all out there so clearly.
-
Fixed. Thanks.
-
-
Great article. Thanks very much.