Character Encodings

My extensive coverage of a complex topic all programmers should study a little.

5

APR
2009

Ruby 1.9's Three Default Encodings

I suspect early contact with the new m17n (multilingualization) engine is going to come to Rubyists in the form of this error message:

invalid multibyte char (US-ASCII)

Ruby 1.8 didn't care what you stuck in a random String literal, but 1.9 is a touch pickier. I think you'll see that the change is for the better, but we do need to spend some time learning to play by Ruby's new rules.

That takes us to the first of Ruby's three default Encodings.

The Source Encoding

In Ruby's new grown up world of all encoded data, each and every String needs an Encoding. That means an Encoding must be selected for a String as soon as it is created. One way that a String can be created is for Ruby to execute some code with a String literal in it, like this:

str = "A new String"

That's a pretty simple String, but what if I use a literal like the following instead?

str = "Résumé"

What Encoding is that in? That fundamental question is probably the main reason we all struggle a bit with character encodings. You can't tell just from looking at that data what Encoding it is in. Now, if I showed you the bytes you may be able to make an educated guess, but the data just isn't wearing an Encoding name tag.

That's true of a frightening lot of data we deal with every day. A plain text file doesn't generally say what Encoding the data inside is in. When you think about that, it's a miracle we can successfully read a lot of things.

When we're talking about program code, the problem gets worse. I may want to write my code in UTF-8, but some Japanese programmer may want to write his code in Shift JIS. Ruby should support that and, in fact, 1.9 does. Let's complicate things a bit more though: imagine that I bundle up that UTF-8 code I wrote in a gem and the Japanese programmer later uses it to help with his Shift JIS code. How do we make that work seamlessly?

The Ruby 1.8 strategy of one global variable won't survive a test like this, so it was time to switch strategies. Ruby 1.9's answer to this problem is the source Encoding.

All Ruby source code now has some Encoding. When you create a String literal in your code, it is assigned the Encoding of your source. That simple rule solves all the problems I just described pretty nicely. As long my source Encoding is UTF-8 and the Japanese programmer's source Encoding is Shift JIS, my literals will work as I expect and his will work as he expects. Obviously if we share any data, we will need to establish some rules about our shared formats using documentation or code that can adapt to different Encodings, but we should have been doing that all along anyway.

Thus the only question becomes, what's my source Encoding and how do I change it?

There are a few different ways Ruby can select a source Encoding. Here are the options:

$ cat no_encoding.rb 
p __ENCODING__
$ ruby no_encoding.rb 
#<Encoding:US-ASCII>

$ cat magic_comment.rb 
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment.rb 
#<Encoding:UTF-8>
$ cat magic_comment2.rb 
#!/usr/bin/env ruby -w
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment2.rb 
#<Encoding:UTF-8>

$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'p __ENCODING__'
#<Encoding:UTF-8>

$ ruby -KU no_encoding.rb 
#<Encoding:UTF-8>

The first example shows us two important things. The first is the main rule of source Encodings: source files receive a US-ASCII Encoding, unless you say otherwise. [Update: this was changed to UTF-8 in Ruby 2.0 and up.] This is where I expect programmers to run into the error I mentioned earlier. If you place any non-ASCII content in a String literal without changing the source Encoding, Ruby will die with that error. Thus you need to change the source Encoding to work with any non-ASCII data. The second thing we see here is the new __ENCODING__ keyword that can be used to get the source Encoding that's active where it is executed.

The second example shows the preferred way to set your source Encoding and it's called a magic comment. If the first line of your code is a comment that includes the word coding, followed by a colon and space, and then an Encoding name, the source Encoding for that file is changed to the indicated Encoding. If your code has a shebang line, the magic comment must come on the second line, with no spacing between them. Once set, all String literals you create in that file will have that Encoding attached to them.

The third example shows an exception to the rule for your convenience. When you feed Ruby some code on the command-line using the -e switch, it gets a source Encoding from your environment. I have UTF-8 set in the LC_CTYPE environment variable, but some people also use the LANG variable for this. This makes scripting easier since Ruby will (hopefully) match the Encoding of any other commands you chain together.

The fourth example is another interesting exception to the rule. Ruby 1.9 still supports the -K* style switches from Ruby 1.8 including the -KU switch I've recommended so heavily in this series. These switches have a couple of effects, but of particular note they are the only non-magic comment way to modify the source Encoding. This is good news for backwards compatibility, because some Ruby 1.8 code may be able to run on Ruby 1.9 without Encoding problems thanks to this. I must stress that this is just for backwards compatibility though, and magic comments are the future.

With magic comments the code will include its Encoding data. It will probably seem a little tedious to add them to all your source files at first, but it's really not that big of a change. In the past, I've recommended we stick the following shebang line at the top of our files:

#!/usr/bin/env ruby -wKU

Now, for Ruby 1.9, I'm recommending we switch to something like this:

#!/usr/bin/env ruby -w
# encoding: UTF-8

Note that the magic comment format rules are pretty loose and all of following examples would work the same:

# encoding: UTF-8

# coding: UTF-8

# -*- coding: UTF-8 -*-

This is nice for support in some text editors that also read such comments.

If we all get into that habit of adding magic comments, our code can work together regardless of the various Encodings we personally favor. Ruby will know how to handle each separate file. As an added bonus, we programmers also get to see these comments and know more about the code we are working with. That makes it a good habit to get into, I think.

The Default External and Internal Encodings

There's another way Strings are commonly created and that's by reading from some IO object. It doesn't make sense to give those Strings the source Encoding because the external data doesn't have to be related to your source code. Also, you really need to know how data is encoded to read it correctly. Even a simple concept like reading the next line of data changes if you are talking about UTF-8 or UTF-16LE (the LE stands for a Little Endian byte order) data. Thus, it makes sense for IO objects to have at least one Encoding attached to them. Ruby 1.9 is generous and gives them two: the external Encoding and the internal Encoding.

The external Encoding is the Encoding the data is in inside the IO object. That affects how data will be read and this is the Encoding data will be returned in as long as the internal Encoding isn't set (more on that in a bit). Let's look at an example of how this plays out in practice:

$ cat show_external.rb 
open(__FILE__, "r:UTF-8") do |file|
  puts file.external_encoding.name
  p    file.internal_encoding
  file.each do |line|
    p [line.encoding.name, line]
  end
end
$ ruby show_external.rb 
UTF-8
nil
["UTF-8", "open(__FILE__, \"r:UTF-8\") do |file|\n"]
["UTF-8", "  puts file.external_encoding.name\n"]
["UTF-8", "  p    file.internal_encoding\n"]
["UTF-8", "  file.each do |line|\n"]
["UTF-8", "    p [line.encoding.name, line]\n"]
["UTF-8", "  end\n"]
["UTF-8", "end\n"]

There are four things to notice in this example:

  1. I set the external Encoding by tacking :UTF-8 onto the end of my mode String when I opened the File
  2. You can use external_encoding() to check the external Encoding as I have here
  3. internal_encoding() works the same for the internal Encoding, which will be nil unless you explicitly set it
  4. Note how each String created as I read the data is given the external_encoding()

The internal Encoding just adds one more twist. When set, data will still be read in the external Encoding, but transcoded to the internal Encoding as the String is created. It's a convenience for you as the programmer. Watch how that changes things:

$ cat show_internal.rb 
open(__FILE__, "r:UTF-8:UTF-16LE") do |file|
  puts file.external_encoding.name
  puts file.internal_encoding.name
  file.each do |line|
    p [line.encoding.name, line[0..3]]
  end
end
$ ruby show_internal.rb 
UTF-8
UTF-16LE
["UTF-16LE", "o\x00p\x00e\x00n\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00f\x00i\x00"]
["UTF-16LE", " \x00 \x00 \x00 \x00"]
["UTF-16LE", " \x00 \x00e\x00n\x00"]
["UTF-16LE", "e\x00n\x00d\x00\n\x00"]

There are a couple differences here:

  1. A second added Encoding on the mode String (my :UTF-16LE in this example) sets the internal_encoding() as I show with the second puts()
  2. This little change gets Ruby to translate all of the data for me (I just shortened the output because UTF-16LE is noisy)

The external Encoding works the same when writing. It still represents the Encoding in the IO object, or the Encoding data is going to. However, you don't need to specify an internal Encoding when writing. Ruby will automatically use the Encoding of a String you output as the internal Encoding and transcode as needed to reach the external Encoding. For example:

$ cat write_internal.rb 
# encoding: UTF-8
open("data.txt", "w:UTF-16LE") do |file|
  puts file.external_encoding.name
  p    file.internal_encoding
  data = "My data…"
  p [data.encoding.name, data]
  file << data
end
p File.read("data.txt")
$ ruby write_internal.rb 
UTF-16LE
nil
["UTF-8", "My data…"]
"M\x00y\x00 \x00d\x00a\x00t\x00a\x00& "

Note how my data was transcoded before it was written even though the internal_encoding() was nil. Ruby used the String's Encoding to decide what was needed.

Both of those IO Encodings should be pretty straight forward. The only question left about them is: what happens if you don't set them? The answer is that the IO inherits the default external Encoding and/or the default internal Encoding whenever one isn't set. Now we need to know how Ruby chooses those defaults.

The default external Encoding is pulled from your environment, much like the source Encoding is for code given on the command-line. Have a look:

$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'puts Encoding.default_external.name'
UTF-8
$ LC_CTYPE=ja_JP.sjis ruby -e 'puts Encoding.default_external.name'
Shift_JIS

The default internal Encoding is simply nil. You must actively change it to get anything else.

Both default IO Encodings have a global setter: Encoding.default_external=() and Encoding.default_internal=(). You can set them to an Encoding object or just the String name of an Encoding.

You can also change these default Encodings using some command-line switches. The new -E switch can be used to set one or both of the IO Encodings:

$ ruby -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, nil]
$ ruby -E Shift_JIS \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, nil]
$ ruby -E :UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, #<Encoding:UTF-16LE>]
$ ruby -E Shift_JIS:UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, #<Encoding:UTF-16LE>]

As you can see, the argument for this switch is just like what you would append to a mode String in a call to File.open().

There's one more command-line switch shortcut for those of us who prefer to just use UTF-8 everywhere. The new -U switch sets Encoding.default_internal() to UTF-8. Using that, you can just set the external Encoding for your IO objects, or let it default from your environment, and all Strings you read will be transcoded to the preferred UTF-8.

Probably the most important thing to note about Encoding.default_external() and Encoding.default_internal() is that you should really just treat them as shortcuts for your own scripting. Pulling Encodings from the environment or command-line switches can be handy when you're in control of where the code runs, but you're going to need to be more explicit for code you intend for others to run. When in doubt, set the external and internal Encodings the way you want them for each IO object. It's a bit more tedious, but also safer in that it won't mysteriously be changed by some outside force. Also remember that the defaults are global settings affecting all loaded code, including any libraries you require(). That can be a boon or bane, so just remember to factor it into your thinking when you're wondering, "Where does this String get its Encoding from?"

Comments (41)
  1. James Edward Gray II
    James Edward Gray II April 6th, 2009 Reply Link

    It's probably worth noting that using the default Encoding setters will trigger warnings:

    $ ruby -we 'Encoding.default_internal = Encoding.default_external = "UTF-8"'
    -e:1: warning: setting Encoding.default_external
    -e:1: warning: setting Encoding.default_internal
    

    That makes sense, as it's really too late to set these in code after IO objects may have already been created.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  2. Saimon Moore
    Saimon Moore April 7th, 2009 Reply Link

    James,

    A possible typo:

    "If the first line of your code is a comment that includes the word 'coding' <== (shouldn't this be 'encoding'), followed by a colon and space"

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. Saimon Moore
      Saimon Moore April 7th, 2009 Reply Link

      I see that both 'coding' and 'encoding' are valid but since the example shown immediately before that paragraph used 'encoding' and I had my finger on the trigger....

      My apologies..

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  3. Nathan de Vries
    Nathan de Vries April 16th, 2009 Reply Link

    When using File#open, is it still possible to specify the file mode using the integer values available through the constants of the File class? How would you represent encoding intentions while specifying a mode of File::WRONLY | File::CREAT | File::TRUNC, for example?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 16th, 2009 Reply Link

      It's easy to use an Integer mode with an Encoding. Most open()-like methods now take an optional Hash of arguments at the end where you can set things like :mode, :external_encoding, or :internal_encoding. Thus your example could be written as:

      $ cat modes_and_encoding.rb 
      open( "utf16.txt", File::WRONLY | File::CREAT | File::TRUNC,
                         external_encoding: "UTF-16BE" ) do |f|
        f.puts "Some data."
      end
      $ ruby modes_and_encoding.rb 
      $ ruby -e 'p File.binread("utf16.txt")'
      "\x00S\x00o\x00m\x00e\x00 \x00d\x00a\x00t\x00a\x00.\x00\n"
      

      I do talk about this later in the series. I just had to spread some of these topics out a bit because there's a lot to cover and the articles where already very long.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
      2. Nathan de Vries
        Nathan de Vries April 17th, 2009 Reply Link

        Thanks for answering such an obvious question, James. I should have read the RDoc for IO#new, which clearly describes the API changes.

        1. Reply (using GitHub Flavored Markdown)

          Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

          Ajax loader
        2. James Edward Gray II
          James Edward Gray II April 17th, 2009 Reply Link

          No worries. My hope is that we are making things better for all by talking this stuff out.

          1. Reply (using GitHub Flavored Markdown)

            Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

            Ajax loader
  4. James Edward Gray II
    James Edward Gray II August 7th, 2009 Reply Link

    It's worth noting, Ruby currently requires that a source Encoding be ASCII compatible.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  5. Julio Fernández
    Julio Fernández March 31st, 2010 Reply Link

    What with class Net::HTTP?
    It take always #<Encoding:US-ASCII>

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II March 31st, 2010 Reply Link

      I believe Net::HTTP leaves it to the programmer to manage the conversions. Thus, it will likely always return content in something like ASCII-8BIT and leave it to you to call force_encoding() using information you pull out of the headers, documentation for the service, or whatever.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
      2. Julio Fernández
        Julio Fernández March 31st, 2010 Reply Link

        return US-ASCII, no force_encoding.

        1. Reply (using GitHub Flavored Markdown)

          Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

          Ajax loader
        2. James Edward Gray II
          James Edward Gray II March 31st, 2010 Reply Link

          Well, it obviously can't return US-ASCII for all cases. What does it do if the data contains extended characters, like UTF-8?

          1. Reply (using GitHub Flavored Markdown)

            Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

            Ajax loader
          2. Julio Fernández
            Julio Fernández March 31st, 2010 Reply Link

            OK
            It returns US-ASCII if all the bytes in the content is < 128
            but return ASCII-8BIT if I put any char >128 in the content.

            1. Reply (using GitHub Flavored Markdown)

              Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

              Ajax loader
            2. James Edward Gray II
              James Edward Gray II March 31st, 2010 Reply Link

              Right. At that point, you will need to call force_encoding(), as I was referring to earlier, to set the proper encoding for your data.

              1. Reply (using GitHub Flavored Markdown)

                Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

                Ajax loader
  6. Julio Fernández
    Julio Fernández April 4th, 2010 Reply Link

    Sorry.

    StringIO.new not seem to have an option to set external_encoding at its constructor; It says StringIO#string.encoding is CP850 in mi pc.

    I had to set Encoding.default_external="UTF-8" and now StringIO#string.encoding is UTF-8.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 4th, 2010 Reply Link

      I think StringIO was a m17n enhanced a little later in the 1.9 conversion game:

      >> require "stringio"
      => true
      >> sio = StringIO.open("", "w:UTF-8")
      => #<StringIO:0x00000100848ae0>
      >> sio << "abc"
      => #<StringIO:0x00000100848ae0>
      >> sio.string.encoding
      => #<Encoding:UTF-8>
      >> RUBY_VERSION
      => "1.9.2"
      

      It doesn't seem to support the Hash-style arguments in my version though.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
      2. Julio Fernández
        Julio Fernández April 4th, 2010 Reply Link

        Thanks.

        Not running in 1.9.1 version.

        irb(main):001:0> require 'stringio'
        => true
        irb(main):002:0> sio = StringIO.open("", "w:UTF-8")
        => #<StringIO:0x29d4a00>
        irb(main):003:0> sio << "abc"
        => #<StringIO:0x29d4a00>
        irb(main):004:0> sio.string.encoding
        => #<Encoding:CP850>
        irb(main):005:0> RUBY_VERSION
        => "1.9.1"
        irb(main):006:0>
        
        1. Reply (using GitHub Flavored Markdown)

          Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

          Ajax loader
  7. Daniel
    Daniel October 16th, 2010 Reply Link

    Hey James,

    thanks so much for the 'magic comment' hint.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  8. raulparolari@gmail.com
    raulparolari@gmail.com November 21st, 2010 Reply Link

    James, thanks for this memorable ride across the enigmatic world of Unicode.

    I have a couple of observations on the script write_internal.rb:

    (1) The sentence note how my data was transcoded before it was written is not clear (it cannot mean "in the block, before it is written to the file", as the printout shows that the data is of course still in Utf-8); we only see it transcoded when we read from the file (or running an od -cx data.txt from the shell), so I lost what that meant.

    (2) But the real problem was presented by the format of the string read from the file and printed at the end; I could not make sense of it. Finally I realized that while the string content is encoded as UTF-16LE, Ruby assigned to the string encoding UTF-8 (as per the script coding line); thus, the apparent oddity of the string derives from the fact that ruby is representing in UTF-8 an UTF-16LE string.

    Only applying force_encode("UTF-16LE") to the string read, the string made sense (the unicode triple dot is shown via its unicode codepoint, and all those zero bytes disappeared in the printout). And then when encoding the previous result to UTF-8, we find the exact string we had at the beginning.

    It all makes sense (although I confess that I had thought, even without realizing it, that Ruby would do at least the first step above, i.e. read the encoding from the file and present a string whose encoding matched the content).

    If you have a chance, let me know if I am correct in interpreting this, or if miss something. In any case, thanks again for this extremely useful series.

    Raul

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II November 21st, 2010 Reply Link

      I did mean, "before it was written to the file." Just think of in terms of encode() instead of encode!(), with the returned result being written into the file.

      As for your other comment, it's not always possible for Ruby to know the encoding from just the data, so it leaves it to us to specify what is intended.

      But yeah, it sounds like you have it pretty figured out to me.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  9. Raul Parolari
    Raul Parolari November 22nd, 2010 Reply Link

    A note on the script show_internal.rb; for each line, it prints the correct encoding (UTF-16LE), but then instead of the expected text it prints the internal byte structure (like it did not recognize the encoding).

    I got the same result running 1.9.1, but with 1.9.2 the problem does not occur and the text is the expected one. Just in case you can update it (perhaps printing the text and an unpack of the line to show the structure).

    Thanks again for this tutorial

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  10. Bob Gustafson
    Bob Gustafson January 19th, 2011 Reply Link

    I have a collection of input files in several different formats - usascii, iso-8859-1, and utf-8

    I need to read them all into a utf-8 encoding for further processing (regex, split, etc).

    My solution, which seems to be working at the moment, is:

    [user1@hoho6 ~]$ cat james.rb
    # coding: utf-8
    puts `ruby -v`
    
    files =  `ls -1 /home/user1/Accounts/Kto*.sta`
    files << `ls -1 /home/user1/Accounts/Kto*.scn`
    files << `ls -1 /home/user1/Accounts/umsMT940*.txt`
    
    pflag = true
    
    files.each_line {|fname|
      fname.chomp!
      enc = `file -bi "#{fname}"`.chomp.split('=')[1]
    # puts enc
      mode = "r"
      mode << ":#{enc}:utf-8" if enc != 'utf-8'
      File.open(fname,mode) {|f|
        f.each_line {|line|
          line.chomp!
          begin
            fields = line.split(':')
          rescue ArgumentError => e
            puts e.message
            puts fname
            puts line
            exit(1)
          end
          # <further processing of fields>
          if /UEBERZ/.match(line) != nil
            puts line if pflag
            pflag = false
          end
        }
      }
    } 
    [user1@hoho6 ~]$ 
    
    [user1@hoho6 ~]$ ruby james.rb
    ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-linux]
    :86:809?00UEBERZ.-ZINS?10999116?20ÜBERZIEHUNGSZINSEN?21ZURZEIT       
    [user1@hoho6 ~]$
    

    I determine the encoding of each input file prior to opening by using the Unix command file -bi

    mode is used to avoid spurious messages when trying to open using "r:utf-8:utf-8"

    The split command was my original problem when porting from Ruby 1.8.6 Rails 2.3.5 to Ruby 1.9.2 Rails 3.0.2. Testing each line of each file with a rescue around the split gives me some confidence that I can process these files.

    Any suggestions for code simplification would be appreciated. It would be nice if it could be done without the external Unix command.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II January 19th, 2011 Reply Link

      You could definitely replace shelling out to ls with a call to Dir.glob(), but guessing encodings is a lot trickier. The rchardet gem can help figure it out, I believe.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  11. 3ануда
    3ануда March 21st, 2011 Reply Link

    Hello,

    for now, in Ruby 1.9.2p136, the setting LC_CTYPE or LANG method isn't working. Do you know why?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II March 21st, 2011 Reply Link

      Probably not. If you are having trouble it's best to discuss on the Ruby Core mailing list where you can get help from several people smarter than me.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  12. jgpawletko
    jgpawletko April 9th, 2011 Reply Link

    Thank you for this great article.

    Regarding magic comments:
    my version of GNU Emacs 22.1.1 complained about an "Invalid coding system" when using this version of the magic comment:

    # -*- coding: UTF-8 -*-
    

    However, Emacs was happy with this:

    # -*- coding: utf-8 -*-
    

    Just thought I'd pass along the info.

    Thanks again for your post.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  13. Tobias Cohen
    Tobias Cohen April 18th, 2011 Reply Link

    I guess somebody never heard of convention over configuration.

    I mean would it really kill those Japanese gem developers (or the rest of us) to set text editors to UTF8 instead of SJIS (or US-ASCII)? The whole point of Unicode is that it's universal - a unique ID for every symbol of every language on earth. Instead of that, now every Ruby file on earth needs to have an extra line of garbage at the top.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 18th, 2011 Reply Link

      Even if all the Japanese switch the UTF-8 today, they will have plenty of legacy data to contend with. They also have reasons for the slow adoption.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  14. skawley
    skawley May 9th, 2011 Reply Link

    Sorry to get in so late on this.

    I tried putting # encoding: UTF-8 to my file but I still get the error "invalid multibyte char (US-ASCII) (SyntaxError)". syntax error, unexpected '|'

    It doesn't seem to like "|"(pipe) character. My Ruby version is ruby 1.9.2p180 (2011-02-18) [i386-mingw32]

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II

      The encoding line must be the very first line of the file, or the second if you have a shebang line.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
      2. skawley
        skawley May 10th, 2011 Reply Link

        Thanks Edward, it worked. I had few comments in there.

        1. Reply (using GitHub Flavored Markdown)

          Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

          Ajax loader
  15. Kaushal Kishore
    Kaushal Kishore August 3rd, 2012 Reply Link

    Great article, since long I am trying to understand the encoding of IO, gone through multiple articles but not convinced with anyone of them. Finally got my hand on this article. Now things are pretty clear.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  16. Colin Kelley
    Colin Kelley August 4th, 2012 Reply Link

    Thanks for a great series! Very helpful.

    I just noticed a typo:

    That makes sense, as it's really to late

    should be

    That makes sense, as it's really too late

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II August 4th, 2012 Reply Link

      Fixed. Thanks.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  17. Emil Rehnberg
    Emil Rehnberg June 28th, 2014 Reply Link

    This is a really good reminder of (seemingly obvious) strats for solving problems as a team. Respect to you for sharing your relearned lessons, Mr Gray!

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  18. HerbCSO
    HerbCSO December 16th, 2012 Reply Link

    Great series of articles, really clarifies the whole encoding mess for me.

    The only contribution I can offer at this time is an insignificant typo to note. In point 2. of four things to note in the example, you start with "Use can use", but I think you meant to write "You can use".

    Thanks for putting this all out there so clearly.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II December 16th, 2012 Reply Link

      Fixed. Thanks.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  19. Jonah Burke
    Jonah Burke December 28th, 2012 Reply Link

    Great article. Thanks very much.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
Leave a Comment (using GitHub Flavored Markdown)

Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

Ajax loader