Gray Soft / Character Encodings / Ruby 1.9's Three Default Encodings

The 41st Comment on "Ruby 1.9's Three Default Encodings"

2014-03-27T01:38:28Z

Great article. Thanks very much.

The 40th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:30:32Z

Fixed. Thanks.

The 39th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:30:32Z

Great series of articles, really clarifies the whole encoding mess for me.

The only contribution I can offer at this time is an insignificant typo to note. In point 2. of four things to note in the example, you start with "Use can use", but I think you meant to write "You can use".

Thanks for putting this all out there so clearly.

The 38th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:29:55Z

The default internal and external encodings apply to IO based communication, not to the encoding of the source code. That's the third encoding type, separate from the other two.

The source encoding is controlled via the "(en)coding" comments. Also, as you've noted, -K can change it as well. That's really just for backwards compatibility though and you should be using the comments.

Hope that helps.

The 37th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:29:16Z

The line endings where still there, but these comments are int Markdown. I reformatted your comment to indent the code.

The 36th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:29:16Z

Sorry for the formatting of the previous comment. The form ate my linefeeds pasted from terminal.

The 35th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:29:55Z

Thanks much for this!

using ruby 1.9.3, it does not appear that default_external applies to requiring files:

$ cat test.rb

puts 'internal:'
puts Encoding.default_internal
puts 'external:'
puts Encoding.default_external
require './test2'

$ cat test2.rb 

def test_str
  return "a string"
end

$ ruby -E UTF-8:UTF-8 test.rb 
internal:
UTF-8
external:
UTF-8
a string
US-ASCII

However, the -Ku option does seem to work:

$ ruby -E UTF-8:UTF-8 -Ku test.rb 
internal:
UTF-8
external:
UTF-8
a string
UTF-8

Any idea if this is by design or a bug?

The 34th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:26:50Z

Fixed. Thanks.

The 33rd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:26:50Z

Thanks for a great series! Very helpful.

I just noticed a typo:

That makes sense, as it's really to late

should be

That makes sense, as it's really too late

The 32nd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:26:07Z

Great article, since long I am trying to understand the encoding of IO, gone through multiple articles but not convinced with anyone of them. Finally got my hand on this article. Now things are pretty clear.

The 31st Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:25:34Z

Thanks Edward, it worked. I had few comments in there.

The 30th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:25:34Z

The encoding line must be the very first line of the file, or the second if you have a shebang line.

The 29th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:25:34Z

Sorry to get in so late on this.

I tried putting # encoding: UTF-8 to my file but I still get the error "invalid multibyte char (US-ASCII) (SyntaxError)". syntax error, unexpected '|'

It doesn't seem to like "|"(pipe) character. My Ruby version is ruby 1.9.2p180 (2011-02-18) [i386-mingw32]

The 28th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:24:03Z

Even if all the Japanese switch the UTF-8 today, they will have plenty of legacy data to contend with. They also have reasons for the slow adoption.

The 27th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:24:03Z

I guess somebody never heard of convention over configuration.

I mean would it really kill those Japanese gem developers (or the rest of us) to set text editors to UTF8 instead of SJIS (or US-ASCII)? The whole point of Unicode is that it's universal - a unique ID for every symbol of every language on earth. Instead of that, now every Ruby file on earth needs to have an extra line of garbage at the top.

The 26th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:22:44Z

Thank you for this great article.

Regarding magic comments:
my version of GNU Emacs 22.1.1 complained about an "Invalid coding system" when using this version of the magic comment:

# -*- coding: UTF-8 -*-

However, Emacs was happy with this:

# -*- coding: utf-8 -*-

Just thought I'd pass along the info.

Thanks again for your post.

The 25th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:21:38Z

Probably not. If you are having trouble it's best to discuss on the Ruby Core mailing list where you can get help from several people smarter than me.

The 24th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:21:38Z

Hello,

for now, in Ruby 1.9.2p136, the setting LC_CTYPE or LANG method isn't working. Do you know why?

The 23rd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:20:59Z

You could definitely replace shelling out to ls with a call to Dir.glob(), but guessing encodings is a lot trickier. The rchardet gem can help figure it out, I believe.

The 22nd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:20:59Z

I have a collection of input files in several different formats - usascii, iso-8859-1, and utf-8

I need to read them all into a utf-8 encoding for further processing (regex, split, etc).

My solution, which seems to be working at the moment, is:

[user1@hoho6 ~]$ cat james.rb
# coding: utf-8
puts `ruby -v`

files =  `ls -1 /home/user1/Accounts/Kto*.sta`
files << `ls -1 /home/user1/Accounts/Kto*.scn`
files << `ls -1 /home/user1/Accounts/umsMT940*.txt`

pflag = true

files.each_line {|fname|
  fname.chomp!
  enc = `file -bi "#{fname}"`.chomp.split('=')[1]
# puts enc
  mode = "r"
  mode << ":#{enc}:utf-8" if enc != 'utf-8'
  File.open(fname,mode) {|f|
    f.each_line {|line|
      line.chomp!
      begin
        fields = line.split(':')
      rescue ArgumentError => e
        puts e.message
        puts fname
        puts line
        exit(1)
      end
      # <further processing of fields>
      if /UEBERZ/.match(line) != nil
        puts line if pflag
        pflag = false
      end
    }
  }
} 
[user1@hoho6 ~]$ 

[user1@hoho6 ~]$ ruby james.rb
ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-linux]
:86:809?00UEBERZ.-ZINS?10999116?20ÜBERZIEHUNGSZINSEN?21ZURZEIT       
[user1@hoho6 ~]$

I determine the encoding of each input file prior to opening by using the Unix command file -bi

mode is used to avoid spurious messages when trying to open using "r:utf-8:utf-8"

The split command was my original problem when porting from Ruby 1.8.6 Rails 2.3.5 to Ruby 1.9.2 Rails 3.0.2. Testing each line of each file with a rescue around the split gives me some confidence that I can process these files.

Any suggestions for code simplification would be appreciated. It would be nice if it could be done without the external Unix command.

The 21st Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:17:28Z

A note on the script show_internal.rb; for each line, it prints the correct encoding (UTF-16LE), but then instead of the expected text it prints the internal byte structure (like it did not recognize the encoding).

I got the same result running 1.9.1, but with 1.9.2 the problem does not occur and the text is the expected one. Just in case you can update it (perhaps printing the text and an unpack of the line to show the structure).

Thanks again for this tutorial

The 20th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:15:29Z

I did mean, "before it was written to the file." Just think of in terms of encode() instead of encode!(), with the returned result being written into the file.

As for your other comment, it's not always possible for Ruby to know the encoding from just the data, so it leaves it to us to specify what is intended.

But yeah, it sounds like you have it pretty figured out to me.

The 19th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:15:29Z

James, thanks for this memorable ride across the enigmatic world of Unicode.

I have a couple of observations on the script write_internal.rb:

(1) The sentence note how my data was transcoded before it was written is not clear (it cannot mean "in the block, before it is written to the file", as the printout shows that the data is of course still in Utf-8); we only see it transcoded when we read from the file (or running an od -cx data.txt from the shell), so I lost what that meant.

(2) But the real problem was presented by the format of the string read from the file and printed at the end; I could not make sense of it. Finally I realized that while the string content is encoded as UTF-16LE, Ruby assigned to the string encoding UTF-8 (as per the script coding line); thus, the apparent oddity of the string derives from the fact that ruby is representing in UTF-8 an UTF-16LE string.

Only applying force_encode("UTF-16LE") to the string read, the string made sense (the unicode triple dot is shown via its unicode codepoint, and all those zero bytes disappeared in the printout). And then when encoding the previous result to UTF-8, we find the exact string we had at the beginning.

It all makes sense (although I confess that I had thought, even without realizing it, that Ruby would do at least the first step above, i.e. read the encoding from the file and present a string whose encoding matched the content).

If you have a chance, let me know if I am correct in interpreting this, or if miss something. In any case, thanks again for this extremely useful series.

Raul

The 18th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:09:53Z

Hey James,

thanks so much for the 'magic comment' hint.

The 17th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:09:30Z

Thanks.

Not running in 1.9.1 version.

irb(main):001:0> require 'stringio'
=> true
irb(main):002:0> sio = StringIO.open("", "w:UTF-8")
=> #<StringIO:0x29d4a00>
irb(main):003:0> sio << "abc"
=> #<StringIO:0x29d4a00>
irb(main):004:0> sio.string.encoding
=> #<Encoding:CP850>
irb(main):005:0> RUBY_VERSION
=> "1.9.1"
irb(main):006:0>

The 16th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:09:30Z

I think StringIO was a m17n enhanced a little later in the 1.9 conversion game:

>> require "stringio"
=> true
>> sio = StringIO.open("", "w:UTF-8")
=> #<StringIO:0x00000100848ae0>
>> sio << "abc"
=> #<StringIO:0x00000100848ae0>
>> sio.string.encoding
=> #<Encoding:UTF-8>
>> RUBY_VERSION
=> "1.9.2"

It doesn't seem to support the Hash-style arguments in my version though.

The 15th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:09:30Z

Sorry.

StringIO.new not seem to have an option to set external_encoding at its constructor; It says StringIO#string.encoding is CP850 in mi pc.

I had to set Encoding.default_external="UTF-8" and now StringIO#string.encoding is UTF-8.

The 14th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

Right. At that point, you will need to call force_encoding(), as I was referring to earlier, to set the proper encoding for your data.

The 13th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

OK
It returns US-ASCII if all the bytes in the content is < 128
but return ASCII-8BIT if I put any char >128 in the content.

The 12th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

Well, it obviously can't return US-ASCII for all cases. What does it do if the data contains extended characters, like UTF-8?

The 11th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

return US-ASCII, no force_encoding.

The 10th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

I believe Net::HTTP leaves it to the programmer to manage the conversions. Thus, it will likely always return content in something like ASCII-8BIT and leave it to you to call force_encoding() using information you pull out of the headers, documentation for the service, or whatever.

The 9th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:05:58Z

What with class Net::HTTP?
It take always #<Encoding:US-ASCII>

The 8th Comment on "Ruby 1.9's Three Default Encodings"

2014-03-27T01:38:26Z

It's worth noting, Ruby currently requires that a source Encoding be ASCII compatible.

The 7th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:03:58Z

No worries. My hope is that we are making things better for all by talking this stuff out.

The 6th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:03:58Z

Thanks for answering such an obvious question, James. I should have read the RDoc for IO#new, which clearly describes the API changes.

The 5th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:03:58Z

It's easy to use an Integer mode with an Encoding. Most open()-like methods now take an optional Hash of arguments at the end where you can set things like :mode, :external_encoding, or :internal_encoding. Thus your example could be written as:

$ cat modes_and_encoding.rb 
open( "utf16.txt", File::WRONLY | File::CREAT | File::TRUNC,
                   external_encoding: "UTF-16BE" ) do |f|
  f.puts "Some data."
end
$ ruby modes_and_encoding.rb 
$ ruby -e 'p File.binread("utf16.txt")'
"\x00S\x00o\x00m\x00e\x00 \x00d\x00a\x00t\x00a\x00.\x00\n"

I do talk about this later in the series. I just had to spread some of these topics out a bit because there's a lot to cover and the articles where already very long.

The 4th Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T18:03:58Z

When using File#open, is it still possible to specify the file mode using the integer values available through the constants of the File class? How would you represent encoding intentions while specifying a mode of File::WRONLY | File::CREAT | File::TRUNC, for example?

The 3rd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T17:59:44Z

I see that both 'coding' and 'encoding' are valid but since the example shown immediately before that paragraph used 'encoding' and I had my finger on the trigger.... My apologies..

The 2nd Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T17:59:44Z

James, A possible typo: "If the first line of your code is a comment that includes the word 'coding' <== (shouldn't this be 'encoding'), followed by a colon and space"

The 1st Comment on "Ruby 1.9's Three Default Encodings"

2014-04-18T17:59:08Z

It's probably worth noting that using the default Encoding setters will trigger warnings:

$ ruby -we 'Encoding.default_internal = Encoding.default_external = "UTF-8"'
-e:1: warning: setting Encoding.default_external
-e:1: warning: setting Encoding.default_internal

That makes sense, as it's really too late to set these in code after IO objects may have already been created.

Ruby 1.9's Three Default Encodings

2014-04-18T18:40:50Z

I suspect early contact with the new m17n (multilingualization) engine is going to come to Rubyists in the form of this error message:

invalid multibyte char (US-ASCII)

Ruby 1.8 didn't care what you stuck in a random String literal, but 1.9 is a touch pickier. I think you'll see that the change is for the better, but we do need to spend some time learning to play by Ruby's new rules.

That takes us to the first of Ruby's three default Encodings.

The Source Encoding

In Ruby's new grown up world of all encoded data, each and every String needs an Encoding. That means an Encoding must be selected for a String as soon as it is created. One way that a String can be created is for Ruby to execute some code with a String literal in it, like this:

str = "A new String"

That's a pretty simple String, but what if I use a literal like the following instead?

str = "Résumé"

What Encoding is that in? That fundamental question is probably the main reason we all struggle a bit with character encodings. You can't tell just from looking at that data what Encoding it is in. Now, if I showed you the bytes you may be able to make an educated guess, but the data just isn't wearing an Encoding name tag.

That's true of a frightening lot of data we deal with every day. A plain text file doesn't generally say what Encoding the data inside is in. When you think about that, it's a miracle we can successfully read a lot of things.

When we're talking about program code, the problem gets worse. I may want to write my code in UTF-8, but some Japanese programmer may want to write his code in Shift JIS. Ruby should support that and, in fact, 1.9 does. Let's complicate things a bit more though: imagine that I bundle up that UTF-8 code I wrote in a gem and the Japanese programmer later uses it to help with his Shift JIS code. How do we make that work seamlessly?

The Ruby 1.8 strategy of one global variable won't survive a test like this, so it was time to switch strategies. Ruby 1.9's answer to this problem is the source Encoding.

All Ruby source code now has some Encoding. When you create a String literal in your code, it is assigned the Encoding of your source. That simple rule solves all the problems I just described pretty nicely. As long my source Encoding is UTF-8 and the Japanese programmer's source Encoding is Shift JIS, my literals will work as I expect and his will work as he expects. Obviously if we share any data, we will need to establish some rules about our shared formats using documentation or code that can adapt to different Encodings, but we should have been doing that all along anyway.

Thus the only question becomes, what's my source Encoding and how do I change it?

There are a few different ways Ruby can select a source Encoding. Here are the options:

$ cat no_encoding.rb 
p __ENCODING__
$ ruby no_encoding.rb 
#<Encoding:US-ASCII>

$ cat magic_comment.rb 
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment.rb 
#<Encoding:UTF-8>
$ cat magic_comment2.rb 
#!/usr/bin/env ruby -w
# encoding: UTF-8
p __ENCODING__
$ ruby magic_comment2.rb 
#<Encoding:UTF-8>

$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'p __ENCODING__'
#<Encoding:UTF-8>

$ ruby -KU no_encoding.rb 
#<Encoding:UTF-8>

The first example shows us two important things. The first is the main rule of source Encodings: source files receive a US-ASCII Encoding, unless you say otherwise. [Update: this was changed to UTF-8 in Ruby 2.0 and up.] This is where I expect programmers to run into the error I mentioned earlier. If you place any non-ASCII content in a String literal without changing the source Encoding, Ruby will die with that error. Thus you need to change the source Encoding to work with any non-ASCII data. The second thing we see here is the new __ENCODING__ keyword that can be used to get the source Encoding that's active where it is executed.

The second example shows the preferred way to set your source Encoding and it's called a magic comment. If the first line of your code is a comment that includes the word coding, followed by a colon and space, and then an Encoding name, the source Encoding for that file is changed to the indicated Encoding. If your code has a shebang line, the magic comment must come on the second line, with no spacing between them. Once set, all String literals you create in that file will have that Encoding attached to them.

The third example shows an exception to the rule for your convenience. When you feed Ruby some code on the command-line using the -e switch, it gets a source Encoding from your environment. I have UTF-8 set in the LC_CTYPE environment variable, but some people also use the LANG variable for this. This makes scripting easier since Ruby will (hopefully) match the Encoding of any other commands you chain together.

The fourth example is another interesting exception to the rule. Ruby 1.9 still supports the -K* style switches from Ruby 1.8 including the -KU switch I've recommended so heavily in this series. These switches have a couple of effects, but of particular note they are the only non-magic comment way to modify the source Encoding. This is good news for backwards compatibility, because some Ruby 1.8 code may be able to run on Ruby 1.9 without Encoding problems thanks to this. I must stress that this is just for backwards compatibility though, and magic comments are the future.

With magic comments the code will include its Encoding data. It will probably seem a little tedious to add them to all your source files at first, but it's really not that big of a change. In the past, I've recommended we stick the following shebang line at the top of our files:

#!/usr/bin/env ruby -wKU

Now, for Ruby 1.9, I'm recommending we switch to something like this:

#!/usr/bin/env ruby -w
# encoding: UTF-8

Note that the magic comment format rules are pretty loose and all of following examples would work the same:

# encoding: UTF-8

# coding: UTF-8

# -*- coding: UTF-8 -*-

This is nice for support in some text editors that also read such comments.

If we all get into that habit of adding magic comments, our code can work together regardless of the various Encodings we personally favor. Ruby will know how to handle each separate file. As an added bonus, we programmers also get to see these comments and know more about the code we are working with. That makes it a good habit to get into, I think.

The Default External and Internal Encodings

There's another way Strings are commonly created and that's by reading from some IO object. It doesn't make sense to give those Strings the source Encoding because the external data doesn't have to be related to your source code. Also, you really need to know how data is encoded to read it correctly. Even a simple concept like reading the next line of data changes if you are talking about UTF-8 or UTF-16LE (the LE stands for a Little Endian byte order) data. Thus, it makes sense for IO objects to have at least one Encoding attached to them. Ruby 1.9 is generous and gives them two: the external Encoding and the internal Encoding.

The external Encoding is the Encoding the data is in inside the IO object. That affects how data will be read and this is the Encoding data will be returned in as long as the internal Encoding isn't set (more on that in a bit). Let's look at an example of how this plays out in practice:

$ cat show_external.rb 
open(__FILE__, "r:UTF-8") do |file|
  puts file.external_encoding.name
  p    file.internal_encoding
  file.each do |line|
    p [line.encoding.name, line]
  end
end
$ ruby show_external.rb 
UTF-8
nil
["UTF-8", "open(__FILE__, \"r:UTF-8\") do |file|\n"]
["UTF-8", "  puts file.external_encoding.name\n"]
["UTF-8", "  p    file.internal_encoding\n"]
["UTF-8", "  file.each do |line|\n"]
["UTF-8", "    p [line.encoding.name, line]\n"]
["UTF-8", "  end\n"]
["UTF-8", "end\n"]

There are four things to notice in this example:

I set the external Encoding by tacking :UTF-8 onto the end of my mode String when I opened the File
You can use external_encoding() to check the external Encoding as I have here
internal_encoding() works the same for the internal Encoding, which will be nil unless you explicitly set it
Note how each String created as I read the data is given the external_encoding()

The internal Encoding just adds one more twist. When set, data will still be read in the external Encoding, but transcoded to the internal Encoding as the String is created. It's a convenience for you as the programmer. Watch how that changes things:

$ cat show_internal.rb 
open(__FILE__, "r:UTF-8:UTF-16LE") do |file|
  puts file.external_encoding.name
  puts file.internal_encoding.name
  file.each do |line|
    p [line.encoding.name, line[0..3]]
  end
end
$ ruby show_internal.rb 
UTF-8
UTF-16LE
["UTF-16LE", "o\x00p\x00e\x00n\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00p\x00u\x00"]
["UTF-16LE", " \x00 \x00f\x00i\x00"]
["UTF-16LE", " \x00 \x00 \x00 \x00"]
["UTF-16LE", " \x00 \x00e\x00n\x00"]
["UTF-16LE", "e\x00n\x00d\x00\n\x00"]

There are a couple differences here:

A second added Encoding on the mode String (my :UTF-16LE in this example) sets the internal_encoding() as I show with the second puts()
This little change gets Ruby to translate all of the data for me (I just shortened the output because UTF-16LE is noisy)

The external Encoding works the same when writing. It still represents the Encoding in the IO object, or the Encoding data is going to. However, you don't need to specify an internal Encoding when writing. Ruby will automatically use the Encoding of a String you output as the internal Encoding and transcode as needed to reach the external Encoding. For example:

$ cat write_internal.rb 
# encoding: UTF-8
open("data.txt", "w:UTF-16LE") do |file|
  puts file.external_encoding.name
  p    file.internal_encoding
  data = "My data…"
  p [data.encoding.name, data]
  file << data
end
p File.read("data.txt")
$ ruby write_internal.rb 
UTF-16LE
nil
["UTF-8", "My data…"]
"M\x00y\x00 \x00d\x00a\x00t\x00a\x00& "

Note how my data was transcoded before it was written even though the internal_encoding() was nil. Ruby used the String's Encoding to decide what was needed.

Both of those IO Encodings should be pretty straight forward. The only question left about them is: what happens if you don't set them? The answer is that the IO inherits the default external Encoding and/or the default internal Encoding whenever one isn't set. Now we need to know how Ruby chooses those defaults.

The default external Encoding is pulled from your environment, much like the source Encoding is for code given on the command-line. Have a look:

$ echo $LC_CTYPE
en_US.UTF-8
$ ruby -e 'puts Encoding.default_external.name'
UTF-8
$ LC_CTYPE=ja_JP.sjis ruby -e 'puts Encoding.default_external.name'
Shift_JIS

The default internal Encoding is simply nil. You must actively change it to get anything else.

Both default IO Encodings have a global setter: Encoding.default_external=() and Encoding.default_internal=(). You can set them to an Encoding object or just the String name of an Encoding.

You can also change these default Encodings using some command-line switches. The new -E switch can be used to set one or both of the IO Encodings:

$ ruby -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, nil]
$ ruby -E Shift_JIS \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, nil]
$ ruby -E :UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:UTF-8>, #<Encoding:UTF-16LE>]
$ ruby -E Shift_JIS:UTF-16LE \
> -e 'p [Encoding.default_external, Encoding.default_internal]'
[#<Encoding:Shift_JIS>, #<Encoding:UTF-16LE>]

As you can see, the argument for this switch is just like what you would append to a mode String in a call to File.open().

There's one more command-line switch shortcut for those of us who prefer to just use UTF-8 everywhere. The new -U switch sets Encoding.default_internal() to UTF-8. Using that, you can just set the external Encoding for your IO objects, or let it default from your environment, and all Strings you read will be transcoded to the preferred UTF-8.

Probably the most important thing to note about Encoding.default_external() and Encoding.default_internal() is that you should really just treat them as shortcuts for your own scripting. Pulling Encodings from the environment or command-line switches can be handy when you're in control of where the code runs, but you're going to need to be more explicit for code you intend for others to run. When in doubt, set the external and internal Encodings the way you want them for each IO object. It's a bit more tedious, but also safer in that it won't mysteriously be changed by some outside force. Also remember that the defaults are global settings affecting all loaded code, including any libraries you require(). That can be a boon or bane, so just remember to factor it into your thinking when you're wondering, "Where does this String get its Encoding from?"