Gray Soft / Tags / Hidden Features

Rich Methods

2015-05-23T05:18:20Z

Some APIs provide collections of dirt simple methods that just do one little thing.

This approach in less common in Ruby though, especially in the core and standard library of the language itself. Ruby often gives us rich methods with lots of switches we can toggle and half hidden behaviors.

Let's look at some examples of what I am talking about.

Get a Line at a Time

I suspect most Rubyists have used gets() to read lines of input from some kind of IO. Here's the basic usage:

>> require "stringio"
=> true
>> f = StringIO.new(<<END_STR)
<xml>
  <tags>Content</tags>
</xml>
END_STR
=> #<StringIO:0x007fd5a264fa08>
>> f.gets
=> "<xml>\n"
>> f.gets
=> "  <tags>Content</tags>\n"

I didn't want to mess with external files for these trivial examples, so I just loaded StringIO from the standard library. It allows us to wrap a simple String (defined in this example using the heredoc syntax) in the IO interface. In other words, I'm calling gets() here for a String just as I could with a File or $stdin.

As the last two calls show, gets() reads until it finds a "\n" and then returns the content read. Actually, that's what it does by default, but you can tell gets() what character to read to, if you prefer:

>> f.rewind
=> 0
>> f.gets(">")
=> "<xml>"
>> f.gets(">")
=> "\n  <tags>"
>> f.gets(">")
=> "Content</tags>"

When you're working with XML documents, newlines don't really mean much. You don't actually care where they are. What you do care about are tags. Reading from tag to tag is like reading one of those great books that skip the boring bits to give you interesting scene after interesting scene.

As you can see above, one tiny change to the gets() call, specifying the character to read to as the tag ending ">", can make this happen.

"But wait, there's more!"

>> f = StringIO.new("One\n\nTwo\n\nThree")
=> #<StringIO:0x007fd5a260efa8>
>> f.gets("")
=> "One\n\n"
>> f.gets("")
=> "Two\n\n"

The empty String ("") is a magic value for the character to read to, since it makes no sense as that value. This turns on paragraph mode and in that mode Ruby will read one paragraph at a time. For this purpose a paragraphs are defined as being separated by two consecutive newlines (or a blank line in word processor terms).

These aren't even all the features of gets(). It can do more. For example, you can provide an upper limit of bytes to read, to prevent wonky input from forcing your program to allocate the tons of memory to hold large Ruby String objects.

Let's look at another method.

`Hash` Merging

Many Ruby methods sneak their rich functionality in through the use of blocks. Deferring some decision to the caller by allowing them to provide custom code for handling it makes some methods crazy flexible.

To show what I mean, let's play with good old merge():

>> {a: 1, b: 2}.merge(c: 3, d: 4)
=> {:a=>1, :b=>2, :c=>3, :d=>4}

Most Rubyists run into examples like this pretty early in their studies. The code just returns a fresh Hash containing the keys and values of both the receiver and the Hash passed as an argument to merge().

How are ties handled?

>> {a: 1, b: 2}.merge(b: :two, c: 3)
=> {:a=>1, :b=>:two, :c=>3}

The Hash passed as an argument to merge() wins. Again, I doubt this is much of a surprise to anyone.

However, I don't think everyone knows that you can take control of this merging process. During a merge() any conflict will be passed to a block, if provided, and the block can return what to store in the new Hash:

>> {a: 1, b: 2}.merge(b: :two, c: 3) { |_, old, new| Array(old) + Array(new) }
=> {:a=>1, :b=>[2, :two], :c=>3}

You can throw away either item, log the conflict, combine them as I have done here, or do whatever else you can think of, all because merge() takes a block.

Can you guess how ActiveSupport implements reverse_merge!() now?

Easy Tokenizing

Let's do one last method with a rich interface (even though Ruby has many more):

>> "1,2,3".split(",")
=> ["1", "2", "3"]

This is another very common method. It turns a String into an Array by dividing up the contents everywhere the passed separator is encountered. I used a String separator above but a Regexp is also allowed:

>> "1, 2, 3".split(/\s*,\s*/)
=> ["1", "2", "3"]

This makes it easier to handle complex separators. For example, the Regexp above permits optional whitespace characters on either side of the comma.

But a Regexp can include capture groups. How are they handled?

>> "1, 2, 3".split(/\s*(,)\s*/)
=> ["1", ",", "2", ",", "3"]

Easy enough: the captured value(s) are returned with the separated contents.

The real question this raises for me is, "What the heck is this feature good for?" Well, one thing I have found over the years is that this usage of split() can make dividing some input into tokens pretty darn easy:

>> "<xml><tags>Content</tags></xml>".split(/(<[^>]+>)/)
=> ["", "<xml>", "", "<tags>", "Content", "</tags>", "", "</xml>"]

You can use this one feature as a backbone for a moderately complex parser. Mote does just that.

I made a video explaining how this parsing trick (and more) are accomplished in detail. You can use the coupon BLOGREADER for $3 off if you want to check it out.

Random Access Terminal

2017-07-18T21:40:47Z

I've recently been playing around with fancy terminal output in Ruby. I've learned quite a bit about this arcane magic. I've also realized that the documentation is pretty spotty. I want to see if I can improve that with a few blog posts, so let's dive right in.

Output

Program output typically happens in a linear order from top to bottom. For example, this code:

puts "onez"
puts "twos"
puts "threes"

generates this output:

onez
twos
threes

But what if you need to change some output? Could you replace the z above with an s if you needed to? Yes, but it can get a little involved.

ANSI Escape Codes

In many cases, we just push some characters to $stdout (the stream Kernel#puts is writing to above) and your terminal program happily shows them to the user. However, your terminal is watching these characters for special sequences that it understands. Some of those sequences of characters can cause your terminal to take actions other than just writing some output to the screen.

For example, there are sequences that move the cursor (the point where output is written) to different locations. Using those special codes, we could change the z in our example:

CSI = "\e["

$stdout.puts "onez"
$stdout.puts "twos"
$stdout.puts "threes"

$stdout.write "#{CSI}3A"  # move up three lines
$stdout.write "#{CSI}3C"  # move right three characters

$stdout.write "s"         # overwrite the 'z'

$stdout.write "#{CSI}3B"  # move down three lines (after our output)
$stdout.write "#{CSI}4D"  # move back four characters (to the start of the line)

If you run that code, it will initially produce the output from the previous example. It will then move the cursor to the z, overwrite it with an s, and return the cursor to where it was. This likely happens too fast for you to see the change, but you can insert sleep() calls if you want to watch it work.

All of the escapes used in this example are commands used to move the cursor around by a specific number of lines or characters in an indicated direction. Note that I switched to using the write() method when I started using escapes. puts() would have printed my escape followed by a newline (moving me away from where I wanted to be).

I used relative positioning commands above, meaning that I moved from where I knew the cursor was to where I wanted it be. I couldn't move to some absolute coordinates, because I don't know what else is on your screen when the program is run. You probably have at least a command prompt above the output and I have no idea how big that is. Now, if I clear the screen before I generate output, I could go on to treat it like a known grid of coordinates:

CSI = "\e["

$stdout.write "#{CSI}2J"    # clear screen
$stdout.write "#{CSI}1;1H"  # move to top left corner

$stdout.puts "onez"
$stdout.puts "twos"
$stdout.puts "threes"

$stdout.write "#{CSI}s"     # save cursor position

$stdout.write "#{CSI}1;4H"  # move to line 1, character 4
$stdout.write "s"           # overwrite the 'z'

$stdout.write "#{CSI}u"     # restore cursor position

This is the same trick, but using absolute positioning and some other fancy codes to save and restore the cursor's position. The movement escape used here, CSIy;xH, allows us to jump directly to any position on the screen. The top left corner is 1;1 with y counting lines down and x counting characters to the right.

Clearing the screen can be a little tricky. First, you can clear part or all of it. I chose all here with the 2 in CSI2J. You also need to remember though that clearing the screen doesn't usually change the position of the cursor. That's why I sent it back to the top left corner after the clear.

This code was pretty eye opening to me. It helped me to realize that a terminal doesn't have to be treated as a sequence of lines. It can alternately be treated like a drawing canvas with coordinates used to represent each spot that a character can be placed.

WARNING: terminals vary in exactly which codes they support. Most of the examples I'm showing will probably work on most UNIX-like terminals. I've tested them on my own UNIX box, but even there I found minor differences. For example, the code above works as expected in my normal terminal, but the terminal emulation mode I often use in Emacs doesn't seem to honor the save and restore cursor codes. It can be quite a bit of work to properly support the various kinds of terminals you can encounter.

Colors

It probably goes without saying that there are codes to change the color of your output and do other fancy graphic tricks. These can get slightly complex because they require a little math to indicate choice of foreground or background colors. They can also involve multiple codes at once when you want to select the bright version of a color. Here's a pretty trivial example:

CSI    = "\e["
COLORS = {
  black:   0,
  red:     1,
  green:   2,
  yellow:  3,
  blue:    4,
  magenta: 5,
  cyan:    6,
  white:   7
}

def color(content, foreground, background = nil)
  colored  = color_name_to_escape_code(foreground, 30)
  colored << color_name_to_escape_code(background, 40) if background
  colored << content
  colored << "#{CSI}0m"
end

def color_name_to_escape_code(name, layer)
  short_name = name.to_s.sub(/\Abright_/, "")
  color      = COLORS.fetch(short_name.to_sym)
  escape     = "#{CSI}#{layer + color}"
  escape    << ";1" if short_name.size < name.size
  escape    << "m"
  escape
end

[ ["C", :red],
  ["O", :bright_red],
  ["L", :bright_yellow],
  ["O", :bright_green],
  ["R", :bright_blue],
  ["S", :blue],
  ["!", :bright_magenta] ].each do |char, color|
  $stdout.write color(char, color, :white)
end
$stdout.puts

If you run that, you should see not-quite-a-rainbow of colored letters on white, regardless of the default color settings in your terminal.

It's pretty important to remember to issue a reset code (CSI0m) after playing with graphic settings like this. If your program exits without doing so, the terminal could be left in some pretty garish states. You can see that my color() method sets some colors, writes the desired content, then immediately resets. This process minimizes the chances of me bleeding over graphics changes into other output.

Visit that link I gave earlier on ANSI escape codes for specifics about any of the sequences I've used here, a good description of what the CSI is, and more.

The Status Line Trick

While ANSI codes allow you to do pretty much any output trick you can dream up with enough effort, there's another trick that just so easy it's worth knowing for the times when it can save you some work.

We usually end lines of output with a newline character (\n). Kernel#puts does this for us automatically. The exact effect of that character depends on the mode your terminal is currently in. We'll cover that more below, but there's another way to end lines. You could use the carriage return (\r) character. (In truth, using both characters is also an option. Again, I'll go into that below.)

A \r does move the cursor back to the beginning of the line, but it does not advance to the next line. This gives you a simple way to overwrite the content you previously placed on the last line only. That can come in handy for updating a status line or progress bar.

Here's a contrived example that just shows progress while it pretends to do some work:

chunks = 0

$stdout.puts "Time left:"
loop do
  $stdout.write "|#{'#' * chunks}#{' ' * (10 - chunks)}|\r"

  if chunks == 10
    $stdout.puts
    break
  end

  next_chunk = rand(1..(10 - chunks))
  sleep next_chunk
  chunks += next_chunk
end

Sample output from partway through a run looks like this:

Time left:
|#####     |

You may need to run it to appreciate the result, but the last line just keeps updating with the bar of #'s, growing each time. Notice that there is a final puts() when we exit the loop(), so the new command prompt won't erase the progress bar. You may or may not wish to do that in your own programs.

Input

I'll be honest and admit that I covered output first because it's easier. Buckle up and we'll see if can survive the jump into input.

I've mentioned before that programmers often ask how to read a single character from the keyboard and it can get tricky. This time I'm going to try to explain what's really going on without hiding behind a library that simplifies the process. I'll also show you some fancy new ways to accomplish the task.

Terminal Modes

Your terminal can operate in a variety of modes. The default cooked mode has several effects. One of those effects is that input is buffered until you press the return key. This buffering allows you to edit content you are entering—using arrow keys, backspace/delete keys, etc.—without all programs needing to handle all of those special cases. That's a great feature, right up to the point where you just want to read one character.

Watch what happens when I try:

loop do
  char = $stdin.getc
  puts "You typed:  #{char.inspect}"
  break if char == ?q
end

Here's some output from a run of that program:

a
You typed:  "a"
You typed:  "\n"
q
You typed:  "q"

What you need to catch here is that I pushed an a, but my program wouldn't acknowledge it, until I pressed return too. Then it read both characters. The q was treated the same way, the program just ended before showing the second newline.

If we really want to do this right, we've got to get out of cooked mode (or at least disable its line buffering feature). In older Ruby code, you had three choices for doing this: shell out to stty (tricky to get right), make very arcane C-style calls to ioctl() (a nightmare), or use a gem that hid the details from you (your best bet, by far). Now you understand my old advice to just use HighLine. But I have good news! Ruby is older and wiser now.

A New Standard Library

The easiest approach to reading just one character is to switch into raw mode (or a similar mode) and try again. However, this change will affect your program's output as well due to other features of cooked mode. Given that, your best bet is to jump into raw mode, grab a character, and return to cooked mode immediately.

Ruby now ships with a standard library that will do exactly that:

require "io/console"

loop do
  char = $stdin.getch
  puts "You typed:  #{char.inspect}"
  break if char == ?q
end

Behold the magic:

You typed:  "a"
You typed:  "q"

There are no newlines this time because the computer is now responding the instant I press a key. Victory. Sort of.

Even this tiny dip of our toes into raw mode has changed other things. You don't see what I typed echoed before the program provides its own output as it was before. This is another feature of cooked mode and you'll need to do your own echoing when reading like this, if you need it.

Here's another change that happens when I push control-C:

You typed:  "\u0003"

As you can see, it no longer interrupts my process. In fact, all of the control sequences have lost their magic powers (control-D to close $stdin and control-Z to suspend the program, for example). You will now need to handle these keystrokes yourself if you want them to have some effect. (Please do support control-C at the very least!)

Terminals have another mode that helps with this control keystroke issue, called cbreak mode. It gives you a near raw mode experience, but with these low-level keystrokes still handled. It makes me very sad to report that io/console doesn't yet support this handy mode.

Still, io/console is great to have. You can read about what it can do in the API documentation and its README. Here's one more simple example of how easy io/console can make things, like finding the screen size:

require "io/console"

rows, cols = $stdout.winsize
puts "Columns:  #{cols}, Rows:  #{rows}"

Full Raw Mode

Dipping into raw mode to grab characters is great when it works for your needs, but the code I've showed so far has a drawback. io/console's getch() is a blocking method. It will wait for input. This is a problem if you need to manage your own event loop, take user input when it's available, but continue making things happen when it's not.

If there's a way to solve this problem with just io/console, I haven't been able to derive the proper incantation. However, we can pull it off by pulling one more standard library into the mix.

To show how this works, imagine that we want to prompt a user with what to do if they don't take any action for a few seconds. Here's some code:

require "io/console"
require "io/wait"

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = io.ready? && io.sysread(1)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}"
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character."
        prompted = true
      end
      sleep 0.1
    end
  end
end

This time I used io/console to put the whole program into raw mode. Cooked mode is restored when the block passed to raw() exits.

The other library I used is io/wait. It adds a ready?() method to $stdin that will return true when there's content waiting to be read. I pull that content out a byte at a time using the low-level sysread() to avoid any buffering.

This kind of works, but we've picked up a new issue:

Please type a character.
                        You typed:  "a"
                                       You typed:  "q"

See how my output is now staggered?

Yet another feature of cooked mode is that newlines in your program's output are translated to a carriage return newline sequence (\r\n). Rubyists are used to puts()ing out some content, with a newline added, and seeing the cursor move to the beginning of the next line. But a newline doesn't really do that by itself.

The newline character means move down a line. As we saw earlier, a carriage return sends you to the beginning of the current line. So it's really the combined effect of both that we're used to being normal.

My output above has only the newline effect, because we're in raw mode and the translation to a two character sequence isn't active. That explains the problem and gives us a way around it. In raw mode, you'll need to manually end lines with \r\n when you want to move to the beginning of the next line.

Here's a fixed version of the code above:

require "io/console"
require "io/wait"

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = io.ready? && io.sysread(1)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}\r\n"  # added \r\n
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character.\r\n"   # added \r\n
        prompted = true
      end
      sleep 0.1
    end
  end
end

The comments show the two minor changes to the code.

And now the output looks more typical:

Please type a character.
You typed:  "a"
You typed:  "q"

We've handled a lot of cases, but there are still issues with a raw approach like this. I'll show you one more. Watch what happens when I press an arrow key while running our latest program:

You typed:  "\e"
You typed:  "["
You typed:  "A"

Ugh. That's all one keystroke, but we pulled it out as separate bytes. This means we wouldn't be able to have conditionals in our program check for this keypress.

Can we fix it? Sure. We can add some more code to detect these escape sequences and combine them into a single String. Here's what that might look like:

require "io/console"
require "io/wait"

CSI = "\e["

def get_char_or_sequence(io)
  if io.ready?
    result = io.sysread(1)
    while ( CSI.start_with?(result)                        ||
            ( result.start_with?(CSI)                      &&
              !result.codepoints[-1].between?(64, 126) ) ) &&
          (next_char = get_char_or_sequence(io))
      result << next_char
    end
    result
  end
end

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = get_char_or_sequence(io)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}\r\n"
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character.\r\n"
        prompted = true
      end
      sleep 0.1
    end
  end
end

All of the new code here is in the get_char_or_sequence() method. The while loop in there checks to see if a read character looks like the start of a CSI sequence and keeps reading, assuming there's more input, until we've seen the full CSI. Then, if what we've read starts with a CSI and there's still more input, we keep reading until we see a termination character for the sequence.

The end result? We can now treat arrow keys as a single unit (as long as our terminal returns them as ANSI escape sequences with a leading CSI):

You typed:  "\e[A"

That's progress. There's definitely more we could handle though, like non-CSI escape sequences and larger Unicode characters. This rabbit hole goes pretty deep. You'll need to decide how many such cases you need to handle for your programs needs.

If you have to deal with many of the various topics covered by this post, you may want to consider using a gem that can handle some of them for you. They can really smooth over some of these edge cases. The details of that are another blog post, but my hope is that this article starts to at least get you familiar with what's involved in fancy terminal I/O.

The Three Tick Sort

2014-11-11T19:21:30Z

Yesterday I showed a newer programmer some code like scores.sort_by(&:reverse). This provoked a comment about how they where going to look up sort_by() later to figure out what magic is involved here. It made me sad to realize how many cool tricks they weren't going to see in that bit of documentation.

Allow me to enumerate those tricks for you, but first let's flesh out an example. Consider this code:

scores = {
  fifteen:         2,
  five_card_run:   5,
  five_card_flush: 5,
  four_card_run:   4,
  four_card_flush: 4,
  his_nobs:        1,
  pair:            2,
  three_card_run:  3,
}
scores.sort_by(&:reverse).each do |name, score|
  puts "Score #{score} for #{name}."
end
# >> Score 1 for his_nobs.
# >> Score 2 for fifteen.
# >> Score 2 for pair.
# >> Score 3 for three_card_run.
# >> Score 4 for four_card_flush.
# >> Score 4 for four_card_run.
# >> Score 5 for five_card_flush.
# >> Score 5 for five_card_run.

In this case, the magic method call (scores.sort_by(&:reverse)) has reordered a list of Cribbage hands first by point value and then alphabetically ("ASCIIabetically" in truth). How this happens is a pretty interesting journey though.

Let's take it a piece at a time.

First you need to know that Hash#each iterates over pairs. (All other iterators, like sort_by(), use each() under the hood, remember.) A pair is a two element Array containing key then value. If your block takes one argument, you'll get this pair Array. If it takes more, Ruby will extract the values out of the Array due to its "multiple assignment" rules. In other words, you can do this:

hash.each do |key, value|
  # ...
end

or this:

hash.each do |pair|
  # ...
end

This means I could have used sort_by { |key, value| [value, key] }, but I used the Array interface instead: sort_by { |pair| pair.reverse }.

That leads us to the second part of this magic trick: you can sort_by() an Array? Yes, you can.

sort_by() works with any object that defines the spaceship operator (<=>), which Array does:

>> [0] <=> [0]
=> 0
>> [1] <=> [2]
=> -1
>> [2] <=> [1]
=> 1

As you probably work out from the code above, Array objects compare themselves based on their contents. Later elements are used to break ties from earlier comparisons:

>> [0, 1] <=> [0, 2]
=> -1
>> [0, 2] <=> [0, 1]
=> 1

By calling reverse() on the pairs in the sorting Array, I put the score into the primary sorting role and used the hand name as a tie breaker.

That pretty much explains what the block I passed to the iterator did, so there's just one problem left: I didn't actually use a block!

When you have a block of the form method { |object| object.thing_i_care_about } you can rewrite it as method(&:thing_i_care_about). Again we need to break this trick down into pieces to understand it.

First, :thing_i_care_about is a Symbol:

>> :thing_i_care_about
=> :thing_i_care_about
>> :thing_i_care_about.class
=> Symbol

Next, Ruby will try to convert the final argument to a method into a Proc object, if it's proceeded by a &:

>> def m(&block) block end
=> :m
>> m(&Object.new)
TypeError: wrong argument type Object (expected Proc)
    from (irb):4
    from /Users/jeg2/.rubies/ruby-2.1.3/bin/irb:11:in `<main>'

Then you have to know that any object can choose to play ball as a Proc by defining a to_proc() method:

>> o = Object.new
=> #<Object:0x007fc5841fb278>
>> def o.to_proc; proc { puts "Howdy." } end
=> :to_proc
>> m(&o)
=> #<Proc:0x007fc5841e6d28@(irb):8>

Now, you can probably guess that Symbol defines to_proc. Watch what it does:

>> symbol_to_proc = :to_s.to_proc
=> #<Proc:0x007fc5841e2750>
>> symbol_to_proc[42]
=> "42"
>> symbol_to_proc[Object.new]
=> "#<Object:0x007fc5841c1a50>"

See how it just calls the method named after the Symbol? (It's essentially using send() under the hood.)

Combine all of those bits together and you can do some pretty fancy sorting with very little code.

Delaying Decisions

2014-04-29T03:29:32Z

I love playing with Ruby's Hash. I think it has a neat API and experimenting with it can actually help you understand how to write good Ruby. Let's dig into this idea to see what I mean.

The nil Problem

In Destroy All Software #9 Gary chooses to show an example in Python because, unlike Ruby's Hash, it will raise an error for a non-existent key. Ruby just returns nil, he explains.

What Gary said isn't really true, but I'm guessing he just didn't know that at the time. He was in the process of switching to Ruby from Python and I'm guessing he just didn't have a deep enough understanding of Ruby's Hash yet. I bet he does know how it works now.

But assume he was right. What's he saying and why does it matter? Consider some code like this:

class SearchesController < ApplicationController
  def show
    terms = params[:terms]
    SomeModel.search(terms)
    # ...
  end
end

This is what Gary doesn't like, and rightfully so. Because I indexed into params here with the []() method, I will indeed get a nil if the :terms key wasn't in params.

The problem with nil is that it's a ticking time bomb just waiting to go off. It has a very limited interface by design, so calling methods on it may raise errors. It's probably not the interface your code is expecting. For example, in the code above I likely intended to get a String in terms, but if I got nil instead and I later try to call String methods on it, it's going to blow up.

This gets worse because nil values tend to bounce around in the code a bit. This means the error may finally manifest far from the line where I actually assigned terms, which is the real problem that needs fixing. For example, let's say the model code I handed off to looks something like this:

class SomeModel < ActiveRecord::Base
  def self.search(terms)
    words = terms.scan(/\S+/)
    # ...
  end
end

This is where the error would show up. The call to scan() would toss a NoMethodError since nil doesn't have scan(). The stack trace would lead us here, but, as I said before, this isn't what needs fixing.

Now, we could fix it here. One way to do that would be to insert a conversion before the scan() call:

words = terms.to_s.scan(/\S+/)

That's probably not a good idea though. We've now said that calling SomeModel.search(nil) is OK and that doesn't really make much sense.

A slightly better fix would be to add a guard before the call to scan()

fail ArgumentError, "You must pass search terms" if terms.nil?
words = terms.scan(/\S+/)

This would get us a better error message that actually tells us what went wrong. But the stack trace still wouldn't be ideal. It's going to bring us here first even though the real issue is in the controller.

My Favorite Method to the Rescue

This whole problem started because I allowed the Hash to give me a nil. I didn't want a nil, so I shouldn't have said it was OK.

That brings us to fetch(). Watch this:

>> hash = { }
=> {}
>> hash[:missing]
=> nil
>> hash.fetch(:missing)
KeyError: key not found: :missing
from (pry):3:in `fetch'

You can think of []() as saying give me some key, if it exists. You get a nil if it doesn't. Now fetch(), on the other hand, is more like saying I need this key, no substitutions allowed. This forces fetch() to raise a KeyError when the key is missing, since nil isn't allowed.

The best fix to the problem I started with would be to replace the initial assignment in the controller with this:

terms = params.fetch(:terms)

You may also want to add some code to handle the KeyError, but the important thing is that the problem now triggers from the proper place. This is the most helpful error we could work with in this scenario.

I Don't Know What You Need

Let's take a step back for a moment to the overall Hash. What exactly is a Hash in your code? The only right answer is that I have no idea.

You might be creating a typical key-to-value mapping:

>> name = {first: "James", last: "Gray"}
=> {:first=>"James", :last=>"Gray"}
>> name[:last]
=> "Gray"

Alternately, you could be tracking some counts:

>> counts = Hash.new(0)
=> {}
>> counts[:one] += 1
=> 1
>> 3.times do counts[:three] += 1 end
=> 3
>> counts.values_at(:zero, :one, :three)
=> [0, 1, 3]

Or you could be keeping some named buckets for data:

>> buckets = Hash.new { |hash, key| hash[key] = [ ] }
=> {}
>> buckets[:one] << 1
=> [1]
>> buckets[:three] << 1 << 2 << 3
=> [1, 2, 3]
>> buckets
=> {:one=>[1], :three=>[1, 2, 3]}

Perhaps it's even a deeply nested structure that you wish to track:

>> tree = Hash.new { |hash, key| hash[key] = Hash.new(&hash.default_proc) }
=> {}
>> tree[:deeply][:nested][:structure] = 42
=> 42
>> tree[:deeply][:nested][:branch] = 1
=> 1
>> tree
=> {:deeply=>{:nested=>{:structure=>42, :branch=>1}}}

These are all very different, but valid, uses of a Hash in Ruby to model various data structures and there are other options too. Ruby can't know what you intend to do. But it was wisely designed so that it doesn't have to know.

Note the progression above that makes this possible. You can create a normal Hash. You can pass a default object, like an initialization value for counters. Or you can go all the way to defining the default behavior with custom code inside of a block as I did in the last two examples.

Ruby delays the decision of default behavior so that it can leave that decision to you. You know what you need better than Ruby does.

And it doesn't stop there.

Back to My Favorite

Being able to set some default behavior at Hash construction time is great, but what if I don't know everything I need to know even then? What if I need to delay the decision even more, until key lookup time? What if what I actually need is different at different times? The answer is that fetch() can handle those cases too.

>> hash = { }
=> {}
>> hash.fetch(:missing, 0)
=> 0
>> hash.fetch(:missing) { |key| key.to_s.split("").sort.join }
=> "giimnss"

You can see that fetch() also supports default return values for a missing keys and it will even allow you to provide custom code to handle the situation however you need to.

If you only get one thing out of this article, make it Hash#fetch(). It's a powerhouse tool that enables you to do all kinds of fancy tricks. (It's worth noting that Hash#delete() is similarly flexible. Look it up.) But that's not really why I'm showing you these methods.

The Pattern is the Thing

The real reason to play with these methods is that they represent a great tactic for Ruby programming in general. It's important to remember that you won't know exactly what the users of your code will need. Delay making that decision whenever you can.

In fact, delay it as long as you can, ideally just pushing the decision off on the user when they finally need to make it. They will know what they need better than you do. Let them raise an error, do a conversion, or whatever else makes sense for them.

In a lot of ways, this discussion is about Ruby's blocks. That's the great tool Ruby gives us to allow us to delay these decisions. Consider even a simple method like this:

def do_some_io_operations_for_me
  # ... IO code here...
rescue IOError, Errno::EACCES #, ...
  block_given? ? yield : raise
end

This is a pretty perfect setup, in my opinion. This code tries to push through some IO operations. Those could fail for any number of reasons, so the code intelligently traps the errors that apply. If a users gives a block, they can control what happens on error without even needing to know which errors the IO code could trigger. However, the decision of exactly what to do is left to the caller.

Of course, the biggest limitation on this strategy is that we only get one block in Ruby. Once you delay one decision with it, you won't have it for other purposes. Try not to let that stop you though! Ruby doesn't.

Consider the find() iterator. With find() the block is already tied up for the test, but Ruby wants to let you decide what happens when nothing is found. Returning nil is not enough for those cases, because nil could legitimately be what the block found. Because of that, find() cheats to expose kind of a second block:

>> (0..9).find(-> { fail ArgumentError, "Not found" }) { |n| n >= 10 }
ArgumentError: Not found
from (pry):28:in `block in <main>'

I think these kind of tricks come out a little cleaner with the new "stabby lambda()" (->) syntax that I used above.

Further Investigation

Fully getting the hang of when and where to use blocks in Ruby seems to take almost as long to get a feel for as the rest of the language does. Here are some sources that might shave a little time off of the journey though:

I wrote an article on blocks a long time ago. I think it has aged fairly well and still provides some insight into why blocks exist and what they really are.
Rake is a great example of the delayed decisions I am advocating in this article in many ways. When you think about it, Rakes is really just an executable dependency graph of delayed decisions. For a more concrete example though, check out Rake's FileTask. It's a delayed decision about how to build a file from one or more dependencies any time the content is no longer fresh.
The 2.10.0 release of RSpec brings with it some matchers for yielded control to a block. This can make testing all of those block methods I'm asking you to write a little easier.

The Secret Shell Helper

2014-04-11T22:02:14Z

Someone pops onto the Ruby Talk mailing list fairly regularly asking how to break up content like:

one "two" "a longer three"

They expect to end with a three element Array, where the third item will contain spaces. They generally expect the quotes will have been removed as well.

If your needs are very, very simple you may be able to handle this with a regular expression:

data = 'one "two" "a longer three"'
p data.scan(/"([^"]*)"|(\S+)/).flatten.compact
# >> ["one", "two", "a longer three"]

That just searches for either a set of quotes with some non-quote characters between them or a run of non-whitespace characters. Those are the two possibilities for the fields. Note that the two separate capture here mean scan() will returns contents in the form:

[[nil, "one"], ["two", nil], ["a longer three", nil]]

That's why I added a flatten() and compact() to get down to the actual matches.

The regular expression approach can get pretty complex though if any kind of escaping for quotes is involved. When that happens, you may need to step up to a parser.

One choice for that would be to abuse a CSV parser to get it to divide up the data for you. Here's how you would do that with FasterCSV:

require "rubygems"
require "faster_csv"

data = 'one "two" "a longer three"'
p data.parse_csv(:col_sep => " ")
# >> ["one", "two", "a longer three"]

As you see, replacing the column separator (traditionally a comma) with a simple space gets FasterCSV to break down this data correctly.

This parser will handle escaping, though it's CSV style escaping. That means that quotes will need to be doubled:

require "rubygems"
require "faster_csv"

data = 'simple "embedded ""quote"" characters"'
p data.parse_csv(:col_sep => " ")
# >> ["simple", "embedded \"quote\" characters"]

I doubt that fits the data well too often. I suspect that an escaped quote is more often \" than "". Why is that? Well, data of this type isn't typically CSV data. Which leads us to the natural question, what kind of data are we really working with here?

I'm guessing it's shell data more often that not. Most shells handle quoting like this:

$ ruby -e 'p ARGV' one "two" "a longer three"
["one", "two", "a longer three"]

If that's really the case, we're going to need a shell oriented parser. It's sadly not well known, I assume because it's strangely absent from http://ruby-doc.org/, but Ruby ships with such a parser. The standard Shellwords library will break these down for you:

require "shellwords"

data = "one 'two' 'a longer three'"
p Shellwords.shellwords(data)
# >> ["one", "two", "a longer three"]

If your data really is shell content, you'll be glad to know that Shellwords will handle all the special cases:

require "shellwords"

s = lambda { |shell| p Shellwords.shellwords(shell) }

s[%Q{"escaped \\"quote\\" characters"}]
s[%Q{escaped\\ spaces}]
s[%Q{'back to'" back quoting"}]
# >> ["escaped \"quote\" characters"]
# >> ["escaped spaces"]
# >> ["back to back quoting"]

Shellwords has some new features in Ruby 1.9 as well:

shellsplit() is added as an alias for shellwords()
shellescape() was added to escape a String for use in Bash
shelljoin() was added to escape and join an Array of arguments
shellsplit() and shellescape() are added to String and shelljoin() is added to Array for easier access

Hopefully this helps you find the right parser for your data.

All About Struct

2014-04-11T19:49:24Z

I build small little data classes all the time and there's a reason for that: Ruby makes it trivial to do so. That's a big win because we all know that what is a trivial data class today will be tomorrow's super object, right? If I start out using a simple Array or Hash, I'll probably end up redoing most of the logic at both ends eventually. Or I can start with the trivial class and grow it naturally.

The key to all this though is that I don't write those classes myself! That's what Ruby is for. More specifically, you need to learn to love Struct. Allow me to show you what I mean.

Imagine I need a basic class to represent a Contact. Ruby gives us so many shortcuts that the class could be very small even without Struct:

class Contact
  def initialize(first, last, email)
    @first = first
    @last  = last
    @email = email
  end

  attr_accessor :first, :last, :email
end

You could shorten that up more with some multiple assignment if you like, but that's the basics. Now using Struct is even easier:

Contact = Struct.new(:first, :last, :email)

To be fair, that's not 100% the same as the previous code. My original class required that three arguments get passed to the constructor whereas Struct is more lenient:

p Contact.new(*%w[James Gray james@grayproductions.net])
# >> #<struct Contact first="James",
#                     last="Gray",
#                     email="james@grayproductions.net">
p Contact.new(*%w[James Gray])
# >> #<struct Contact first="James", last="Gray", email=nil>
p Contact.new("James")
# >> #<struct Contact first="James", last=nil, email=nil>
p Contact.new
# >> #<struct Contact first=nil, last=nil, email=nil>

As you can see, all arguments to the constructor Struct builds for you are optional. It will just fill in the passed values, from left to right. This may or may not be an advantage for your needs.

Now lets look once more at how I built that Struct:

ClassName = Struct.new(...)

Struct::new() builds and returns Class objects. If you then assign that to a constant, you can pretty much treat it like any other class you build. Ruby usually just handles the constant assignment for you.

You can forgo the assignment to a constant if you pass the first argument as a constant name in String form:

Struct.new("Contact", :first, :last, :email)

This would not define a top-level Contact, but instead a Struct::Contact. Given that, your name must be unique among all Structs defined when using this approach.

Getting back to our fledgling Contact, it's important to note that it does have all the getter and setter methods for the attributes:

c = Contact.new
p c
# >> #<struct Contact first=nil, last=nil, email=nil>
c.first = "James"
c.last  = "Gray"
p c
# >> #<struct Contact first="James", last="Gray", email=nil>
p c.last
# >> "Gray"

Again, Struct always defines both getter and setter for all attributes. That may or may not work for you.

You can also set values using a Hash like syntax:

c[:email] = "james@grayproductions.net"
p c["email"]
# >> "james@grayproductions.net"

String and Symbol keys are interchangeable.

Another awesome feature is that Struct gives you other ways to go through this data. Here are three of my personal favorites:

p c.members
# >> ["first", "last", "email"]
p c.values
# >> ["James", "Gray", "james@grayproductions.net"]
c.each_pair do |name, value|
  puts "#{name}: #{value}"
end
# >> first: James
# >> last: Gray
# >> email: james@grayproductions.net

We get a lot of functionality for free. That's obvious. But eventually you are always going to want to add your own. In times past, the following was a common idiom for that:

class Contact < Struct.new(:first, :last, :email)
  # ...
end

This is a neat example because it shows how flexible Ruby is. The parent Class for a Class definition doesn't have to be just a constant name. It can actually be any code that results in a Class object. Then we can just inherit from it and add to it as we like.

However, projects like Rails have shown the error of this approach. Because Rails is often dynamically reloading code, Class definitions will be rerun. That means the Struct call will happen again, resulting in a fresh parent Class object (which happens to have the exact same behavior). Ruby will see the new parent for an existing definition and choke with an error:

superclass mismatch for class Contact (TypeError)

The good news is that this isn't much of an issue because Struct plans for it. It's prepared to accept a block during definition and the contents of that block will be evaluate within your new Class. Thus it's trivial to add methods:

Contact = Struct.new(:first, :last, :email) do
  def to_hash
    Hash[*members.zip(values).flatten]
  end
end

# ...

p c.to_hash
# >> {"last"=>"Gray",
#     "first"=>"James",
#     "email"=>"james@grayproductions.net"}

There is a gotcha when defining methods this way though. Struct cheats on the internal implementation and doesn't actually place values in real instance variables. Thus, you will need to stick to accessing your data through the method interface:

Contact = Struct.new(:first, :last, :email) do
  def full
    "#{first} #{last}".strip
  end
end

# ...

p c.full
# >> "James Gray"

If Struct isn't dynamic enough for you, you may want to examine the standard OpenStruct library. It's essentially a Hash with a method interface, allowing you to change attributes as needed. You can also initialize it with a Hash, if needed:

require "ostruct"

name = OpenStruct.new(:first => "James", :last => "Gray")
p name.last
# >> "Gray"

name.suffix = "II"  # add an attribute
p name.suffix
# >> "II"

Sadly, OpenStruct is missing most of the niceties of Struct. Because of that, I don't feel it buys you much over a Hash.

Keep Struct in mind next time you need a simple data object. It's hardly any effort to setup, it comes fully loaded with options, and it can grow as your needs do.

I'm Addicted to the Word Array

2014-04-11T18:53:50Z

Continuing with my recent trend of showing of fun uses of Ruby syntax, I have a confession to make: I'm addicted to Ruby's "word Array." I really am.

I suspect most of you know this, but the word Array is a shortcut that can lessen the quote-comma-quote syndrome of simple a simple Array like:

["a", "wordy", "Array"]

You can create the same Array with the word Array syntax:

%w[a wordy Array]

That's essentially just a String that will automatically be split() on whitespace to build an Array. You can use any amount of space any place you like, so you can layout the data in whatever way makes the most sense for you:

require "pp"
pp %w[ one   two   three
       four  five  six
       seven eight nine
             zero        ]
# >> ["one",
# >>  "two",
# >>  "three",
# >>  "four",
# >>  "five",
# >>  "six",
# >>  "seven",
# >>  "eight",
# >>  "nine",
# >>  "zero"]

Note that you can chose the punctuation characters used at either ends of the Array, some of which are paired while others just repeat:

%w(a wordy Array)
%w{a wordy Array}
%w<a wordy Array>

%w!a wordy Array!
%w%a wordy Array%
%w#a wordy Array#

I tend to just stick with %w[ … ] though because I like how it parallels the [ … ] for a normal Array. It's pretty smart too and will allow nested brackets, as long as they are paired properly:

p %w[ [[]] abc[1] [start end] ]
# >> ["[[]]", "abc[1]", "[start", "end]"]

With just that much, this feature is crazy useful. Here are examples with Rails validations and filters, platform-safe relative path building, and simple iteration just to give you some ideas:

STATUSES = %w[Pending Requested Booked Declined Submitted]
validates_inclusion_of :status, :in => STATUSES

before_filter :find_user, :only => %w[show edit update destroy]

$LOAD_PATH << File.join(File.dirname(__FILE__), *%w[.. lib])

Card  = Struct.new(:face, :suit)
cards = [ ]
%w[heart spade club diamond].each do |suit|
  %w[A 2 3 4 5 6 7 8 9 T J Q K].each do |face|
    cards << Card.new(face, suit)
  end
end

Everyone of those examples was pulled out of real code I've written. I'm a serious 12 stepper on this issue.

That's the basics. However, like most delights in Rubyland, the word Array hides some lesser known surprises. First, did you know that you can embed entries with spaces in them?

SORT_OPTIONS = %w[ Default
                   Name
                   Price
                   Average\ Rating ]
p SORT_OPTIONS
# >> ["Default", "Name", "Price", "Average Rating"]

As you can see, a simple backslash escapes spaces. Now this is a technique you can definitely abuse and if you find yourself escaping every other space, you should probably just break down and build a normal Array. This is great for the occasional multiword entry though.

It doesn't stop there either. Remember when I said the word Array is essentially a String? Well, you can even mix in some interpolation, if you switch to the capital W:

MODE = Object.const_defined?(:Test) ? :test : :data
PATH = File.join(File.dirname(__FILE__), *%W[.. .. #{MODE} db.sqlite])

This is code from a trivial SQLite wrapper I made. It determines what MODE to run in with a simple test that will pass after Test::Unit has been loaded. It then builds a PATH to the database using the selected MODE. It's interpolation in the word Array that makes that easy to do.

It's probably worth mentioning that interpolated values are not subject to internal splitting:

var = "three four"
ary = %W[one two #{var}]
p ary
# >> ["one", "two", "three four"]

The capital W version supports some other String escapes as well, though I confess that I've never made use of this:

p %W[\t \x20 \n]
# >> ["\t", " ", "\n"]

Well, that's all the word Array goodness I can pack into a single blog post. Hopefully you are sold and I'll be seeing a lot more of you at the Word Array Anonymous User meetings.

Working With Multiline Strings

2014-04-11T18:54:39Z

I imagine most Rubyists are aware that Ruby has "heredocs," but do you really know all they can do? Let's find out.

A "here document" is a literal syntax for a multiline String. In the most basic form, they look like this:

p <<END_HEREDOC
This is a
  multiline,
as is String!
END_HEREDOC
# >> "This is a\n  multiline,\nas is String!\n"

The <<NAME syntax introduces the heredoc, but it actually begins at the start of the following line. It continues until NAME occurs again, at the beginning of a line. Note the trailing newline in the example above. All of the data between start and finish is packaged up into a String and dropped in where the original <<NAME designator appeared.

There are some important details in that description, namely that the String begins on the next line and that it's inserted where the heredoc was started. This means that the rest of the line where the heredoc is started can have normal Ruby code (though your editor may syntax highlight it badly):

p <<END_SQL.gsub(/\s+/, " ").strip
SELECT * FROM     users
         ORDER BY users.id DESC
END_SQL
# >> "SELECT * FROM users ORDER BY users.id DESC"

The spacing in the above example was to make it easier for a human to understand, but I use gsub() and strip() to normalize the actual String. You can do that since the heredoc doesn't begin until the next line.

Taking it one step further, the content of the rest of the line can include another heredoc. The second one will begin on the line after the first ends. This continues on down for however many you care to make:

def send_messages(*messages)
  messages.each { |m| p m }
end

send_messages(<<END_MESSAGE_ONE, <<END_MESSAGE_TWO)
This is message one.

...
END_MESSAGE_ONE
Another message.

....
END_MESSAGE_TWO
# >> "This is message one.:\n\n...\n"
# >> "Another message.\n\n...\n"

Another interesting thing to know about heredocs is that they are double-quoted Strings by default. You can use any escapes allowed there as well as interpolation:

p <<DOUBLE_QUOTED
Tricks:
\tindented
\t#{100 + 23}
DOUBLE_QUOTED
# >> "Tricks:\n\tindented\n\t123\n"

However, if you would prefer single-quoted behavior, you can just surround the heredoc name with single quotes:

p <<'SINGLE_QUOTED'
Tricks:
\tindented
\t#{100 + 23}
SINGLE_QUOTED
# >> "Tricks:\n\\tindented\n\\t\#{100 + 23}\n"

There's one more trick with regard to heredoc syntax. If you begin with <<-NAME, the end marker can be indented on the line it appears on. This is mostly useful when you want to inline some code, for example:

module HateMacro
  def self.generate_hate(target)
    module_eval <<-END_RUBY
      def self.hate_#{target}
        puts "#{target.to_s.capitalize} sucks!"
      end
    END_RUBY
  end
end

HateMacro.generate_hate(:emacs)  # Just for Jim Weirich!
HateMacro.hate_emacs
# >> Emacs sucks!

That String of code will have a bunch of whitespace at the beginning of each line. The space before the end marker is not counted though. This doesn't affect the code of course, but you need to keep it in mind for other content.

A final point of interest that may be of value to TextMate users: TextMate will properly syntax highlight the contents of <<-SQL, <<-HTML, and <<-CODE_FOR_EVAL (as Ruby) heredocs. You do need to use the indented form though even if you don't indent the content.

Hopefully all of this gives you some new ideas for ways you might handle multiline Strings with Ruby. You don't need heredocs everywhere, but they can clean things up with the right usage.

I Just Want One Character!

2014-04-03T22:46:33Z

Every so often a person asks the question on Ruby Talk, "How can I get just one character from the keyboard (without needing the user to hit return)?" Everyone is always quick to post solutions, but sadly there are some issues with almost every one of them.

The general consensus is that this is a tough problem to solve correctly. I say that's the exact reason to let HighLine handle this for you:

#!/usr/bin/env ruby -w

require "highline/system_extensions"
include HighLine::SystemExtensions

print "Enter one character:  "
char = get_character
puts char.chr

That doesn't look too tough, does it?

What's terrific about this solution is that under-the-hood HighLine will check your platform and libraries and then try to use the solution that makes the most sense for your environment. The code is really pretty robust too, because people a lot smarter than me have been sending in patches for over a year, slowly eliminating all of those tricky edge cases.

As you can see, I've split this functionality of HighLine into a separate module so you don't even need to load the full HighLine system. This was done just because this is such a real and common problem. This section of HighLine is one pure Ruby file, so feel free to vendor it if the external dependency is an issue.

Trust me, reading individual characters from the keyboard doesn't have to be that tough. You just need the right tool for the job.

PStore Meets YAML

2014-04-03T20:00:39Z

I love the PStore standard library. It's a very graceful interface to get some fairly robust serialized mini-database handling in just a few lines. With it you get:

Transactions with commit and rollbacks (automatic on exception).
File locking, shared and exclusive.
Multiprocessing safety.

PStore does even more, including some file mode checking and MD5 hashing to avoid unneeded writes, but the above are the major selling points for me.

Now, if I had to level any one complaint at PStore, it would be that because it uses Marshal under the hood it doesn't create files you can easily browse or tweak by hand. (Marshal is a feature, don't get me wrong. It's fast, which is very helpful.) Sometimes though I want PStore protection with the YAML file format.

I'm embarrassed to admit that I use to use a hack for this:

require "pstore"
require "yaml"
class PStore; Marshal = YAML; end

That just redefines the Marshal constant in a scope that should only alter PStore. The library only uses dump() and load() and those methods work the same with Marshal and YAML.

Ready for the punch-line?

I learned today that my fragile hack has been in vain, no matter how clever it may be. YAML ships with a file that will load and modify PStore for you. Usage is as simple as:

require "yaml/store"

From there just replace your PStore.new() calls with YAML::Store.new() and you're in business. YAML::Store is a subclass of PStore, so you won't need to change one bit of the API to get PStore robustness with YAML output.