Gray Soft / Tags / Object Orientation

Objectified Beer

2014-05-30T22:09:39Z

I got another thing out of my recent conversation with Katrina Owen: a will-not-let-go itch to try a programming exercise that she mentioned. I'm such a sucker for a challenge.

Katrina spoke of an assignment that her and Sandi Metz have used in their object orientation trainings. She said that they build an OO implementation of the 99 Bottles of Beer song, "mainly removing ifs and such." There may be more to the actual task than this, but I ran with that brief explanation.

As I often say, you'll probably learn more by trying the exercise for yourself before you read through my thoughts about it. Give it a go if you can spare the time.

Diff Driven Development

I decided to throw some scaffolding into the master branch of a Git repository. I figured I could then branch off of that with each idea, to keep trying things out.

I started by constructing a trivial framework for running and verifying the song. That consisted of an executable:

#!/usr/bin/env ruby -w

require_relative "../lib/bottles_of_beer"

verses = ARGV.first =~ /\A\d+\z/ ? ARGV.shift.to_i : 99
BottlesOfBeer::Song.new(verses).sing($stdout)

a copy and paste implementation:

# -*- coding: utf-8 -*-

module BottlesOfBeer
  class Song
    def initialize(verses = 99)
      @verses = verses
    end

    attr_reader :verses
    private     :verses

    def sing(target = $stdout)
      target << <<END_SONG
99 bottles of beer on the wall, 99 bottles of beer.
Take one down and pass it around, 98 bottles of beer on the wall…

…
END_SONG
    end
  end
end

and a Rake task to check the output against a static file I copied from the Web as my definitive source of the lyrics:

desc "Verifies the song output."
task :spec do
  print "\e[31m"
  system "bash -c 'diff -u <(bin/bottles_of_beer) spec/full_song.txt'"
  print "\e[0m"
  if $?.success?
    puts "\e[32mPerfect match.\e[0m"
  end
end

task default: :spec

This system of validation was very important to me. I wanted it to be easy to see how close to correct I was and a diff sums that up beautifully. However, I did not want to test drive my playing around. That would add overhead to my experimentation that I just don't need for a problem that I can keep in my head like this.

A Swing and a Miss

With the setup done, I was ready to try a solution.

My first idea was pretty straightforward. Build up the needed verses and then write them out:

require_relative "verse"
require_relative "penultimate_verse"
require_relative "ultimate_verse"

module BottlesOfBeer
  class Song
    def initialize(verses = 99)
      @verses  = Array.new(verses - 1) { |i| Verse.new(verses - i) }
      @verses << PenultimateVerse.new
      @verses << UltimateVerse.new(verses)
    end

    attr_reader :verses
    private     :verses

    def sing(target = $stdout)
      verses.each do |verse|
        target << verse
      end
    end
  end
end

Most verses are represented by a fairly predicable object:

# -*- coding: utf-8 -*-

module BottlesOfBeer
  class Verse
    def initialize(bottles)
      @bottles = bottles
    end

    attr_reader :bottles
    private     :bottles

    def to_s
      "#{bottles} bottles of beer on the wall, " +
      "#{bottles} bottles of beer.\n"            +
      "Take one down and pass it around, "       +
      "#{bottles - 1} bottles of beer on the wall…\n\n"
    end
  end
end

However, the one bottle verse is an edge case:

require_relative "verse"

module BottlesOfBeer
  class PenultimateVerse < Verse
    def initialize
      super(1)
    end

    def to_s
      super.gsub("1 bottles", "1 bottle")
    end
  end
end

The last verse is also special:

# -*- coding: utf-8 -*-

require_relative "verse"

module BottlesOfBeer
  class UltimateVerse < Verse
    def to_s
      "No more bottles of beer on the wall, " +
      "no more bottles of beer.\n"            +
      "Go to the store and buy some more, "   +
      "#{bottles} bottles of beer on the wall…\n"
    end
  end
end

This solution gets close, but it doesn't quite work:

--- /dev/fd/63  2014-05-30 13:21:03.000000000 -0500
+++ spec/full_song.txt  2014-05-30 09:34:28.000000000 -0500
@@ -290,10 +290,10 @@
 Take one down and pass it around, 2 bottles of beer on the wall…

 2 bottles of beer on the wall, 2 bottles of beer.
-Take one down and pass it around, 1 bottles of beer on the wall…
+Take one down and pass it around, 1 bottle of beer on the wall…

 1 bottle of beer on the wall, 1 bottle of beer.
-Take one down and pass it around, 0 bottles of beer on the wall…
+Take one down and pass it around, no more bottles of beer on the wall…

 No more bottles of beer on the wall, no more bottles of beer.
 Go to the store and buy some more, 99 bottles of beer on the wall…

This was an interesting decision point for me in the exercise. Obviously, I'm not green. Eventually, I need to get there. But I'm really close. Only two lines are wrong and I can see what the problem is. Mainly, when I transition to new verses, I don't pick up on the edge cases. I was starting to have ideas about how I might address this issue, but I wasn't too happy with the rest of the code yet anyway. I decided to attempt to improve what I had built a little first, before I tackled the obvious bug. I feel like my diff-based validation gave me the confidence that I could do this, since I would notice by the line count alone if I was drifting further off track.

Can I get a side of objects with that?

OK, my first major complaint was that I didn't land on a very OO solution. Since that's the point of the exercise, I wanted to address it. More specifically I think overriding a method to call gsub() on its output largely misses to point of sharing an implementation, as objects do. Similarly, my UltimateVerse pretty much ignores everything from its superclass. I minted a new branch and broke out the polish.

I began by extracting a few methods in Verse so I could override them in subclasses:

# -*- coding: utf-8 -*-

module BottlesOfBeer
  class Verse
    def initialize(bottles)
      @bottles = bottles
    end

    attr_reader :bottles
    private     :bottles

    def to_s
      "#{count_with_units} of beer on the wall, " +
      "#{count_with_units.downcase} of beer.\n"   +
      "#{action}, "                               +
      "#{bottles - 1} bottles of beer on the wall…\n\n"
    end

    private

    def count_with_units
      "#{bottles} bottles"
    end

    def action
      "Take one down and pass it around"
    end
  end
end

That allowed me to clean up the content editing in the one bottle verse:

require_relative "verse"

module BottlesOfBeer
  class PenultimateVerse < Verse
    def initialize
      super(1)
    end

    def count_with_units
      "1 bottle"
    end
  end
end

It also allowed me to turn the no more bottles verse into a more proper subclass that doesn't just override the work of the superclass:

require_relative "verse"

module BottlesOfBeer
  class UltimateVerse < Verse
    def to_s
      super.strip + "\n"
    end

    private

    def count_with_units
      "No more bottles"
    end

    def action
      "Go to the store and buy some more"
    end
  end
end

While things are definitely more object oriented in this version, I've actually worsened the results by one more failing line:

--- /dev/fd/63  2014-05-30 13:36:26.000000000 -0500
+++ spec/full_song.txt  2014-05-30 09:34:28.000000000 -0500
@@ -290,10 +290,10 @@
 Take one down and pass it around, 2 bottles of beer on the wall…

 2 bottles of beer on the wall, 2 bottles of beer.
-Take one down and pass it around, 1 bottles of beer on the wall…
+Take one down and pass it around, 1 bottle of beer on the wall…

 1 bottle of beer on the wall, 1 bottle of beer.
-Take one down and pass it around, 0 bottles of beer on the wall…
+Take one down and pass it around, no more bottles of beer on the wall…

 No more bottles of beer on the wall, no more bottles of beer.
-Go to the store and buy some more, 98 bottles of beer on the wall…
+Go to the store and buy some more, 99 bottles of beer on the wall…

I'm still quite close to correct, but I am seeing that drift I decided to watch for. This told me it was time to get serious about reaching a real solution.

Go Green or Go Home

My problems come from the fact that the verses transition into the next part of the song. That leads me to another thing that bothered me with my initial code: the need for bottles - 1 in Verse. If Song is keeping track of the current Verse, it feels like they shouldn't need to peek ahead internally. This made me feel like a layer was missing and I set out to add it in a new branch.

I decided to introduce this new concept of verses transitioning to the next verse:

# -*- coding: utf-8 -*-

module BottlesOfBeer
  class VerseWithTransition
    def initialize(verse, next_verse)
      @verse      = verse
      @next_verse = next_verse
    end

    attr_reader :verse, :next_verse
    private     :verse, :next_verse

    def to_s
      "#{verse}, #{next_verse.opening.downcase}…\n"
    end
  end
end

Of course, that required some changes in the Song to make use of these new transitions:

require_relative "verse"
require_relative "penultimate_verse"
require_relative "ultimate_verse"
require_relative "verse_with_transition"

module BottlesOfBeer
  class Song
    def initialize(verses = 99)
      @verses  = Array.new(verses - 1) { |i| Verse.new(verses - i) }
      @verses << PenultimateVerse.new
      @verses << UltimateVerse.new(verses)
    end

    attr_reader :verses
    private     :verses

    def sing(target = $stdout)
      verses.each_cons(2) do |verse, next_verse|
        target << VerseWithTransition.new(verse, next_verse) << "\n"
      end
      target << VerseWithTransition.new(verses.last, verses.first)
    end
  end
end

The base Verse itself could now quit worrying about the next number, as long as it made its opening() available for reuse:

module BottlesOfBeer
  class Verse
    def initialize(bottles)
      @bottles = bottles
    end

    attr_reader :bottles
    private     :bottles

    def opening
      "#{count_with_units} of beer on the wall"
    end

    def to_s
      "#{opening}, "                            +
      "#{count_with_units.downcase} of beer.\n" +
      action
    end

    private

    def count_with_units
      "#{bottles} bottles"
    end

    def action
      "Take one down and pass it around"
    end
  end
end

The edge cases were largely unchanged in this round, save that I pulled out an override from UltimateVerse that twiddled whitespace. It felt more natural to handle that at the Song level. The low number of changes needed here made me feel like I was making better use of subclasses now.

Also, good news, my output was finally validating.

Things That Bug Me

Now that I was green, I wanted to attack other concerns I had with the code. My biggest remaining complaint was this line in Song:

      @verses  = Array.new(verses - 1) { |i| Verse.new(verses - i) }

This represents multiple problems, in my opinion. First, I don't like that I'm pregenerating all of the verses. What if we want to use a huge number of verses? Why should that be memory inefficient? I would prefer to do the work as it comes in, when possible.

Also, there's some very non-obvious index math in this code, if you ask me. I had to lower the size of the initial Array to make room for the edge cases and I had to reverse the indexes inside the block, because the lyrics count down.

Another branch led to another round of fixes.

Here's a more clean iteration through the verses, in my opinion:

require_relative "verse"
require_relative "penultimate_verse"
require_relative "ultimate_verse"
require_relative "verse_with_transition"

module BottlesOfBeer
  class Song
    def initialize(verses = 99)
      fail ArgumentError, "Cannot sing negative verses" if verses < 0

      @verses = verses
    end

    attr_reader :verses
    private     :verses

    def sing(target = $stdout)
      verses.downto(2).each_cons(2) do |bottles, next_bottles|
        write_verse(target, Verse.new(bottles), Verse.new(next_bottles))
      end
      write_verse(target, Verse.new(2),         PenultimateVerse.new) \
        if verses >= 2
      write_verse(target, PenultimateVerse.new, UltimateVerse.new) \
        if verses >= 1
      write_verse(target, UltimateVerse.new,    Verse.new(verses), "")
    end

    private

    def write_verse(target, verse, next_verse, padding = "\n")
      target << VerseWithTransition.new(verse, next_verse) << padding
    end
  end
end

The main difference here is that I feel the iterators downto() and each_cons() make it easier to see how I am moving through the data. There's a lot more object construction in this version, but that's a concern for the garbage collector (which has improved a ton in recent releases). I try not to be afraid of using lots of objects, until I have some reason to be.

Just for the record, there was one more tiny change that I didn't show here. I finally dropped the argument to UltimateVerse's constructor, since that's now handled by transitions. This required that I override the base constructor to ignore it.

The Big Questions

Now that I had a working solution, I found myself wondering how much extra work I had done to objectify this code that could have a pretty short solution. I pulled a fresh branch off of master and solved the problem again for comparison purposes.

I rewrote the Song using the least complex proceedural code that I could dream up:

# -*- coding: utf-8 -*-

module BottlesOfBeer
  class Song
    def initialize(verses = 99)
      fail ArgumentError, "Cannot sing negative verses" if verses < 0

      @verses = verses
    end

    attr_reader :verses
    private     :verses

    def sing(target = $stdout)
      verses.downto(0) do |count|
        action  = count.zero? ? "Go to the store and buy some more"
                              : "Take one down and pass it around"
        target << "#{bottles(count)} of beer on the wall, " +
                  "#{bottles(count).downcase} of beer.\n"   +
                  "#{action}, "                             +
                  "#{bottles(count - 1).downcase} of beer on the wall…\n"
        target << "\n" unless count.zero?
      end
    end

    private

    def bottles(count)
      if count < 0
        bottles(verses)
      elsif count.zero?
        "No more bottles"
      else
        "#{count} bottle#{'s' unless count == 1}"
      end
    end
  end
end

I'll be honest and admit that I expected to like this code better. This is a small problem and I can keep the whole thing in my head when working at this level. I figured a short chunk of code that reflected that would be pretty digestible.

After seeing the two solutions side-by-side though, I'm not sure I really do feel that way. Sure, this one is shorter and you can easily view the whole thing at once, but there's a ton of edge cases mashed into this small space that make it harder to reason about. At least that's what I think. Notice the three separate count.zero? checks and the return of count - 1 even though the iterator is pushing us through the verses. I'm pretty sure I would rather fix bugs in the objectified code, rather than reason out which rules are in effect here at any given time.

Thanks for the mental workout Katrina.

Decorators Verses the Mix-in

2014-04-30T02:31:45Z

It is a neat time to be involved in the Ruby community, if you ask me. A large portion of us are currently studying the techniques for doing good object oriented development. We are looking at the ideas that have come before and trying to decide the best ways to apply those ideas to our favorite language. This leads to blog posts, forum threads, and conference talks about what we are learning. No matter what, we all gain from explorations like this. Everybody wins as our collective knowledge grows. We all deserve gold stars.

So far, there's one point pretty much everyone agrees on: composition should typically be preferred to inheritance. The trickier part of that discussion though is deciding what composition looks like in Ruby. Generally you see Rubyists comparing the merits of decorators and mix-ins. [Note: the comments correctly pointed out that this was a bad use of the word "composition" on my part, to describe mix-ins.] There's a very representative thread on the excellent Objects on Rails mailing list.

Much of time, decorators seem to come out ahead, but comparing the two is pretty complex. In this article I want to add my opinions to the discussion. Let's start by looking at some examples.

Decoration

Let's say that I have a simple server that just receives messages and displays what was sent:

require "socket"

server = TCPServer.new("127.0.0.1", 61676)
Socket.accept_loop(server) do |connection, _|
  while (object_sent = connection.gets)
    puts "Received:  %p" % [object_sent]
  end
end

This is pretty crude. It doesn't even handle concurrent connections, but it will work fine for the purposes of this example.

We also need a simple client to send some messages to our server:

require "socket"

server         = TCPSocket.new("127.0.0.1", 61676)
object_to_send = "A message."
server.puts object_to_send

At this point I can:

Start the server in one shell
Run the client in a separate shell
See output in the server's shell

The end result looks like this from the server's shell:

$ ruby server.rb
Received:  "A message.\n"

That's nice, but let's say it's not quite what I want. I can really just send String messages using this simple setup, but perhaps I would prefer to be able to send other Ruby objects, like an Array or Hash.

Let's add that capability to our client and server. I'll start that process by creating a serialization.rb file and defining a couple of decorators in there:

class Serializer
  def initialize(socket)
    @socket = socket
  end

  def puts(*objects)
    objects.each do |object|
      serialized = Marshal.dump(object)
      @socket.write([serialized.length, serialized].pack("NA*"))
    end
  end
end

class Deserializer
  def initialize(socket)
    @socket = socket
  end

  def gets
    if (message = @socket.read(4))
      object_size = message.unpack("N").first
      Marshal.load(@socket.read(object_size))
    end
  end
end

Both of these objects take a Socket, store it away for later use, then wrap the simple IO methods that my client and server are using to communicate. The wrappers manage a very simple serialization process where we convert a Ruby object to a binary representation and then send the length in bytes followed by those bytes. The other end can peek at the length, read that many bytes, and rehydrate the object.

I've skimped on the error handling here to keep the code simple. A more robust version of this code would need to make sure the serialized object length doesn't overflow the size of the number we're sending and handle the various networking errors that can occur.

Now let's update the client to support serialization:

require "socket"

require_relative "serialization"

server         = TCPSocket.new("127.0.0.1", 61676)
server         = Serializer.new(server)  # decorate the socket
object_to_send = %w[A more complex message.]
server.puts object_to_send

Notice that, aside from the added require_relative() call and the switch to a more complex object to show off serialization, there's really just one new line here. That line wraps the socket connection to the server in our decorator. This replaces the default puts() logic with our more involved process. That change actually takes effect on the last line of the client, but we didn't need to modify that code. That's the whole idea behind decoration.

The server is just as easy to update:

require "socket"

require_relative "serialization"

server = TCPServer.new("127.0.0.1", 61676)
Socket.accept_loop(server) do |connection, _|
  connection = Deserializer.new(connection)  # decorate the socket
  while (object_sent = connection.gets)
    puts "Received:  %p" % [object_sent]
  end
end

Again, all we need to do is wrap the connection to our client with the decorator. That replaces the gets() operation and we still didn't need to touch that code.

If I run the server and client again, we can see that the Array does come through:

$ ruby server.rb
Received:  ["A", "more", "complex", "message."]

It's time to admit that I purposefully selected this example to be something decorators are just perfect for. They really excel at these layering-on-functionality tasks. Java's IO class hierarchy is built much like the code I just showed for exactly this reason.

For comparison's sake though, let's switch to an example where I don't think decorators are as good of a fit, at least in Ruby.

Mixing It Up

I've shown an example very similar to this next one before, in an article about mix-ins, but let me flesh it out a bit more this time so we can really discuss why I handle it the way I do.

Let's say that I have some not-quite-CSV data in a file:

"Name (last, first)",Job
"Gray, James \"JEG2\"",Developer
"Gray, Dana",Full-time Mommy
"Gray, Summer","\"Cutie Pie\""

That data isn't really valid, according to the CSV standard. It should be doubling quotes to escape them (""), not using backslashes (\").

I would like to read that data with the standard CSV library so I can take advantage of the things it can do for me, like parsing out the headers. I want to use some code like the following:

require "csv"

CSV.foreach("not_quite_csv.csv", headers: true) do |row|
  p row.to_hash
end

But the CSV library doesn't recognize this format:

$ ruby read_csv.rb
…: Missing or stray quote in line 2 (CSV::MalformedCSVError)
…

To handle that, I would use a mix-in that fixes the data between the time when it is read and the time when CSV tries to parse it. Let's code that up:

require "csv"

module CSVNormalizer
  def gets(*args)
    line = super
    if line
      line.gsub!(/"(?:\\\\|\\"|[^"])*"/) { |field|
        field.gsub(/(?<!\\)\\"/) { '""' }
      }
    end
    line
  end
end

open("not_quite_csv.csv") do |file|
  file.extend(CSVNormalizer)
  csv = CSV.new(file, headers: true)
  csv.each do |row|
    p row.to_hash
  end
end

CSVNormalizer is pretty straight forward. It just wraps gets() and gives me a place to do a few simple substitutions.

Don't lose too much sleep over understanding the regular expressions I used there. They just replace the escapes with the format CSV expects.

In the lower chunk of code, I switched how I do the hand-off to CSV. This lets me get a hold of the File object and extend() it before CSV tries to read from it.

This solution isn't really complete, because it doesn't yet handle the multi-line fields that real CSV data can contain. It does work for the data in my example though:

$ ruby read_csv.rb
{"Name (last, first)"=>"Gray, James \"JEG2\"", "Job"=>"Developer"}
{"Name (last, first)"=>"Gray, Dana", "Job"=>"Full-time Mommy"}
{"Name (last, first)"=>"Gray, Summer", "Job"=>"\"Cutie Pie\""}

The Decision

I said that the second example is better as a mix-in. But why?

I mean you could choose to solve it with a decorator, right? Of course. Let's do that to see how it goes.

At first glance, it looks like we could get away with a few minor changes:

require "csv"

class CSVNormalizer  # switch to a class
  # store the IO for later use
  def initialize(io)
    @io = io
  end

  def gets(*args)
    line = @io.gets(*args)  # forward to the IO
    if line
      line.gsub!(/"(?:\\\\|\\"|[^"])*"/) { |field|
        field.gsub(/(?<!\\)\\"/) { '""' }
      }
    end
    line
  end
end

open("not_quite_csv.csv") do |file|
  file = CSVNormalizer.new(file)  # decorate
  csv  = CSV.new(file, headers: true)
  csv.each do |row|
    p row.to_hash
  end
end

If you run that code, it will probably seem to run the same. But it's not equivalent. In order to show you why, I need to modify my data to use different line endings:

$ ruby -pi -e 'gsub "\n", "\r\n"' not_quite_csv.csv

I'm on Unix, so I've switched the line endings to something I might have received from a Windows machine. You would need to go the other way if you are on Windows, but the end result is that our decorator is now broken:

$ ruby read_csv.rb  # using the decorator
…: Unquoted fields do not allow \r or \n (line 1). (CSV::MalformedCSVError)
…

That's interesting, because when I switch to the mix-in, it can read this data just fine:

$ ruby read_csv.rb  # using the mix-in
{"Name (last, first)"=>"Gray, James \"JEG2\"", "Job"=>"Developer"}
{"Name (last, first)"=>"Gray, Dana", "Job"=>"Full-time Mommy"}
{"Name (last, first)"=>"Gray, Summer", "Job"=>"\"Cutie Pie\""}

To know why this happens, you have to understand a little about how CSV works internally. As you can see, this whole line ending issue is kind of annoying. CSV tries to save you from needing to worry about that. By default, it will attempt to guess your line endings. You can always specify the ending manually if you need to, but a lot of times the guessing means that your program will just work without you taking any action.

That's what's happening with the mix-in version. No matter which line endings I feed it, CSV correctly guesses them and adapts. It just works.

Obviously we broke that in the decorator version, but how? Well, under the hood, CSV has to peek ahead at the data to guess the endings. It expects to be passed an IO object, so it calls on some IO methods to fetch some data and then reset the position pointer to where it was.

Of course, this feature doesn't always work. Sometimes you can't skip around in a stream like this. Consider STDIN or a Socket for example. So, if CSV cannot guess, it just defaults to the normal line endings for the current platform. That's the best we can do with no information to go on and the user can always specify the line ending manually as needed.

Now you can probably deduce why the decorator broken. It's not a full IO stand-in. We wrapped the method CSV needs to read data, but not the methods it uses to do the line ending guessing. That means it has to give up a go with the platform default.

All of this leads to the question: could we fix that? Sure. One option is that we could delegate the needed methods manually. Multiple methods are needed though, and if I wasn't the guy who wrote CSV, I wouldn't really know what they are. Plus, they could change some day. (That has actually happened. The algorithm was changed at one point to use different methods so it would work with more IO-like objects.) Given that, it's probably better to just forward any messages I don't want to wrap to the underlying IO object. That's easy enough to do:

require "csv"

class CSVNormalizer < BasicObject  # make sure we delegate most calls
  def initialize(io)
    @io = io
  end

  def gets(*args)
    line = @io.gets(*args)
    if line
      line.gsub!(/"(?:\\\\|\\"|[^"])*"/) { |field|
        field.gsub(/(?<!\\)\\"/) { '""' }
      }
    end
    line
  end

  # forward all other messages to the IO object
  def method_missing(method, *args, &block)
    @io.send(method, *args, &block)
  end
end

open("not_quite_csv.csv") do |file|
  file = CSVNormalizer.new(file)
  csv  = CSV.new(file, headers: true)
  csv.each do |row|
    p row.to_hash
  end
end

As the comments show, I only made two changes. First, I explicitly inherited from BasicObject instead of accepting the default Object parent. This means that my decorator responds to very few messages. Almost all incoming messages will hit method_missing() because of that change. The second change was to provide that method_missing() and have it push everything down to the underlying IO. This does restore the line ending guessing functionality and get the decorator fully working:

$ ruby read_csv.rb  # using the decorator again with tricky line endings
{"Name (last, first)"=>"Gray, James \"JEG2\"", "Job"=>"Developer"}
{"Name (last, first)"=>"Gray, Dana", "Job"=>"Full-time Mommy"}
{"Name (last, first)"=>"Gray, Summer", "Job"=>"\"Cutie Pie\""}

We now know what it takes build the equivalent decorator. If we compare the two, I say the mix-in is superior in this case. Here are my reasons:

The whole point of the BasicObject and method_missing() trick is to do our own method dispatching here. Why do that? Ruby is better at that kind of thing than we are, so I would rather leave that task to the language whenever we can. What do I mean by "better?" Well, for one thing, we can use super when Ruby is doing the job. For another, Ruby is going to be much faster when dispatching. Rack (another good example of decorators) has a little of this latter problem. As frameworks that use Rack, like Rails, move more and more functionality into middleware, the call stack just gets longer and longer. This is the primary reason that they filter backtraces in Rails and it has slowed things down a little as incoming requests must pass through all of the layers.
The decorator object I ended up creating lies about what it really is. Consider methods like class(), is_a?(), and ancestors(). These are all going to tell me that I have a File object in the case above. That's only sort of accurate though. I could probably drop BasicObject, in this case, but then the lies just change to leave out any mention of the hidden File object. Compare that with the mix-in version which is fully reflection compliant. It can tell you all about its "types" no matter how many there are.

Because of this, I tend to prefer mix-ins for cases where I am going to delegate an entire API. I feel like the advantages are pretty clear for those cases.

Arguments for and Against Decorators

I'm not trying to say there's no good uses for decorators. I've already provided some in this article. Greg Brown gives an even more spirited defense of them in Practicing Ruby 3.7, which is probably the most detailed, Ruby focused discussion of these issues to date. Greg is discussing higher level inheritance concepts in that article, but his examples largely boil down to an analysis of mix-ins and decorators.

I think Greg has some good and not-so-good points.

The best argument in favor of decorators, by far in my opinion, is the encapsulation issue. I think we normally envision encapsulation as hiding our data from the outside world, but in this case we're talking more about hiding it from ourselves. When you have some object with several mix-ins, how do we know that none of those methods have instance variable conflicts? Heck, the methods could even be accidentally overriding each other with unrelated functionality if we're not careful with naming. This could create some maddening bugs where things like the load order matter. These are very real and tough issues.

How often are we bitten by this encapsulation issue? I wish I knew. I think a strong test suite can help to reduce it. I also imagine it's less of a problem when most of the API is written by a single team. I'm guessing they would be more likely to catch misuses or impose naming conventions the protect themselves from these issues. But in the world where we embrace the value of sending Pull Requests on GitHub to fix most problems, I'm not sure how much of this safety we can count on.

I am less sold on the "bloated contracts" or "too many entry points" arguments that I see in many places, including Greg's post and the Objects on Rails thread I mentioned earlier. The example everyone throws around is ActiveRecord::Base which does have a ton of mix-ins. However, languages, libraries, and frameworks (like ActiveRecord) have different goals than we do for our application code.

We want to keep application code as minimal as possible and expose even less than we write whenever possible. This helps with many, many aspects of application development.

As I mentioned though, languages, libraries, and frameworks have almost the opposite goals. They want to cast their nets far and wide. They expose a lot and let your code decide which pieces it needs to do its work.

This makes sense when you think about it. The more a library does for us, the less application code we end up writing. That satisfies both aims.

Even if you sympathize with the bloated contract argument, I'm not convinced that decorators do much for it. Each individual object may respond to less messages, but is that going to save you from reading the documentation of all of those decorator classes? It didn't save me a ton of memorization when I passed the Java certification, I can tell you that much. The big API is still there, even if we do a better job of hiding it.

Testability is another complaint I often see leveled at mix-ins that I'm skeptical of. By way of example, let's test the mix-in I made earlier. I created a csv_spec.rb file and added this code to it to get started:

module CSVNormalizer
  def gets(*args)
    line = super
    if line
      line.gsub!(/"(?:\\\\|\\"|[^"])*"/) { |field|
        field.gsub(/(?<!\\)\\"/) { '""' }
      }
    end
    line
  end
end

describe CSVNormalizer do
  let(:message_queue) { [ ] }

  def gets(*_)
    message_queue.shift
  end

  before do
    extend(CSVNormalizer)
  end

  it "translates escaped quotes as they are read" do
    message_queue << '"a \"quoted\" field"'
    gets.should eq('"a ""quoted"" field"')
  end

  # ...
end

As you can see, I just gave my example group the expected gets() method and then extended it with my CSVNormalizer. From there I can test normally. I don't feel like I'm jumping through a lot of extra hoops here compared to testing a normal object, but you be the judge.

The Winner Is…

I don't think we can definitively say that one approach is superior to the other in all cases. I hope given compelling use cases for both in this article.

Mix-ins seems to be the underdog in most of these discussions, but I'm not quite ready to write them off. I hope I've shown that there are at least some things going for them.

As with most things in programming, I think it pays to try to understand as many of these angles as we can and make the best choices we can on a case by case basis.

The biggest question remaining for me is if we can do better at modeling large API's, like ActiveRecord, with the tools Ruby gives us. Is there a good way to use composition there without the mix-ins? I'm not sure. Hopefully some Rubyist will improve on the formula and enlighten us.

Single Method Classes

2014-04-29T03:10:09Z

[Update: I've changed my mind about some of the following due to this excellent counter argument.]

In the words of Dennis Miller, "I don't want to get off on a rant here, but…"

There's something that drives me crazy and I see it in so much Ruby code. I see it in the documentation for our key projects; I see Rubyists of all skill levels doing it; it's just everywhere.

Let's talk about when the use of a Class is and is not appropriate.

The Chained new()

Here's an example of one form of code that bugs me:

class Adder
  def add(n)
    40 + n
  end
end
p Adder.new.add(2)

The problem here is that a Class has been used, probably because as Rubyists that's always our default choice, but it's the wrong fit for this code. A Class is for state and behavior. The example above is just using behavior. No state is maintained.

A handy tip for sniffing out this problem is watching for a call to new() in the middle of method chaining as we have here. If you always use a Class like that, it's not really a Class. Put another way, if an instance never gets assigned to a variable, something has likely gone wrong with the design.

Here's a slightly trickier version:

class Adder
  def initialize(n)
    @n = n
  end

  def add(n)
    @n + n
  end
end

Does this code have the same problem? It depends on how it gets used. If it's always used with a chained new() then it's the same thing we looked at above:

p Adder.new(40).add(2)

State isn't really being tracked here. It's just an illusion.

However, it may be fine if the Adder is sometimes used like this:

forty = Adder.new(40)
puts "40 + 1 = #{forty.add(1)}"
puts "40 + 2 = #{forty.add(2)}"
puts "40 + 3 = #{forty.add(3)}"

Note the variable assignment. As I said before, this is our hint that things are on the up and up. In this case, some value is locked-in so we can then run some experiments on it. That initial value is state, so this is a fine use for a Class.

This isn't the only form of the problem we need to watch out for though.

The Unused new()

Some programmers see my first example and think, "No problem. Let's remove the new()." They will often change the code to be something like this:

class Adder
  def self.add(n)
    40 + n
  end
end
p Adder.add(2)

This isn't any better. Yes, the chained call to new() is gone. However, now we have a new problem. What does this mean?

what_am_i = Adder.new

A Class is a factory for manufacturing instances of some type. With this code, I am still allowed to make those instances, but they are completely pointless. There's no meaningful state or behavior for my new instance.

We could used some Ruby tricks to make new() uncallable, but that's not really better. Introspection would still tell us that Adder is a Class. That's a lie. It's not for manufacturing instances.

That's why this problem bugs me when I see it in code, by the way. Remember, the primary purpose of code is to communicate with the reader. Always. Period. Code that uses Classes in this way does not do that well. When I see a Class, I am thinking about the instances it will be used to manufacture. If that's not its purpose, I have been lead astray. We want our code to communicate better.

The One Word Fix

We have multiple choices for how to improve code like this, when we see it. The easiest is to make a one word change to my last example:

module Adder
  def self.add(n)
    40 + n
  end
end
p Adder.add(2)

Now we are working with a Module instead of a Class. The rules have changed. A Module is not used to manufacture instances. It can't do that. It's not really for maintaining state (though I admit that there are minor exceptions to this).

The important point is that I won't be thinking about instances when I read this code. It communicates better.

Is it the best we can do though?

Dual Natured

While a Module is not an instance factory, it does fill multiple roles in Ruby. This isn't your fault. A Module is a bit of an overloaded concept in the language.

First, a Module is often used as a namespace in Ruby. It can group together related constants and methods. We are fine with that meaning here. It just says that the add() method is part of the Adder namespace. No problem.

A Module is also a place to hang methods that don't belong on a Class, typically because they don't maintain any state. Conversion methods are a great example of this. Think of ERB::Util.h(). Again, this meaning is fine. We actually switched to a Module for exactly this reason.

Finally, a Module can be used as a mix-in in Ruby. This essentially allows plugging it into any other scope. It will add its methods and constants to that scope. This last meaning is a bit of a problem for our "fixed" code. The reason is that it's possible to write this code using our last example:

class SomeMathyThing
  include Adder
end

This is similar to our unused new() problem. While we can mix this Module in (because you can do that with any Ruby Module), it has no meaning. Only instance-level methods and constants matter when mixing a Module in and we didn't define either of those.

I want to stress that this isn't your fault. This is a quirk of how Ruby is designed. Switching to a Module is still an acceptable fix.

The question is, could we do even better? Could we bring our code in line with all three meanings of Ruby's Module. If we can, it might just communicate even better. That's a noble goal, so let's try it.

It turns out that Ruby includes a not-too-well-known tool just for this reason. A Module that uses this tool can be used both as a bag of methods and a mix-in. For example, take a look at the built-in Math Module:

module SomeMathyThing
  extend Math

  def self.do_math
    sqrt(4)
  end
end
p Math.sqrt(4)
p SomeMathyThing.do_math

As you can see we can call methods on Math directly and we can also add Math to other scopes when it makes sense. It fits all three definitions of a Module in Ruby.

The Kernel Module also behaves this way. If you were in some context that defined a p() method, you could still get to the one in Kernel if you needed it. The code for that would be ::Kernel.p(). The leading :: forces the constant lookup to happen from the top level, where Kernel lives. That ensures you will get the right Kernel, even if another constant with the same name is available in your current scope. Then we just call p() on the Module normally.

This is done with the help of module_function() and we can use it ourselves:

module Adder
  module_function

  def add(n)
    40 + n
  end
end

module SomeMathyThing
  extend Adder

  def self.do_add
    add(2)
  end
end

p Adder.add(2)
p SomeMathyThing.do_add

Notice that I just defined normal instance-level methods after calling module_function(). module_function() is like Ruby's public() and private() in that it will affect methods that follow, when used without an argument. (It does support passing arguments to affect specific methods as well.)

Controlling Access

Transforming a method with module_function() does a couple of things. First, the instance level methods are copied up to the Module level. This is what allows for the dual interface.

Another effect of module_function() is that the instance level methods are made private(). This prevents them from adding to any external interface when they are used as a mix-in. This is also why things like Kernel's p() cannot be called with a receiver, except in the Module level access case I showed earlier. In other words, this code will throw a "private method called" error even with our last example:

Object.new.extend(Adder).add(2)

This is usually desirable, but if you have a case where you would prefer to control access for the methods you define, you're probably going to lock horns with module_function().

There's a trick you can use to get around even that. Checkout this example:

module AccessControlled
  extend self

  def share_me
    hide_me.sub("private", "public")
  end

  private

  def hide_me
    "I am a private interface."
  end
end

p AccessControlled.share_me
p Object.new.extend(AccessControlled).share_me

The magic line is extend self which mixes the Module into itself at the Module level. This gives the same kind of dual interface we had before, but it also respects the access control we have setup for the methods. You can see that share_me() does add to the public interface of objects it is mixed into. Also, both of the following lines are errors, assuming the code above:

AccessControlled.hide_me
Object.new.extend(AccessControlled).hide_me

I tend to reach for module_function() first, preferring to keep as much private() as I can get away with. If I find myself in a situation where I need more control over visibility though, I'll switch to extend self.

The One True Interface

I want to cover one last semi-related point.

A lot of API's ask us to pass an object that responds to a certain method. If you are expecting just one method, make it call().

I think the downside to this approach is that programmer's worry it's not as expressive. While that's the right concern to have, I don't think it's as bad we think. It basically comes to this code validater.valid?(whatever) verses validater.call(whatever). While the former does clearly express what is happening, the second still comes out pretty OK in plain English, "Mr. Validater I am now calling upon you to do your job."

Plus there's just too much gain for going with call(). It's already been decided: call() is the preferred one method interface in Ruby. This makes you compatible with Proc (and lambda()) and Rack, just to give some key examples. That allows me to skip defining a Class/Module altogether if my needs are simple or possibly to serve whatever I am defining in a Web application. That's just because you chose to go with call(). Standard interfaces are handy.

If I haven't swayed you, at least support both, with some code like this:

if validater.respond_to?(:valid?)
  validater.valid?(whatever)
else
  validater.call(whatever)
end

Help Me End This Bad Habit

You now know that every time I see these single method stateless Classes in the wild I die a little inside. Have pity on my aging heart! Here's what you can do to help:

Check your code. I know you were reading this whole article thinking, "I don't do that." However, you probably do. Find code like this and improve it.
Ignore bad documentation. Whenever a Ruby project's documentation tells you to define these silly Classes, try it my way first. It pretty much always works just fine.
Fix the code and documentation of others. Start sending patches. Share a link to this article when you do. Let's make this less common. I'll live longer.

Learn to Love Mix-ins

2014-04-27T15:37:55Z

The road to mastering Ruby is paved with understanding some key Ruby concepts. Mix-ins are one of those concepts. I'm sure everyone reading this knows the mechanics of how mix-ins work, but it pays to spend some time really thinking about all that mix-ins imply. Let's do just that.

Adding a Type

One of the primary reasons that Ruby needs mix-ins is that it does not support multiple inheritance. That leaves mix-ins as our only option for modeling hybrid objects. It's the way Ruby programmers can add another type.

That's a good way to think about it too: adding a type.

Take pagination, for example. Pagination methods are usually defined to return an object like this:

class PaginatedCollection < Array
  # ... paginated helpers defined here ...
end

That's never really felt right to me though.

First, inheriting from Ruby's core classes can come back to bite you in some scenarios. The reason is that Ruby makes some performance tradeoffs to keep the core classes fast, but those tradeoffs mean that those classes don't always perfectly follow Ruby's rules.

You can get around some of these issues by delegating to an Array, instead of inheriting from it. But that creates even more structure for this setup when it seems that Ruby has a totally workable built-in solution.

The name of the created Class is another hint that it's not ideal, in my opinion: PaginatedCollection. Collection is just another way to say Array, right? So why can't we just say Array? In truth, Collection implies a lot more than just an Array, but solutions like this don't generally adapt to other types.

That leaves just Paginated, which is used as an adjective here. Adjectives are generally better modeled by a Module than a Class. It's almost like Ruby is telling us the right way to model this concept, if we listen closely enough.

To me, it's telling us to build this:

module Paginated
  def paginate(page = 1, items_per_page = size, total_items = size)
    @page           = page
    @items_per_page = items_per_page
    @total_items    = total_items
  end

  attr_reader :page, :items_per_page, :total_items

  def total_pages
    (total_items / items_per_page.to_f).ceil
  end

  # etc...
end

collection = %w[first second third]

collection.extend(Paginated)
collection.paginate(1, 3, 10)

puts "There are #{collection.total_pages} pages."
puts "The collection is an Array."           if collection.is_a?(Array)
puts "The collection is a Paginated object." if collection.is_a?(Paginated)

That seems more natural. The item is an Array. It also happens to be Paginated. We can understand that duality and so can Ruby.

Semantics aside though, this has actual benefits. The code works just fine if I change this line:

collection = %w[first second third]

to:

require "set"
collection = Set.new(%w[first second third])

or even:

collection = {first: 1, second: 2, third: 3}

In other words, we can now paginate whatever we want, including our own classes. We can also mix Paginated into any result set we need to, without needing to use some special collection and a replace() method (as you sometimes see in Rails code using will_paginate).

There's even another plus to this approach though: you can make these type decisions as the code runs. Let's dig into what that can do for us.

What do I Need Now?

Another great aspect of using mix-ins to build up these types is that objects can be different things at different times. We'll start with a somewhat silly, but illustrative, example:

class Human
  def initialize(name, strength)
    @name     = name
    @strength = strength
  end

  attr_reader :name, :strength

  def move
    "#{name} walks."
  end
end

module Werewolf
  module HybridForm
    def strength
      super + 5
    end

    def move
      super.sub("walks", "lopes")
    end
  end

  module WolfForm
    def strength
      2
    end

    def move
      super.sub("walks", "runs on all fours")
    end
  end
end

human = Human.new("James", 3)
puts "Normally, #{human.name} is a #{human.class} " +
     "with a strength of #{human.strength}.  #{human.move}"
puts

werewolf = human.dup.extend(Werewolf::HybridForm)
puts "But when there's a full moon, " +
     "strength raises to #{werewolf.strength} and #{werewolf.move}"
puts

wolf = human.dup.extend(Werewolf::WolfForm)
puts "Some also claim to have seen him as a normal wolf, " +
     "with a strength of #{wolf.strength}."
puts "In this form #{wolf.move}"

That code outputs:

Normally, James is a Human with a strength of 3.  James walks.

But when there's a full moon, strength raises to 8 and James lopes.

Some also claim to have seen him as a normal wolf, with a strength of 2.
In this form James runs on all fours.

As you can see, we're able to change the human object as needed. We can add in the abilities of the Werewolf::HybridForm or the Werewolf::WolfForm. The object then has those behaviors.

Believe it or not, this does have practical value outside of building fantasy games. For example, I once worked on a Rails application that had some pretty complex reporting needs. The queries used to do the reporting were a bit sluggish, so we wanted to make sure that kind of thing never got run during a normal request. Thus we had a model and a separate Module for the reporting code. They looked something like:

class User < ActiveRecord::Base
  # no generate_report method here
end

module Reportable
  def generate_report
    # build and return a processing intensive report
  end
end

The Module wasn't even loaded by our normal application. Instead, we ran a nightly background process, on a different server, that used our backup dump of the database to do the reporting. That code is where the Module actually lived. The reporting process then used it in a loop like this:

User.find_each do |user|
  user.extend(Reportable)
  report = user.generate_report
  # report sending code here...
end

This has other benefits besides safety. For example, it really helps with code organization. The reporting code was quite complex and it just didn't apply to the normal workings of the application. With this setup, we only had to dig through that code when it was relevant to what we were doing. The rest of the time we could pretty much forget it was there. (We did have tests to ensure User kept the contract Reportable counted on, so we wouldn't accidentally break it.)

I would like to see Rails move towards supporting a system like this. I'll build a plugin at some point that lets you add various mix-ins for the models. They could be stored in the application like this:

app/models/user.rb
app/models/user/authenticatible.rb
app/models/user/reportable.rb
…

Then I'll add a method for mixing these into the query results they apply to. For example:

user_logging_in = User.behavior(Authenticatible).find_by_email(email)

and:

User.behavior(Reportable).find_each do |user|
  # ... generate reports...
end

You get the idea.

OK, let's consider modules in one more way to round out this exercise.

Monkey Patching is Generally Silly

We always say that we love how dynamic Ruby is. It allows us to rewrite the rules as we go. The ultimate expression of this is how we can just reopen a class and replace some of its methods wholesale. We lovingly refer to this practice as "monkey patching."

When you think about it though, the practice is almost silly for most of the cases we use it for.

To explain that claim, we need to take a quick detour and discuss Ruby's method lookup. Instead of explaining how it works, let's just ask Ruby to show us:

class Object
  def show_yourself
    puts "In Object."
  end
end

class Parent
  def show_yourself
    puts "In Parent."
    puts "Handing up the line..."
    puts
    super
  end
end

class Child < Parent
  def show_yourself
    puts "In Child."
    puts "Handing up the line..."
    puts
    super
  end
end

o = Child.new
o.show_yourself

That code prints:

In Child.
Handing up the line...

In Parent.
Handing up the line...

In Object.

Hopefully, that's pretty straightforward. We called a method on a Child instance. That method then handed up to the same method defined in Parent, because Child inherits from Parent. That method then hands up to Object, because anything that doesn't declare a superclass, like Parent, gets Object.

In Ruby though, that's not the full story. Let's change the last chunk of code to this:

# ...

o = Child.new
class << o
  def show_yourself
    puts "In the hidden singleton class."
    puts "Handing up the line..."
    puts
    super
  end
end
o.show_yourself

Now we get this output:

In the hidden singleton class.
Handing up the line...

In Child.
Handing up the line...

In Parent.
Handing up the line...

In Object.

Each instance in Ruby has its own Class, called the singleton class. (No, we're not talking about the Singleton design pattern here. This is a different use of the term.) This is why we can modify individual objects no matter what their Class is. The specializations end up in the singleton class. Ruby mostly hides this class from us as an implementation detail, which is why I had to open the object's singleton class explicitly to get it to show up above.

We can now see that method lookup is a straight line in Ruby. You start at the bottom, with the object's singleton class and go straight up until you find a matching method.

That leads us to a question though: how do mix-ins play into this? The answer is pretty simple: they are inserted into the call chain behind a Class. [Update: Ruby 2's prepend() gives us the ability to insert mix-ins in front of a Class as well.] To see what I mean, let's mix something into Child:

# ...

module Mixin
  def show_yourself
    puts "In Mixin."
    puts "Handing up the line..."
    puts
    super
  end
end

class Child < Parent
  include Mixin

  def show_yourself
    puts "In Child."
    puts "Handing up the line..."
    puts
    super
  end
end

# ...

That prints:

In the hidden singleton class.
Handing up the line...

In Child.
Handing up the line...

In Mixin.
Handing up the line...

In Parent.
Handing up the line...

In Object.

See how Mixin just hopped in line behind Child? That's what they do.

Well, that's what include() does. Isn't extend() special? Not really. The fact is that extend() is just a shortcut that translates to this:

def extend(mixin)
  singleton_class.class_eval do
    include mixin
  end
end

As you can see, extend() just does an include() under the hood, but it does it on the singleton class. Using that knowledge, we can modify our example one last time to show off a superior form of monkey patching:

# ...

module SuperiorMonkeyPatch
  def show_yourself
    puts "In SuperiorMonkeyPatch."
    puts "Handing up the line..."
    puts
    super
  end
end

o = Child.new
o.extend(SuperiorMonkeyPatch)
class << o
  def show_yourself
    puts "In the hidden singleton class."
    puts "Handing up the line..."
    puts
    super
  end
end
o.show_yourself

Now the code prints:

In the hidden singleton class.
Handing up the line...

In SuperiorMonkeyPatch.
Handing up the line...

In Child.
Handing up the line...

In Mixin.
Handing up the line...

In Parent.
Handing up the line...

In Object.

Note the SuperiorMonkeyPatch is behind the singleton class, because mix-ins are always inserted behind a Class. But that puts it in front of Child, which is right where we need it to be if we want to override some behavior.

To sum this up, the key insights are:

Ruby's method lookup is just a straight line
Using mix-ins, we can insert code at various points along that line

This is the ideal way to override behavior. Why? Well, if you monkey patch a Class, you have changed it for the whole world. We've already seen though that mix-ins give us the choice of when we want the changes and when we don't.

Furthermore, you have to be very careful when monkey patching if you want to keep the old code. This involves a dance of aliasing (really copying) the old method, replacing it, and then referring to the renamed code. (That's how the infamous alias_method_chain() in Rails works.) With a mix-in though, super can be used normally.

This makes using a Module safer. We're not all playing in one namespace and aliasing a bunch of names that could eventually collide.

To give a realistic example of how you can use this technique in practice, let's talk about a support email I received recently. The programmer who wrote me was working with a database that spit out some pretty goofy CSV. It would escape quotes with \", even though the CSV format calls for "". It also escaped other things, like null characters as \0. Because this created a non-standard data format, he couldn't find anything that could read it. That made him write me and ask if there were some way he could modify the CSV library to read it.

"No need," I told him. CSV is a simple wrapper over a normal Ruby IO object and it mostly just counts on one method gets(). When you call CSV#gets, it pulls some data from IO#gets, then parses it and returns it to you. That gives us a familiar method lookup line:

 IO#gets
   ^
   |
CSV#gets

All we need to do is get some code in the middle of that line. That code can translate from the broken CSV format to the correct format the library expects and everything else will just work. No monkey patching is required. Here's the code I sent to him:

module CSVNormalizer
  def gets(*args)
    result = super
    if result
      # FIXME:  Improve escape handling
      result.gsub!(/\\(["0])/) { $1 == "0" ? "\0" : '""' }
    end
    result
  end
end

if __FILE__ == $PROGRAM_NAME
  require "csv"

  abort "#{$PROGRAM_NAME} FILE_PATH" unless ARGV.first

  open(ARGV.first) do |io|
    io.extend(CSVNormalizer)
    csv = CSV.new(io)
    csv.each do |row|
      # use row here...
    end
  end
end

My unescaping code may not be perfect, if there are other scenarios than those he described, but that's not the point. Note how easy this was to setup: I defined a module that uses super to get the data, transforms it, and returns the fixed data; I manually setup the IO object, instead of letting CSV do it for me; I used my mix-in to make that object return good data; then I hand the now safe-to-use object on to CSV. This is a clean patch job that won't affect any other code, even if we use it in a project that also processes some normal CSV data. Perfect.

Keep Modules in Mind

Ruby gives us these different ways to construct types using mix-ins. We can combine them, use them conditionally, and even build up modified behaviors with them, as needed.

A lot of this power comes the elegance of how Ruby handles method lookup. Once you realize that message passing happens down a straight line and that you can make interceptions at points along that line, you gain a lot of control over what is happening in your programs.

I strongly encourage to play with these techniques in your programs, or just on the side. It can take a while for all of this to click, but it really expands your understanding of Ruby when you get there.

Learn More About Modules

If you enjoyed this discussion of how to make mix-ins work for you, I'm pretty sure you would also enjoy:

Module Magic is a presentation I gave at the Lone Star Ruby Conference in 2009. I showed some of the tricks discussed above (in less detail) and some other handy tricks you can use. My slides are also available.
The book Crafting Rails Applications is a detailed explanation of how Rails 3 works under the hood. One of the most significant changes from Rails 2 to Rails 3, was to rework the framework from using old style monkey patching (with alias_method_chain()) to the Module approach I've covered here. This had many benefits which this book covers very well. We discussed this book on the Ruby Rogues podcast.
Dave Thomas did a screencast series a while back on The Ruby Object Model and Metaprogramming. Though it uses an old version of Ruby, and misses out on some niceties added later (like the singleton_class() method), it's easily one of the best discussions of Ruby's method lookup system and a lot more. If you have struggled to understand the tricks we are using when metaprogramming, this is the place to get those questions answered.

The Right Ruby Mix

2014-04-25T21:40:34Z

Ruby is a melting pot language. It borrows ideas from many things that came before. It combines several different programming philosophies.

This aspect of the language can be a plus. It means that Ruby is suited to multiple applications. It also opens up some pragmatic shortcuts. Even better, it sometimes encourages us to think about problems using a different lens of thought.

Of course, this cuts both ways. Ruby living at the intersection of many ideas does have some downsides. First, there's more to learn than you find with some simpler languages. There's a cost for the extra knowledge we have to track. Even worse though, in my opinion, is that it's sometimes hard to know exactly what Ruby's style really is.

Going Off Script

One culture Ruby borrowed heavily from is that of the so called "Scripting Languages." The main source of these features was Perl, in my opinion, but you can also find influences from Bash and other sources. I found this comforting since I came to Ruby from Perl, but the truth is that it bothers some people.

What's up with that test() method, really?

One thought is that Ruby has it to make Bash programmers more comfortable. In other words, this Bash code:

$ if [[ -e "Gemfile" ]]; then echo "This project uses Bundler."; fi
This project uses Bundler.

can be ported to similar Ruby code:

$ ruby -e 'if test ?e, "Gemfile" then puts "This project uses Bundler." end'
This project uses Bundler.

That's not a very Rubyish way to check for a file though. Typically you would see it written more like this:

$ ruby -e 'puts "This project uses Bundler." if File.exist?("Gemfile")'
This project uses Bundler.

That's shorter and it reads better, so it's no surprise that Rubyists prefer it. We generally love code that reads well.

The interface to test() is still a little bizarre though, don't you think? I guess we wanted something close to -e and Ruby's character syntax ?e was deemed close enough. The truth is that we could get closer to the Bash syntax, if we really wanted to:

$ ruby -r ./bash -e 'if bash_test [[ -e, "Gemfile" ]] then puts "..." end'
...
$ cat bash.rb
%w[A b c C d e f g G k l M o O p r R s S u w W x X z].each do |arg|
  flag = -arg.getbyte(0)
  if arg.downcase == arg
    Object.class_eval do
      define_method(arg) { flag }
    end
  else
    Object.const_set(arg, flag)
  end
end

def bash_test(*args)
  test(*args.flatten)
end

Why didn't Ruby go this far? I assume the reasons are pretty obvious, but let's discuss them briefly.

First, it pollutes the top level namespace with several methods and constants. This already bit me in the creation of this tiny example. I had a bug in my first attempt at this code and I tried to debug it with p() only to realize that I had replaced that method with a much less helpful chunk of code. (If you ever run into that problem, you can still get to it with Kernel.p() thanks to the magic of module_function().) The added methods and constants return almost senseless values, so it's unlikely they would be good for much else.

Another problem is that I couldn't support all of the operators test() accepts using this trick. It also understands ?-, ?=, ?<, and ?>.

We can see that test() was a compromise, to get close to some Bash-like support without doing too much damage to Ruby itself. Was it worth it? Tough call. I favor methods like File.exist?() and the File::Stat module when I do checks, because I think they better reveal my intentions and they don't make the code a lot longer.

Now, I do use several of the borrowed Perl shortcuts. You probably know that, since I have written about them before. I find that using Ruby on the command-line with -n or -p, $_, and the flip-flop operator does save me a lot of code and time. In those cases, the tradeoff feels worth it to me.

But perhaps this just shows that I was more of a Perl guy than a Bash guy when I came to Ruby.

Do I use these crazy shortcuts when I'm writing full Ruby programs? Not usually. I am not above shelling out to a command when I think it's easier though. For example, I've seen several complex and slow ways to generate a UUID in Ruby over the years, but I usually favor this cheat:

$ ruby -e 'uuid = `uuidgen`.strip; p uuid'
"6196D647-F6C4-4764-90A8-1775E22400B9"

It's been fast enough for any need I've ever had and I haven't had to run the code anywhere that didn't support this trick. (Windows probably wouldn't out of the box.)

This is the real win of Ruby's scripting heritage, in my opinion. It allows us to think about some problems in a different way. Instead of being forced to think, "I need to find a library for generating UUID's," Ruby makes it easy to substitute, "Doesn't Unix have a standard tool for this that I can talk to?" Would you try the same trick in a language like Java? Probably not, just because scripting support is a lot less robust there. That's why they tend to favor the libraries.

I vote we keep the I-can-always-fall-back-to-scripting attitude.

Life After Functional Programming

Does Ruby inherit from the functional programming languages? I would say that it definitely does. The functional languages are really where concepts like iterators originated. In fact, I believe Ruby's blocks in general were inspired by functional concepts.

Does that mean Ruby is a functional programming language? That question is a lot tougher, but I lean towards answering, "No." There are definitely folks that disagree with me on this point though.

Ruby methods aren't quite first-class functions. However, you can pretty much simulate that style of programming if you stick to using lambda(). That allows us to store "functions" in variables:

multiply = lambda { |l, r| l * r }
double   = lambda { |n| multiply[n, 2] }
triple   = lambda { |n| multiply[n, 3] }

(1..5).each_with_index do |n, i|
  puts unless i.zero?
  puts "      N: #{n}"
  puts "Doubled: #{double[n]}"
  puts "Tripled: #{triple[n]}"
end

Because the block passed to lambda() is a closure, we can later refer to those functions by name.

We can also return functions from functions:

build_multiplier = lambda { |multiplier|
  lambda { |n| multiplier * n }
}

double = build_multiplier[2]
triple = build_multiplier[3]

# ...

Of course, we can also pass functions to functions:

poor_mans_curry = lambda { |arg, f|
  lambda { |*rest| f[arg, *rest] }
}

double = poor_mans_curry[2, lambda { |l, r| l * r }]
triple = poor_mans_curry[3, lambda { |l, r| l * r }]

# ...

The example above also shows that anonymous functions are supported.

Ruby even has features to support this kind of work in some ways. I could have written the first example as:

multiply = lambda { |l, r| l * r }
double   = multiply.curry[2]
triple   = multiply.curry[3]

# ...

Since we've established that Ruby supports all of these functional features, why do I say it's not a functional language? Well, it's true that we can compose with functions in Ruby. That's pretty much the gateway to functional programming. But things don't end there.

Once you have those features, you start to do things in a certain way to maximize the benefits of using them. For example, it's ideal to use immutable data structures and treat variables more like constants. This allows you to guarantee that the same function called with the same arguments always returns the same results.

It's also very common to write recursive functions when programming functionally, so it's nice for the language to support that.

These are examples of areas where Ruby stops scoring so well. Of course, you can make immutable data structures and write recursive functions. However, there are downsides to using these tactics. Ruby's GC isn't really tuned for immutable data structures and you can definitely take some performance hits while the language struggles to clean up after you. Similarly, Ruby doesn't do tail-call optimization by default, so it's easy to exhaust the stack with a recursive function.

In my opinion, details like these make it clear that Ruby wasn't designed to favor this style of programming, even if it does make it possible. That's why I say that it's not really a functional language.

In Ruby, we could write a function to wrap another function and add some feature: say memoization. But the fact is that Ruby often has other ways to accomplish these tasks. For example, here's a dirt simple memoization technique for Ruby:

def fibonacci(nth, series = method(:fibonacci))
  if nth < 2
    nth
  else
    series[nth - 2] + series[nth - 1]
  end
end

# The easiest memoization in the world:
fibs = Hash.new { |series, nth| series[nth] = fibonacci(nth, series) }

p fibs[100]

The "function" above is originally recursive, though probably not as you normally see them written. I allow for the source of the series to be passed as an argument. That was just to make it trivial to swap it out later and that argument defaults to a reference of the method (Ruby's terminology) itself. This means it's really normal recursion (in disguise).

The main point here is that the default block of a Hash is actually just a memoization function already written for us in optimized C. We can just use a Hash to achieve that effect. It's clean and lightning quick.

I've seen multiple Rubyists over the years take detours into the functional languages, then come back and try to apply the concepts they have learned to Ruby itself. I did the same thing myself and wrote a lot about it. I never ended up finishing that series, because I realized that it wasn't really working. If I followed the functional code too closely, the resulting Ruby had issues. In other cases I really liked the code I came up with, but that's just because I translated the underlying concepts to how Ruby would better handle them (leaving functional programming behind).

The moral is that Ruby did inherit a lot from the functional programming languages, but you are probably already using the important bits. Work with blocks and iterators where they make sense and you will gain the most benefit from that style. Going any further than that may work for some cases, but the path is full of pitfalls.

It's Objects All the Way Down

Did I save the best for last? You bet!

Ruby clearly inherits from object oriented languages, like Smalltalk. That's a major focus of the design of the language.

The fact is that it really paid off too. Ruby has a gorgeous object model due to features like:

Almost everything is an object
Classes are objects too
Class definitions are just plain Ruby code being executed
Classes are open
Hooks like method_missing() and const_missing()
Per object behavior via singleton classes
Mix-ins

Sure, the system isn't perfect. We traded method overloading for dynamic typing (which is good and bad, in my opinion). We also don't really have keyword arguments. [Update: I am finally satisfied with Ruby's keyword arguments as of Ruby 2.1.] We do have techniques for getting around these deficiencies that generally seem to be good enough though.

What about multiple inheritance? Some say it's a feature. Others say it's a bug. It's definitely debatable, but Ruby seems dynamic enough to get by without it, in either case.

The fact is, whether by accidental or purposeful design, Ruby really has a terrific object model. The more I've learned about it over the years, the more I have come to appreciate it. I mean it's almost a zen moment when you finally realize that Ruby's method lookup is just a straight line and tools like classes, singleton classes, and mix-ins allow you to define methods at desired points on that line. It all fits together so well.

What's more, as I have been a Rubyist, I have studied some of the more classical material on object orientation. The more I learn from that world, the better my Ruby seems to get. It's almost exactly the opposite of trying to apply functional techniques to the language in that Ruby seems to welcome such ideas.

For example, look at a simple pattern like Pluggable Selector in Java, a mainstream object oriented language. Now observe the same pattern in Ruby:

class PluggableSelector
  def initialize(selector)
    @selector = selector
  end

  def run
    send("run_#{@selector}")
  end

  def run_foo
    "foo"
  end

  def run_bar
    "bar"
  end
end

result = PluggableSelector.new(:foo).run
puts "Ran #{result}"

Features like Symbol objects, interpolation, and send() just make this kind of work trivial. Ruby was made to do this.

Object thinking should always remain a big part of the Ruby we do.

What's Our Style?

I've argued that Ruby isn't just a scripting language, a functional language, or even just an object oriented language. It even has some other influences that I didn't visit here.

So, what kind of language is Ruby? What's the kind of programming that we do with it called?

I don't know that there is a name for it, but clearly it's a blend of many ideas. The trick is to find the right mix to make great code.

How do we define "great code?" Code's first priority is always to communicate with the reader. Start there.

Here's the mix I recommend for great Ruby:

74.41% Object Oriented Programming: the classic techniques enhanced by Ruby's dynamic nature
17.21% Functional Programming: mostly the bits Ruby has adopted, like blocks and iterators
9.38% Scripting: for quick hacks and pragmatic shortcuts

You can clearly see by my exact figures and excellent math that this is a highly scientific calculation not to be argued with.

Infinite Streams

2014-03-28T21:17:43Z

I've tried to summarize this chapter a couple of times now, but I keep getting tripped up over syntax. So, let's talk about that…

Functional Perl

Obviously, the examples in the book can be more or less directly translated. Here's a sample from the first couple of pages:

#!/usr/local/bin/ruby -w

### Stream Methods ###

def node(head, tail)
  [head, tail]
end

def head(stream)
  stream.first
end

def tail(stream)
  tail = stream.last
  if tail.is_a?(Proc)
    tail.call
  else
    tail
  end
end

def drop(stream)
  head                 = head(stream)
  stream[0], stream[1] = Array(tail(stream))
  head
end

def show(stream, limit = nil)
  while head(stream) && (limit.nil? or (limit -= 1) > -1)
    print drop(stream), $, || " "
  end
  print $/
end

### Examples ###

def upto(from, to)
  return if from > to
  node(from, lambda { upto(from + 1, to) })
end
show(upto(3, 6))  # => 3 4 5 6

def upfrom(start)
  node(start, lambda { upfrom(start + 1) })
end
show(upfrom(7), 10)  # => 7 8 9 10 11 12 13 14 15 16

Already though this is feeling very un-Rubyish, and it will only get worse if we keep going down this path.

I suspect the main reason MJD took such a functional approach with his book is that Perl's objects can be heavy and awkward. That doesn't make for great book examples. Ruby isn't like that though and the previous code is just crying out to be objectified.

Ruby's Object Orientation

We will start at the beginning. Clearly we need a constructor. Now the whole point of all this is building lazy streams, so we're always going to be using some head value and a promise for the tail. A promise is a chunk of code, and in Ruby we call that a block:

module LazyStream
  class Node
    def initialize(head, &promise)
      @head, @tail = head, promise
    end
  end
end

That's already an improvement, since it will allow us to drop the calls to Kernel#lambda. The name is a bit lengthy, but we can easily fix that after we have the rest of this down. For now, we're just tucking our work into a safe namespace.

The next step is to restore access to the head and tail. These are pretty simple readers in this form and while we are building them, let's add some Rubyish aliases:

module LazyStream
  class Node
    attr_reader  :head
    alias_method :current, :head

    def tail
      if @tail.is_a?(Proc)
        @tail.call
      else
        @tail
      end
    end
    alias_method :next, :tail
  end
end

Now, adding back support for drop is where things start to get a little tricky. If a call to LazyStream::Node#tail returns a LazyStream::Node, we want that to replace the current LazyStream::Node. If we get anything else, we'll just assign it normally. Again, here's the code, with aliases:

module LazyStream
  class Node
    def drop
      result, next_stream = head, tail
      @head, @tail        = if next_stream.is_a?(self.class)
        next_stream.instance_eval { [@head, @tail] }
      else
        Array(next_stream)
      end
      result
    end
    alias_method :tail!, :drop
    alias_method :next!, :drop
  end
end

When using our objectified LazyStream, users won't see a nil return for streams that terminate. Because of that, we should add methods for checking if we are at the end of a stream:

module LazyStream
  class Node
    def end?
      @tail.nil?
    end

    def next?
      !end?
    end
  end
end

Let's add a quick LazyStream::Node#show and see how we have done. We will again toss in an alias, an I'm sorry, but the ugly Perl variables have to go:

module LazyStream
  class Node
    def show(*limit_and_options)
      options = {:sep => " ", :end => "\n"}.merge!(
        limit_and_options.last.is_a?(Hash) ? limit_and_options.pop : Hash.new
      )
      limit = limit_and_options.shift

      while head && (limit.nil? or (limit -= 1) > -1)
        print drop, options[:sep]
      end
      print options[:end]
    end
    alias_method :display, :show
  end
end

On with the examples:

#!/usr/local/bin/ruby -w

require "lazy_stream"

def upto(from, to)
  return if from > to
  LazyStream::Node.new(from) { upto(from + 1, to) }
end
upto(3, 6).show  # => 3 4 5 6

def upfrom(start)
  LazyStream::Node.new(start) { upfrom(start + 1) }
end
upfrom(7).show(10)  # => 7 8 9 10 11 12 13 14 15 16

We're making progress. This is starting to feel more like Ruby. Time for me to make good on my promise for a shorter constructor though. We can just add this to the end of lazy_stream.rb:

module Kernel
  def lazy_stream(*args, &block)
    LazyStream::Node.new(*args, &block)
  end
end

That allows us to trim our functional usage to:

#!/usr/local/bin/ruby -w

require "lazy_stream"

def upto(from, to)
  return if from > to
  lazy_stream(from) { upto(from + 1, to) }
end
upto(3, 6).show  # => 3 4 5 6

def upfrom(start)
  lazy_stream(start) { upfrom(start + 1) }
end
upfrom(7).show(10)  # => 7 8 9 10 11 12 13 14 15 16

Even better though, this is Ruby and we can make full objects:

#!/usr/local/bin/ruby -w

require "lazy_stream"

class Upto < LazyStream::Node
  def initialize(from, to)
    if from > to
      super(nil, &nil)
    else
      super(from) { self.class.new(from + 1, to) }
    end
  end
end
Upto.new(3, 6).show  # => 3 4 5 6

class Upfrom < LazyStream::Node
  def initialize(from)
    super(from) { self.class.new(from + 1) }
  end
end
Upfrom.new(7).show(10)  # => 7 8 9 10 11 12 13 14 15 16

Now we can have our interface any way we like it. Some things still need work though…

Inside-out Iteration

So far, I've just copied MJD's external iterator interface. Those are certainly helpful with lazy streams, because you may want to pull values for a while, stop to do something else, and come back to the stream later.

However, Ruby just doesn't feel right without an #each method and Enumerable mixed-in. In this particular case, we really want two forms of iteration, one each for LazyStream::Node#tail and LazyStream::Node#drop so we can advance the stream, or just peek ahead.

It's easier to build the destructive version first. In fact, we already have the implementation coded up inside LazyStream::Node#show. Let's even keep the limit as a handy way to keep from going too deep:

module LazyStream
  class Node
    def each!(limit = nil)
      loop do
        break unless limit.nil? || (limit -= 1) > -1

        yield(drop)

        break if end?
      end

      self
    end
  end
end

Then we can use that to build the standard iterator:

module LazyStream
  class Node
    def each(limit = nil, &block)
      clone.each!(limit, &block)

      self
    end
    alias_method :peek, :each
    include Enumerable
  end
end

That let's us use standard Ruby iteration, for the most part. We can even set a limit to keep from going too far. We can also use methods like Enumerable#find on an infinite stream, since it will stop as soon as it finds a match.

Where this system has trouble is if we want to use something like Enumerable#map on an infinite stream. The problem here is that we have no good way to pass the limit down. It's also tricky to use LazyStream::Node#each! with the Enumerable methods. Let's fix both issues:

require "enumerator"

module LazyStream
  class Node
    def limit(max_depth = nil)
      enum_for(:each, max_depth)
    end

    def limit!(max_depth = nil)
      enum_for(:each!, max_depth)
    end
  end
end

There's a little black magic there we need to see in action, but before we go back to examples we can simplify LazyStream::Node#show to use the new iterators:

module LazyStream
  class Node
    def show(*limit_and_options)
      options = {:sep => " ", :end => "\n"}.merge!(
        limit_and_options.last.is_a?(Hash) ? limit_and_options.pop : Hash.new
      )
      limit = limit_and_options.shift

      each(limit) { |cur| print cur, options[:sep] }
      print options[:end]
    end
    alias_method :display, :show
  end
end

Okay, let's see what we have created. Here's an infinite stream iterator, using Enumerable#map:

#!/usr/local/bin/ruby -w

require "lazy_stream"

class Step < LazyStream::Node
  def initialize(step, start = 1)
    super(start) { self.class.new(step, start + 1) }

    @step = step
  end

  def next_group(count = 10)
    limit!(count).map { |i| i * @step }
  end
end

evens = Step.new(2)

puts "The first ten even numbers are:"
puts evens.next_group.join(" ")  # => 2 4 6 8 10 12 14 16 18 20

# later...

puts
puts "The next ten even numbers are:"
puts evens.next_group.join(" ")  # => 22 24 26 28 30 32 34 36 38 40

puts
puts "The current index for future calculations is:"
puts evens.current # => 21

That feels a lot more like Ruby to me. Now we can get back to MJD's examples.

Adding `#filter` and `#transform`

MJD adds both a #filter and #transform method next. Technically, my last example is a transformation of the stream and we could do filtering the same way. Still it's nice to have these built-in and we should add them.

The #filter is easy enough, we just have to keep dropping items until we find a match and make sure setting a filter is viral to future nodes:

module LazyStream
  class Node
    def tail
      result = if @tail.is_a?(Proc)
        @tail.call
      else
        @tail
      end

      result.filter(@filter) unless @filter.nil? || !result.is_a?(self.class)

      result
    end

    def filter(pattern = nil, &block)
      @filter = pattern || block

      drop until matches_filter?(@head)

      self
    end

    private

    def matches_filter?(current)
      case @filter
      when nil
        true
      when Proc
        @filter[current]
      else
        @filter === current
      end
    end
  end
end

We can add #transform by defining an actual getter for @head, instead of the attr_reader shortcut I've been using and, again, making the setting viral:

module LazyStream
  class Node
    def head
      @transformer.nil? ? @head : @transformer[@head]
    end

    def tail
      result = if @tail.is_a?(Proc)
        @tail.call
      else
        @tail
      end

      result.filter(@filter)          unless @filter.nil? ||
                                      !result.is_a?(self.class)
      result.transform(&@transformer) unless @transformer.nil? ||
                                      !result.is_a?(self.class)

      result
    end

    def transform(&transformer)
      @transformer = transformer

      self
    end
  end
end

Here are trivial examples with both:

#!/usr/local/bin/ruby -w

require "lazy_stream"

def letters(letter)
  lazy_stream(letter) { letters(letter.succ) }
end

letters("a").filter(/[aeiou]/).show(10)           # => a e i o u aa ab ac ad ae
letters("a").filter { |l| l.size == 2 }.show(3)   # => aa ab ac 

letters("a").transform { |l| l + "..." }.show(3)  # => a... b... c...

Recursive Streams

I think we can fix the recursive issues MJD discusses with ease in our Ruby version, just by passing the stream head to the promise block:

module LazyStream
  class Node
    def tail
      result = if @tail.is_a?(Proc)
        @tail.call(head)
      else
        @tail
      end

      unless result == @tail
        result.filter(@filter)          unless @filter.nil? ||
                                        !result.is_a?(self.class)
        result.transform(&@transformer) unless @transformer.nil? ||
                                        !result.is_a?(self.class)
      end

      @tail = result
    end
  end
end

With that, MJD's example is trivial:

#!/usr/local/bin/ruby -w

require "lazy_stream"

class Powers < LazyStream::Node
  def initialize(of, start = 1)
    super(start) { |last| self.class.new(of, last * of) }
  end
end

powers_of_two = Powers.new(2)
powers_of_two.show(10)  # => 1 2 4 8 16 32 64 128 256 512

In all honesty though, you seldom need that with the blocks being closures. You can just reference the local variables instead.

You also don't need merging to solve the Hamming Sequence. Here's a simple example:

#!/usr/local/bin/ruby -w

require "lazy_stream"

class Hamming < LazyStream::Node
  def initialize(seq = [1])
    cur = seq.shift
    super(cur) do 
      self.class.new((seq << cur * 2 << cur * 3 << cur * 5).uniq.sort)
    end
  end
end
Hamming.new.show(20)  # => 1 2 3 4 5 6 8 9 10 12 15 16 18 20 24

Regexp String Generation

We don't need too many modifications to be able to create this example. First, let's build a LazyStream::union:

module LazyStream
  def self.union(*streams)
    current = streams.shift or return
    Node.new(current.head) { union(*(streams << current.tail)) }
  end
end

We can even add a little syntax sugar for calling that:

module LazyStream
  class Node
    def +(other_stream)
      LazyStream.union(self, other_stream)
    end
  end
end

Finally, because I made LazyStream::Node#transform modify the stream, we will need a LazyStream::Node#dup to build MJD's #concat method:

module LazyStream
  class Node
    def dup
      if tail.nil? || tail.is_a?(Proc)
        self.class.new(head, &tail)
      else
        self.class.new(head) { tail }
      end
    end
  end
end

From there we can build a RegexpMatchesGenerator:

#!/usr/local/bin/ruby -w

require "lazy_stream"

module RegexpMatchesGenerator
  extend self

  def literal(string)
    lazy_stream(string)
  end

  def concat(stream1, stream2)
    lazy_stream(stream1.head + stream2.head) do
      combinations = Array.new
      unless stream1.end?
        combinations << stream1.tail.dup.transform { |str| str + stream2.head }
      end
      unless stream2.end?
        combinations << stream2.tail.dup.transform { |str| stream1.head + str }
      end
      unless stream1.end? || stream2.end?
        combinations << concat(stream1.tail, stream2.tail)
      end
      LazyStream.union(*combinations)
    end
  end

  def star(stream, head = "")
    lazy_stream(head) { star(stream, head + stream.head) }
  end

  def char_class(string)
    LazyStream.union(*string.split("").map { |str| literal(str) })
  end

  def plus(stream)
    star(stream, stream.head)
  end
end

if __FILE__ == $0
  include RegexpMatchesGenerator

  # /^(a|b)(c|d)$/
  concat(literal("a") + literal("b"), literal("c") + literal("d")).show

  # /^(HONK)*$/
  puts star(literal("HONK")).limit(6).map { |str| str.inspect }.join(" ")

  # /^ab*$/
  puts(concat(literal("a"), star(literal("b"))).limit(10).to_a)
end

The Rest

That should be enough example translation to address most of the highlights from this chapter. All those math problems at the end make me drowsy, so I'll leave those as an exercise for the reader…

Pathname and Enumerator

2014-03-28T13:52:33Z

I'm always digging around in Ruby's standard library looking for new toys. It's one-stop-shopping for geeks. I love it.

There really are some great tools in there that can help you do your work easier. Let me tell you a little about two I am using on a current application.

Pathname

Pop Quiz: Which would you rather work with?

if File.exist? some_dir
  data = File.read( File.join( File.dirname(some_dir),
                               "another_dir",
                               File.basename(some_file) ) )
  # ...
end

Or:

if some_dir.exist?
  data = (some_dir.dirname + "another_dir" + some_file.basename).read
  # ...
end

If you prefer the second version, the Pathname library is for you.

Ruby's file manipulations are usually fine for normal work, but anytime I start messing with directories I really start feeling the pain. For that kind of work, I find Pathname to be a superior interface. My current project makes great use of the methods in the above example as well as Pathname#entries, Pathname#relative_path_from, and more.

Look up the documentation if the above has you curious and you won't be sorry. You're only one require away from a smooth directory interface.

Enumerator

Pop Quiz: Ever wish Ruby's Strings iterated over characters instead of numbers?

unless "TEAM".enum_for(:each_byte).find { |c| c == ?I }
  puts "There's no I in T-E-A-M!"
end

Pop Quiz: Ever needed map_with_index()?

table_rows.enum_for(:each_with_index).map do |row, i|
  [row, i % 2 == 0 ? :even : :odd]
end

The magic enum_for() returns an Enumerator, which is a standard Enumerable object using the passed method as each(). This gives you the power of all the other iterators (including find() and map()), but on the data of your choosing.

This is so handy, Enumerator has been moved from the standard library to the language core, as of Ruby 1.9. Not only does this allow you to drop the require statement, the core version of Enumerator has been enhanced. All of the standard iterators will now return an Enumerator, if called without a block. That makes map_with_index() even easier to create:

table_rows.each_with_index.map do |row, i|
  # ...
end

As an added bonus, the Enumerator library adds two more handy iterators to Enumerable:

tic_tac_toe = "XO  X  OX".split("")
tic_tac_toe.each_slice(3) do |row|
  puts " " + row.join(" | ")
  puts "-" * 11 unless row == tic_tac_toe[-3..-1]
end

As you can see, each_slice() let's you grab a passed number of elements at a time. The other method, each_cons(), is similar, but will keep consecutive elements together:

"by James Gray".split.each_cons(2) do |words|
  if words.all? { |word| word =~ /^[A-Z]/ }
    puts "#{words.join(' ')} is an author."
  end
end

Note that each_slice() wouldn't have found my name, since it would have tried ["by", "James"] and ["Gray"], but not ["James", "Gray"].

Again, see the documentation for more details. Advanced Enumerator usage is Ruby Voodoo and it will help you impress your friends and coworkers.