Rubies in the Rough

Dreamy Testing (Part 2)

In Part 1 of this article, I began building out my ideal testing interface, or at least my best attempt at such a thing.

In that article, I worked primarily on the "assertion" interface: a file full of calls to ok() with a block that returns true or false to pass or fail tests. I also built some standard test printers to show us familiar output.

As I wrapped up, I was running this code in example/basic_test.rb:

ok("Is true")  { true        }
ok("Is false") { false       }
ok("Is error") { fail "Oops" }

and seeing these results:

$ ruby -I lib -r ok example/basic_test.rb 
Running tests:
.FE

0) Failure: Is false
  example/basic_test.rb:2:in `<main>'
1) Error: Is error
  example/basic_test.rb:3:in `block in <main>'
  example/basic_test.rb:3:in `<main>'

Finished tests in 0.000300s
3 tests, 1 failure, 1 error

Of course, there was still a lot missing in my code. Let's work on adding some of the other must have features and perhaps a nicety or two.

Running Tests

In the first article, I spent a lot of time talking about how all of the references to things other than my code in tests are a distraction. I wanted to remove as much of that as possible. We have done pretty well on that front.

The context of a subclass or describe() block are gone. That doesn't have to mean that we don't have contexts any more, as we will see later. It just means we don't have to explicitly declare them.

Also, I haven't been using require() statements. I'm currently getting around that with command-line switches and we know that can't last, but it does tell us that it should be reasonable to roll without them, if we think it through correctly.

Special methods with a test_ prefix (or calls to it()) have been replaced by ok(). They aren't gone. But, we dramatically simplified the idea of assertions, so I say we can count this as at least a partial win.

Realistically, are we going to get rid of test/ (or spec/) directories or files names with suffixes like _test.rb (or _spec.rb)? Probably not. A directory just makes sense to have. We need somewhere for tests to live. It also makes sense to test a file like printer.rb with printer_test.rb for multiple reasons:

• The file tells you what it tests
• Having a different file name is nice in stack traces

To put it another way, these are conventions of testing and if we can't beat them, we should join them. That's the "Convention Over Configuration" idea from Rails, right?

Given all of the above, I'm wanting to build an executable. It can hunt for my tests for me since we know they are probably in test/ (or spec/). If nothing else, they should have a _test.rb (or _spec.rb) suffix.

Of course, I like my executables to be all but empty. They should just load some code and kick-off a method. This makes everything easier to test. An executable is just another interface to a method. That means I can write it and forget about it.

So, we need some global thing that serves as our entry point interface. The word global suggests a global variable, but we aren't suppose to use those. Don't worry, object oriented systems still have global variables, they are just renamed to singletons classes (I mean the pattern, not the special Ruby construct). Someone is surely mad at me by now, for speaking so poorly of this pattern, but I'm afraid it gets worse. How we even make a singleton class is debatable in Ruby. I don't want to go too far down this rabbit hole, so I'll leave my position as this:

• Some state is meant to be global
• I think this is one of those cases
• module_function isn't all bad and I think it's the right tool for this job

Therefore, I added this code in lib/ok/tester.rb:

module OK
  module Tester
    module_function

    def test(paths)
      paths = Array(paths)
      paths = [%w[spec test .].find { |path| File.exist? path }] \
        if paths.empty?

      tests = [ ]
      while (path = paths.shift)
        if File.directory?(path)
          new_paths = Dir.entries(path)
                         .reject { |entry| entry =~ /\A\./        }
                         .map    { |entry| File.join(path, entry) }
          paths.unshift(*new_paths)
        elsif File.basename(path) =~ /\A.+_(?:spec|test)\.rb\z/
          tests << path
        end
      end

      tests.each do |test|
        load test
      end
    end
  end
end

The first chunk of the method above makes sure we have a list of things to search for tests, even defaulting the content of that list if it was empty. The middle chuck then walks that list recursively, hunting for test files. Those files are added to an Array and all sequentially invoked at the end of the method.

Then I added this code as bin/ok:

#!/usr/bin/env ruby

require_relative "../lib/ok"

OK::Tester.test(ARGV)

Like I said, just load and call. I also made that file executable with a quick:

chmod +x bin/ok

With that code in place, we now have a way to run the examples I have been creating:

$ bin/ok example
example/basic_test.rb
  Is true
  Failed: Is false
    bin/ok:6:in `<main>'
    example/basic_test.rb:2:in `<top (required)>'
    example/basic_test.rb:2:in `<top (required)>'
  Error: Is error
    example/basic_test.rb:3:in `block in <top (required)>'
    example/basic_test.rb:3:in `<top (required)>'
    bin/ok:6:in `<main>'
example/compound_test.rb
  Is compound
example/setup_test.rb
  Is using setup
  Is also using setup

Finished specs in 0.000878s
6 specs, 1 failure, 1 error

Or we could run a specific file:

$ bin/ok example/compound_test.rb
example/compound_test.rb
  Is compound

Finished specs in 0.000210s
1 spec, 0 failures, 0 errors

Providing no argument at all defaults it to the test/ directory I created in the last article, but that's still empty and not worth showing.

Now this may be hard to believe, but that simple test runner just solved a lot of our problems. Before, I was relying on some loose code stuck in one file and a clumsy at_exit() hook to control the flow of execution. But now we have an utterly straightforward place to deal with such things. We're back in control of this cycle.

Tying in the Printer

Remember the code that setup printing from the last article? It looked like this:

$printer = OK::Printer::SpecPrinter.new
$printer.print_running
at_exit { $printer.print_summary }

In addition to the awkward flow control I mentioned before, this needs a global variable and has a hard-coded printer. Let's clean that whole mess up with one more pass refining the Tester:

module OK
  module Tester
    module_function

    def printer=(type_or_printer)
      @printer = case type_or_printer.to_s
                 when *%w[dot test] then OK::Printer::DotPrinter.new
                 when "spec"        then OK::Printer::SpecPrinter.new
                 else                    type_or_printer
                 end
    end

    def printer
      @printer ||= ENV["RUBY_OK_PRINTER"] =~ /\ASpec\z/i ?
                   OK::Printer::SpecPrinter.new          :
                   OK::Printer::DotPrinter.new
    end

    def warnings=(boolean)
      @warnings = boolean
    end

    def warnings
      @warnings ||= ENV["RUBY_OK_NO_WARNINGS"] ? false : true
    end

    def test(paths)
      # ... same as before ...

      tests = [ ]
      while (path = paths.shift)
        if File.directory?(path)
          # ... same as before ...
        elsif File.basename(path) =~ /\A.+_(spec|test)\.rb\z/
          self.printer  = :spec if not defined?(@printer) and $1 == "spec"
          tests        << path
        end
      end

      Object.send(:include, OK::DSL) unless Object < OK::DSL
      $VERBOSE = true if warnings
      printer.print_running
      tests.each do |test|
        load test
      end
      printer.print_summary
    end
  end
end

The printer=() method just gives us a few shortcuts for naming printers. The Reader defaults the printer as needed and allows that default to be changed by an environment variable.

The warnings=() and warnings() methods produce a similar interface for another setting: whether or not to run your tests with warnings turned on. I believe you should. I realize I could have used attr_reader() here, but I thought the code symmetry came out better with the long hand form of the method.

The changes to test() accomplish several things:

• Switch to a spec printer if the test files look like specs
• Prep a test run by installing our DSL (I made that up!)
• Turn on warnings if desired
• Call the Printer hooks at the correct times

The mention of OK::DSL above was more programming by wishful thinking, so it's time to make those wishes come true. I added this code in a file called lib/ok/dsl.rb:

module OK
  module DSL
    def ok(name, &test)
      OK::Tester.printer.print_test(OK::Test.new(name, test))
    end
  end
end

You've seen this method before. The only change is that it now fetches the Printer from the Tester.

Those changes allow us to remove a lot of junk from the original lib/ok.rb, like the clunky printing code and the global method definition. It's now just a list of require statements:

require_relative "ok/printer"
require_relative "ok/printer/dot_printer"
require_relative "ok/printer/spec_printer"
require_relative "ok/test"
require_relative "ok/dsl"
require_relative "ok/tester"

Notice how the code is cleaning itself as we add this feature? That's how we know we are on the right track!

A Trick of Interfacing

We've really rounded out our test framework. We have a defined way to run tests, our minimal DSL for writing tests, and multiple printers that can be swapped out.

If you want to change the printer (or the warnings setting), you can set the environment variable RUBY_OK_PRINTER (or RUBY_OK_NO_WARNINGS for the latter). Of course, we could also modify them programmatically in something like test_helper.rb (or spec_helper.rb) with code like:

OK::Tester.printer = :spec

I'll be honest though, I'm not in love with that. It takes us out of the pretty DSL. It's like when they shatter your suspension of disbelief at the movies. Not cool.

Of course, I don't want to stick a ton of methods on object to handle all of the settings we will someday have. I love how ok() is all the interface we have needed to write tests and I'm not ready to give that up.

The answer? Let's teach ok() a new trick:

module OK
  module DSL
    def ok(name = nil, &test)
      OK::Tester.printer.print_test(OK::Test.new(name, test)) if name
      OK::Tester
    end
  end
end

I've changed the code to only create a test if one was given (by name). It also returns the Tester to the caller, for manipulation.

Beauty is restored. We can now set a printer with:

ok.printer = :spec

I really like the look of that. It opens the door for any configuration we need going forward, but doesn't really complicate our interface.

Ready for Some Extras

My new testing framework has a long way to go if it's going to someday catch the powerful tools that have been around a while, like RSpec. They have a lot of nice features. Obviously, I can't build all of those right now, but let's do one as an example.

Everyone love's RSpec's let() feature. Can we make that happen? I sat down to code it up and finally ran into the bad idea I've been waiting to share with you.

At first, I thought I could define variables with some code like:

@test.binding.eval("#{var} = ...")

That code runs, but those variables don't end up being in scope when the block is finally called. In the end, I had to resort to a horrific use of method_missing() to fake the variable.

This gets pretty ugly, so I'll show the changes in chunks. First, I had to give myself a way to add let() definitions, so I added these methods to lib/ok/test.rb:

module OK
  class Test
    def self.let_definitions
      @let_definitions ||= Hash.new { |paths, path| paths[path] = { } }
    end

    def self.let_definition(path, name, initializer)
      scope = if File.basename(path) =~ /\A(?:spec|test)_helper\.rb\z/
                File.dirname(path)
              else
                path
              end
      let_definitions[scope][name] = initializer
    end
  end
end

I defined let_definitions() as a Hash of initializer blocks stored by the path they affect and the variable name. Adding definitions just has one twist, I scope anything in the previously mentioned test_helper.rb (or spec_helper.rb) file to the directory that contains it. This allows you to put definitions in there that affect all test files below that point.

I'll be honest and say that I'm not sure putting let() calls in test_helper.rb (or spec_helper.rb) is even a good idea. The point was just to show that we should also support the other convention we identified in any way that we can.

Now I needed the other side of the equation, a way to call those definitions as they apply and cache the value for the rest of the test run, so I added these methods as well:

module OK
  class Test
    def self.let_values
      @let_values ||= { }
    end

    def self.let_value(*path_and_name)
      return let_values[path_and_name] if let_values.include?(path_and_name)

      scope = path_and_name.first
      loop do
        if (initializer = OK::Test.let_definitions[scope][path_and_name.last])
          let_values[path_and_name] = initializer.call
          return let_values[path_and_name]
        end
        break if %w[. /].include?(scope)
        scope = File.split(scope).first
      end

      yield
    end
  end
end

Again, let_values() are hashed by their path and name. If a value is already set, I just reuse it. Otherwise, I need to see if there is an initializer that can build it. I search for those up the path. Believe it or not, that allows for nested contexts. Directories higher up can setup all that follows beneath them.

The final yield allows calling code to decide how to handle the error case where no initializer could be found. This is another great strategy I learned from Exceptional Ruby.

I also had to rework the actual execution of tests, so I could clear any cached values before each test is run. Here's the code for that:

module OK
  class Test
    def initialize(name, test)
      @name   = name
      @test   = test
      @result = run_test
    end

    private

    def run_test
      self.class.let_values.clear
      !!@test.call
    rescue Exception => error
      error
    end
  end
end

The only new element here, is the added call to clear(). I just pushed the code down into a method the keep it focused on one job in each place: initialization and running tests.

With Test supporting the feature, it's easy to add an interface for setting the values. I added this method to lib/ok/tester.rb:

module OK
  module Tester
    module_function

    def let(name, &initializer)
      OK::Test.let_definition(caller.first[/\A[^:]+/], name, initializer)
    end
  end
end

This wrapper just hands off to the code we already wrote. It exists to keep our ok() interface being the one place to set things.

I also tweaked the test runner to load test_helper.rb (or spec_helper.rb) files. This will look like a lot of code, but you've seen most of it before. I just restructured it a bit:

module OK
  module Tester
    module_function

    def test(paths)
      # ... same as before ...

      Object.send(:include, OK::DSL) unless Object < OK::DSL
      $VERBOSE = true if warnings

      tests = [ ]
      while (path = paths.shift)
        if File.directory?(path)
          new_paths = Dir.entries(path)
                         .reject  { |entry| entry =~ /\A\./               }
                         .map     { |entry| File.join(path, entry)        }
                         .sort_by { |child| [ File.basename(child, ".rb"),
                                              File.file?(child) ? 0 : 1 ] }
          paths.unshift(*new_paths)
        elsif File.basename(path) =~ /\A.+_(spec|test)\.rb\z/
          # ... same as before ...
          directory     = File.dirname(path)
          if File.exist?(helper = File.join(directory, "spec_helper.rb")) ||
             File.exist?(helper = File.join(directory, "test_helper.rb"))
            require File.expand_path(helper)
          end
        end
      end

      # ... same as before ...
    end
  end
end

Here are the changes:

• The DSL loading and warning setting code isn't new, it just moved up so it would be loaded before any helper code
• Directory entries are sorted to best load nested contexts
• When a test file is found, we look for a matching helper file to require (remember that require only loads a file once)

Now for the major sin required to support this feature. I added this evil code to lib/ok/dsl.rb:

module OK
  module DSL
    def method_missing(method, *args, &block)
      super if method == :to_path  # GROSS!!!
      OK::Test.let_value(caller.first[/\A[^:]+/], method) { super }
    end
  end
end

This method pulls the variables from the values Hash as needed, but it has two significant flaws:

• As part of our DSL, this gets mixed into Object and affects all code!
• I had to add a :to_path guard because my Hash lookup triggers it!

Yuck and double yuck. This is just not safe. Eventually, this hack is going to break some code. Heck, I had to work hard not to break Ruby itself! I was blowing the stack before I figured out the :to_path trick.

To test my work, I created an example/let_test.rb and stuck this in it:

ok.let(:forty_two) { 42 }

ok("Is using let()") { forty_two == 42 }

All of that was for this: initialize a value and use it latter. That's nice, but we did the same thing with a local variable earlier.

That works as you would expect, but I hope I've shown that it just wasn't a good idea. The code was fighting me at every step of the implementation. You have to learn to listen to that. It was telling me, "Don't do this." If this had been a real project, it would be time for a git reset --hard. Then you take a step back and rethink what happened.

What Did I Learn?

The point of this whole exercise was to learn something. I did. A lot actually. I learned:

• It wasn't tough at all to improve assertion syntax, in my opinion
• It was also easy to come up with a reasonable test running framework
• That let() didn't work well with the system I built
• That let() failed because it needs one of those contexts I ripped out
• This idea was too long and complex to make a good article (Sorry about that!)

There may be ways to get that missing context back and make supporting let() easier. For example, load() can wrap the loaded code in an anonymous module. I could then patch that module with methods as needed and avoid the painful method_missing() hack.

Interestingly, before(:each) is similar, but much easier to support even if we don't use anonymous modules. You can use similar code to the let() definition code, skip the value stuff, and execute them with an easy loop. No hacks are required.

Maybe this just means that let() doesn't fit here. Honestly, I'm not totally convinced that it is needed. We may be able to use local variables in the file, if the value doesn't change. Even changing values are pretty easy to handle with a small method.

I'll need to play with this code for a while to finish learning all that it has to teach me about interfaces.

Broken Rules

In this Breaking All of the Rules miniseries of articles I've been pretty critical of a lot of the rules we live by in software. I did that because of what the exercise can teach us. I don't really want us to live in a world without rules.

The key is to keep thinking critically. Get in the habit of asking yourself why a rule exists and if it applies to you in the current situation. There's no way raised awareness won't make you a stronger programmer. I promise.