Gray Soft

The programming blog of James Edward Gray II (JEG2).
  • 16


    Lists and Sets in Redis

    [Update: though all of the techniques I show here still apply, many methods of the Redis gem have changed names to match the actual Redis commands they call. There are also easier and more powerful ways to do some of what I show in here, thanks to additions to Redis.]

    Redis adds one huge twist to traditional key-value storage: collections. Supporting both lists and sets through some very powerful atomic operations allows for advanced key-value usage.


    Redis allows a single key to hold a list of values. This is your typical ordered list with the operations you would expect: appending, indexed access, and access to a range of values.

    This has many potential uses. I'll cover two that I think will be very common. First, if you are going to use Redis as a full database, you store things that are naturally a list of items, like comments, in a real list. Let's look at some code:

    #!/usr/bin/env ruby -wKU
    require "redis"
    CLEAR = `clear`
    # create an article to comment on
    db                =
    article_id        = db.incr("global:next_article_id")
    article           = "article:#{article_id}"
    class << article
      def method_missing(field, *args, &blk)
        return super unless field.to_s !~ /[!?=]\z/ && args.empty? && blk.nil?
    db[article.title] = "My Favorite Language"
    db[article.body]  = "I love Ruby!"
    # initialize some session details
    comments_per_page = 2
    comment_page      = 1
    login             = ARGV.shift || "JEG2"
    loop do
      # show article
      print CLEAR
      puts "#{db[article.title]}:"
      puts "  #{db[article.body]}"
      # paginate comments
      start      =  comments_per_page * (comment_page - 1)
      finish     =  start + comments_per_page - 1
      comments   =  db.list_range(article.comments, start, finish)
      pagination =  Array( ? nil : "(p)revious")
      pagination << "(n)ext" if db.list_length(article.comments) - 1 > finish
      # show comments
      comments.each do |comment|
        posted, user, body = comment.split("|", 3)
        puts "----"
        puts "  #{body}"
        puts "  posted by #{user} on #{posted}"
      # handle commands
      print "Command? [#{(%w[(c)omment (q)uit] + pagination).join(', ')}]  "
      case (command = gets)
      when /\Ac(?:omment)?\Z/i  # add a comment
        print "Your comment?  "
        comment = gets or break
        posted  ='%m/%d/%Y at %H:%M:%S')
        db.push_tail(article.comments, "#{posted}|#{login}|#{comment.strip}")
      when /\Ap(?:revious)?\Z/i  # view previous page of comments
        if pagination.first =~ /\A\(p\)/
          comment_page -= 1
          puts "You are on the first page of comments."
          gets or break
      when /\An(?:ext)?\Z/i  # view next page of comments
        if pagination.last =~ /\A\(n\)/
          comment_page += 1
          puts "You are on the last page of comments."
          gets or break
      when /\Aq(?:uit)?\Z/i, nil  # exit program

    Read more…

  • 15


    Using Redis as a Key-Value Store

    [Update: though all of the techniques I show here still apply, many methods of the Redis gem have changed names to match the actual Redis commands they call.]

    Redis is a first and foremost a server providing key-value storage. As such, the primary features of any client library are for connecting to the server and manipulating those key-value pairs.

    Connecting to the Server

    Connecting to the Redis server can be as simple as, thanks to some defaults in both the server and Ezra's Ruby client library for talking to that server. I won't pass any options to the constructor calls below, but you can use any of the following as needed:

    • :host if you need to connect to an external host instead of the default
    • :port if you need to use something other than the default port of 6379
    • :password if you configured Redis to require a password on connection
    • :db if you want to select one of the multiple configured databases, other than the default of 0 (databases are identified by a zero-based index)
    • :timeoeut if you want a different timeout for Redis communication than the default of 5 seconds
    • :logger if you want the library to log activity as it works

    Read more…

  • 14


    Setting up the Redis Server

    Before we can play with Redis, you will need to get the server running locally. Luckily, that's very easy.

    Installing Redis

    Building Redis is a simple matter of grabbing the code and compiling it. Once built, you can place the executables in a convenient location in your PATH. On my box, I can do all of that with these commands:

    curl -O
    tar xzvf redis-1.0.tar.gz 
    cd redis-1.0
    sudo cp redis-server redis-cli redis-benchmark /usr/local/bin

    Those commands build version 1.0 of the server, which is the current stable release as of this writing. You may need to adjust the version numbers down the road to get the latest releases though.

    I also copied the executables to where I prefer to have them: /usr/local/bin. Feel free to change that directory in the last command to whatever you prefer.

    If you will be talking to Redis from Ruby, as I will show in all of my examples, you are going to need a client library. I recommend Ezra Zygmuntowicz's redis-rb. You can install that gem with:

    Read more…

  • 14


    Using Key-Value Stores From Ruby

    I've been playing with a few different key-value stores recently. My choices are pretty popular and you can find documentation for them. However, it can still be a bit of work to relate everything to Ruby specific usage, which is what I care about. Given that, here are my notes on the systems I've used.


    1. Setting up the Redis Server
    2. Using Redis as a Key-Value Store
    3. Lists and Sets in Redis
    4. Where Redis is a Good Fit

    Tokyo Cabinet, Tokyo Tyrant, and Tokyo Dystopia

    1. Installing the Tokyo Software
    2. Tokyo Cabinet as a Key-Value Store
    3. Tokyo Cabinet's Key-Value Database Types
    4. Tokyo Cabinet's Tables
    5. Threads and Multiprocessing With Tokyo Cabinet
    6. Tokyo Tyrant as a Network Interface
    7. The Strengths of Tokyo Cabinet
  • 5


    eval() Isn't Quite Pure Evil

    I was explaining method lookup to the Lone Star Ruby Conference attendees and needed to show some trivial code like this:

    module B
      def call
        puts "B"
    module C
      def call
        puts "C"
    module D
      def call
        puts "D"

    Unfortunately, my space was limited due to the code being on slides and me needing to show more than just what you see above. I cracked under the pressure and committed two major programming sins. First, I collapsed the code with eval():

    %w[B C D].each do |name|
      eval <<-END_RUBY
      module #{name}
        def call
          puts "#{name}"

    Then I really blew it when I jokingly apologized to the audience for using eval().

    I got the first email with corrected code before I even finished the speech. OK, the email was from a friend and he wasn't mean, but he still instantly needed to set me straight. I had clearly turned from the light.

    Only, his corrected code didn't quite work. It got close, but it had bugs. Still the approach was sound and it could be made to work. Let me fix the bugs and show you what was recommended:

    Read more…

  • 11


    Load an EC2 GUI on Your Mac OS X Box

    Using straight shell access on EC2 servers works just fine, of course, but there are images available that include the full desktop environment. If you use one of those, you can activate GUI programs on the EC2 server. Now you can't plug a monitor into your EC2 instance, so you will need to tunnel the GUI bits down to your local box. Luckily, Unix pretty much just handles all of this for you and if you can SSH into an EC2 instance and you have Apple's X11 installed, you are all set to try this.

    I know at one time Apple's X11 environment was an optional install. I can't remember if it still is, but you can see if you have it by looking for a program called X11 in /Applications/Utilities/. Or, just fire up a Terminal and enter the command xterm. If a rather plain white shell window eventually appears, you are in business. If you do need to add the optional install, it should be on the disk that came with your computer (or the OS install disks).

    Once you've confirmed X11 is ready, you need to fire up an EC2 instance. Create a Key Pair and configure a Security Group to at least allow SSH access. Then just launch an EC2 instance (I use the AWS Managment Console for that) with the needed desktop environment tools. I chose one of the wonderful Ubuntu Images by Eric Hammond.

    Read more…

  • 2


    Getting Ready for Ruby 1.9

    We've all been waiting for Ruby 1.9 to reach maturity for some time now. We've complained about things like Ruby's speed and weak character encoding support. We knew 1.9 could improve things, but it brings pretty big changes and a lot of Ruby 1.8 code needs updating before it can really be used there. For these and other reasons, the official production release came and went while most of us have stuck with 1.8 for our day to day needs.

    I think we're reaching the tipping point though. Rails runs on 1.9 now and many other libraries are beginning to become compatible. We may not yet have the full range of 1.8 goodies available on the new platform, but many of the staples are moving over and it's looking like we can now do some serious work there.

    Which means it's finally time for us to learn this 1.9 stuff.

    There are several good sources of Ruby 1.9 information now, so you have choices. I'm going to tell you about three I like. Be warned, this is a super biased list, but I really hope it will be helpful to others.

    Read more…

  • 18


    What Ruby 1.9 Gives Us

    In this final post of the series, I want to revisit our earlier discussion on encoding strategies. Ruby 1.9 adds a lot of power to the handling of character encodings as you have now seen. We should talk a little about how that can change the game.

    UTF-8 is Still King

    The most important thing to take note of is what hasn't changed with Ruby 1.9. I said a good while back that the best Encoding for general use is UTF-8. That's still very true.

    I still strongly recommend that we favor UTF-8 as the one-size-almost-fits-all Encoding. I really believe that we can and should use it exclusively inside our code, transcode data to it on the way in, and transcode output when we absolutely must. The more of us that do this, the better things will get.

    As we've discussed earlier in the series, Ruby 1.9 does add some new features that help our UTF-8 only strategies. For example, you could use things like the Encoding command-line switches (-E and -U) to setup auto translation for all input you read. These shortcuts are great for simple scripting, but I'm going to recommend you just be explicit about your Encodings in any serious code.

    Read more…

  • 15


    Miscellaneous M17n Details

    We've now discussed the core of Ruby 1.9's m17n (multilingualization) engine. String and IO are where you will see the big changes. The new m17n system is a big beast though with a lot of little details. Let's talk a little about some side topics that also relate to how we work with character encodings in Ruby 1.9.

    More Features of the Encoding Class

    You've seen me using Encoding objects all over the place in my explanations of m17n, but we haven't talked much about them. They are very simple, mainly just being a named representation of each Encoding inside Ruby. As such, Encoding is a storage place for some tools you may find handy when working with them.

    First, you can receive a list() of all Encoding objects Ruby has loaded in the form of an Array:

    $ ruby -e 'puts Encoding.list.first(3), "..."'

    If you're just interested in a specific Encoding, you can find() it by name:

    $ ruby -e 'p Encoding.find("UTF-8")'
    $ ruby -e 'p Encoding.find("No-Such-Encoding")'
    -e:1:in `find': unknown encoding name - No-Such-Encoding (ArgumentError)
        from -e:1:in `<main>'

    Read more…

  • 5


    Ruby 1.9's Three Default Encodings

    I suspect early contact with the new m17n (multilingualization) engine is going to come to Rubyists in the form of this error message:

    invalid multibyte char (US-ASCII)

    Ruby 1.8 didn't care what you stuck in a random String literal, but 1.9 is a touch pickier. I think you'll see that the change is for the better, but we do need to spend some time learning to play by Ruby's new rules.

    That takes us to the first of Ruby's three default Encodings.

    The Source Encoding

    In Ruby's new grown up world of all encoded data, each and every String needs an Encoding. That means an Encoding must be selected for a String as soon as it is created. One way that a String can be created is for Ruby to execute some code with a String literal in it, like this:

    str = "A new String"

    That's a pretty simple String, but what if I use a literal like the following instead?

    str = "Résumé"

    What Encoding is that in? That fundamental question is probably the main reason we all struggle a bit with character encodings. You can't tell just from looking at that data what Encoding it is in. Now, if I showed you the bytes you may be able to make an educated guess, but the data just isn't wearing an Encoding name tag.

    Read more…