Terminal Tricks

A Curses Application

I've now written two articles covering low-level curses and some higher abstractions. We know what curses can do at this point, but we haven't really seen how to put everything together and build a full application. Let's do that today by examining an example of moderate size.

I have written the beginnings of a command-line Twitter client using curses. To make this easier, I developed a super simple wrapper over the raw curses API, called Rurses. Rurses, for Ruby curses, provides more Ruby-like abstractions over the clunky C API. Here's how the Twitter client looks in action:

┌─JEG2─────────────────────────────────┐  @
│                                     ↑│
│Alex Harms @onealexh… 20/03/2015 14:57│ Clayton Flesher @Cal… 19/03/2015 20:52
│RT @postsecret: http://t.co/LrV0IIYgUM│ @TrevorBramble its inspired by, as in
│                                      │ I played it once for about ten minutes
│SimKlabnik 2000 @ste… 20/03/2015 14:57│ and then @JEG2 said 'go make your
│the closure docs are finally flowing  │ version of that'.
│from my fingertips, it seems          │
│                                      │ Mandy Moore @theruby… 19/03/2015 18:31
│ashe dryden @ashedry… 20/03/2015 14:57│ Thanks to @JEG2 I now want a MiP robot
│Can anyone recommend  an interestingly│ in the worst kind of way!!!
│written (not dry) history of Haiti?   │
│Preferably written by a Haitian.      │ Sam Livingston-Gray … 19/03/2015 14:23
│                                      │ @avdi @JEG2 hush! Keep this up and
│Jessica Lord @jllord  20/03/2015 14:57│ EVERYONE WILL KNOW
│RT @PDX44: Welcome to Portland, where │ https://t.co/deJBBjoOTV
│the old airport carpet is dressed up  │
│as a human and named Grand Marshal of │ Josh Susser @joshsus… 19/03/2015 14:06
│a parade. http://t.co/aTCicqSzEI      │ @geeksam @jeg2 @avdi POLS issue. No
│                                      │ standard == no way to avoid surprising
│Garann Means @garannm 20/03/2015 14:56│ many developers
│RT @verge: Robyn launches tech        │
│                                     ↓│                                      ↓
└──────────────────────────────────────┘

I don't want to take you through every line of code that I wrote. A lot of it doesn't really relate to curses anyway. The curious can dig into GitHub and fiddle with the code as much as they like, but let me give you the dime tour.

Curses Programs

If you've read the previous articles in this series, you know there's always a fair bit of boilerplate in any curses example that I show. Step one was make this feel more Rubyish. Here's the curses invocation in my Twitter client, Bird of Paradise, thanks to the helper library Rurses:

module BirdOfParadise
  class UI
    # ...

    def show(screen_name: , timeline: , mentions: )
      Rurses.program(
        modes: %i[c_break no_echo keypad non_blocking hide_cursor]
      ) do |screen|
        @screen = Screen.new(screen, event_q, screen_name, timeline, mentions)

        listen_for_events
        keyboard.read

        wait_for_exit
      end
    end

    # ...
  end
end

The most obvious change here is the use of the block. Rurses will setup curses, call the block, and clean up afterwords. This greatly reduces the amount of boilerplate code we have to use.

The other notable change in this code is the invocation of the various modes that curses provides. They are now passed as simple arguments to Rurses.program(). I've also cleaned up the mode names a tiny bit.

The code that provides these niceties isn't very complex. Here's the main entry point of the Rurses library:

module Rurses
  # ...

  module_function

  def curses
    FFI::NCurses
  end

  def program(modes: [ ])
    @stdscr = Window.new(curses_ref: curses.initscr, standard_screen: true)
    @stdscr.change_modes(modes)
    yield(@stdscr)
  ensure
    curses.endwin
  end

  # ...
end

You should recognize a couple of curses calls in here: initscr() and endwin(). We'll take more about the Window object that I wrapped stdscr in later, but know that I wanted this new API to favor the use of objects where it makes sense.

You can see here that the modes are passed into a change_modes() method of Window. This is a judgment call that I made while building this new API. Some modes are window specific, some aren't, and at least one takes a window parameter that is ignored. I didn't want to build three different systems for changing modes and force users to remember which to use where. Because of that, I've pushed all modes into the Window objects. If you want to set global modes, just make the call on stdscr as I do here. Or better yet, don't make any calls to change_modes() manually and just pass what you want as arguments to program().

Here's the actual mode changing code:

module Rurses
  class Window
    MODE_NAMES = {
      c_break:      :cbreak,
      no_echo:      :noecho,
      keypad:       [:keypad, :window, true],
      hide_cursor:  [:curs_set, 0],
      non_blocking: [:timeout, 0]
    }

    # ...

    def change_modes(modes)
      modes.each do |name|
        mode = Array(MODE_NAMES[name] || name)
        Rurses.curses.send(*mode.map { |arg| arg == :window ? curses_ref : arg })
      end
    end

    # ...
  end
end

The MODE_NAMES mapping links my slightly more readable aliases with the details for invoking that mode. Obviously, this doesn't handle all of curses modes yet. I've only mapped what I needed so far.

I've decided to prefer descriptive names (:non_blocking) to magic arguments (timeout(0)), but this doesn't handle the full range of curses capabilities. It's totally viable to call timeout(100) or timeout(500) and I can't make up sensible names for all possible combinations. Eventually, some exceptions would need to be made for these modes.

The :window flag is special. It gets replaced with the curses pointer as the mode is invoked. Again, not all modes need this, but we have to support those that do.

Do Several Things At Once

I chose my example project carefully. One of the primary reasons to work with a library like curses is that you can't afford to wait on keyboard input. A Twitter client is a good example of this. Twitter offers streaming APIs that you can connect to. As new tweets come in, they will be pushed down to your connection. This means tweets can come in at any time and you need to be ready for them. Twitter will close the connection if you don't keep up.

But we also need to watch the keyboard so the user can press keys to navigate the content we have already displayed. The user doesn't want to wait for the next tweet to come in before some instruction from the keyboard is honored. We must pay attention to both needs at the same time.

Let's dig into the code that waits for each of these data sources. First, the streaming code for Twitter:

module BirdOfParadise
  class Stream
    # ...

    def read
      @thread = Thread.new(streaming_client) do |stream|
        stream.user do |message|
          case message
          when Twitter::Streaming::FriendList
            update_followings(message)
          when Twitter::Tweet
            queue_tweet(message)
          end
        end
      end
    end

    private

    def update_followings(list)
      @followings = list
    end

    def queue_tweet(tweet)
      q << Event.new(name: :add_tweet, details: build_tweet(tweet))
    end

    # ...
  end
end

Stream.read() could block regularly, waiting on the next call of the block passed to user() (for a user's tweets). We sidestep this issue by wrapping the procedure in a Thread. We don't care how much time it spends waiting since it will just tie up that Thread and the rest of our code can keep running.

I should mention that this Twitter streaming code is incomplete. Twitter can send other events, like instructions to delete a tweet, that a full client does need to handle.

Notice that incoming tweets are just pushed onto an event queue. Whenever you have multiple incoming events, it's usually best to funnel them into the same pipeline and have some other chunk of code work through making the needed changes. This means several channels of execution won't be manipulating shared resources like the screen at the same time.

The keyboard code looks quite similar:

module BirdOfParadise
  class Keyboard
    # ...

    def read
      @thread = Thread.new(q, key_reader) do |events, keys|
        loop do
          case keys.call
          when "\t"
            events << Event.new(name: :switch_column)
          when :UP_ARROW, "k", "p", "w"
            events << Event.new(name: :move_up)
          when :DOWN_ARROW, "j", "n", "s"
            events << Event.new(name: :move_down)
          when "q"
            events << Event.new(name: :exit)
          end
          sleep 0.1
        end
      end
    end
  end
end

Again, we tuck some code in a Thread that just loops over incoming keys and turns them into events in a queue. It's not shown here, but the key_reader is a tiny wrapper over Rurses.get_key().

Before we look at that code, let's talk about the gotcha in the code above. See how my Thread includes a call to sleep()? Earlier, I turned on :non_blocking (timeout(0)) mode, so the key_reader doesn't block. Here I add the small pause to keep this loop from pegging a CPU core. However, you may be wondering, who cares if we block inside that Thread? Won't the rest of the code keep running? In this case, no, it won't.

As you may have heard, Ruby has a Global Interpreter Lock (GIL). This means only one Thread can truly execute at once. This protects us from some problems we could run into, especially with C extensions. If the Thread above pauses, waiting on a key, the GIL will prevent other code from running while we wait.

However, Twitter's Thread didn't hit this limitation, did it? If you dug down into Twitter's client code, you would find that somewhere deep in the stack it's based on Ruby's IO primitives. Those tools are aware of the GIL and they know a few tricks to avoid it. For example, when they are about to block waiting on input, they release the GIL so that other code may run. When some input finally arrives, they politely wait their turn to reacquire the GIL and only then return the input to your code.

curses is different. Remember that we're using ffi-ncurses to make calls into a C API at the lowest levels. FFI doesn't know which calls it could safely release the GIL for, so it never does. That's why we need :non_blocking mode and some pauses. We're giving other code some time to run before we make another C call.

Enough about tricky threading details. Here's the Rurses.get_key() code that's indirectly used above:

module Rurses
  SPECIAL_KEYS = Hash[
    FFI::NCurses::KeyDefs
      .constants
      .select { |name| name.to_s.start_with?("KEY_") }
      .map    { |name|
        [ FFI::NCurses::KeyDefs.const_get(name),
          name.to_s.sub(/\AKEY_/, "").to_sym ]
      }
  ]

  module_function

  # ...

  def get_key
    case (char = curses.getch)
    when curses::KeyDefs::KEY_CODE_YES..curses::KeyDefs::KEY_MAX
      SPECIAL_KEYS[char]
    when curses::ERR
      nil
    else
      char.chr
    end
  end

  # ...
end

The getch() function from curses can only return integers, so it uses different numbers to mean different things. Some are ASCII codes for keys on the keyboard, others are special constants for things like arrow keys, and one even means there's no available input right now (when you're in :non_blocking mode and don't wish to wait).

In Ruby, I can return different objects for the different cases and still sort out the comparisons with a simple case statement. Given that, I premap all of the SPECIAL_KEYS to Symbol names, use nil for no input, and transform the ASCII codes into actual String characters. You can scroll back up to see how each type is handled, if you need a reminder.

Controlling Where Output Goes

Rurses encourages code to work with Window objects, which wrap curses windows. The idea behind Ruby's version of the Window is straightforward: keep a reference to the curses window pointer and pass it to functions called as needed. Here's how that looks in practice:

module Rurses
  class Window
    # ...

    def initialize(**details)
      @curses_ref      = details.fetch(:curses_ref) {
        Rurses.curses.newwin(
          details.fetch(:lines),
          details.fetch(:columns),
          details.fetch(:y),
          details.fetch(:x)
        )
      }
      @standard_screen = details.fetch(:standard_screen) { false }
      @subwindows      = { }
    end

    attr_reader :curses_ref, :subwindows

    def standard_screen?
      @standard_screen
    end

    def cursor_x
      Rurses.curses.getcurx(curses_ref)
    end

    def cursor_y
      Rurses.curses.getcury(curses_ref)
    end

    def cursor_xy
      y, x = Rurses.curses.getyx(curses_ref)
      {x: x, y: y}
    end

    # ...
  end
end

You get the idea. You can either pass a curses_ref when you create the Window (as Rurses.program() does for stdscr) or just pass dimensions and coordinates to have the underlying structure created for you. Have a look at the cursor_*() methods to see how the reference is used.

Now, I also want to have an easy way to manage curses subwindows. You can see that the code above allocates a Hash for them and sets up a reader. Here's a couple more methods in Window for managing subwindows:

module Rurses
  class Window
    # ...

    def create_subwindow( name: , top_padding:   0, left_padding:   0,
                                  right_padding: 0, bottom_padding: 0 )
      s                = size
      xy               = cursor_xy
      subwindows[name] =
        self.class.new(
          curses_ref: Rurses.curses.derwin(
            curses_ref,
            s[:lines]   - (top_padding  + bottom_padding),
            s[:columns] - (left_padding + right_padding),
            xy[:y]      + top_padding,
            xy[:x]      + left_padding
          )
        )
    end

    def subwindow(name)
      subwindows[name]
    end
  end
end

create_subwindow() constructs another Window object, using some relative coordinate math, and adds it to the Hash by name. You can then later access any subwindow by name using the subwindow() method.

My Twitter client uses this combination of Window objects and their attached subwindows to divide the screen into columns. It also separates content from bordered regions to avoid any overwriting. Here's the code that arranges the screen, called on start and in the event of a terminal resize operation:

module BirdOfParadise
  class Screen
    # ...

    private

    def layout
      timeline_width, mentions_width, lines = calculate_column_sizes
      [
        { x: 0,              columns: timeline_width, feed: timeline },
        { x: timeline_width, columns: mentions_width, feed: mentions }
      ].each do |details|
        window = Rurses::Window.new(
          lines:   lines,
          columns: details[:columns],
          x:       details[:x],
          y:       0
        )
        window.create_subwindow(
          name:           :content,
          top_padding:    1,
          left_padding:   1,
          right_padding:  1,
          bottom_padding: 1
        )
        panels.add(window)
        details[:feed].window = window
      end
    end

    # ...
  end
end

This method builds two Window objects, adds a :content subwindow to each, and adds them as the canvas that a couple of not-yet-shown Feed objects will draw their output on. Let's have a look at that drawing code:

module BirdOfParadise
  class Feed
    # ...

    def redraw
      if changed?
        content = window.subwindow(:content)
        size    = content.size
        content.clear
        tweets
          .lines(count: size[:lines], width: size[:columns])
          .each do |line, cursor_location|
            if line
              attributes = selected? && cursor_location ? %i[bold] : [ ]
              content.style(*attributes) do
                content.draw_string_on_a_line(line)
              end
            else
              content.skip_line
            end
          end
      end

      @changed = false
    end
  end
end

This process is pretty basic. The :content area is cleared, the visible chunk of tweets is rendered as some lines, and the lines are added to the :content area one by one. I realize that I haven't show you all of the methods used here, but I bet you can guess what most of them do.

What you don't see here is any cursor moving code. The reason for that is that I'm using some methods in Window that add some sensible cursor management to standard operations. Have a look:

module Rurses
  class Window
    # ...

    def move_cursor(x: , y: )
      Rurses.curses.wmove(curses_ref, y, x)
    end

    def draw_string(content)
      Rurses.curses.waddstr(curses_ref, content)
    end

    def draw_string_on_a_line(content)
      old_y = cursor_y
      draw_string(content)
      new_y = cursor_y
      move_cursor(x: 0, y: new_y + 1) if new_y == old_y
    end

    def skip_line
      move_cursor(x: 0, y: cursor_y + 1)
    end

    def clear(reset_cursor: true)
      Rurses.curses.wclear(curses_ref)
      move_cursor(x: 0, y: 0) if reset_cursor
    end

    # ...
  end
end

See how draw_string_on_a_line() bumps the cursor down (assuming it didn't wrap) after each add? clear() also restores the cursor to the top left corner by default. This can save user code from needing to do a lot of manual move commands, in some cases.

Managing Redraw

If you were paying close attention earlier, you may have caught this line that I never explained:

        # ...

        panels.add(window)

        # ...

As each Window is constructed in the Twitter client, it is added to a Rurses::PanelStack. This wraps curses panels much like Rurses::Window wraps windows. However, there's just this one call, even with subwindows involved. That's because PanelStack also has some added niceties:

module Rurses
  class PanelStack
    def initialize
      @window_to_panel_map = { }
    end

    attr_reader :window_to_panel_map
    private     :window_to_panel_map

    def add(window, add_subwindows: true)
      window_to_panel_map[window] = Rurses.curses.new_panel(window.curses_ref)
      if add_subwindows
        window.subwindows.each_value do |subwindow|
          add(subwindow, add_subwindows: add_subwindows)
        end
      end
    end
    alias_method :<<, :add

    def remove(window, remove_subwindows: true)
      if remove_subwindows
        window.subwindows.each_value do |subwindow|
          remove(subwindow, remove_subwindows: remove_subwindows)
        end
      end
      window.clear
      Rurses.curses.del_panel(window_to_panel_map[window])
      Rurses.curses.delwin(window.curses_ref)
    end

    def refresh_in_memory
      Rurses.curses.update_panels
    end
  end
end

Both add() and remove() handle any subwindows by default. This object also keeps a mapping of Ruby Window objects to curses panels, so user code doesn't need to worry about tracking another set of references. I've found that just this much support makes panel management all but invisible.

The final piece of the puzzle is the method that updates the screen. It is called after each event is pulled from the queue and processed. Here's the code:

module BirdOfParadise
  class Screen
    # ...

    def update
      timeline.redraw
      mentions.redraw
      panels.refresh_in_memory
      Rurses.update_screen
    end

    # ...
  end
end

We have now looked at every call in there except the very last one and it just wraps doupdate() from curses.

This completes our tour of the primary curses interactions, but, as I said before, the full application is on GitHub. Interested parties are encourages to explore the code further.