Terminal Tricks

Random Access Terminal

I've recently been playing around with fancy terminal output in Ruby. I've learned quite a bit about this arcane magic. I've also realized that the documentation is pretty spotty. I want to see if I can improve that with a few blog posts, so let's dive right in.

Output

Program output typically happens in a linear order from top to bottom. For example, this code:

puts "onez"
puts "twos"
puts "threes"

generates this output:

onez
twos
threes

But what if you need to change some output? Could you replace the z above with an s if you needed to? Yes, but it can get a little involved.

ANSI Escape Codes

In many cases, we just push some characters to $stdout (the stream Kernel#puts is writing to above) and your terminal program happily shows them to the user. However, your terminal is watching these characters for special sequences that it understands. Some of those sequences of characters can cause your terminal to take actions other than just writing some output to the screen.

For example, there are sequences that move the cursor (the point where output is written) to different locations. Using those special codes, we could change the z in our example:

CSI = "\e["

$stdout.puts "onez"
$stdout.puts "twos"
$stdout.puts "threes"

$stdout.write "#{CSI}3A"  # move up three lines
$stdout.write "#{CSI}3C"  # move right three characters

$stdout.write "s"         # overwrite the 'z'

$stdout.write "#{CSI}3B"  # move down three lines (after our output)
$stdout.write "#{CSI}4D"  # move back four characters (to the start of the line)

If you run that code, it will initially produce the output from the previous example. It will then move the cursor to the z, overwrite it with an s, and return the cursor to where it was. This likely happens too fast for you to see the change, but you can insert sleep() calls if you want to watch it work.

All of the escapes used in this example are commands used to move the cursor around by a specific number of lines or characters in an indicated direction. Note that I switched to using the write() method when I started using escapes. puts() would have printed my escape followed by a newline (moving me away from where I wanted to be).

I used relative positioning commands above, meaning that I moved from where I knew the cursor was to where I wanted it be. I couldn't move to some absolute coordinates, because I don't know what else is on your screen when the program is run. You probably have at least a command prompt above the output and I have no idea how big that is. Now, if I clear the screen before I generate output, I could go on to treat it like a known grid of coordinates:

CSI = "\e["

$stdout.write "#{CSI}2J"    # clear screen
$stdout.write "#{CSI}1;1H"  # move to top left corner

$stdout.puts "onez"
$stdout.puts "twos"
$stdout.puts "threes"

$stdout.write "#{CSI}s"     # save cursor position

$stdout.write "#{CSI}1;4H"  # move to line 1, character 4
$stdout.write "s"           # overwrite the 'z'

$stdout.write "#{CSI}u"     # restore cursor position

This is the same trick, but using absolute positioning and some other fancy codes to save and restore the cursor's position. The movement escape used here, CSIy;xH, allows us to jump directly to any position on the screen. The top left corner is 1;1 with y counting lines down and x counting characters to the right.

Clearing the screen can be a little tricky. First, you can clear part or all of it. I chose all here with the 2 in CSI2J. You also need to remember though that clearing the screen doesn't usually change the position of the cursor. That's why I sent it back to the top left corner after the clear.

This code was pretty eye opening to me. It helped me to realize that a terminal doesn't have to be treated as a sequence of lines. It can alternately be treated like a drawing canvas with coordinates used to represent each spot that a character can be placed.

WARNING: terminals vary in exactly which codes they support. Most of the examples I'm showing will probably work on most UNIX-like terminals. I've tested them on my own UNIX box, but even there I found minor differences. For example, the code above works as expected in my normal terminal, but the terminal emulation mode I often use in Emacs doesn't seem to honor the save and restore cursor codes. It can be quite a bit of work to properly support the various kinds of terminals you can encounter.

Colors

It probably goes without saying that there are codes to change the color of your output and do other fancy graphic tricks. These can get slightly complex because they require a little math to indicate choice of foreground or background colors. They can also involve multiple codes at once when you want to select the bright version of a color. Here's a pretty trivial example:

CSI    = "\e["
COLORS = {
  black:   0,
  red:     1,
  green:   2,
  yellow:  3,
  blue:    4,
  magenta: 5,
  cyan:    6,
  white:   7
}

def color(content, foreground, background = nil)
  colored  = color_name_to_escape_code(foreground, 30)
  colored << color_name_to_escape_code(background, 40) if background
  colored << content
  colored << "#{CSI}0m"
end

def color_name_to_escape_code(name, layer)
  short_name = name.to_s.sub(/\Abright_/, "")
  color      = COLORS.fetch(short_name.to_sym)
  escape     = "#{CSI}#{layer + color}"
  escape    << ";1" if short_name.size < name.size
  escape    << "m"
  escape
end

[ ["C", :red],
  ["O", :bright_red],
  ["L", :bright_yellow],
  ["O", :bright_green],
  ["R", :bright_blue],
  ["S", :blue],
  ["!", :bright_magenta] ].each do |char, color|
  $stdout.write color(char, color, :white)
end
$stdout.puts

If you run that, you should see not-quite-a-rainbow of colored letters on white, regardless of the default color settings in your terminal.

It's pretty important to remember to issue a reset code (CSI0m) after playing with graphic settings like this. If your program exits without doing so, the terminal could be left in some pretty garish states. You can see that my color() method sets some colors, writes the desired content, then immediately resets. This process minimizes the chances of me bleeding over graphics changes into other output.

Visit that link I gave earlier on ANSI escape codes for specifics about any of the sequences I've used here, a good description of what the CSI is, and more.

The Status Line Trick

While ANSI codes allow you to do pretty much any output trick you can dream up with enough effort, there's another trick that just so easy it's worth knowing for the times when it can save you some work.

We usually end lines of output with a newline character (\n). Kernel#puts does this for us automatically. The exact effect of that character depends on the mode your terminal is currently in. We'll cover that more below, but there's another way to end lines. You could use the carriage return (\r) character. (In truth, using both characters is also an option. Again, I'll go into that below.)

A \r does move the cursor back to the beginning of the line, but it does not advance to the next line. This gives you a simple way to overwrite the content you previously placed on the last line only. That can come in handy for updating a status line or progress bar.

Here's a contrived example that just shows progress while it pretends to do some work:

chunks = 0

$stdout.puts "Time left:"
loop do
  $stdout.write "|#{'#' * chunks}#{' ' * (10 - chunks)}|\r"

  if chunks == 10
    $stdout.puts
    break
  end

  next_chunk = rand(1..(10 - chunks))
  sleep next_chunk
  chunks += next_chunk
end

Sample output from partway through a run looks like this:

Time left:
|#####     |

You may need to run it to appreciate the result, but the last line just keeps updating with the bar of #'s, growing each time. Notice that there is a final puts() when we exit the loop(), so the new command prompt won't erase the progress bar. You may or may not wish to do that in your own programs.

Input

I'll be honest and admit that I covered output first because it's easier. Buckle up and we'll see if can survive the jump into input.

I've mentioned before that programmers often ask how to read a single character from the keyboard and it can get tricky. This time I'm going to try to explain what's really going on without hiding behind a library that simplifies the process. I'll also show you some fancy new ways to accomplish the task.

Terminal Modes

Your terminal can operate in a variety of modes. The default cooked mode has several effects. One of those effects is that input is buffered until you press the return key. This buffering allows you to edit content you are entering—using arrow keys, backspace/delete keys, etc.—without all programs needing to handle all of those special cases. That's a great feature, right up to the point where you just want to read one character.

Watch what happens when I try:

loop do
  char = $stdin.getc
  puts "You typed:  #{char.inspect}"
  break if char == ?q
end

Here's some output from a run of that program:

a
You typed:  "a"
You typed:  "\n"
q
You typed:  "q"

What you need to catch here is that I pushed an a, but my program wouldn't acknowledge it, until I pressed return too. Then it read both characters. The q was treated the same way, the program just ended before showing the second newline.

If we really want to do this right, we've got to get out of cooked mode (or at least disable its line buffering feature). In older Ruby code, you had three choices for doing this: shell out to stty (tricky to get right), make very arcane C-style calls to ioctl() (a nightmare), or use a gem that hid the details from you (your best bet, by far). Now you understand my old advice to just use HighLine. But I have good news! Ruby is older and wiser now.

A New Standard Library

The easiest approach to reading just one character is to switch into raw mode (or a similar mode) and try again. However, this change will affect your program's output as well due to other features of cooked mode. Given that, your best bet is to jump into raw mode, grab a character, and return to cooked mode immediately.

Ruby now ships with a standard library that will do exactly that:

require "io/console"

loop do
  char = $stdin.getch
  puts "You typed:  #{char.inspect}"
  break if char == ?q
end

Behold the magic:

You typed:  "a"
You typed:  "q"

There are no newlines this time because the computer is now responding the instant I press a key. Victory. Sort of.

Even this tiny dip of our toes into raw mode has changed other things. You don't see what I typed echoed before the program provides its own output as it was before. This is another feature of cooked mode and you'll need to do your own echoing when reading like this, if you need it.

Here's another change that happens when I push control-C:

You typed:  "\u0003"

As you can see, it no longer interrupts my process. In fact, all of the control sequences have lost their magic powers (control-D to close $stdin and control-Z to suspend the program, for example). You will now need to handle these keystrokes yourself if you want them to have some effect. (Please do support control-C at the very least!)

Terminals have another mode that helps with this control keystroke issue, called cbreak mode. It gives you a near raw mode experience, but with these low-level keystrokes still handled. It makes me very sad to report that io/console doesn't yet support this handy mode.

Still, io/console is great to have. You can read about what it can do in the API documentation and its README. Here's one more simple example of how easy io/console can make things, like finding the screen size:

require "io/console"

rows, cols = $stdout.winsize
puts "Columns:  #{cols}, Rows:  #{rows}"

Full Raw Mode

Dipping into raw mode to grab characters is great when it works for your needs, but the code I've showed so far has a drawback. io/console's getch() is a blocking method. It will wait for input. This is a problem if you need to manage your own event loop, take user input when it's available, but continue making things happen when it's not.

If there's a way to solve this problem with just io/console, I haven't been able to derive the proper incantation. However, we can pull it off by pulling one more standard library into the mix.

To show how this works, imagine that we want to prompt a user with what to do if they don't take any action for a few seconds. Here's some code:

require "io/console"
require "io/wait"

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = io.ready? && io.sysread(1)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}"
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character."
        prompted = true
      end
      sleep 0.1
    end
  end
end

This time I used io/console to put the whole program into raw mode. Cooked mode is restored when the block passed to raw() exits.

The other library I used is io/wait. It adds a ready?() method to $stdin that will return true when there's content waiting to be read. I pull that content out a byte at a time using the low-level sysread() to avoid any buffering.

This kind of works, but we've picked up a new issue:

Please type a character.
                        You typed:  "a"
                                       You typed:  "q"

See how my output is now staggered?

Yet another feature of cooked mode is that newlines in your program's output are translated to a carriage return newline sequence (\r\n). Rubyists are used to puts()ing out some content, with a newline added, and seeing the cursor move to the beginning of the next line. But a newline doesn't really do that by itself.

The newline character means move down a line. As we saw earlier, a carriage return sends you to the beginning of the current line. So it's really the combined effect of both that we're used to being normal.

My output above has only the newline effect, because we're in raw mode and the translation to a two character sequence isn't active. That explains the problem and gives us a way around it. In raw mode, you'll need to manually end lines with \r\n when you want to move to the beginning of the next line.

Here's a fixed version of the code above:

require "io/console"
require "io/wait"

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = io.ready? && io.sysread(1)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}\r\n"  # added \r\n
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character.\r\n"   # added \r\n
        prompted = true
      end
      sleep 0.1
    end
  end
end

The comments show the two minor changes to the code.

And now the output looks more typical:

Please type a character.
You typed:  "a"
You typed:  "q"

We've handled a lot of cases, but there are still issues with a raw approach like this. I'll show you one more. Watch what happens when I press an arrow key while running our latest program:

You typed:  "\e"
You typed:  "["
You typed:  "A"

Ugh. That's all one keystroke, but we pulled it out as separate bytes. This means we wouldn't be able to have conditionals in our program check for this keypress.

Can we fix it? Sure. We can add some more code to detect these escape sequences and combine them into a single String. Here's what that might look like:

require "io/console"
require "io/wait"

CSI = "\e["

def get_char_or_sequence(io)
  if io.ready?
    result = io.sysread(1)
    while ( CSI.start_with?(result)                        ||
            ( result.start_with?(CSI)                      &&
              !result.codepoints[-1].between?(64, 126) ) ) &&
          (next_char = get_char_or_sequence(io))
      result << next_char
    end
    result
  end
end

$stdin.raw do |io|
  last_read = Time.now
  prompted  = false
  loop do
    char = get_char_or_sequence(io)
    if char
      last_read = Time.now
      prompted  = false
      puts "You typed:  #{char.inspect}\r\n"
      break if char == ?q
    else
      if !prompted && Time.now - last_read > 3
        puts "Please type a character.\r\n"
        prompted = true
      end
      sleep 0.1
    end
  end
end

All of the new code here is in the get_char_or_sequence() method. The while loop in there checks to see if a read character looks like the start of a CSI sequence and keeps reading, assuming there's more input, until we've seen the full CSI. Then, if what we've read starts with a CSI and there's still more input, we keep reading until we see a termination character for the sequence.

The end result? We can now treat arrow keys as a single unit (as long as our terminal returns them as ANSI escape sequences with a leading CSI):

You typed:  "\e[A"

That's progress. There's definitely more we could handle though, like non-CSI escape sequences and larger Unicode characters. This rabbit hole goes pretty deep. You'll need to decide how many such cases you need to handle for your programs needs.

If you have to deal with many of the various topics covered by this post, you may want to consider using a gem that can handle some of them for you. They can really smooth over some of these edge cases. The details of that are another blog post, but my hope is that this article starts to at least get you familiar with what's involved in fancy terminal I/O.