Gray Soft / Tags / Community

Asking Better Questions

2014-09-19T14:52:52Z

I've been playing with some Rust lately and learning a bunch of new concepts.

As part of my experimentation, I built an RPN calculator in the language, just as an exercise that would require a moderate amount of code to be worked out. Honestly, I don't fully understand everything that I had to do to get this code working yet. Maybe 20% of it came about as me following instructions from what seem to be very helpful compiler errors.

I wanted to start attacking these concepts that I didn't understand to increase my knowledge. Of course, I was impatient. I had some working code and I knew what I was missing, so I jumped into IRC, pointed at the code, and asked some not-at-all complete questions. All I got was crickets.

This isn't a failing of the Rust community. It's a lesson I have to relearn every now and then. You have to take the time to present a good enough question that answering it is easy enough and worth it. It's hard work to get your head around 100 lines of code and, even if you do, you'll still be missing plenty of context if the question isn't really well formed. Given that, most people just ignore the question. That's probably for the better too, because any answers provided likely would have missed the points I really needed help with.

What it did do was remind me that I needed to ask better questions. Today I took the time to build a small example that addressed one of my key confusions with Rust so far: Lifetimes (plus Ownership, Borrowing, and References it turns out). Here's the code I whipped up:

// I'm trying to make sure I have my head around "lifetimes"
// after a few readings of the guide:
// http://doc.rust-lang.org/guide-lifetimes.html.  The following is
// a simple example of what I think I now know that I'll try to explain
// in this comment.
//
// Because the letters are "borrowed references," (Do I have my terms right?)
// they could potentially be freed before the Lists struct that contains them.
// Since this possibility exists, Rust requires the explicit lifetime 'a which
// indicates that the lifetime of a created Lists instance is tied to whatever
// references were passed into the letters slot.

#[deriving(Show)]
struct Lists<'a> {
    numbers: Vec<int>,
    letters: Vec<&'a str>
}

fn main() {
    let lists = Lists{numbers: vec![1, 2, 3], letters: vec!["A", "B", "C"]};
    println!("{}", lists);
}

I made a Gist out of that and then went back to IRC to ask people how correct my current understanding was. The effort really paid off. I got some great clarifications:

JEG2
  Can anyone tell me if my understanding of the explicit lifetime
  in this example, as explained by my comment, feels correct?
  https://gist.github.com/JEG2/ae7746544cef14be1606
…
kimundi
  JEG2: That sounds right
…
steveklabnik
  JEG2: oh hey!!!!
  JEG2: good to see you here :)
  <3
Sharp
  JEG2: It's very close to being right (I *think*). Lists and letters
  must have the same lifetime, period. It's not that Lists acquires
  the lifetime of letters when it's created.
kimundi
  JEG2: Not that in this example you are using string literals,
  which have 'static lifetime and thus the references are always valid
JEG2
  steveklabnik: Hey steve. Thanks.
steveklabnik
  JEG2: the lifetime guide is _terrible_ btw, and as soon as my guide
  is wrapped up, which should be this week, it's next on the re-write train
…
kimundi
  Sharp: Eh, that doesn't sound right
…
JEG2
  steveklabnik: Good to know. Thanks.
…
Sharp
  kimundi: Well, actually I think Lists is bound to live at most
  as long as any one of its lifetime parameters. So yeah, you're right.
  I am clearly still confused about lifetimes :P
kimundi
  yeah, that sounds more correct :)
  No, wait
…
kimundi
  Eh, nevermind, its correct. List can not live longer as the 'a lifetime
  because else it would not be able to be created.
…
Dr-Emann
  JEG2: minor thing: everything has a lifetime, whether it
  contains references or not. Rust doesn't just create one
  when there's a possibility of premature frees. It only
  makes you be explicit with them in those cases though.
…
JEG2
  Sharp and Dr-Emann: Good clarifications on lifetimes. Thank you.
Sharp
  You're welcome, glad you could get something out of my confusion :)
…
steveklabnik
  JEG2: i left a comment on your gist too
Dr-Emann
  JEG2: of note is that even though the lifetime of the references
  is defined as the same as the lifetime of the containing struct,
  you can store any reference which lives _at least_ as long as the struct.
steveklabnik
  JEG2: make that two comments
…
JEG2
  Dr-Emann: Ah, I did not know that. So it's legal for my reference
  to outlive the struct?
…
Dr-Emann
  JEG2: they do, in your case, even. You pass literal strings,
  which are references into read only memory in the executable itself.
  They live for the entirety of the program.
…
JEG2
  Dr-Emann: Gotcha.
Dr-Emann
  JEG2: they have the special lifetime 'static. But you can also
  make a string on the stack and include references into it,
  as long as your struct doesn't outlive the String.
JEG2
  steveklabnik: Thanks for the comments. I'm getting a more complete picture.
…
steveklabnik
  JEG2: awesome. I -really really- want to make sure we explain this well,
  but we're not quite there yet.

As Steve Klabnik said, he also posted some very helpful comments on my Gist.

In other words, I got quite a bit of help. That help is also on public IRC logs and Gists, so it might even be of use to others.

It really pays to spend some time developing your question asking skills and then working on remembering to put those skills to use.

Dave Thomas is Definitely the Sammy Sosa of Programming

2014-04-05T16:22:22Z

There is a debate raging in the Ruby community that I don't want any part of. I'm not going to engage in any of the mud slinging and there will be no debate here. Commenters have been warned. What I do want to do is to share some simple uncontested facts about a man I am lucky enough to know.

Sammy Sosa is famous for one thing: hitting home runs. In the entire history of the game of baseball five players have managed to hit over 600 home runs and Sammy is one of them. If that wasn't amazing enough, he has hit at least one home run against every single Major League team and in 44 Major League ballparks. Baseball fans everywhere love to watch Sammy Sosa at bat.

Now if I had to name five programmers who get me as excited about programming as Sammy Sosa does about baseball, Dave Thomas would definitely make the list. Dave does exactly what Sammy always does: continually preforms the hardest tasks of his profession while making it look easy to the fans. Allow me to give a few examples.

Dave Thomas was one of the authors of the Agile Manifesto. I think we tend to forget just how significant that is. If you are a programmer who doesn't spend his day in discussions with 50 coworkers, trying to satisfy the needs of various committees, or reading endless specifications of expected behavior, the odds are good that you have been affected by this document. I, for one, am ecstatic that he helped spare me from that fate. Of course, he didn't stop there.

The Agile Manifesto gave us the theory, but we still needed to develop the practices it embodied. Many have done that for us, including Dave. He was the coauthor of a book called The Pragmatic Programmer: From Journeyman to Master. If you are a programmer who hasn't read this book, you owe it to yourself to pick it up. It gave so much great advice that it made me a better person in addition to a better programmer. I've lost count of how many times I've told someone to "Make Stone Soup" or "Fix Broken Windows." If the Agile Manifesto was a home run, this book was a grand slam.

Then there's Ruby. You can quite literally say that one man lead the western world in the charge to discover the Ruby programming language. He found the language, encouraged other programmers to try it out, cowrote the first English book about it, literally gave that book away for free, designed a documentation tool for the language, and contributed all the documentation he had created for that first book back to the source code. That was long before most of us had even heard of Ruby. These days that book is going into its third edition and that same man has coauthored a book about Ruby's most popular framework, Ruby on Rails. It should be little surprise that Dave did all of that.

Giving us Ruby wasn't enough though, Dave had to teach us how to use it too. It would be a quite a challenge to count the number of conference speeches he has given and workshops he has led. Some of those workshops even raised large amounts of money for multiple charities, just because Dave is also a nice guy. He also cofounded his own publishing company and sought out authors to write great new titles on all things programming. That publishing company has become a trend setter for all of the technical publishers, popularizing concepts like PDF-only distribution and "Beta Books."

Those are just some of the bigger things I can tell you that Dave Thomas has done for programming. I can't even begin to guess at the scope of his smaller deeds. For example, many know that I now maintain the Ruby Talk to comp.lang.ruby gateway, but take a wild guess at who built it in the first place… Yeah, that's the guy.

Dave, all I can say is, keep knocking 'em out of the park. I'll be the fan watching from the stands, eating pistachio nuts, and cheering you on.

I Enjoy the Regional Conferences

2014-04-05T14:43:55Z

I have to say that the Lone Star Rubyconf was just great. I'm hoping that's representative of the other regional Ruby conferences as well and from what I've heard it is.

I was at the official Rubyconf last year and I'm comfortable saying that the Lone Star Rubyconf competed well on content. We had great keynotes from Charles Nutter and Zed Shaw (though he's wrong about that don't love your language point); we had presentations from icons of the community, like Hal Fulton; we had popular topics covered by the experts, like the RSpec presentation from Mr. RSpec, David Chelimsky; and we had the wonderfully practical technical talks, like Evan Short's smooth coverage of Domain Specific Languages.

While you do have to consider it a small minus to miss seeing Matz and a few other key Rubyists, the conference countered with a terrific small community feel. Everyone was open and friendly. You could easily approach anyone and chat them up about nearly any topic. Several people approached me and I loved it.

Oh, and you can say "y'all" in an LSRC talk without getting funny looks. You can't beat that.

So, if you're like me and can't make the official conference this year, consider attending a regional conference instead. I had a very positive experience and I look forward to next year's offering.

The Ruby VM: Episode I

2014-04-04T20:29:56Z

Hello and thank you both for agreeing to answer my questions. To begin, would you please introduce yourselves and tell us about your role in Ruby's development?

Matz:

I am the designer and the first implementer of the Ruby language. My real name is Yukihiro Matsumoto, that sounds something like You-Key-Hero Matz-Motor in English. But it's too long to remember and pronounce, so just call me Matz.

I have been developing Ruby since 1993. It is now quite complicated and has performance problem. I have had vague plan of rewriting the interpreter for long time, but I have never been motivated enough to throw out the current interpreter and start developing new one.

Then Koichi came in with YARV that seemed to have much brighter future than my vaporware - it runs - so I asked him to take a role of the official implementer of the core. Although I enjoy both designing and implementation of the language, I don't think I am gifted for language implementation. So I thought that it might be the time to focus on designing when I saw YARV.

ko1:

Thank you for your interest in YARV and me. BTW, I'm thinking what "YARV" stand for. Because it is not Yet Another. Someone proposed that "YARV ain't RubyVM". If YARV means "YARV ain't RubyVM", what is YARV?

I'm Koichi Sasada. Koichi is given name, and "ichi" means "one" in Japanese. So I use "ko1" as my nick. I'm an assistant at Department (...snip...) of Tokyo. My research interest is systems software, especially operating system, programming language, parallel systems, and so on. And I'm a member of Nihon Ruby no Kai (Ruby Association in Japan). I plan(ed) some Ruby events like RubyKaigi and am an editor of Rubyist Magazine. I also develop(ed) Nadoka, Rava, Rucheme, and some projects. Say, I'm a developer of YARV: Yet Another RubyVM.

My role in Ruby's development? To steal VM hacking pleasure from Matz?

The point of this interview is to talk about the future of Ruby's interpreter. To start that, can you please explain what YARV/Rite is? How is it different in design from the old Ruby interpreter?

Matz:

I have always been more interested in designing the language than implementing it. So Ruby interpreter is always slower than it should be. I think I pruned all low-hanging fruits, so that it seemed to required to re-implement whole core to achieve performance boost. I planned a new interpreter code-named 'Rite' in 2001 or so, but I have never motivated enough to start the project. Maybe I had been too busy, or perhaps too lazy.

Then, Koichi came in, and showed us his YARV. Many had tried implementing Ruby interpreter in the past, but no one but Koichi reached that level of implemented feature set (at the time; now we have JRuby and RubyCLR both compatible with Ruby 1.8). So I asked him to take part in the development of the new core, and he agreed.

January 1st 2007, he checked in YARV in to the trunk of our repository, so it is now official core of the Ruby 1.9. I am still working on old implementation in matzruby branch. Since it is easier for me to experiment new language features on the old interpreter, but I will eventually switch to the new engine.

For YARV implementation detail, Koichi will explain.

Does this mean we are leaving the name Rite behind and keeping YARV? Or will YARV be renamed at some point?

The name Rite will not be used for this generation of the language, unless Koichi ask me. I am not sure Koichi is going to keep YARV, or not, since it already 'the VM' for Ruby.

ko1:

YARV is vanished :)

In fact, I'm removing "yarv" words from structure names, function names, and file names. YARV is only code name that not made by *Matz*. Now, YARV is not "Yet Another". In this article, I use "YARV" words as current Ruby trunk on official repository.

At first, YARV is simple stack machine which run pseudo sequential instructions. Old interpreter (matzruby) *traverses* abstract syntax tree (AST) naively. Obviously it's slow. YARV compile that AST to YARV bytecode and run it.

Secondly, YARV uses native thread (that supported by OS or so) to implement Ruby thread. It means that you can run *blocking* task in extension libraries. (On Ruby's spec, blocking task should be interrupted by Thread#raise. To know details, see [ruby-core:10252].) Because thread creation is slower than matzruby (green thread), you shouldn't make many threads at a time. Supporting native thread *does not* means that you can run Ruby scripts in *parallel* on parallel machine such as Multi-Core CPUs. Current implementation uses Giant VM Lock to avoid synchronization problems. (Many extension libraries doesn't care thread safety. See array.c, string.c, etc.)

Thirdly, I made many optimization like specialized instructions, etc. These features are my purpose of developing YARV. Toy benchmarks run fast because of these optimization techniques.

YARV doesn't change parser/syntax/specs (matz' hobbytask), GC (memory/object management), and extension libraries like String/Array/Hash/Regexp/etc. Therefore your script doesn't run fast on YARV if bottleneck is string processing, or so.

Congratulations to you both for completing Ruby/YARV merger recently. That must have been a lot of work, but I know it has the whole Ruby world very excited. Now that the merger has taken place, how do you see this changing the way Ruby is developed?

Matz:

Congrats should go to Koichi who has done a lot of work. I am moving my developing from matzruby (a branch for my old interpreter) to trunk (the yarv). Recently I have implemented some new features on the trunk, for example, class local instance variables and new local variable scope. The transition will complete pretty soon.

Since the trunk is originally Koichi's work, I need more help from others especially from Koichi than before. I know everything about the previous interpreter (well, most of them), but there are still mysteries in the new one. I am well satisfied with new one. It's clearer, well-formed, and faster.

ko1:

Thank you. I'm a newbie of Ruby developer (in fact, I didn't have CVS account to commit any ruby codes). So I can't say how change on ruby development :)

When will the first production release of Ruby running on YARV by available for all Rubyists to play with?

Matz:

Short answer: now.

Longer answer: the YARV is already publicly avaiblabe via our Subversion repository. You can fetch and play with it now. But the first public "release" from us will be Christmas 2007, if we are as diligent as we should be. Knowing how lazy I am, I will try not to be a stumbling block for the release. ;-)

The Ruby VM Serial Interview

2014-04-04T21:40:57Z

I have really enjoyed reading Pat Eyler's Rubinius Serial Interview and Nick Sieger's spun-off JRuby Serial Interview. It's very educational to read what the developers have to say about their projects and ideas.

The more I read though, the more I wanted the equivalent content for the official Ruby VM. I asked Matz and Koichi if they would be willing to answer questions from me and they agreed to do so. We are now ready to share their responses with the community.

This will be a serial interview as Pat Eyler calls them. We will deliver regular episodes until I run out of good questions or Matz and Koichi get sick of me bothering them, whichever comes first. I will ask the questions in the interview, but feel free to make suggestions in the comments to this article.

One last note: we are not promising any kind of schedule for the episodes. Matz and Koichi are heroically providing their answers in English. We want to respect how much work that is and give them all the time they need to do that. Personally, I cannot thank them enough.

With that, I give you the episode index:

In this first episode I ask Matz and Koichi to introduce themselves and their roles as well as to give us an update on where we are with the Ruby VM.
In the second episode I ask Matz and Koichi about for their thoughts on the alternate Ruby implementations and how they see them changing Ruby's development.
In the third episode Matz and Koichi discuss the past, present and future of Ruby threading.
In the fourth episode Matz tells us a little about how m17n is shaping up.
In the fifth episode Koichi gives us the inside story on optimization in the new VM.

news_to_mail.rb

2014-04-04T19:16:17Z

[Note: You need to know what the Gateway is before reading this article.]

The second half of the Ruby Gateway runs as a cron job every five minutes and is expected to move all new newsgroup messages from the NNTP host to the Ruby Talk mailing list. The cron invocation is simply:

ruby /path/to/gateway/bin/news_to_mail.rb /path/to/news_to_mail.log

This side of the Gateway is not piped any messages and exit codes from it are not monitored. It needs to tend its own affairs.

The Code

Here's the source:

GATEWAY_DIR  = File.join(File.dirname(__FILE__), "..").freeze
DATA_DIR     = File.join(GATEWAY_DIR, "data").freeze
LAST_ID_FILE = File.join(DATA_DIR, "last_news_id.txt").freeze
PID_FILE     = File.join(DATA_DIR, "pid.txt").freeze

$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join(GATEWAY_DIR, "lib")

# ...

The script begins by locating the root directory of the Gateway source and the data sub directory, building constants for two external files it will interact with, and adjusting the $LOAD_PATH so that it can require needed resources.

Here are those requires:

# ...

require "servers_config"
require "nntp"

require "net/smtp"
require "logger"
require "timeout"

# ...

The first two requires are the same code used by the other half of the Gateway. The last three are standard Ruby libraries the code will make use of.

# ...

# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "

# ...

This code opens a log file to record its status and adjusts the date and time formatting.

With setup complete, it's now vital to ensure that the current execution is the only copy of the code we are running. Should the Gateway get behind and fail to complete before the cron job launches another copy, duplicate messages could be sent to the list or worse.

# ...

# make sure only one copy is ever running at a time
begin
  File.open(PID_FILE, File::CREAT|File::EXCL|File::WRONLY) { |pid| pid.puts $$ }
  at_exit do
    begin
      File.unlink(PID_FILE)
    rescue
      log.error "Unable to unlink pid file:  #{$!.message}"
    end
  end
rescue
  pid = File.read(PID_FILE).strip.to_i rescue "unknown"
  log.warn "Process #{pid} was already running"
  exit
end

# ...

The script attempts to write its process ID number to a file. The File::CREAT|File:EXCL options ensure that the file is created or an exception is thrown, if it already existed. When the exception comes it is assumed that an earlier execution is still running and the script exits. If able to create the process ID file, the scripts knows it is the only copy running and it is safe to proceed. The code then immediately arranges for Ruby to remove the file on exit.

Note that to check for the file then create it if missing would introduce a race condition. Another execution could create the file between check and creation. As unlikely as that is for a five minute cron job, it's better to play things safe and just try the creation directly.

Having made it this far, the script needs to refresh its memory of where it was in the list of newsgroup messages:

# ...

# find out what the last news item sent was
begin
  last_id_sent = Integer(File.read(LAST_ID_FILE).strip)
rescue
  log.fatal "Unable to get message ID of last news post:  #{$!.message}"
  exit -1
end

# ...

A simple text file is used to hold the message number of the last file seen. The code reads that file back to know where to resume its work.

It's now time to connect to Usenet.

# ...

# 
# connect to NNTP host, switch to newgroup, and see how many messages are
# available to read
# 
begin
  nntp, last_id_available = nil, nil
  Timeout.timeout(30) do
    nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
                          Net::NNTP::NNTP_PORT,
                          ServersConfig::NEWS_USER,
                          ServersConfig::NEWS_PASS )
    last_id_available = nntp.group(ServersConfig::NEWSGROUP)[3].to_i
  end
rescue Timeout::Error
    log.error "The NNTP connection timed out."
    exit
rescue
    log.fatal "Unable to establish connection to NNTP host:  #{$!.message}"
    exit -1
end

# ...

This code tries to establish a connection to the NNTP host and switch to the comp.lang.ruby group being monitored. A reasonable timeout is imposed on this code and will be for all network operations attempted. If the connection is slow for some reason, another attempt can be made by the next cron execution.

The script now advances the NNTP reader to the last message seen, which turns out to be complicated to get right:

# ...

# switch to the last message we sent (or the first real message before that)
begin
  Timeout.timeout(30) do
    nntp.stat(last_id_sent)
  end
rescue Timeout::Error
    log.error "The NNTP message shift timed out."
    exit
rescue
  if $!.message.include?("Bad article number") and last_id_sent.nonzero?
    last_id_sent -= 1
    retry
  else
    log.fatal "Unable to switch to the last message:  #{$!.message}"
    exit -1
  end
end

# ...

The problem here is that Usenet messages can be revoked. When that happens, the last message the script saw may no longer exist. The real goal is the message after the last seen post though and an NNTP NEXT command will skip missing IDs. Given that, the code backs up in the count until it can find an existing message. NEXTing forward from that will skip the vanished IDs and land the reader at the desired later message.

To make that leap forward in the message count, the script enters an infinite loop of message processing:

# ...

# main event loop
loop do
  # advance to the next message, if there is one
  new_last_id, more_messages = nil, true
  begin
    Timeout.timeout(30) do
      new_last_id = nntp.next[1].to_i
    end
  rescue Timeout::Error
      log.error "Advancing the current NNTP message timed out."
      exit
  rescue
    more_messages = false
  end
  break unless more_messages

  # reality check that we are moving forward
  if new_last_id <= last_id_sent
    log.fatal "new_last_id (#{new_last_id}) <= last_id_sent " +
              "(#{last_id_sent}), quitting to prevent sending duplicates"
    exit -1
  end

  # ...

As I said, a NEXT command is sent to the host bringing the reader to an unseen message. A quick sanity check is then made to ensure the count is rising.

Note that this is also the code to determine when there are no more messages. If the NEXT command fails, the message processing loop terminates. You really need to used this combination of an infinite loop and last message detection when working with Usenet. Any method of counting is pointless, thanks to the numbering issues I described.

At this point, it's time to actually start reading the post:

  # ...

  # pull the headers for the current messae
  begin
    head = nil
    Timeout.timeout(30) do
      head = nntp.head(new_last_id)[3].join("\r\n") + "\r\n"
    end
  rescue Timeout::Error
      log.error "Fetching the message headers timed out."
      exit
  rescue
    log.error "Unable to retrieve headers for message ##{new_last_id}, " +
              "will try again later:  #{$!.message}"
    exit
  end

  # 
  # don't send articles that the mail_to_news program has previously forwarded
  # to the newsgroup (or we'd loop)
  # 
  if head =~ /^X-rubymirror:/
    log.info "Skipping message ##{new_last_id}, sent by mail_to_news"
    next
  end

  # ...

This code pulls the message headers and performs the loop check to ensure that messages sent by mail_to_news.rb are skipped.

  # ...

  # pull the body for the current message
  begin
    body = nil
    Timeout.timeout(30) do
      body = nntp.body(new_last_id)[3].join("\r\n") + "\r\n"
    end
  rescue Timeout::Error
      log.error "Fetching the message body timed out."
      exit
  rescue
    log.error "Unable to retrieve body for message ##{new_last_id}, " +
              "will try again later:  #{$!.message}"
    exit
  end

  # scan headers for message data
  subject = head =~ /^Subject:\s+([^\r\n]*)/     ? $1 : "?"
  from    = head =~ /^From:\s+([^\r\n]*)/        ? $1 : "?"

  # ensure the message is properly addressed
  unless head =~ /^To:/
    head = "To: #{ServersConfig::MAILING_LIST}\r\n" + head
  end

  log.info "Sending message ##{new_last_id}:  #{subject} -- #{from}..."

  # ...

The body is then pulled. The script then scans for some message details in the headers to allow us to log accurately about the post. The message is also addressed.

  # ...

  # extracting path information (poster's host, first nntp hop)
  posting_host = head =~ /^NNTP-Posting-Host:\s+(\S[^\r\n]*)/i ? $1 : "unknown"

  path = head =~ /^Path:\s+([^\r\n]*)/i ? $1 : "unknown"
  path = path.split("!").delete_if { |hop| hop !~ /^(([^.]+)\.)+([^.]+)$/ }

  first_hop = path.last

  # adding pseudo "Received:" header and rubymirror header
  head += <<END_RECEIVED.gsub("\n", "\r\n")
Received: from #{first_hop} (#{first_hop})
     by #{ServersConfig::NEWSGROUP} with NTTP id #{new_last_id}
     for <#{ServersConfig::MAILING_LIST}>; #{Time.now.to_s}
Received: from [#{posting_host}]
     by #{first_hop} (unknown) with NNTP id #{new_last_id}
     for <#{ServersConfig::NEWSGROUP}>; #{Time.now.to_s}
Received: from Usenet via a Usenet to mail gateway located at
     #{ServersConfig::NEWSGROUP}.  This service provided as a courtesy
     to the ruby-talk mailing list.  If this message is SPAM, its
     ultimate origin is Usenet, not this gateway program.  All
     subscribers to the ruby-talk mailing list agree to receive the
     Usenet postings made to comp.lang.ruby via this gateway.  Please
     see http://www.ruby-lang.org/ruby-talk-usenet-policy.html.
X-From-Usenet: see Received: header above.
X-rubymirror: yes

END_RECEIVED

  # build final message
  msg = head + body
  log.info "Message looks like: #{msg.inspect}"

  # ...

The above adds a header explaining the service and constructs the final message.

  # ...

  # attempt to send email
  unless $DEBUG
    begin
      Timeout.timeout(30) do
        Net::SMTP.start(ServersConfig::SMTP_SERVER, 25) do |smtp|
          smtp.send_mail( msg,
                          ServersConfig::MAIL_SENDER,
                          ServersConfig::MAILING_LIST )
        end
      end
    rescue Timeout::Error
        log.error "The SMTP connection timed out."
        exit
    rescue
      log.fatal "Unable to send email:  #{$!.message}"
      exit -1
    end
    log.info "...  Sent."
  end

  # ...

Here a connection is established to the SMTP server and the email is sent. The $DEBUG flag allows me to spot check Gateway functionality without actually sending emails to Ruby Talk.

On to the final step:

  # ...

  # record new high-water mark
  begin
    unless $DEBUG or new_last_id == last_id_sent
      File.open(LAST_ID_FILE, "w") { |file| file.puts new_last_id }
    end
  rescue
    log.error "Unable to write message ID to file:  #{$!.message}"
  end
  last_id_sent = new_last_id
  log.info "New last message sent ID:  #{last_id_sent}"
end

Getting this far means the script sent an email successfully, so it records the new last seen ID for future runs. That final end signals the end of the message processing event loop.

Possible Improvements

This code is pretty feature complete.

About the only enhancement I can think of would be to DRY up some of the timeout and error handling code. That's trickier that you might guess because of the extra layers of scoping plus the error messages and exit codes. We almost need a multi-block syntax here. If someone poses a reasonable alternative I'll consider it, but I've tried to keep the code pretty straight forward and the truth is that is just may not be worth the effort.

Any comments or suggestions for the Gateway can be left as comments below.

mail_to_news.rb

2014-04-04T19:20:41Z

[Note: You need to know what the Gateway is before reading this article.]

There are two halves to the Ruby Gateway. One half runs as a qmail filter for an email address on the Ruby Talk mailing list. Every message sent to that address is piped through this filter with a shell script like:

ruby /path/to/gateway/bin/mail_to_news.rb /path/to/mail_to_news.log

The email is piped to the filter via the standard input and the code is expected to handle the message by posting it to comp.lang.ruby or choosing to ignore it. If the filter exits normally, qmail considers the matter handled. A non-zero exit code will cause the filter to be called with that same message again later.

The Code

Let's dive right into the source of this half of the Gateway:

GATEWAY_DIR = File.join(File.dirname(__FILE__), "..").freeze

$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join(GATEWAY_DIR, "lib")

# ...

The code above just sets things up so this script can require some other files in the project normally. Here are those requires:

# ...

require "servers_config"
require "nntp"

require "net/smtp"
require "logger"
require "timeout"

# ...

The last three requires are standard Ruby libraries. The first two are not.

The servers_config.rb file sets up a ServerConfig Module with information needed to connect to our email and Usenet hosts. I will not show this file, but the references should be obvious when see them.

The nntp.rb is pretty much a vendored copy of the net-nntp library. I've made some minor changes for debugging output purposes, but the library functions the same.

We're now ready to initiate logging. Here's the code that starts that process:

# ...

# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "

# ...

That just builds a Logger object and cleans up the default date and time formatting.

Now it's time to start setting variables in preparation for the coming email parse:

# ...

# only allow certain headers through
VALID_HEADERS    = %w[ From Subject References In-Reply-To Message-Id Content-Type
                       Content-Transfer-Encoding Date X-ML-Name X-Mail-Count
                       X-X-Sender ]
valid_headers_re = /^(?:#{VALID_HEADERS.join("|")}):/i

# ...

The Gateway passes through only a subset of the email headers for the Usenet post. The above is the list of those headers and the Regexp that will locate them.

Now, there are two types of messages we do not wish to forward: spam and a message sent to Ruby Talk by the other half of the Gateway (causing an infinite loop of sending). The following code prepares flags for these conditions:

# ...

# message flags
spam     = false
mirrored = false

# ...

The following code allocates variables to hold key header information parsed from the message:

# ...

# header data
msg_id   = "unknown"
subject  = "unknown"
from     = "unknown"
reply_to = "unknown"
ref      = "unknown"
head     = <<END_RECEIVED  # build received header, including loop flag
Newsgroups: #{ServersConfig::NEWSGROUP}
X-received-from: This message has been automatically forwarded from the
   ruby-talk mailing list by a gateway at #{ServersConfig::NEWSGROUP}. If it is
   SPAM, it did not originate at #{ServersConfig::NEWSGROUP}. Please report the
   original sender, and not us. Thanks!
   Please see http://hypermetrics.com/rubyhacker/clrFAQ.html#tag24 too.
X-rubymirror: yes
END_RECEIVED

# ...

Note that we get the headers started with an X-received-from explaining our service and add the X-rubymirror flag the other half of the Gateway will use to detect that this half of the Gateway sent this new post we are creating.

Now we need to parse the email headers:

# ...

# process message headers
valid_header = false
$stdin.each do |line|
  case line
  when /^X-Spam-Status: Yes/      # flag message as spam
    spam = true
  when /^X-rubymirror: yes/       # flag messages from news_to_mail
    mirrored = true
  when /^\s*$/                    # end of headers
    break
  when /^\s/                      # continuation line
    head << line if valid_header  # only allow after valid headers
  when valid_headers_re           # valid header
    valid_header = true

    # parse header data
    case line
    when /Message-Id:\s+(.*)/i
      msg_id = $1.sub(/\.+>$/, ">")
    when /In-Reply-To:\s*(.*)/i
      reply_to = $1
    when /References:\s*(.*)/i
      ref = $1
    when /^Subject:\s*(.*)/i
      subject = $1
    when /^From:\s*(.*)/i
      from = $1
    end

    head << line
  else                            # invalid header, discard
    valid_header = false
  end
end

# ...

There's nothing too tricky in the above code. We match headers with simple expressions, pulling the information we need into variables. We also set flags as appropriate and add to the headers we have started for the newsgroup post. This code stops reading at the blank line signaling the end of the email headers.

The code above didn't address flagged messages immediately, because we wanted to be able to log the key details about them. We now have those details, so it's time to address the flags:

# ...

# skip any flagged messages
if mirrored
  log.info "Skipping message ##{msg_id}, sent by news_to_mail"
  exit
elsif spam
  log.info "Ignoring Spam ##{msg_id}:  #{subject} -- #{from}"
  exit
end

# ...

As you can see, flagged messages are noted in the log and we exit cleanly without further processing.

The Gateway does some final header doctoring in an attempt to set a reasonable References header and also includes the Ruby Talk message id for reader reference:

# ...

# doctor headers for Ruby Talk
if ref.nil?
    if reply_to.nil?
        if subject =~ /^Re:/
            head << "In-Reply-To: <this_is_a_dummy_message-id@rubygate>\n"
            head << "References: <this_is_a_dummy_message-id@rubygate>\n"
        end
    else
        head << "References: #{reply_to}\n"
    end
end
head << "X-ruby-talk: #{msg_id}\n"

# ...

We are finally ready to construct a complete Usenet post:

# ...

# construct final message
body = $stdin.read
msg  = head + "\n" + body
msg.gsub!(/\r?\n/, "\r\n")

log.info "Sending message ##{msg_id}:  #{subject} -- #{from}..."
log.info "Message looks like: #{msg.inspect}"

# ...

The above code just joins the headers and existing message body, cleans up the newlines and logs our progress.

Actually sending the message is a two-step process. First we connect to our Usenet host:

# ...

# connect to NNTP host
begin
  nntp = nil
  Timeout.timeout(30) do
    nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
                          Net::NNTP::NNTP_PORT,
                          ServersConfig::NEWS_USER,
                          ServersConfig::NEWS_PASS )
  end
rescue Timeout::Error
    log.error "The NNTP connection timed out."
    exit -1
rescue
    log.fatal "Unable to establish connection to NNTP host:  #{$!.message}"
    exit -1
end

# ...

Above you can see several references to the ServerConfig Module I spoke of earlier. These constants contain exactly what their names indicate.

Note that we exit with an error code if anything goes wrong here, assuming the problem is temporary and allowing qmail to try again later.

The final step is to send the message:

# ...

# attempt to send newsgroup post
unless $DEBUG
  begin
    result = nil
    Timeout.timeout(30) { result = nntp.post(msg) }
  rescue Timeout::Error
      log.error "The NNTP post timed out."
      exit -1
  rescue
    log.fatal "Unable to post to NNTP host:  #{$!.message}"
    exit -1
  end
  log.info "...  Sent.  nntp.post() result = #{result}"
end

The above code makes the post and logs what we have accomplished. Again we exit with error codes if something goes wrong, to signal retries. The $DEBUG check allows me to test Gateway operation with everything but the actual post send when needed.

Possible Improvements

Usenet and email are two different worlds with opposing rules. Our Usenet host, like many, does not allow the posting of multipart/alternative messages (used to send HTML email). Some have expressed a desire for the Gateway to convert these messages into a Usenet safe format. This could possibly be done by using the text/plain variant of content, when provided, and stripping the HTML when it is not. This change is of low importance to me, since I don't believe posters should be sending HTML email to Ruby Talk.

In a similar vein, some Usenet hosts reject certain types of multipart/mixed messages (used to send email attachments), generally those that have Base 64 encoded portions to avoid allowing binary content through. Our host allows such posts, but they may not be well circulated on Usenet for these reasons. Again we might be able to inline the content for these files, but this could get pretty tricky for some attachments. For example, imagine a post with a zip archive of files. This problem interests me more than HTML email.

The first step to either off these changes is probably to switch to a real email parsing library. I imagine the original code didn't use one because the choices weren't convenient when the Gateway was designed. I just cleaned up the code in my rewrite and don't have enough experience with such libraries to select the proper replacement. Odds are this could simplify a fair portion of the Gateway code though, if we find the right one. We are looking for a library that:

Makes it easy to read email headers.
Allow us to set new headers, for things like the no-mirror flag.
Allows us to remove unwanted headers. Alternately I guess we could build a new message object and copy over the headers we wish to keep.
Supports easy manipulation of multipart/alternative and multipart/mixed content.

If you have experience with such a library, please leave a comment below showing how this could be used to simplify the code above.

Hacking the Gateway

2014-04-04T14:45:37Z

[Update: The Ruby Gateway was retired in June of 2011. Our community simply grew past the point were we needed to combine the various groups, in my opinion.]

Though I rewrote the current Gateway and I handle the maintenance, it really belongs to the Ruby community. Because of that, I'm going to release the two primary source files on this blog for all to view and critique. This may have value to those who want to know how the Gateway works, those who would like to implement similar technologies, and those who would like to purpose changes to the Gateway code.

I do welcome purposed changes to the Gateway, but let's set some ground rules for the right way to make suggestions:

I will show the important elements of the Gateway code and do my best to explain it as I go. In return, please take the time to read what I write about the code and try to understand how it works. Poorly developed change requests increase my maintenance time with the Gateway, which all comes out of my free time, so please be considerate.
You purpose changes to the Gateway by commenting on the code articles. This is intended to be a public discussion with all of us working together. Don't email me or Ruby Talk ideas, I'm monitoring the comments here.
Show code in your requests. I don't want to throw the Gateway in a publicly accessible Subversion repository and start taking patches for several reasons. If you want a change, convince me to implement it. The best way to do that is to throw around some code showing me how we would build your request and how it would make the Gateway better.
I am thinking about some elements of the Gateway you are not, like the fact that I run this code on a server provided by my work where security is a consideration and the level of maintenance a change will inflict on me. I ask only that you keep this in mind as we debate changes. In return, I will be as open minded to improvements as possible.
Gateway changes will not happen overnight. (See note about free time above.) Please be patient.

What is the Ruby Talk Gateway?

2014-04-04T01:49:45Z

[Update: The Ruby Gateway was retired in June of 2011. Our community simply grew past the point were we needed to combine the various groups, in my opinion.]

The Ruby community makes use of both email and Usenet communication, in addition to other resources. The primary mailing list is Ruby Talk and the primary Usenet group is comp.lang.ruby. These two services are joined by the Ruby Gateway.

In 2001 The Pragmatic Programmers wrote the initial version of the Ruby Gateway to ferry messages back and forth between these two resources. Emails sent to Ruby Talk are posted as Usenet messages and Usenet posts are forwarded to Ruby Talk by the Gateway. The Gateway has had a few guardians and code changes since then, but the functionality remains the same.

I'm am the current caretaker of the Ruby Gateway. Highgroove Studios generously provides hosting for it and I monitor the system for problems. I also wrote the current version of the Gateway.

You are free to report Gateway problems for me to look into. Before you do though, please read the following notes:

I rewrote the entire Gateway after I assumed control of it. My code was deployed on December 4th, 2006, so anything before that is history. If you raise issues, please make sure they involve posts after that date.
I now have very detailed logs on everything the Gateway does, so please be specific. For example, please send me links to exact messages that appeared on one side of the Gateway, but not the other.
Our Usenet host does not allow us to post HTML emails (multipart/alternative). This is not changing. These messages are not supported. (Yes, that means you should not be sending HTML email to Ruby Talk.)
Our Gateway is an NNTP ↔ email Gateway. There has been at least one instance of a Usenet post using ancient header formatting predating NNTP. These messages are not supported.