Gray Soft / The Gateway

news_to_mail.rb

2014-04-04T19:16:17Z

[Note: You need to know what the Gateway is before reading this article.]

The second half of the Ruby Gateway runs as a cron job every five minutes and is expected to move all new newsgroup messages from the NNTP host to the Ruby Talk mailing list. The cron invocation is simply:

ruby /path/to/gateway/bin/news_to_mail.rb /path/to/news_to_mail.log

This side of the Gateway is not piped any messages and exit codes from it are not monitored. It needs to tend its own affairs.

The Code

Here's the source:

GATEWAY_DIR  = File.join(File.dirname(__FILE__), "..").freeze
DATA_DIR     = File.join(GATEWAY_DIR, "data").freeze
LAST_ID_FILE = File.join(DATA_DIR, "last_news_id.txt").freeze
PID_FILE     = File.join(DATA_DIR, "pid.txt").freeze

$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join(GATEWAY_DIR, "lib")

# ...

The script begins by locating the root directory of the Gateway source and the data sub directory, building constants for two external files it will interact with, and adjusting the $LOAD_PATH so that it can require needed resources.

Here are those requires:

# ...

require "servers_config"
require "nntp"

require "net/smtp"
require "logger"
require "timeout"

# ...

The first two requires are the same code used by the other half of the Gateway. The last three are standard Ruby libraries the code will make use of.

# ...

# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "

# ...

This code opens a log file to record its status and adjusts the date and time formatting.

With setup complete, it's now vital to ensure that the current execution is the only copy of the code we are running. Should the Gateway get behind and fail to complete before the cron job launches another copy, duplicate messages could be sent to the list or worse.

# ...

# make sure only one copy is ever running at a time
begin
  File.open(PID_FILE, File::CREAT|File::EXCL|File::WRONLY) { |pid| pid.puts $$ }
  at_exit do
    begin
      File.unlink(PID_FILE)
    rescue
      log.error "Unable to unlink pid file:  #{$!.message}"
    end
  end
rescue
  pid = File.read(PID_FILE).strip.to_i rescue "unknown"
  log.warn "Process #{pid} was already running"
  exit
end

# ...

The script attempts to write its process ID number to a file. The File::CREAT|File:EXCL options ensure that the file is created or an exception is thrown, if it already existed. When the exception comes it is assumed that an earlier execution is still running and the script exits. If able to create the process ID file, the scripts knows it is the only copy running and it is safe to proceed. The code then immediately arranges for Ruby to remove the file on exit.

Note that to check for the file then create it if missing would introduce a race condition. Another execution could create the file between check and creation. As unlikely as that is for a five minute cron job, it's better to play things safe and just try the creation directly.

Having made it this far, the script needs to refresh its memory of where it was in the list of newsgroup messages:

# ...

# find out what the last news item sent was
begin
  last_id_sent = Integer(File.read(LAST_ID_FILE).strip)
rescue
  log.fatal "Unable to get message ID of last news post:  #{$!.message}"
  exit -1
end

# ...

A simple text file is used to hold the message number of the last file seen. The code reads that file back to know where to resume its work.

It's now time to connect to Usenet.

# ...

# 
# connect to NNTP host, switch to newgroup, and see how many messages are
# available to read
# 
begin
  nntp, last_id_available = nil, nil
  Timeout.timeout(30) do
    nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
                          Net::NNTP::NNTP_PORT,
                          ServersConfig::NEWS_USER,
                          ServersConfig::NEWS_PASS )
    last_id_available = nntp.group(ServersConfig::NEWSGROUP)[3].to_i
  end
rescue Timeout::Error
    log.error "The NNTP connection timed out."
    exit
rescue
    log.fatal "Unable to establish connection to NNTP host:  #{$!.message}"
    exit -1
end

# ...

This code tries to establish a connection to the NNTP host and switch to the comp.lang.ruby group being monitored. A reasonable timeout is imposed on this code and will be for all network operations attempted. If the connection is slow for some reason, another attempt can be made by the next cron execution.

The script now advances the NNTP reader to the last message seen, which turns out to be complicated to get right:

# ...

# switch to the last message we sent (or the first real message before that)
begin
  Timeout.timeout(30) do
    nntp.stat(last_id_sent)
  end
rescue Timeout::Error
    log.error "The NNTP message shift timed out."
    exit
rescue
  if $!.message.include?("Bad article number") and last_id_sent.nonzero?
    last_id_sent -= 1
    retry
  else
    log.fatal "Unable to switch to the last message:  #{$!.message}"
    exit -1
  end
end

# ...

The problem here is that Usenet messages can be revoked. When that happens, the last message the script saw may no longer exist. The real goal is the message after the last seen post though and an NNTP NEXT command will skip missing IDs. Given that, the code backs up in the count until it can find an existing message. NEXTing forward from that will skip the vanished IDs and land the reader at the desired later message.

To make that leap forward in the message count, the script enters an infinite loop of message processing:

# ...

# main event loop
loop do
  # advance to the next message, if there is one
  new_last_id, more_messages = nil, true
  begin
    Timeout.timeout(30) do
      new_last_id = nntp.next[1].to_i
    end
  rescue Timeout::Error
      log.error "Advancing the current NNTP message timed out."
      exit
  rescue
    more_messages = false
  end
  break unless more_messages

  # reality check that we are moving forward
  if new_last_id <= last_id_sent
    log.fatal "new_last_id (#{new_last_id}) <= last_id_sent " +
              "(#{last_id_sent}), quitting to prevent sending duplicates"
    exit -1
  end

  # ...

As I said, a NEXT command is sent to the host bringing the reader to an unseen message. A quick sanity check is then made to ensure the count is rising.

Note that this is also the code to determine when there are no more messages. If the NEXT command fails, the message processing loop terminates. You really need to used this combination of an infinite loop and last message detection when working with Usenet. Any method of counting is pointless, thanks to the numbering issues I described.

At this point, it's time to actually start reading the post:

  # ...

  # pull the headers for the current messae
  begin
    head = nil
    Timeout.timeout(30) do
      head = nntp.head(new_last_id)[3].join("\r\n") + "\r\n"
    end
  rescue Timeout::Error
      log.error "Fetching the message headers timed out."
      exit
  rescue
    log.error "Unable to retrieve headers for message ##{new_last_id}, " +
              "will try again later:  #{$!.message}"
    exit
  end

  # 
  # don't send articles that the mail_to_news program has previously forwarded
  # to the newsgroup (or we'd loop)
  # 
  if head =~ /^X-rubymirror:/
    log.info "Skipping message ##{new_last_id}, sent by mail_to_news"
    next
  end

  # ...

This code pulls the message headers and performs the loop check to ensure that messages sent by mail_to_news.rb are skipped.

  # ...

  # pull the body for the current message
  begin
    body = nil
    Timeout.timeout(30) do
      body = nntp.body(new_last_id)[3].join("\r\n") + "\r\n"
    end
  rescue Timeout::Error
      log.error "Fetching the message body timed out."
      exit
  rescue
    log.error "Unable to retrieve body for message ##{new_last_id}, " +
              "will try again later:  #{$!.message}"
    exit
  end

  # scan headers for message data
  subject = head =~ /^Subject:\s+([^\r\n]*)/     ? $1 : "?"
  from    = head =~ /^From:\s+([^\r\n]*)/        ? $1 : "?"

  # ensure the message is properly addressed
  unless head =~ /^To:/
    head = "To: #{ServersConfig::MAILING_LIST}\r\n" + head
  end

  log.info "Sending message ##{new_last_id}:  #{subject} -- #{from}..."

  # ...

The body is then pulled. The script then scans for some message details in the headers to allow us to log accurately about the post. The message is also addressed.

  # ...

  # extracting path information (poster's host, first nntp hop)
  posting_host = head =~ /^NNTP-Posting-Host:\s+(\S[^\r\n]*)/i ? $1 : "unknown"

  path = head =~ /^Path:\s+([^\r\n]*)/i ? $1 : "unknown"
  path = path.split("!").delete_if { |hop| hop !~ /^(([^.]+)\.)+([^.]+)$/ }

  first_hop = path.last

  # adding pseudo "Received:" header and rubymirror header
  head += <<END_RECEIVED.gsub("\n", "\r\n")
Received: from #{first_hop} (#{first_hop})
     by #{ServersConfig::NEWSGROUP} with NTTP id #{new_last_id}
     for <#{ServersConfig::MAILING_LIST}>; #{Time.now.to_s}
Received: from [#{posting_host}]
     by #{first_hop} (unknown) with NNTP id #{new_last_id}
     for <#{ServersConfig::NEWSGROUP}>; #{Time.now.to_s}
Received: from Usenet via a Usenet to mail gateway located at
     #{ServersConfig::NEWSGROUP}.  This service provided as a courtesy
     to the ruby-talk mailing list.  If this message is SPAM, its
     ultimate origin is Usenet, not this gateway program.  All
     subscribers to the ruby-talk mailing list agree to receive the
     Usenet postings made to comp.lang.ruby via this gateway.  Please
     see http://www.ruby-lang.org/ruby-talk-usenet-policy.html.
X-From-Usenet: see Received: header above.
X-rubymirror: yes

END_RECEIVED

  # build final message
  msg = head + body
  log.info "Message looks like: #{msg.inspect}"

  # ...

The above adds a header explaining the service and constructs the final message.

  # ...

  # attempt to send email
  unless $DEBUG
    begin
      Timeout.timeout(30) do
        Net::SMTP.start(ServersConfig::SMTP_SERVER, 25) do |smtp|
          smtp.send_mail( msg,
                          ServersConfig::MAIL_SENDER,
                          ServersConfig::MAILING_LIST )
        end
      end
    rescue Timeout::Error
        log.error "The SMTP connection timed out."
        exit
    rescue
      log.fatal "Unable to send email:  #{$!.message}"
      exit -1
    end
    log.info "...  Sent."
  end

  # ...

Here a connection is established to the SMTP server and the email is sent. The $DEBUG flag allows me to spot check Gateway functionality without actually sending emails to Ruby Talk.

On to the final step:

  # ...

  # record new high-water mark
  begin
    unless $DEBUG or new_last_id == last_id_sent
      File.open(LAST_ID_FILE, "w") { |file| file.puts new_last_id }
    end
  rescue
    log.error "Unable to write message ID to file:  #{$!.message}"
  end
  last_id_sent = new_last_id
  log.info "New last message sent ID:  #{last_id_sent}"
end

Getting this far means the script sent an email successfully, so it records the new last seen ID for future runs. That final end signals the end of the message processing event loop.

Possible Improvements

This code is pretty feature complete.

About the only enhancement I can think of would be to DRY up some of the timeout and error handling code. That's trickier that you might guess because of the extra layers of scoping plus the error messages and exit codes. We almost need a multi-block syntax here. If someone poses a reasonable alternative I'll consider it, but I've tried to keep the code pretty straight forward and the truth is that is just may not be worth the effort.

Any comments or suggestions for the Gateway can be left as comments below.

mail_to_news.rb

2014-04-04T19:20:41Z

[Note: You need to know what the Gateway is before reading this article.]

There are two halves to the Ruby Gateway. One half runs as a qmail filter for an email address on the Ruby Talk mailing list. Every message sent to that address is piped through this filter with a shell script like:

ruby /path/to/gateway/bin/mail_to_news.rb /path/to/mail_to_news.log

The email is piped to the filter via the standard input and the code is expected to handle the message by posting it to comp.lang.ruby or choosing to ignore it. If the filter exits normally, qmail considers the matter handled. A non-zero exit code will cause the filter to be called with that same message again later.

The Code

Let's dive right into the source of this half of the Gateway:

GATEWAY_DIR = File.join(File.dirname(__FILE__), "..").freeze

$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join(GATEWAY_DIR, "lib")

# ...

The code above just sets things up so this script can require some other files in the project normally. Here are those requires:

# ...

require "servers_config"
require "nntp"

require "net/smtp"
require "logger"
require "timeout"

# ...

The last three requires are standard Ruby libraries. The first two are not.

The servers_config.rb file sets up a ServerConfig Module with information needed to connect to our email and Usenet hosts. I will not show this file, but the references should be obvious when see them.

The nntp.rb is pretty much a vendored copy of the net-nntp library. I've made some minor changes for debugging output purposes, but the library functions the same.

We're now ready to initiate logging. Here's the code that starts that process:

# ...

# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "

# ...

That just builds a Logger object and cleans up the default date and time formatting.

Now it's time to start setting variables in preparation for the coming email parse:

# ...

# only allow certain headers through
VALID_HEADERS    = %w[ From Subject References In-Reply-To Message-Id Content-Type
                       Content-Transfer-Encoding Date X-ML-Name X-Mail-Count
                       X-X-Sender ]
valid_headers_re = /^(?:#{VALID_HEADERS.join("|")}):/i

# ...

The Gateway passes through only a subset of the email headers for the Usenet post. The above is the list of those headers and the Regexp that will locate them.

Now, there are two types of messages we do not wish to forward: spam and a message sent to Ruby Talk by the other half of the Gateway (causing an infinite loop of sending). The following code prepares flags for these conditions:

# ...

# message flags
spam     = false
mirrored = false

# ...

The following code allocates variables to hold key header information parsed from the message:

# ...

# header data
msg_id   = "unknown"
subject  = "unknown"
from     = "unknown"
reply_to = "unknown"
ref      = "unknown"
head     = <<END_RECEIVED  # build received header, including loop flag
Newsgroups: #{ServersConfig::NEWSGROUP}
X-received-from: This message has been automatically forwarded from the
   ruby-talk mailing list by a gateway at #{ServersConfig::NEWSGROUP}. If it is
   SPAM, it did not originate at #{ServersConfig::NEWSGROUP}. Please report the
   original sender, and not us. Thanks!
   Please see http://hypermetrics.com/rubyhacker/clrFAQ.html#tag24 too.
X-rubymirror: yes
END_RECEIVED

# ...

Note that we get the headers started with an X-received-from explaining our service and add the X-rubymirror flag the other half of the Gateway will use to detect that this half of the Gateway sent this new post we are creating.

Now we need to parse the email headers:

# ...

# process message headers
valid_header = false
$stdin.each do |line|
  case line
  when /^X-Spam-Status: Yes/      # flag message as spam
    spam = true
  when /^X-rubymirror: yes/       # flag messages from news_to_mail
    mirrored = true
  when /^\s*$/                    # end of headers
    break
  when /^\s/                      # continuation line
    head << line if valid_header  # only allow after valid headers
  when valid_headers_re           # valid header
    valid_header = true

    # parse header data
    case line
    when /Message-Id:\s+(.*)/i
      msg_id = $1.sub(/\.+>$/, ">")
    when /In-Reply-To:\s*(.*)/i
      reply_to = $1
    when /References:\s*(.*)/i
      ref = $1
    when /^Subject:\s*(.*)/i
      subject = $1
    when /^From:\s*(.*)/i
      from = $1
    end

    head << line
  else                            # invalid header, discard
    valid_header = false
  end
end

# ...

There's nothing too tricky in the above code. We match headers with simple expressions, pulling the information we need into variables. We also set flags as appropriate and add to the headers we have started for the newsgroup post. This code stops reading at the blank line signaling the end of the email headers.

The code above didn't address flagged messages immediately, because we wanted to be able to log the key details about them. We now have those details, so it's time to address the flags:

# ...

# skip any flagged messages
if mirrored
  log.info "Skipping message ##{msg_id}, sent by news_to_mail"
  exit
elsif spam
  log.info "Ignoring Spam ##{msg_id}:  #{subject} -- #{from}"
  exit
end

# ...

As you can see, flagged messages are noted in the log and we exit cleanly without further processing.

The Gateway does some final header doctoring in an attempt to set a reasonable References header and also includes the Ruby Talk message id for reader reference:

# ...

# doctor headers for Ruby Talk
if ref.nil?
    if reply_to.nil?
        if subject =~ /^Re:/
            head << "In-Reply-To: <this_is_a_dummy_message-id@rubygate>\n"
            head << "References: <this_is_a_dummy_message-id@rubygate>\n"
        end
    else
        head << "References: #{reply_to}\n"
    end
end
head << "X-ruby-talk: #{msg_id}\n"

# ...

We are finally ready to construct a complete Usenet post:

# ...

# construct final message
body = $stdin.read
msg  = head + "\n" + body
msg.gsub!(/\r?\n/, "\r\n")

log.info "Sending message ##{msg_id}:  #{subject} -- #{from}..."
log.info "Message looks like: #{msg.inspect}"

# ...

The above code just joins the headers and existing message body, cleans up the newlines and logs our progress.

Actually sending the message is a two-step process. First we connect to our Usenet host:

# ...

# connect to NNTP host
begin
  nntp = nil
  Timeout.timeout(30) do
    nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
                          Net::NNTP::NNTP_PORT,
                          ServersConfig::NEWS_USER,
                          ServersConfig::NEWS_PASS )
  end
rescue Timeout::Error
    log.error "The NNTP connection timed out."
    exit -1
rescue
    log.fatal "Unable to establish connection to NNTP host:  #{$!.message}"
    exit -1
end

# ...

Above you can see several references to the ServerConfig Module I spoke of earlier. These constants contain exactly what their names indicate.

Note that we exit with an error code if anything goes wrong here, assuming the problem is temporary and allowing qmail to try again later.

The final step is to send the message:

# ...

# attempt to send newsgroup post
unless $DEBUG
  begin
    result = nil
    Timeout.timeout(30) { result = nntp.post(msg) }
  rescue Timeout::Error
      log.error "The NNTP post timed out."
      exit -1
  rescue
    log.fatal "Unable to post to NNTP host:  #{$!.message}"
    exit -1
  end
  log.info "...  Sent.  nntp.post() result = #{result}"
end

The above code makes the post and logs what we have accomplished. Again we exit with error codes if something goes wrong, to signal retries. The $DEBUG check allows me to test Gateway operation with everything but the actual post send when needed.

Possible Improvements

Usenet and email are two different worlds with opposing rules. Our Usenet host, like many, does not allow the posting of multipart/alternative messages (used to send HTML email). Some have expressed a desire for the Gateway to convert these messages into a Usenet safe format. This could possibly be done by using the text/plain variant of content, when provided, and stripping the HTML when it is not. This change is of low importance to me, since I don't believe posters should be sending HTML email to Ruby Talk.

In a similar vein, some Usenet hosts reject certain types of multipart/mixed messages (used to send email attachments), generally those that have Base 64 encoded portions to avoid allowing binary content through. Our host allows such posts, but they may not be well circulated on Usenet for these reasons. Again we might be able to inline the content for these files, but this could get pretty tricky for some attachments. For example, imagine a post with a zip archive of files. This problem interests me more than HTML email.

The first step to either off these changes is probably to switch to a real email parsing library. I imagine the original code didn't use one because the choices weren't convenient when the Gateway was designed. I just cleaned up the code in my rewrite and don't have enough experience with such libraries to select the proper replacement. Odds are this could simplify a fair portion of the Gateway code though, if we find the right one. We are looking for a library that:

Makes it easy to read email headers.
Allow us to set new headers, for things like the no-mirror flag.
Allows us to remove unwanted headers. Alternately I guess we could build a new message object and copy over the headers we wish to keep.
Supports easy manipulation of multipart/alternative and multipart/mixed content.

If you have experience with such a library, please leave a comment below showing how this could be used to simplify the code above.

Hacking the Gateway

2014-04-04T14:45:37Z

[Update: The Ruby Gateway was retired in June of 2011. Our community simply grew past the point were we needed to combine the various groups, in my opinion.]

Though I rewrote the current Gateway and I handle the maintenance, it really belongs to the Ruby community. Because of that, I'm going to release the two primary source files on this blog for all to view and critique. This may have value to those who want to know how the Gateway works, those who would like to implement similar technologies, and those who would like to purpose changes to the Gateway code.

I do welcome purposed changes to the Gateway, but let's set some ground rules for the right way to make suggestions:

I will show the important elements of the Gateway code and do my best to explain it as I go. In return, please take the time to read what I write about the code and try to understand how it works. Poorly developed change requests increase my maintenance time with the Gateway, which all comes out of my free time, so please be considerate.
You purpose changes to the Gateway by commenting on the code articles. This is intended to be a public discussion with all of us working together. Don't email me or Ruby Talk ideas, I'm monitoring the comments here.
Show code in your requests. I don't want to throw the Gateway in a publicly accessible Subversion repository and start taking patches for several reasons. If you want a change, convince me to implement it. The best way to do that is to throw around some code showing me how we would build your request and how it would make the Gateway better.
I am thinking about some elements of the Gateway you are not, like the fact that I run this code on a server provided by my work where security is a consideration and the level of maintenance a change will inflict on me. I ask only that you keep this in mind as we debate changes. In return, I will be as open minded to improvements as possible.
Gateway changes will not happen overnight. (See note about free time above.) Please be patient.

What is the Ruby Talk Gateway?

2014-04-04T01:49:45Z

[Update: The Ruby Gateway was retired in June of 2011. Our community simply grew past the point were we needed to combine the various groups, in my opinion.]

The Ruby community makes use of both email and Usenet communication, in addition to other resources. The primary mailing list is Ruby Talk and the primary Usenet group is comp.lang.ruby. These two services are joined by the Ruby Gateway.

In 2001 The Pragmatic Programmers wrote the initial version of the Ruby Gateway to ferry messages back and forth between these two resources. Emails sent to Ruby Talk are posted as Usenet messages and Usenet posts are forwarded to Ruby Talk by the Gateway. The Gateway has had a few guardians and code changes since then, but the functionality remains the same.

I'm am the current caretaker of the Ruby Gateway. Highgroove Studios generously provides hosting for it and I monitor the system for problems. I also wrote the current version of the Gateway.

You are free to report Gateway problems for me to look into. Before you do though, please read the following notes:

I rewrote the entire Gateway after I assumed control of it. My code was deployed on December 4th, 2006, so anything before that is history. If you raise issues, please make sure they involve posts after that date.
I now have very detailed logs on everything the Gateway does, so please be specific. For example, please send me links to exact messages that appeared on one side of the Gateway, but not the other.
Our Usenet host does not allow us to post HTML emails (multipart/alternative). This is not changing. These messages are not supported. (Yes, that means you should not be sending HTML email to Ruby Talk.)
Our Gateway is an NNTP ↔ email Gateway. There has been at least one instance of a Usenet post using ancient header formatting predating NNTP. These messages are not supported.