Gray Soft / Rails

The Nice New Italian Restaurant of Server Monitoring

2014-04-10T21:25:30Z

David Heinemeier Hanson, the creator of Rails, has made one of his business strategy talks available recently. This is a great talk about how we all might be trying just a little too hard at what can be a fairly simple task.

We all hear advice like this over and over again, but we seem to forget it so fast that we need that constant reminding. Just today I saw a site redesigned to improve a section users haven't even seen yet (as that section isn't yet public). Is that really the top use of resources for an unlaunched portion of the site? How do we know they wouldn't have liked it? How do we know they will like the new version better? I'm sure it won't surprise readers to learn that this site is over budget on time and money.

The point of all of this is that I want to tell you about a new server monitoring tool the company I work for has recently launched. The new service is called Scout and we've done our dead-level best to keep to the simplicity principal both because we agree with David and because we flat out need it to work that way.

We build Rails applications for clients. That involves monitoring servers, naturally. If that process is heavy, it hurts our business in a very real way. We need to spend our time writing Rails code, not fiddling with server configurations. We've used many tools available for this in the past, but none of them hit our sweet spot. In short, we need to be able to:

Setup a new client for monitoring in minutes
Have a centralized interface where we can check up on all of our client machines
Receive timely notification when events we deem important occur (or fail to occur)
Be able to examine seemingly random data points side by side to locate possible trends
Add any new data we can think up to the tracking quickly and easily

That's exactly what Scout does. Installing it on a new client involves filling out a short web form, installing a gem, and setting up a Cron job. You can then check in on all of your client machines at any time through the web interface or even have daily reports emailed to you. Scout can also email you alerts, say when a client fails to check-in or when your free disk space falls below a number you are comfortable with. You can graph any data from the clients that makes sense to graph, separately or with other data points. Oh, and adding a plugin to track anything you can think of is as simple as building a simple little Ruby script.

If that sounds like the low-hassle server monitoring you have been waiting for, bounce on over to the site and test drive one of our free trials. My hope is that you will find, as we did, that this dramatically reduces the work needed to keep tabs on servers.

Five ActiveRecord Tips

2014-04-10T21:19:41Z

This article was written for the Railcasts 100th Episode Contest. I think the idea is great and I look forward to reading great tips from all who decide to participate.

1. `create_or_find_by_…`

I imagine most of you know that ActiveRecord can handle finders like:

MyARClass.find_or_create_by_name(some_name)

This will attempt to find the object that has some_name in its name field or, if the find fails, a new object will be created with that name. It's important to note that the order is exactly as I just listed it: find then create. Here are the relevant lines from the current Rails source showing the process:

record = find_initial(options)

if record.nil?
  record = self.new { |r| r.send(:attributes=, attributes, guard_protected_attributes) }
  #{'yield(record) if block_given?'}
  #{'record.save' if instantiator == :create}
  record
else
  record
end

The above code is inside a String literal fed to class_eval(), which is why you see interpolation being used.

Unfortunately, this process is subject to race conditions because the object could be created by another process (or Thread) between the find and the creation. If that happens, you are likely to run into another hardship in that calls to create() fail quietly (returning the unsaved object). These are some pretty rare happenings for sure, but they can be avoided under certain conditions.

I find myself typically using find_or_create_by_… for things like category groupings or geocoded locations. In my case, there always seems to be two unique properties I can count on:

I don't allow duplicate listings
I don't have to worry about entries being deleted from the database

When both of those conditions are true, you can rewrite the Rails helper method to be multiprocessing safe. The steps are:

Build validations so that attempts to save the duplicate fail
Be sure to use create!() instead of create()
Reverse the order to create!() then find()

Here's the practical example of how this plays out:

class Category < ActiveRecord::Base
  validates_uniqueness_of :name

  def self.create_or_find_by_name(name)
    create!(:name => name) rescue find_by_name(name)
  end
end

The idea here is that we try the create!() first and just let it fail if it doesn't validate (because it is a duplicate). I use create!() because it doesn't fail silently and the truth is that I generally prefer it for that very reason. You could also use create() and check the returned object with new_record?(), if you prefer. When the create!() fails, we know it's safe to call the finder because there is definitely a matching entry in the database.

2. `find_all_in_chunks`

I often want to scan whole tables in a Rails application, usually to generate reports about the data. Of course, for very large tables, a call to find(:all) is not generally a great idea as it uses up too much memory to slurp all of that data into Ruby objects.

I have a tiny little library I drop in most Rails projects now to help with this. It requires will_paginate, so be sure to install that plugin first. Then just drop this code in lib/find_all_in_chunks.rb:

module FindAllInChunks
  def find_all_in_chunks(query = Hash.new, &iterator)
    1.upto(1.0/0.0) do |i|
      records = paginate({:page => i, :per_page => 50}.merge(query))
      records.each(&iterator)
      break unless records.next_page
    end
  end
end

ActiveRecord::Base.extend(FindAllInChunks)

and add this line to config/environment.rb:

require "find_all_in_chunks"

The code is simple. It just fetches the data in chunks by paginating the results and walking through the pages until it reaches the last one. Each entry in the page is then yielded to the block you pass this finder. This makes it seem like you are working with a simple each() iterator, though it will eventually fetch all of the entries. You can pass will_paginate's :per_page parameter to adjust how many entries are retrieved at one time.

3. `FasterCSV` as an Import Tool

If you need to jump start the database by importing some content, FasterCSV can be a big help. It can easily feed CSV data into an ActiveRecord subclass using code like:

FCSV.foreach( csv_path, :headers           => true,
                        :header_converters => :symbol ) do |row|
  begin
    MyARClass.create!(row.to_hash)
  rescue ActiveRecord::RecordNotSaved # in current Rails
    # handle save failures here…
  end
end

This uses the first line of your CSV data as the field names, so be sure they match up to the names in your database. It's also a good idea to make sure the CSV data is in UTF-8 and set $KCODE = "U" when using an older version of Rails that doesn't do this for you.

I tend to just vendor the FasterCSV source and add a require for it in config/environment.rb, so I don't have to worry about installing the gem everywhere.

4. Learn to Love `ActiveRecord`'s `:select` Parameter

I think the most under used feature of ActiveRecord's finders is the :select parameter. You really owe it to yourself to play with this sucker until you get the "Ah ha!" moment.

The concept is simple: :select tells ActiveRecord which columns to fetch from the database.

By default, everything is fetched, but there are great reasons to exclude some fields. For example, if you are grabbing a bunch of objects from a sizable table (meaning that it has many fields) but you only need certain fields, throw the rest out! It will take ActiveRecord less time to fetch the data and convert it to Ruby and use less memory, possibly making it practical for you to fetch more entries at once. That's a lot of great gains!

Another thing to know is that ActiveRecord takes the field names it uses from the names provided here so use SQL's AS to get what you want. You can even add aggregate fields and alias them so they are just like normal fields for this query.

This has a million different applications, but, to give an example, I pull some email campaign statistics by type using a query like:

statistics = Profile.find( :all
  :select => "profiles.type, " +
             "COUNT(DISTINCT users.id) AS emailed, " +
             "COUNT(IF(profiles.viewed_offer = 1, 1, NULL)) AS viewed, " +
             "COUNT(IF(profiles.signed_up = 1, 1, NULL)) AS accepted, " +
             "COUNT(IF(users.email_status = 'Opted Out', 1, NULL)) AS unsubscribed",
  :joins  => "INNER JOIN users ON users.id = profiles.user_id " +
             "INNER JOIN emailings ON emailings.user_id = users.id",
  :group  => "profiles.type"
)

In this database a User has_many Profiles and Emailings are associated with User objects. I use :joins to link all of that together here. (The :include option seems to override your custom :select so I just use :joins instead.) Note that I INNER JOIN these tables to cut down on the data fetched. It's handy to remember that there are more options to SQL than just what ActiveRecord uses.

The :select parameter is the juicy part here. I setup a group of aggregate fields to track the statistics we care about, aliasing each of these to nice method names for the returned objects.

This single query combs through a ton of data very quickly and returns all the vital details we care about. Now it's important to note that what is returned here aren't really Profile objects as we typically think of them. These are more summary objects that have a new set of methods we defined in this query (emailed(), viewed(), accepted(), unsubscribed()).

5. Sneaking in a `HAVING` Clause

ActiveRecord doesn't have a parameter for including a HAVING clause in your database queries and I sometimes find myself needing it. Luckily, you can sneak it in on the end of your :group parameter without needing to resort to a find_by_sql() call. (I don't recall ever using HAVING without GROUP BY, though there probably are some cases where it would make sense to do so.)

As an example, here's a query from one of my Rails applications that finds all duplicate email addresses in the database:

duplicates = User.find( :all,
  :select     => "email, COUNT(email) AS duplicate_count",
  :conditions => "email IS NOT NULL AND email != ''",
  :group      => "email HAVING duplicate_count > 1"
)

What Not to Test

2014-03-27T15:47:39Z

I've now seen multiple claims amounting to something like, "I built this Rails application with just testing." I think it's great that people can do that. They are obviously smarter than me. I need a browser to build a Web application.

I do test my Rails projects, of course. I'm a huge fan of testing and I'm always telling people how much it could help with their work. However, I believe there's a time to test and a time not to, if you can imagine that. Yes, you heard me right, there are things I don't test and I'm sure this next revelation will shock you even more:

I don't test Rails views.

Now before everyone fires up their mail client and pours on the hate mail, let's make sure we are very clear about what I just said. I do write thorough unit tests for all of my model classes, of course. I also test the controllers as much as possible. I make sure the data I expect ends up in the right instance variables with the magic assigns(). I also use the session Hash to validate what I am remembering about the user. If a controller action is a complex logic branch that can end up in several different places, I will also make sure I add some assertions to verify that the right template handled the response.

All I said is that I don't test the view layer. I'm talking about the final HTML pages here. Those are what I don't validate.

Now that we are clear about what I don't test, let's cover why I don't. I assure you this isn't some random choice or even an attempt to avoid work. I have found that not testing the views works out better for me and I want to share my reasons with you.

As I've already hinted at, I'm constantly viewing my Rails work in a Web browser. If there's a problem, I'm going to see it pretty quickly. In fact, I have a much better chance of seeing some browser oddities than I do of successfully coding a test to catch them, I think.
If you are doing Rails correctly, the only programming logic in your templates is flow control, right? (Helpers are a legitimate exception here, but you can test them normally.) That makes your templates little more than data files in my eyes and I don't validate the contents of data files.
HTML pages change rapidly. Practically each time I look at a page, I will tweak something. If that breaks a test, I just have to change two things instead of one. To me, that's the whole point of data files, I can change them without worrying about the code that relies on them.
The main tool for testing views, assert_tag(), is a clever but flawed helper. Have you seen what it spits out when the test starts failing? "I couldn't find tag xyz in this-giant-mess-of-unformatted-HTML." To me, that's like saying, "I couldn't find this needle in this haystack. You try." Even a great tool like unit_diff can't fix that.

All of that adds up to the fact that I am already visually checking what view tests would be checking. I feel I am more thorough and accurate than the tests I know how to write for this process. It's also a low risk area, since I'm not hiding any important functionality in the templates. Given all of the above, I feel I lose nothing and gain speed by not writing those tests.

This is just my system, of course. Obviously, others are doing things differently. I just wanted to share what is working for me.

Add to Subversion

2014-03-28T20:40:02Z

I love Rails generators and I love Subversion, but I'm not so sure they love each other.

Little is worse than generating some scaffold files or a login system, tediously typing svn add ... a bunch of times, and later learning that you missed a file. Oops.

Given that, here's my hack to get the Rails generators and Subversion to kiss and make up:

desc "Add generated files to Subversion"
task :add_to_svn do
  sh %Q{svn status | ruby -nae 'system "svn add \\#{$F[1]}" if $F[0] == "?"'}
end

I put that in "lib/tasks/add_to_svn.rake" for all my Rails projects. You can then access it whenever you need with rake add_to_svn.

I've seen similar tasks posted, but they've always been Unix specific and I work with some Windows programmers. This is my attempt to help both on platforms.

How does it work?

You're probably not use to seeing ugly Ruby code like that, but Ruby supports a lot of command-line shortcuts that it inherited from Perl. Here's a breakdown of the options used:

-e The code to execute directly follows this option.
-n Wrap my code in a standard Unix filter style processing loop.
-a Auto-split each line of input.

Together they mean that Ruby sees the above input as something close to:

ARGF.each_line do |line|
  $F = line.split

  # my custom code is inserted here...
  system "svn add #{$F[1]}" if $F[0] == "?"
end

That just asks Ruby to call svn add ... for each file svn status flagged with a "?". Those will be all the files Subversion doesn't yet recognize.

WARNING: Make sure you've told Subversion to ignore files you don't want it messing with (like Rails's log files), or they will be added!

Gray Soft / Rails

The Nice New Italian Restaurant of Server Monitoring

Five ActiveRecord Tips

1. create_or_find_by_…

2. find_all_in_chunks

3. FasterCSV as an Import Tool

4. Learn to Love ActiveRecord's :select Parameter

5. Sneaking in a HAVING Clause