tag:graysoftinc.com,2014-03-20:/posts/32 2014-04-04T21:54:13Z James Edward Gray II tag:graysoftinc.com,2007-04-18:/comments/106 2014-04-04T21:54:13Z

Oops. Good catch. I have corrected the article.

<p>Oops. Good catch. I have corrected the article.</p> James Edward Gray II tag:graysoftinc.com,2007-04-18:/comments/105 2019-09-10T09:53:35Z

`LightCsv` do not use `StringIO`. It use `StringScanner`.

<p><code>LightCsv</code> do not use <code>StringIO</code>.<br> It use <code>StringScanner</code>.</p> tommy tag:graysoftinc.com,2007-04-16:/posts/32 2019-09-10T09:53:35Z

Some speed demons are faster than me. Let's take a look at the new efforts in CSV parsing.

<p>If your number one concern when working with CSV data in Ruby is raw speed, you might want to know that FasterCSV is no longer the fastest option.</p> <p>There are a couple of new contenders for Ruby CSV processing including a C extension called <a href="https://rubygems.org/gems/simplecsv">SimpleCSV</a> and a pure Ruby library called <a href="https://github.com/smulube/lightcsv">LightCsv</a>. I haven't been able to test <code>SimpleCSV</code> locally, because I can't get it to build on my box, but users do tell me it's faster. I have run some trivial benchmarks for <code>LightCsv</code> though and it too is pretty quick:</p> <pre><code>$ rake benchmark (in /Users/james/Documents/faster_csv) time ruby -r csv -e '6.times { CSV.foreach("test/test_data.csv") { |row| } }' real 0m5.481s user 0m5.468s sys 0m0.010s time ruby -r lightcsv -e \ '6.times { LightCsv.foreach("test/test_data.csv") { |row| } }' real 0m0.358s user 0m0.349s sys 0m0.008s time ruby -r lib/faster_csv -e \ '6.times { FasterCSV.foreach("test/test_data.csv") { |row| } }' real 0m0.742s user 0m0.732s sys 0m0.009s </code></pre> <p>It's important to note that <code>LightCsv</code> is indeed very "light." <code>FasterCSV</code> has grown up into a feature rich library that provides many different ways to look at your data. In contrast, <code>LightCsv</code> doesn't yet allow you to set column or row separators. Given that, it's only an option for vanilla CSV you just need to iterate over. If that's what you have though, and speed counts, it might just be the right choice.</p> <p>For the curious, <code>LightCsv</code> achieves its speed advantage in two ways. First, it uses <code>StringScanner</code> to manage the parsing. <code>StringScanner</code> is a C extension, though it is a standard library installed with Ruby.</p> <p>More importantly, I suspect, <code>LightCsv</code> uses an input buffer for reading while <code>FasterCSV</code> works line by line. I suspect this second difference accounts for the majority of the speed increase since the buffered code will hit the hard drive quite a bit less for the average CSV file. This does require more memory though, of course.</p> <p>Aside from these differences, <code>FasterCSV</code> and <code>LightCsv</code> have very similar parsers.</p> James Edward Gray II