My Projects

The projects that I have spent significant time on.

16

APR
2007

No Longer the Fastest Game in Town

If your number one concern when working with CSV data in Ruby is raw speed, you might want to know that FasterCSV is no longer the fastest option.

There are a couple of new contenders for Ruby CSV processing including a C extension called SimpleCSV and a pure Ruby library called LightCsv. I haven't been able to test SimpleCSV locally, because I can't get it to build on my box, but users do tell me it's faster. I have run some trivial benchmarks for LightCsv though and it too is pretty quick:

$ rake benchmark
(in /Users/james/Documents/faster_csv)
time ruby -r csv -e '6.times { CSV.foreach("test/test_data.csv") { |row| } }'

real    0m5.481s
user    0m5.468s
sys     0m0.010s
time ruby -r lightcsv -e \
'6.times { LightCsv.foreach("test/test_data.csv") { |row| } }'

real    0m0.358s
user    0m0.349s
sys     0m0.008s
time ruby -r lib/faster_csv -e \
'6.times { FasterCSV.foreach("test/test_data.csv") { |row| } }'

real    0m0.742s
user    0m0.732s
sys     0m0.009s

It's important to note that LightCsv is indeed very "light." FasterCSV has grown up into a feature rich library that provides many different ways to look at your data. In contrast, LightCsv doesn't yet allow you to set column or row separators. Given that, it's only an option for vanilla CSV you just need to iterate over. If that's what you have though, and speed counts, it might just be the right choice.

For the curious, LightCsv achieves its speed advantage in two ways. First, it uses StringScanner to manage the parsing. StringScanner is a C extension, though it is a standard library installed with Ruby.

More importantly, I suspect, LightCsv uses an input buffer for reading while FasterCSV works line by line. I suspect this second difference accounts for the majority of the speed increase since the buffered code will hit the hard drive quite a bit less for the average CSV file. This does require more memory though, of course.

Aside from these differences, FasterCSV and LightCsv have very similar parsers.

Comments (2)
  1. tommy
    tommy April 18th, 2007 Reply Link

    LightCsv do not use StringIO.
    It use StringScanner.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 18th, 2007 Reply Link

      Oops. Good catch. I have corrected the article.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    3. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Logan
      Logan September 7th, 2016 Reply Link

      I'm doing an internship acido tranexamico argentina precio She ordered Deutsche to reinstate the traders - known only as Ardalan G, Kai-Uwe K, Markus K and Joerg V because their full names were not made public for legal reasons - and pay them their salaries since firing them in February.
      ramipril 5 mg 100 stck preisvergleich Alarmed by the longest recession since the euro was launched and record unemployment, EU leaders have slowed up the pace of their austerity drive in recent months. And there are signs that activity is picking up.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    4. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Alfonso
      Alfonso September 7th, 2016 Reply Link

      Just over two years precio medicamento xeloda Lisbeth Gronlund, a co-director of UCS's Global Security Program who worked on the report, said in an interview that modernization efforts by NNSA could undermine confidence in the reliability of the arsenal.
      prijs ibuprofen 400 “Bud Selig is a disgrace to the game, to the players and our children,” said the advertisement, which was signed by Hispanics Across America president Fernando Mateo, who has led protests outside MLB’s offices during the arbitration. It’s not clear who picked up the tab for the black-and-white advertisement that appeared in the sports section of the Times. A newspaper industry executive estimated that a full-page ad in the newspaper costs $75,000 or more.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    5. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Jewel
      Jewel September 7th, 2016 Reply Link

      History precio de crema aldara en venezuela John Empson and Jeff Rowbottom will lead the global capitalmarkets structuring and distribution efforts, while Adam Smithwill oversee MerchCap Solutions, a middle market capital marketsjoint venture with Stone Point Capital LLC.
      comment ca marche une mobicarte “At the moment we’re seeing a very encouraging decline in smoking rates. Since the smoke free legislation was introduced in 2007 we’ve seen a fall in asthma admissions, but evidence suggests that it could take several years for the decline to work through and show up in statistics for other forms of lung disease… We haven’t seen the last of the impact of old habits.”

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    6. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Issac
      Issac September 7th, 2016 Reply Link

      Is this a temporary or permanent position? angelique noire kaufen In addition to CGI, the foundation projects include the Clinton Climate Initiative, the Clinton Health Access Initiative, the Alliance for a Healthier Generation — which has focused on ending obesity — and solving poverty, as well as reducing chronic health issues like, AIDs in Africa. CGI, which attracts top-tier participants for a $20,000 registration fee and requires them to make a good-faith effort to meet commitments for the public good by the following year, was turned into a separate entity while Hillary Clinton was at the State Department but was recently folded back in.
      benzoyl peroxide 2.5 kopen Japan' Nikkei share average shed 0.7 percent to itslowest since June 28 after data showed its economy grew at aslower-than-expected pace in April-June, triggering investors tocut their risk exposure.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    7. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Dro4er
      Dro4er September 7th, 2016 Reply Link

      I'd like , please desconto actonel Vargas then went back into his burning apartment and fired 10 to 20 shots from a 9mm pistol into the street. One of the bullets struck 33-year-old Carlos Javier Gavilanes, who was parking his car after returning home from work. Zogby said his body was found next to his vehicle.
      entocort prix en france KANSAS CITY, Mo. - Jerry Reese was so crystal clear on the first day of training camp when he marched to the podium and drew his red line. Anything besides a playoff berth “really is not acceptable,” he said. And he “put everybody on notice” that the Giants' high standards had not recently been met.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    8. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Jerrell
      Jerrell September 7th, 2016 Reply Link

      This is your employment contract cellcept baisse de prix We have been spoiled. New Approach, Sea The Stars, followed closely by the unforgettable Frankel era, with a dramatic cameo by Australia’s Black Caviar thrown in, we have been very, very fortunate.
      prezzo aspirina c 10 compresse "In three months' time, this could be back again," said Chakraborty. "If this kind of pushing it back happens several times, then this comfort that the markets had over the last 20 days that a deal will be reached, that comfort may now be dead."

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    9. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Richie
      Richie September 7th, 2016 Reply Link

      Who's calling? kamagra kopen in een winkel While Mickelson blamed the elevation at The Old White TPC for messing with the distance control of his irons, he will be playing at sea level the next two weeks so altitude will not be an issue. Nor will the health of the World Golf Hall of Famer who’s looking for his fifth major championship. He won his fourth at Augusta in 2010 after learning he had psoriatic arthritis but has never used the ailment as an excuse for less-than-stellar outings.
      efedrina cafeina aspirina comprar A provincial official in Baluchistan, Additional Home Secretary Zahid bin Maqsood, put the death toll at 210 and said 375 people had been injured, while a spokesman for the provincial government, Jan Mohammad Bulaidi, put the death toll at 238 — conflicting figures likely due to the difficulty in contacting local officials and people in the remote region.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    10. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Grady
      Grady September 7th, 2016 Reply Link

      I'd like , please propranolol pirkti But others familiar with his work at D.E. Shaw described him as a team player who offered input into a broad range of projects. "Larry is an original thinker, and it's a lot of fun to work with him," said Max Stone, a D.E. Shaw managing director and a member of its executive committee.
      tamoxifen online bestellen Chinese firms have snapped up Saudi contracts on everything from power stations to railways - even though China has vetoed Saudi-backed U.N. resolutions and has given political support to Iran, Saudi Arabia's principal regional adversary.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    11. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Ezequiel
      Ezequiel September 7th, 2016 Reply Link

      I'm on a course at the moment beli clomid No one died, but eight workers were injured, four critically, including one worker who was hit by a car on a nearby road while fleeing the explosions. Officials said the damage could have been significantly worse if three 30,000-pound propane storage containers had caught fire at the plant that refills propane tanks for gas grills and other home uses. About 50 nearby houses were temporarily evacuated, though none was ultimately damaged.
      atarax 25 mg 30 film tablet fiyat Western diplomats fear a slide into instability for South Sudan where a government mostly made up of former guerrilla commanders has been unable to kickstart development. Basic services such as health and education are mostly provided by the United Nations.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    12. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Jessie
      Jessie September 7th, 2016 Reply Link

      Until August apcalis oral jelly preis "This is a dream come true," Kipsang told The Associated Press. "Ten years ago, I watched (Kenya's) Paul Tergat break the world record in Berlin (in 2:04.55) and now I have achieved the dream. I felt strong, so I attacked at 35K, because the pace had become a little too slow."
      acheter nolvadex en ligne He is understood to be ready for a direct confrontation with the doctors' union, the British Medical Association, over reforms to GPs’ contracts if it is necessary to deliver more convenient and reliable services for patients.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
Leave a Comment (using GitHub Flavored Markdown)

Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

Ajax loader