My Projects

The projects that I have spent significant time on.

16

APR
2007

No Longer the Fastest Game in Town

If your number one concern when working with CSV data in Ruby is raw speed, you might want to know that FasterCSV is no longer the fastest option.

There are a couple of new contenders for Ruby CSV processing including a C extension called SimpleCSV and a pure Ruby library called LightCsv. I haven't been able to test SimpleCSV locally, because I can't get it to build on my box, but users do tell me it's faster. I have run some trivial benchmarks for LightCsv though and it too is pretty quick:

$ rake benchmark
(in /Users/james/Documents/faster_csv)
time ruby -r csv -e '6.times { CSV.foreach("test/test_data.csv") { |row| } }'

real    0m5.481s
user    0m5.468s
sys     0m0.010s
time ruby -r lightcsv -e \
'6.times { LightCsv.foreach("test/test_data.csv") { |row| } }'

real    0m0.358s
user    0m0.349s
sys     0m0.008s
time ruby -r lib/faster_csv -e \
'6.times { FasterCSV.foreach("test/test_data.csv") { |row| } }'

real    0m0.742s
user    0m0.732s
sys     0m0.009s

It's important to note that LightCsv is indeed very "light." FasterCSV has grown up into a feature rich library that provides many different ways to look at your data. In contrast, LightCsv doesn't yet allow you to set column or row separators. Given that, it's only an option for vanilla CSV you just need to iterate over. If that's what you have though, and speed counts, it might just be the right choice.

For the curious, LightCsv achieves its speed advantage in two ways. First, it uses StringScanner to manage the parsing. StringScanner is a C extension, though it is a standard library installed with Ruby.

More importantly, I suspect, LightCsv uses an input buffer for reading while FasterCSV works line by line. I suspect this second difference accounts for the majority of the speed increase since the buffered code will hit the hard drive quite a bit less for the average CSV file. This does require more memory though, of course.

Aside from these differences, FasterCSV and LightCsv have very similar parsers.

Comments (2)
  1. tommy
    tommy April 18th, 2007 Reply Link

    LightCsv do not use StringIO.
    It use StringScanner.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 18th, 2007 Reply Link

      Oops. Good catch. I have corrected the article.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    3. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Claude
      Claude November 15th, 2016 Reply Link

      Who would I report to? generic baclofen images It is understood that two complaints had been made about PC Ali, connected to the probe into sexual exploitation of children in Rotherham and he had been placed on restricted duties while the matter was being considered by the Independent Police Complaints Commission (IPCC).
      interactions between bactrim and coumadin The study was conducted by Douglas Altshuler and Benjamin Goller, two researchers from the University of British Columbia
      lopressor dose for anxiety “If you think about it, it’s probably the right thing to do rather than start out the year and all of a sudden, try to pitch into May, get hurt and miss all of the rest of this year and part of next year
      is meclizine hcl safe for dogs Alternatively, it might have a direct effect on the hormones involved in menstruation, which may be particularly important before the onset of puberty and regular monthly periods," the team from the University of Queensland suggested.
      buy prednisone dog The Yankees chose John Ryan Murphy over Austin Romine as their backup catcher, designating the latter, a one-time prospect, for assignment

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    4. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Hobert
      Hobert November 15th, 2016 Reply Link

      Have you got any experience? prednisone for allergy to hair dye The documents showed many kinds of patent applications were being flagged, with examples ranging from a suicide machine to race-based diagnostics, as well as broad categories such as smartphones and anything "pioneering."
      can i buy trimethoprim online Brendan and his parents sought out the school for themselves, but the majority of students are referred by detox and drug-treatment centres or the court system
      prozac price The state contends the legislature was entitled to invoke police powers to solve a fiscal emergency
      pyridium over counter substitute "The bathhouse owner is accused of turning the facility into a site of immoral and indecent conduct and group homosexuality", the agency quotes Mohamed Hetta, head of the local prosecution office, as saying.
      frumil tablets Although most studies so far have been done in rodents, trials have begun at Stanford University in which patients with early signs of Alzheimer's get infusions of blood from young volunteers

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    5. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Jarrett
      Jarrett November 15th, 2016 Reply Link

      Could you ask him to call me? promethazine with codeine syrup uk "It couldn't have possibly been him," said Houston Rockets spokeswoman Tracey Hughes, who told Reuters that Howard and the Rockets were playing the San Antonio Spurs in Houston on that date
      sulfasalazine prescribing information Bruni noted that Michael Brown’s parents traveled to Switzerland to testify before the torture committee in early November
      order vermox online canada HONG KONG/TAIPEI, Aug 25 (Reuters) - Taiwan's Wei family hasagreed to buy majority control of China Network Systems (CNS),the island's biggest cable TV operator, for about $2.4 billionincluding debt from private equity firm MBK Partners, sourceswith direct knowledge of the matter said.
      is levaquin used for urinary tract infections Are you happy with the care you receive from your GP, hospital or care home? Did your doctor have a good bedside manner? Were you left waiting too long in A&E? Do you have an elderly relative in a care home?
      buy actavis promethazine cough syrup 16oz Making models available in stores - as Xiaomi does in China - could help the fast-growing startup reach more users in the world's third-largest smartphone market as Bharti promotes the handsets to attract subscribers to its 4G service.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    6. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Darryl
      Darryl November 15th, 2016 Reply Link

      Would you like to leave a message? clomid clomiphene He thinks that oil prices will be around $60 to $65 a barrel over the next 18 months and then will recover
      pfizer minipress 1mg prazosin "This is one of many indicators that climate change has not stopped and that it continues to be one of the most important issues facing humanity," said University of Illinois climate scientist Donald Wuebbles.
      prednisone dose poison oak In low-income countries where malaria is endemic, the expensive multi-drug therapy required to treat it is often not an available option so there is a need for new preventative tools.
      robaxin medication side effects Gordon Brown is leading a debate on proposed reforms to trading relationships with Europe
      order glyburide The controversy began on Thursday when Hicks was hired to do a photo shoot with the Clevenger family after the birth of their second child

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    7. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Janni
      Janni November 15th, 2016 Reply Link

      This is your employment contract prazosin hydrochloride tablets side effects Prime Minister Manuel Valls offered up his Socialist government's resignation after accusing the economy minister of crossing a line with his blunt criticism of the government's policies
      frumil water tablets They were then billed for services they didn’t receive, and unfairly charged interest and fees, the CFPB said.
      allopurinol side effects nhs "(The cockpit voice recorder) seems to be under a wing,which is quite heavy," said Supriyadi, operations coordinatorfor the search and rescue agency
      levaquin online pharmacy uk About 25,000 people still live in tents, shelters and bunkhouses more than a year later.
      glipizide versus glyburide elderly He was convicted despite an alibi: He had been at work inside a locked warehouse 15 minutes away during the third shooting.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    8. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Walton
      Walton November 15th, 2016 Reply Link

      Please call back later is adalat a prescription drugs "It might look like there are lots of people visiting these sites but it is difficult to conclude that from this information."
      mebendazole pinworm treatment In each of his previous stops Showalter helped change losing cultures or, in the case of the expansion Diamondbacks, build a winning culture by instilling a certain discipline in the clubhouse while winning the chess game in the dugout with the opposing manager on most nights.
      prednisone 5mg cats side effects Those who hit pedestrians undergo intensive investigation and are subject to internal discipline
      precio del medicamento combivent respimat Selin cautions that after a certain point, most of the health benefits have already been reaped, and additional emissions reductions won’t translate into greater improvements.
      ranitidine 300 mg price I just want to have a chance and I have confidence in my ability to make it happen.' Look what happened

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    9. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Wilton
      Wilton November 15th, 2016 Reply Link

      Do you play any instruments? pyridium phenazopyridine cost So, even if someone has cracked the password, a special one time password will be sent to the registered mobile number to access.
      order carafate online Van Dyken was speeding on her ATV after dinner on a clear June evening in Show Low, Arizona, when she tumbled over a curb and fell about eight feet into the forest below
      what all does bactrim treat Most are loggers themselves, or the families of loggers, others are merchants who rely on loggers to purchase goods from them.
      generic form of sulfasalazine Mahmood claimed MQM sheltered criminals and obstructed life in the neighborhood by placing barricades in the streets.
      nizoral 2 shampoo walgreens This is a type of cookie which is collected by Adobe Flash media player (it is also called a Local Shared Object) - a piece of software you may already have on your electronic device to help you watch online videos and listen to podcasts.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    10. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Johnie
      Johnie November 15th, 2016 Reply Link

      I hate shopping levonorgestrel ethinyl estradiol brand names In Dallas, "I suspect no one was watching to make sure the people who were taking care of the patients were taking care of themselves," he said.
      methylprednisolone iv to prednisone po conversion DAVOS, Switzerland - Central banks have done their best to rescue the world economy by printing money and politicians must now act fast to enact structural reforms and pro-investment policies to boost growth, central bankers said on Saturday.
      bactrim ds acne Steffon Armitage could be the missing piece in the Stuart Lancaster jigsaw, the openside flanker who has wowed and wooed audiences on the other side of the Channel for the past three years in the colours of Toulon.
      order trimethoprim While politics often come up during UN news conferences, as foreign ministers or presidents address questions with a home audience in mind, that stage has never been given to someone with no official role representing a country.
      caverta 25 mg tablet Not long after Obama called Islamic State a "cancer" with a bankrupt ideology, the Pentagon said U.S

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    11. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Carol
      Carol November 15th, 2016 Reply Link

      I'm afraid that number's ex-directory avodart 0.5 mg side effects Tesco has removed a sign that referred to superhero alarm clocks as “fun gifts for boys” after a seven-year-old girl took umbrage with the assumption that only boys would like it.
      sulfasalazine (azulfidine) dosage One of nationalist leader Alex Salmond’s most potent claims has been that only a vote for independence will stop the National Health Service being dismantled by Cameron’s Conservatives.
      caverta 50 review Having ups and downsis a normal part of life, but staying in a slump for too long can have adetrimental effect on you and your family.
      cataflam dosis niños Frederick Bordry, director for accelerators, said it could take time to resolve what he described as an intermittent short-circuit because it was in a cold section of the machine, meaning that part would probably have to be warmed up.
      zantac cost walmart It’s nothing to see a team pass for over 400 yards in a high school game here in Texas and those are 48-minute games.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
    12. Your comment is awaiting moderation. It will appear as you see it below if approved.
      Merlin
      Merlin November 15th, 2016 Reply Link

      I work here mestinon overdose side effects While the government has announced its intention to slash coal imports over the next two to three years, it wants to pump up domestic production from 565 million tonnes last year to one billion tonnes by 2019
      antabuse online uk If we can catch parents with depression earlier and treat them, then there won't be a continuation of symptoms, and maybe even as importantly, their child won't be affected by a parent with depression," commented the study's lead author, Dr Sheehan Fisher, of Northwestern University.
      valtrex without a subscription 3M is the latest company to take voluntary steps to reformits supply chain practices amid pressure from environmentalgroups
      norfloxacin tablets bp 400mg Israel's Foreign Minister Avigdor Lieberman exits a news conference on the sidelines of the 69th United Nations General Assembly at the U.N
      can i order prednisone online "This is surprising because these birds do not look anything like a pipit or wagtail," explained co-author Per Alstrom from the Swedish Species Information Centre, based at the Swedish University of Agricultural Sciences.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
Leave a Comment (using GitHub Flavored Markdown)

Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

Ajax loader