Rectangle 27 0

Why is Ruby CSV file reading very slow?


You'll likely get a massive speed boost by simply updating to a current version of Ruby. in Version 1.9, FasterCSV was integrated as Ruby's standard CSV library.

Note
Rectangle 27 0

Why is Ruby CSV file reading very slow?


Check out the smarter_csv Gem, which has special options for handling huge files by reading data in chunks.

It also returns the CSV data as hashes, which can make it easier to insert or update the data in a database.

Note
Rectangle 27 0

Why is Ruby CSV file reading very slow?


Check out the smarter_csv Gem, which has special options for handling huge files by reading data in chunks.

It also returns the CSV data as hashes, which can make it easier to insert or update the data in a database.

Note
Rectangle 27 0

Why is Ruby CSV file reading very slow?


CSV.foreach

A long time ago I saw this question, and the reason for the slowness of the Ruby is that it loads the entire CSV file into the memory at once. I have seen some people overcome this issue by using the IO class. For example take a look at this gist for its self.perform(url) method.

For the 1.8.7 Ruby that Eric is using, it might be :)

I think that using CSV is little bit overkill for this.

lol I totally overlooked that. I was convinced that CSV.foreach is loading firstly everything to the memory.

Note