Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

CSV.foreach(filename, :headers => true) do |row|
  Moulding.create!(row.to_hash)
end

@AlphaNico Create a question with your problem. That error is unrelated to this, your Model objects seem out of sync.

@Simon: indeed. It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

I have this error, do you know why?: ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

No need for with_indifferent_access or symbolize_keys, and no need to read in the file to a string first.

Simpler version of yfeldblum's answer, that is simpler and works well also with large files:

Thanks Tom, love elegant and easy to understand examples, with a few small modifications it worked perfectly.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

CSV.foreach(filename, :headers => true) do |row|
  Moulding.create!(row.to_hash)
end

@AlphaNico Create a question with your problem. That error is unrelated to this, your Model objects seem out of sync.

@Simon: indeed. It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

I have this error, do you know why?: ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

No need for with_indifferent_access or symbolize_keys, and no need to read in the file to a string first.

Simpler version of yfeldblum's answer, that is simpler and works well also with large files:

Thanks Tom, love elegant and easy to understand examples, with a few small modifications it worked perfectly.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'smarter_csv'
  options = {}
  SmarterCSV.process('input_file.csv', options) do |chunk|
    chunk.each do |data_hash|
      Moulding.create!( data_hash )
    end
  end

@Tass it's also pretty easy to add a series of individual methods, each for a specific purpose and before you know it your application has excessive logic that you have to maintain. If a gem works, is well maintained, and uses little resources or can be quarantined to the relevant environments (i.e. Staging for production tasks) it seems to me always a better option to use the gem. Ruby and Rails are all about writing less code.

As the CSV class is included, I feel it's better to use it instead of adding or installing an additional gem. Granted, you didn't propose that a new gem be added to the application. It's so easy to add a series of individual gems, each for a specific purpose and before you know it your application has excessive dependencies. (I find myself consciously avoiding the addition of any gems. In my shop we need to justify the addition to our teammates.)

I have the following error, do you know why? ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

I tried this on a rake task, console returns: rake aborted! NoMethodError: undefined method `close' for nil:NilClass stackoverflow.com/questions/42515043/

The smarter_csv gem was specifically created for this use-case: to read data from CSV file and quickly create database entries.

You can use the option chunk_size to read N csv-rows at a time, and then use Resque in the inner loop to generate jobs which will create the new records, rather than creating them right away - this way you can spread the load of generating entries to multiple workers.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'smarter_csv'
  options = {}
  SmarterCSV.process('input_file.csv', options) do |chunk|
    chunk.each do |data_hash|
      Moulding.create!( data_hash )
    end
  end

@Tass it's also pretty easy to add a series of individual methods, each for a specific purpose and before you know it your application has excessive logic that you have to maintain. If a gem works, is well maintained, and uses little resources or can be quarantined to the relevant environments (i.e. Staging for production tasks) it seems to me always a better option to use the gem. Ruby and Rails are all about writing less code.

As the CSV class is included, I feel it's better to use it instead of adding or installing an additional gem. Granted, you didn't propose that a new gem be added to the application. It's so easy to add a series of individual gems, each for a specific purpose and before you know it your application has excessive dependencies. (I find myself consciously avoiding the addition of any gems. In my shop we need to justify the addition to our teammates.)

I have the following error, do you know why? ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

I tried this on a rake task, console returns: rake aborted! NoMethodError: undefined method `close' for nil:NilClass stackoverflow.com/questions/42515043/

The smarter_csv gem was specifically created for this use-case: to read data from CSV file and quickly create database entries.

You can use the option chunk_size to read N csv-rows at a time, and then use Resque in the inner loop to generate jobs which will create the new records, rather than creating them right away - this way you can spread the load of generating entries to multiple workers.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

CSV.foreach(filename, :headers => true) do |row|
  Moulding.create!(row.to_hash)
end

@AlphaNico Create a question with your problem. That error is unrelated to this, your Model objects seem out of sync.

@Simon: indeed. It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

I have this error, do you know why?: ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

It doesnt't keep the whole file in memory at once, but reads in line by line and creates a Moulding per line.

No need for with_indifferent_access or symbolize_keys, and no need to read in the file to a string first.

Simpler version of yfeldblum's answer, that is simpler and works well also with large files:

Thanks Tom, love elegant and easy to understand examples, with a few small modifications it worked perfectly.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'smarter_csv'
  options = {}
  SmarterCSV.process('input_file.csv', options) do |chunk|
    chunk.each do |data_hash|
      Moulding.create!( data_hash )
    end
  end

@Tass it's also pretty easy to add a series of individual methods, each for a specific purpose and before you know it your application has excessive logic that you have to maintain. If a gem works, is well maintained, and uses little resources or can be quarantined to the relevant environments (i.e. Staging for production tasks) it seems to me always a better option to use the gem. Ruby and Rails are all about writing less code.

As the CSV class is included, I feel it's better to use it instead of adding or installing an additional gem. Granted, you didn't propose that a new gem be added to the application. It's so easy to add a series of individual gems, each for a specific purpose and before you know it your application has excessive dependencies. (I find myself consciously avoiding the addition of any gems. In my shop we need to justify the addition to our teammates.)

I have the following error, do you know why? ActiveModel::UnknownAttributeError: unknown attribute 'siren;nom_ent;adresse;complement_adresse;cp_ville;pays;region;departement;activite;date;nb_salaries;nom;prenom;civilite;adr_mail;libele_acti;categorie;tel' for Transaction

I tried this on a rake task, console returns: rake aborted! NoMethodError: undefined method `close' for nil:NilClass stackoverflow.com/questions/42515043/

The smarter_csv gem was specifically created for this use-case: to read data from CSV file and quickly create database entries.

You can use the option chunk_size to read N csv-rows at a time, and then use Resque in the inner loop to generate jobs which will create the new records, rather than creating them right away - this way you can spread the load of generating entries to multiple workers.

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

csv_text = File.read('...')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
  Moulding.create!(row.to_hash)
end

It worked perfectly. However I have a beginner-level question - when I tried to browse described methods in Ruby and Rails API documentation I was unable to find them on place (I looked on official Ruby and Rails sites, API docs). E.g. I couldn't find what object returns CSV.parse(), I didn't find to_hash() and with_indifferent_access() methods... Maybe I looked in wrong place or missed some basic principle on how to traverse Ruby & Rails API docs. Can anyone share the best practice how to read Ruby API docs?

You can put it in a Rake task, or in a controller action, or anywhere you like....

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

csv_text = File.read('...')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
  Moulding.create!(row.to_hash)
end

Sign up for our newsletter and get our top new questions delivered to your inbox (see an example).

It worked perfectly. However I have a beginner-level question - when I tried to browse described methods in Ruby and Rails API documentation I was unable to find them on place (I looked on official Ruby and Rails sites, API docs). E.g. I couldn't find what object returns CSV.parse(), I didn't find to_hash() and with_indifferent_access() methods... Maybe I looked in wrong place or missed some basic principle on how to traverse Ruby & Rails API docs. Can anyone share the best practice how to read Ruby API docs?

You can put it in a Rake task, or in a controller action, or anywhere you like....

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'csv'    

csv_text = File.read('...')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
  Moulding.create!(row.to_hash)
end

Sign up for our newsletter and get our top new questions delivered to your inbox (see an example).

It worked perfectly. However I have a beginner-level question - when I tried to browse described methods in Ruby and Rails API documentation I was unable to find them on place (I looked on official Ruby and Rails sites, API docs). E.g. I couldn't find what object returns CSV.parse(), I didn't find to_hash() and with_indifferent_access() methods... Maybe I looked in wrong place or missed some basic principle on how to traverse Ruby & Rails API docs. Can anyone share the best practice how to read Ruby API docs?

You can put it in a Rake task, or in a controller action, or anywhere you like....

Note
Rectangle 27 0

Ruby on Rails Import Data from a CSV file?


require 'upsert' # add this to your Gemfile
require 'csv'    

u = Upsert.new Moulding.connection, Moulding.table_name
CSV.foreach(file, headers: true) do |row|
  selector = { name: row['name'] } # this treats "name" as the primary key and prevents the creation of duplicates by name
  setter = row.to_hash
  u.row selector, setter
end

If this is what you want, you might also consider getting rid of the auto-increment primary key from the table and setting the primary key to name. Alternatively, if there is some combination of attributes that form a primary key, use that as the selector. No index is necessary, it will just make it faster.

You might try Upsert:

Note