Giter Site home page Giter Site logo

laravel-csv-seeder's Introduction

CSV Seeder

Latest Version on Packagist Software License Build Status Quality Score Total Downloads

Seed your database with CSV files

This package allows CSV based seeds.

Installation

Require this package in your composer.json and run composer update (or run composer require flynsarmy/csv-seeder:2.* directly):

For PHP 7.4+

"flynsarmy/csv-seeder": "2.0.*"

For older PHP versions

"flynsarmy/csv-seeder": "1.*"

Usage

Your CSV's header row should match the DB columns you wish to import. IE to import id and name columns, your CSV should look like:

id,name
1,Foo
2,Bar

Seed classes must extend Flynsarmy\CsvSeeder\CsvSeeder, they must define the destination database table and CSV file path, and finally they must call parent::run() like so:

use Flynsarmy\CsvSeeder\CsvSeeder;

class StopsTableSeeder extends CsvSeeder {

	public function __construct()
	{
		$this->table = 'your_table';
		$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	}

	public function run()
	{
		// Recommended when importing larger CSVs
		DB::disableQueryLog();

		// Uncomment the below to wipe the table clean before populating
		DB::table($this->table)->truncate();

		parent::run();
	}
}

Drop your CSV into /database/seeds/csvs/your_csv.csv or whatever path you specify in your constructor above.

Configuration

In addition to setting the database table and CSV filename, the following configuration options are available. They can be set in your class constructor:

  • connection (string '') Connection to use for inserts. Leave empty for default connection.
  • insert_chunk_size (int 500) An SQL insert statement will trigger every insert_chunk_size number of rows while reading the CSV
  • csv_delimiter (string ,) The CSV field delimiter.
  • hashable (array [password]) List of fields to be hashed before import, useful if you are importing users and need their passwords hashed. Uses Hash::make(). Note: This is EXTREMELY SLOW. If you have a lot of rows in your CSV your import will take quite a long time.
  • offset_rows (int 0) How many rows at the start of the CSV to ignore. Warning: If used, you probably want to set a mapping as your header row in the CSV will be skipped.
  • mapping (array []) Associative array of csvCol => dbCol. See examples section for details. If not specified, the first row (after offset) of the CSV will be used as the mapping.
  • should_trim (bool false) Whether to trim the data in each cell of the CSV during import.
  • timestamps (bool false) Whether or not to add created_at and updated_at columns on import.
    • created_at (string current time in ISO 8601 format) Only used if timestamps is true
    • updated_at (string current time in ISO 8601 format) Only used if timestamps is true

Examples

CSV with pipe delimited values:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
}

Specifying which CSV columns to import:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    5 => 'age',
	];
}

Trimming the whitespace from the imported data:

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    5 => 'age',
	];
	$this->should_trim = true;
}

Skipping the CSV header row (Note: A mapping is required if this is done):

public function __construct()
{
	$this->table = 'users';
	$this->csv_delimiter = '|';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
	$this->offset_rows = 1;
	$this->mapping = [
	    0 => 'first_name',
	    1 => 'last_name',
	    2 => 'password',
	];
	$this->should_trim = true;
}

Specifying the DB connection to use:

public function __construct()
{
	$this->table = 'users';
	$this->connection = 'my_connection';
	$this->filename = base_path().'/database/seeds/csvs/your_csv.csv';
}

Migration Guide

2.0

  • $seeder->hashable is now an array of columns rather than a single column name. Wrap your old string value in [].

License

CsvSeeder is open-sourced software licensed under the MIT license

laravel-csv-seeder's People

Contributors

danatfh avatar fabiosato avatar flynsarmy avatar jackfruh avatar mtakahashi1977 avatar slakbal avatar teodortalov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

laravel-csv-seeder's Issues

comments

Hello,

What about comments in csv files?
eg I have this csv file:

St. James the Greater;2016-07-25;true;false;true

An comment

Constitution Day;2016-12-06;true;false;true
National Holiday;2016-03-13;true;false;true #Vatikan

What I mean is that all lines after # should be ignored.

Add a function for insert.

I have an excel of users and an excel of addresses with the user full name, how I can seed the database converting the full name into the ID of the users?

where can I add a function that queries the id of the user with the name?

insert_chunk_size issue

I have this issue where I have a csv with 7740 rows. If I set the insert_chuck_size = 1 all rows are inserted into the db table. But as soon as I start to increase the insert_chuck_size it starts to drop off some of the records at the end of the csv file and don't insert them.

I suspect there is an counter/indexing problem on the chunking mechanism, as I increase the chunk size the more records it leaves out from import from the bottom of the csv.

Un-quote imported HTML Code

I try to import html code. The csv has a , as delimiter and " as enclosure. Within the code there are linebreaks by \n and some escaped quotes like "

After import the quotes are still escaped.

as \ is the default escape character of fgetcsv I am not sure why.

make Hash work for Lumen

Hello and thank you for such a great plugin!

I'm experiencing problem trying to use your plugin in Lumen enviroment. I have the following seeder:

<?php
  
use Flynsarmy\CsvSeeder\CsvSeeder;

class UserSeeder extends CsvSeeder
{   
    public function __construct()
    {
        $this->table = 'user';
        $this->csv_delimiter = "\t";
        $this->filename = base_path().'/database/seeds/csv/user.csv';
        $this->should_trim = true;
    }

    public function run()
    {
        parent::run();
    }
}

When i run ./artisan db:seed --class=UserSeeder i get the following:

# ./artisan db:seed --class=UserSeeder

In CsvSeeder.php line 286:
                          
  Class 'Hash' not found  
                          

I don't have mush experience with Laravel framework, but it seems to me Hash namespace is only available in Laravel framework and IS NOT available in Lumen framework.

Isn't it better to change "use" directive in file CsvSeeder.php from:

use Hash;

to

use Illuminate\Support\Facades\Hash;

? When I change CsvSeeder.php to use Illuminate\Support\Facades\Hash everything works fine in Lument enviroment:

# ./artisan db:seed --class=UserSeeder
Database seeding completed successfully.

Specifying connection

There is no way to specify the database connection to be used by the seeder. This makes it useless in environments with multiple connections.

Mutiple Table and CSV

I would like to ask how to populate multiple tables with every csv.
For example

$this->table = 'table_1';
$this->filename = base_path().'/database/seeds/csvs/table1.csv';

$this->table = 'table_2';
$this->filename = base_path().'/database/seeds/csvs/table2.csv';

Row callback function

This is a feature request.

Would be useful to have a callback function we can hook into each time a row in the CSV is being processed. For example in my case I have a CSV that:

  • has empty rows which would be good to be able to skip instead of getting DB insert errors
  • some data that has to be changed before being inserted

array_pull() helper method deprecated

Overview

Issue when using laravel-csv-seeder in Laravel v7.

Expected Behavior

Expect to seed database using csv seeder.

Actual Behavior

Receive the following error when trying to seed with artisan migrate --seed:

 Call to undefined function Flynsarmy\CsvSeeder\array_pull()

  at vendor/flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php:168
    164|
    165|                 // skip csv columns that don't exist in the database
    166|                 foreach($mapping  as $index => $fieldname){
    167|                     if (!DB::getSchemaBuilder()->hasColumn($this->table, $fieldname)){
  > 168|                        array_pull($mapping, $index);
    169|                     }
    170|                 }
    171|             }
    172|             else

Resolution

It turns out the array_pull() method was deprecated in Laravel v5.8. These methods are now preferred as static methods: Arr::pull()

The deprecated methods have been separated into a laravel/helpers package. Including this package resolved the issue:

composer require laravel/helpers

You may want to update the library to use the static method, make laravel/helpers a peer dependency, or simply update the README with the requirements. I'd be happy to make a PR if you're interested.

Thanks for the library, by the way. It's very handy!

doesn't work at all

I am trying this one to seed my DB from CSV.
I run it and nothing happens.
Also please could you explain exactly the syntax used to set the delimiter?
Can you also explain what happens to the first line, the header, in the csv?
Is it used to set the columns for seeding in the table?
Is it used at all?
I mean, what does the package expects from the csv file? Same number or columns? as in the db table?

empty values issue

Hello. I had trouble inserting value 0 into my database. I figured out that this happen because of if statement around 115th row in the foreach statement. As stated in PHP docs for empty statement in here it will return true for zero as well as for empty string, array etc. It can happen (in my case there was order number in table which was 0) and this should be escaped. I remade code simply by deleting this statement and I want to know why that statement was there in the first place?

Not all entries make it into the database.

Hello,

I am having a bit of an issue importing a csv full of email addresses.

As per your example here is the code I am using:

public function __construct()
{
    $this->table = 'users';
    $this->csv_delimiter = '|';
    $this->filename = base_path().'/database/seeds/csvs/list.csv';
    $this->mapping = [
        0=> 'email',
        1 => 'language'
    ];
}

public function run()
{
    // Recommended when importing larger CSVs
    DB::disableQueryLog();

    // Uncomment the below to wipe the table clean before populating
    DB::table($this->table)->truncate();

    parent::run();
}

My CSV looks something like this:

[email protected]|nl
[email protected]|en
[email protected]|fr
...

I originally have 11033 entries in the csv, but after the import only 8883 make it, and if I use the the insert_chunk_size option I only get a little over 5000 entries.

I have noticed that there seems to be rows that don't make it or get deleted since the end id of the last row does match the number of entries, i.e. row 8883 in mysql has ID 11033

For example in the table the ids jump from 7550 -> 7701 from one row to the next. i.e

id email language
7550 [email protected] en
7701 [email protected] fr

Thanks for the plugin!

New release?

Any chance you would want to do a new release? I am currently stuck as I am in need of this fix :)

4f50c9d

I will gladly help anyway I can.

seeder skips columns with legitimate "0" values

CSV file has "0"s intermixed in one column

One bit of logic is this in the csvseeder class

                foreach ($header as $key) {
                    if (!empty($row[$i])) {
                        $row_values[$key] = $row[$i];
                    }
                    $i++;
                }

php will evaluate empty() on the 0 and return true. So things become problematic when you really want the value to be zero. Ideally it shouldn't care about whether its a blank string or not and should just be a isset check to ensure the array element exists at all(proper number of columns read in row).

Does not show error message when database throws an error

I got the "Seeded:" message but the csv file was not seeded because of a database error(forgotten to make a column nullable). I wasted lot of time to figure out the error. It would be wonderful it it can throw if there is any database error.

PS: Thanks for providing this library for free. It was really helpful.

hook into up() and down()?

Does your class hook into up() and down() of Migration, so rather than using run(), can I put the code in up() and put a delete-all statement in down() to clean up on rollback?

Syntax error at CsvSeeder.php:22

syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)

at C:\xampp\htdocs\docstasapp\vendor\flynsarmy\csv-seeder\src\Flynsarmy\CsvSeeder\CsvSeeder.php:22
18| * DB table name
19| *
20| * @var string
21| */

22| public string $table = '';
23|
24| /**
25| * CSV filename
26| *

FatalThrowableError : syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)

I am using Php 7.2, Laravel 6.18 and i chacked against 2.0.0->2.0.2 versions and I am always getting this error even if I have an almost empty CsvSeeder

Symfony\Component\Debug\Exception\FatalThrowableError  : syntax error, unexpected 'string' (T_STRING), expecting function (T_FUNCTION) or const (T_CONST)

  at /var/www/vendor/flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php:22
    18|      * DB table name
    19|      *
    20|      * @var string
    21|      */
    22|     public string $table;
    23|
    24|     /**
    25|      * CSV filename
    26|      *

  Exception trace:

  1   Composer\Autoload\includeFile("/var/www/vendor/composer/../flynsarmy/csv-seeder/src/Flynsarmy/CsvSeeder/CsvSeeder.php")
      /var/www/vendor/composer/ClassLoader.php:322
  2   Composer\Autoload\ClassLoader::loadClass("Flynsarmy\CsvSeeder\CsvSeeder")
      [internal]:0
...

This is the CsvSeeder:

<?php
use Flynsarmy\CsvSeeder\CsvSeeder;
class Comuni201901Seeder extends CsvSeeder
{
    public function __construct()
    {
        $this->table = 'comuni201901s';
        //$this->csv_delimiter = ',';
        $this->filename = base_path().'/database/seeds/csv/comuni_2019_01.csv';
        //$this->should_trim = true;
    }

    public function run()
    {
        // Recommended when importing larger CSVs
        DB::disableQueryLog();
        // Uncomment the below to wipe the table clean before populating
        //DB::table($this->table)->truncate();
        //parent::run();
    }
}

Export from Excel on Mac needs to export as Windows CSV

Exporting from a Mac using Comma Separated File save causes it to only process the first row of the CSV (I think due to the end of line delimiters only having \r and not \r\n) - for it to work , you have to export as a Windows Comma Separated csv file.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.