horker/import-csv2

A PowerShell module for reading CSV files.

import-csv2's Issues

How to map only certain fields into a record?

I have a CSV with over 200 columns. I want to select only some of them into a record.

$typeMap = @{ 
    "Artikel"        = [string];
    "Artikelgruppe"  = [string];
    "Hersteller"     = [string];
    "Bezeichnung1"   = [string];
    "Verkaufspreis1" = [Nullable[decimal]];
    "Verkaufspreis2" = [Nullable[decimal]];
    "Verkaufspreis3" = [Nullable[decimal]];
    "Verkaufspreis4" = [Nullable[decimal]];
    "Verkaufspreis5" = [Nullable[decimal]];
    "Verkaufspreis6" = [Nullable[decimal]];
    "Verkaufspreis7" = [Nullable[decimal]];
    "Verkaufspreis8" = [Nullable[decimal]];
    "Verkaufspreis9" = [Nullable[decimal]];
    "Preiseinheit"   = [string];
}

$cMap = @{  "Artikel" = "Artikel" }

class RawArtikel {
    [string]$Artikel
}

Import-Csv2 -Path "mycv.csv" -Delimiter ";" -Encoding $([System.Text.Encoding]::UTF8) -IgnoreQuote -RecordType ([RawArtikel])    -ColumnTypes $typeMap -ColumnNameMap $cMap -Culture $([System.Globalization.CultureInfo]::InvariantCulture) 

It fails with

CsvHelper.ReaderException: An unexpected error occurred. ---> System.ArgumentException: Der Typ "System.Void*" darf nicht 
als Typargument verwendet werden.
   bei System.RuntimeType.ThrowIfTypeNeverValidGenericArgument(RuntimeType type)
   bei System.RuntimeType.SanityCheckGenericArguments(RuntimeType[] genericArguments, RuntimeType[] genericParamters)
   bei System.RuntimeType.MakeGenericType(Type[] instantiation)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapConstructorParameters(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapConstructorParameters(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMapMembers(ClassMap map, Configuration configuration, LinkedList`1 mapParents, Int32 indexStart)
   bei CsvHelper.Configuration.ClassMap.AutoMap(Configuration configuration)
   bei CsvHelper.Configuration.Configuration.AutoMap(Type type)
   bei CsvHelper.Expressions.ObjectRecordCreator.CreateCreateRecordDelegate(Type recordType)
   bei CsvHelper.Expressions.RecordCreator.GetCreateRecordDelegate(Type recordType)
   bei CsvHelper.Expressions.RecordCreator.Create(Type recordType)
   bei CsvHelper.CsvReader.GetRecord(Type type)
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei CsvHelper.CsvReader.GetRecord(Type type)
   bei Horker.CsvHelper.ImportCsv2.LoadFile(CsvLoader loader) in C:\Users\msumi\work\import-csv2\source\Horker.CsvHelper.PowerShell\ImportCsv2.cs:Zeile 263.
   bei Horker.CsvHelper.ImportCsv2.BeginProcessing() in C:\Users\msumi\work\import-csv2\source\Horker.CsvHelper.PowerShell\ImportCsv2.cs:Zeile 172.
   bei System.Management.Automation.Cmdlet.DoBeginProcessing()
   bei System.Management.Automation.CommandProcessorBase.DoBegin()

Update to CsvHelp 27.x

Would it be possible to integrate the batch/core changes Wardrop implemented and upgrade the CSvHelper to latest (27.x)?

Add batch processing

First of all, great library. Very handy for importing data from CSV into SQL.

A problem I'm having is that for larger datasets (CSV's around ~500MB) it naturally gets very memory hungry and slow. If a batch option could be added, that would be awesome for handling large CSV's. Something like "-BatchSize" and "-BatchAction", where BatchSize is the number of rows per batch, and BatchAction is a code block that is run for every batch of rows.

Just a suggestion anyway.

Cheers