Giter Site home page Giter Site logo

kotlin-grass's Introduction

Kotlin-Grass

Kotlin Parser: 0.8.0 Core: 0.8.0 License: Apache License 2.0 CodeFactor CodeFactor

Csv File to Kotlin Data Class Parser
Currently, it requires to have @ExperimentalStdlibApi on the class/method using this Library.
Requires kotlin-csv by doyaaaaaken for reading csv file.

Features

1. Simple And Direct

  • No hard configuration
  • No invasive annotations to data class
  • Custom mapping
  • Nullable Data Types

2. Primitive Types

  • Short
  • Int
  • Long
  • Float
  • Double
  • Boolean
  • String

3. Support for Java 8 Date Time Apis

  • LocalTime
  • LocalDateTime
  • LocalDate
  • Custom Formatting

Usage

Gradle DSL:

//doyaaaaaken's kotlin-csv
implementation("com.github.doyaaaaaken:kotlin-csv-jvm:0.15.2")
//kotlin-grass
implementation("io.github.blackmo18:kotlin-grass-core-jvm:1.0.0")
implementation("io.github.blackmo18:kotlin-grass-parser-jvm:0.8.0")

Maven:

<dependency>
    <groupId>com.github.doyaaaaaken</groupId>
    <artifactId>kotlin-csv-jvm</artifactId>
    <version>0.15.2</version>
</dependency>
<dependency>
    <groupId>io.github.blackmo18</groupId>
    <artifactId>kotlin-grass-core-jvm</artifactId>
    <version>1.0.0</version>
</dependency>
<dependency>
    <groupId>io.github.blackmo18</groupId>
    <artifactId>kotlin-grass-parser-jvm</artifactId>
    <version>0.8.0</version>
</dependency>

Examples

CSV file

short int long float double boolean string
0 1 2 3.0 4.0 true hello

Declaring data class

    data class PrimitiveTypes(
        val short: Short,
        val int: Int,
        val long: Long,
        val float: Float,
        val double: Double,
        val boolean: Boolean,
        val string: String
    )

Nullable Data Types

If a variable in your data class is a nullable, all you have to do is mark it with ?

    data class NullableData(
        val nullableString: String?,
        val nullableInt: Int? = null,
        ...
    )

Parsing to data class

    val csvContents = csvReader().readAllWithHeader(file)
    val dataClasses = grass<PrimitiveTypes>().harvest(csvContents)

Parsing to data class using Kotlin Flow

    val contents = File("file/path").inputStream()
    val dataClasses: Flow<PrimitiveTypes> = csvReader().openAsync(contents) {
        val data = readAllWithHeaderAsSequence().asFlow()
        grass<PrimitiveTypes>().harvest(data)
    }

Custom Configuration

Option default value description
dateFormat yyyy-MM-dd date format
timeFormat HH:mm time format
dateTimeSeparator (space) date time separator
trimWhiteSpace true trims white spaces on csv entries
ignoreUnknownFields false ignore unknown / unmapped fields in input
caseSensitive true case sensitive header matching
customKeyMap null Map<String,String> custom key mapping, priority if not empty or null
customKeyMapDataProperty null Map<String, KProperty<*>> custom key mapping

Java Date Time API Support

csv file

time datetime date
12:00 2020-12-31 12:00 2020-12-31

Date and Time Types

Import the following extension library

implementation("io.github.blackmo18:kotlin-grass-date-time-jvm:0.8.0")
    data class DateTimeTypes(
        val time: LocalTime,
        val datetime: LocalDateTime,
        val date: LocalDate,
    )

Customize Formatting

    val grass = grass<DateTimeTypes> {
        dateFormat = "MM-dd-yyyy"
        timeFormat = "HH:mm:ss"
        dateTimeSeparator = "/"
        customDataTypes = arrayListOf(Java8DateTime)
    }

Custom Mapping Support

CSV file

hour birthdate
12:00 2020-12-31

Code

    data class DateTime(
        val time: LocalTime,
        val date: LocalDate,
    )

    val grass = grass<DateTimeTypes> {
        customKeyMap = mapOf("hour" to "time", "birthdate" to "date")
    }

    // or

    val grass = grass<DateTimeTypes> {
        customKeyMapDataProperty = mapOf("hour" to DateTime::time, "birthdate" to DateTime::date)
    }

🀝 Contributing

Contributions, issues and feature requests are welcome!
Feel free to check issues page.

Changelog

Show your support

Give a ⭐️ if this project helped you!

πŸ“ License

Copyright Β© 2020 blackmo18.
This project is Apache License 2.0 licensed.


This project inspired ❀️ by kotlin-csv

kotlin-grass's People

Contributors

blackmo18 avatar jonashelgemo avatar micheljung avatar winterhate avatar xeruf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

kotlin-grass's Issues

Enhancemen Custom Key Map

Update custom key mapping that reference directly to the property of Target data class
dev.to suggestion

val grass = grass<DateTimeTypes> {
    customKeyMap = mapOf("hour" to DateTimeTypes::time, "birthdate" to DateTimeTypes::date)
}

Cannot be parsed if LocalDate or LocalDateTime exists when read

Describe the bug

Cannot be parsed if LocalDate or LocalDateTime exists when read

how can read LocalDate and LocalDateTime parse?

To Reproduce
Steps to reproduce the behavior.
Attach code snippet which reproduce the bug.

https://github.com/momosetkn/csv_performance_test

Run below main function
src/main/kotlin/com/github/momosetkn/csv/confirmbug/ForConfirmBug.kt

Expected behavior
Can parsed LocalDate and LocalDateTime

Environment

  • kotlin-csv version [e.g. 0.10.0]
  • com.github.doyaaaaaken:kotlin-csv: 1.9.2
  • io.github.blackmo18:kotlin-grass-core-jvm: 1.0.0
  • io.github.blackmo18:kotlin-grass-parser-jvm: 0.8.0
  • java version 21.0.1-tem
  • OS: Linux

Screenshots
execute log

/home/momose/.sdkman/candidates/java/21-tem/bin/java -javaagent:/home/momose/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate-4/lib/idea_rt.jar=33193:/home/momose/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate-4/bin -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.stderr.encoding=UTF-8 -classpath /home/momose/IdeaProjects/csv_performance_test/build/classes/kotlin/main:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.9.21/5570877dec93532519eda165abb3674ea1e07cbc/kotlin-reflect-1.9.21.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.9.21/17ee3e873d439566c7d8354403b5f3d9744c4c9c/kotlin-stdlib-1.9.21.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.orangesignal/orangesignal-csv/2.2.1/feaf95f21b32cb9d01a20bfc37146bd5543a9fda/orangesignal-csv-2.2.1.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.github.mygreen/super-csv-annotation/2.3/63fe1a156c0dde4847dcaddb234ec00cf254da27/super-csv-annotation-2.3.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.univocity/univocity-parsers/2.9.1/81827d186e42129f23c3f1e002b757ad4b4e769/univocity-parsers-2.9.1.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-csv/2.15.3/5f83ba44f2fd4b1cccf82a48c8fb75c7212b6a9b/jackson-dataformat-csv-2.15.3.jar:/home/momose/.gradle/caches/modules-2/files-2.1/io.github.blackmo18/kotlin-grass-core-jvm/1.0.0/eca0f3559497902e6aa357b8642ebc514823e945/kotlin-grass-core-jvm-1.0.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/io.github.blackmo18/kotlin-grass-parser-jvm/0.8.0/96f8e0887e20d2dc64085b36ac1bc76516698bdd/kotlin-grass-parser-jvm-0.8.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.opencsv/opencsv/5.9/284ea0b60a24b71a530100783185e7d547ab5339/opencsv-5.9.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-csv/1.10.0/8669bee353424c3223c93723291b5c3753260c1c/commons-csv-1.10.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.8.0-RC2/7bb4e7056dbe2bab0e5584928cde6b0a4e03f264/kotlinx-coroutines-core-jvm-1.8.0-RC2.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.github.doyaaaaaken/kotlin-csv-jvm/1.9.2/afb1e3abef98fe4cf020aa17dc30fe8aa7ec6aa9/kotlin-csv-jvm-1.9.2.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/23.0.0/8cc20c07506ec18e0834947b84a864bfc094484e/annotations-23.0.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/net.sf.supercsv/super-csv/2.4.0/17f8708c929029dde48bc298deaf3c7ae2452d3/super-csv-2.4.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-jexl/2.1.1/6ecc181debade00230aa1e17666c4ea0371beaaa/commons-jexl-2.1.1.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.15.3/a734bc2c47a9453c4efa772461a3aeb273c010d9/jackson-databind-2.15.3.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.15.3/79baf4e605eb3bbb60b1c475d44a7aecceea1d60/jackson-annotations-2.15.3.jar:/home/momose/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.15.3/60d600567c1862840397bf9ff5a92398edc5797b/jackson-core-2.15.3.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-text/1.11.0/2bb044b7717ec2eccaf9ea7769c1509054b50e9a/commons-text-1.11.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.13.0/b7263237aa89c1f99b327197c41d0669707a462e/commons-lang3-3.13.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/commons-beanutils/commons-beanutils/1.9.4/d52b9abcd97f38c81342bb7e7ae1eee9b73cba51/commons-beanutils-1.9.4.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-collections4/4.4/62ebe7544cb7164d87e0637a2a6a2bdc981395e8/commons-collections4-4.4.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.8.0/ed04f49e186a116753ad70d34f0ac2925d1d8020/kotlin-stdlib-jdk8-1.8.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.2/4bfc12adfe4842bf07b657f0369c4cb522955686/commons-logging-1.2.jar:/home/momose/.gradle/caches/modules-2/files-2.1/commons-collections/commons-collections/3.2.2/8ad72fe39fa8c91eaaf12aadb21e0c3661fe26d5/commons-collections-3.2.2.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.8.0/3c91271347f678c239607abb676d4032a7898427/kotlin-stdlib-jdk7-1.8.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-debug/1.8.0-RC2/acee451a516103251dc507fde1a66f440df24dde/kotlinx-coroutines-debug-1.8.0-RC2.jar:/home/momose/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-test/1.5.21/c07c15adca227480638197afb4e3b1dda7e8dfef/kotlin-test-1.5.21.jar:/home/momose/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna-platform/5.9.0/c535a5bda553d7d7690356c825010da74b2671b5/jna-platform-5.9.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.9.0/8f503e6d9b500ceff299052d6be75b38c7257758/jna-5.9.0.jar:/home/momose/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.10.9/2c03f15cd1131692feca93f190ed81412a3de961/byte-buddy-1.10.9.jar:/home/momose/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.10.9/cbbeffa557e6b1b4cbb181b0782436921c523699/byte-buddy-agent-1.10.9.jar com.github.momosetkn.com.github.momosetkn.csv.confirmbug.ForConfirmBugKt
Exception in thread "main" java.lang.NullPointerException
	at io.blackmo18.kotlin.grass.pot.Root.createObject(Root.kt:58)
	at io.blackmo18.kotlin.grass.pot.Stem.harvestData(Stem.kt:48)
	at io.blackmo18.kotlin.grass.pot.Plant.harvest(Plant.kt:42)
	at com.github.momosetkn.csv.individual.KotlinCsvExample.readEach(KotlinCsvTypedCsvData.kt:21)
	at com.github.momosetkn.com.github.momosetkn.csv.confirmbug.ForConfirmBugKt.read(ForConfirmBug.kt:24)
	at com.github.momosetkn.com.github.momosetkn.csv.confirmbug.ForConfirmBugKt.main(ForConfirmBug.kt:15)

Process finished with exit code 1

New version released with old tag

Hello @blackmo18 , it looks like you released new version without changing tag name (still 0.5.0). Therefore my Gradle does not detect the change in artifact repository and the new feature is not available to me or anybody who was using 0.5.0 before release.

import csv throw NullPointerException.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
NPE

Expected behavior
Test failed
java.lang.NullPointerException
at io.blackmo18.kotlin.grass.pot.Root.createObject(Root.kt:70)
at io.blackmo18.kotlin.grass.pot.Stem.harvestData(Stem.kt:48)
at io.blackmo18.kotlin.grass.pot.Plant.harvest(Plant.kt:42)

Environment

  • kotlin-csv version:1.2.0 、 0.15.2
  • java version: 17
  • OS: Windows11\MacOS\Windows10

Screenshots
image
image

Ignore unknown fields

There should be a switch to allow ignore unmaped fields in input data.

            "parse to primitive data type with unmapped field" {
                val expected =
                    PrimitiveTypes(0, 1, 2, 3.0f, 4.0, true, "hello")
                val contents = readTestFile("/primitive-with-unmapped-field.csv").asSequence()
                val grass = grass<PrimitiveTypes>() {
                    ignoreUnknownFields = true
                }
                val parsed = grass.harvest(contents)
                val actual = parsed.first()

                assertTrue { expected == actual }
            }

where primitive-with-unmapped-field.csv is

short,int,long,float,double,boolean,string,unmapped
0,1,2,3.0,4.0,true,hello,unmapped1
5,6,7,8.0,9.0,true,hi,unmapped2

@blackmo18 I already implemented the feature, but I am unable to push the branch for review due to insufficient permissions:

12:05:33.738: [kotlin-grass] git -c core.quotepath=false -c log.showSignature=false push --progress --porcelain origin refs/heads/enhancement/ignore-unknown-field-option:enhancement/ignore-unknown-field-option --set-upstream
remote: Permission to blackmo18/kotlin-grass.git denied to winterhate.
fatal: unable to access 'https://github.com/blackmo18/kotlin-grass.git/': The requested URL returned error: 403

ClassNotFoundException : DateTimeFormatter

ERROR
Caused by: java.lang.ClassNotFoundException: Didn't find class "java.time.format.DateTimeFormatter" on path: DexPathList[[zip file "/data/app/[app_name]/base.apk"],nativeLibraryDirectories=[/data/app/[app_name]/lib/arm, /system/lib, /vendor/lib]]

Environment

  • kotlin-csv version 0.11.0
  • java version java8
  • OS: Windows/Android

Custom transformers

Hello!

First of all, thank you very much for this awesome project! I am having a lot of fun with it. πŸ˜‰

I would like to suggest a feature: custom transformers.

Background: I have an "interesting" CSV which contains a column (field) holding "booleans". The catch is: they are not described in terms of true and false, but yes and no. I was wondering if grass could include some sort of custom transformers, so one could save post-processing data and skip the use of an intermediate data class. πŸ˜‰

Thanks!

Parsing error when a nullable field has a custom key name

Describe the bug
When a field key name is mapped using customKeyMap, the parser fails with java.lang.NumberFormatException: For input string: ""

For bug details see below. I attempted to create a fix as well, it fixes the test case I discovered and does not break any other, but... ⚠️ !!

Stack trace:

java.lang.NumberFormatException: For input string: ""
	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68)
	at java.base/java.lang.Integer.parseInt(Integer.java:662)
	at java.base/java.lang.Integer.parseInt(Integer.java:770)
	at com.vhl.blackmo.grass.vein.PrimitiveType$Companion$int$1.invoke(PrimitiveType.kt:21)
	at com.vhl.blackmo.grass.vein.PrimitiveType$Companion$int$1.invoke(PrimitiveType.kt:19)
	at com.vhl.blackmo.grass.pot.Root.createObject(Root.kt:66)
	at com.vhl.blackmo.grass.pot.Stem.harvestData(Stem.kt:41)
	at com.vhl.blackmo.grass.pot.Plant.harvest(Plant.kt:29)

To Reproduce
Test case:

            "parse null values with custom names" {
                val expected0 = NullableDataTypesCustomNames(null, null, null, null, null, null, null)
                val contents = readTestFile("/primitive-empty.csv")
                val parser = grass<NullableDataTypesCustomNames>() {
                    customKeyMap = mapOf(
                        "short" to "shortCustom", "int" to "intCustom", "long" to "longCustom",
                        "float" to "floatCustom", "double" to "doubleCustom", "boolean" to "booleanCustom",
                        "string" to "stringCustom"
                    )
                }
                val parsed = parser.harvest(contents)

                assertTrue { expected0 == parsed.first() }
            }

where NullableDataTypesCustomNames is

data class NullableDataTypesCustomNames(
        val shortCustom: Short?,
        val intCustom: Int?,
        val longCustom: Long?,
        val floatCustom: Float?,
        val doubleCustom: Double?,
        val booleanCustom: Boolean?,
        val stringCustom: String?
)

Expected behavior
Parser should not fail. Parser should detect nullable type in the data class and use empty string as an null value indicator.

Environment
Git revision 329f7f0

Screenshots
Not needed...

Fix_parsing_error_when_a_nullable_field_has_a_custom_key_name.patch.gz

Case Insensitive Headings

A nice improvement would be to not require the CSV headers to match case as the Class.

This is currently achievable by using the Custom Key Map but it would be nice if there was a just a flag.

Improve handling of missing target fields in CSV

I was recently parsing a CSV into a target class, where one of the fields of said class wasn't present in the CSV. This resulted in this exception being thrown:

java.lang.ClassCastException: Cannot cast kotlin.Unit to java.lang.String

Upon further analysis this seems to happen because when creating the array of parameters for the constructor of the target class in
Root.kt an empty default value block is specified (the Unit in the exception).

I think this is a corner case with no clear-cut solution, at the same time I think the behaviour towards the user could be improved (it took me a debugging session to understand what was going on).

A couple of ideas:

  • setting null as the default value for constructor parameters. Since anyway the type of actualParams is Array<Any?> this wouldn't cause any inconsistency or incompatibility, allowing for target classes with optional parameters not present in the source CSV to still be initialized correctly
  • doing a pass over the same array and remove all the values that have Unit type. This would allow two useful things: supporting target classes with default values on parameters that might not be in the source CSV, and otherwise showing a more meaningful error to the user (as in this case, the final actualParams won't have a required argument, hence resulting in Kotlin reflection to clearly state which field is missing rather than throwing the more generic ClassCastException)

For now I can solve the problem in my code by removing the offending fields, but I'd love to help alleviate this problem.
Please let me know your thoughts, I'd be happy to PR a solution afterwards.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.