hasnep/dataskimmer.jl

📊 A Julia package that summarises tabular data in the REPL

DataSkimmer.jl

A Julia package that gives a summary of a tabular dataset in the REPL. Based on the skimr R package.

Installation

From the Julia REPL, type ] to enter the Pkg REPL, then run:

add DataSkimmer

Or run from the Julia REPL:

import Pkg
Pkg.add("DataSkimmer")

Example

# Load some data
using RDatasets
iris = RDatasets.dataset("datasets", "iris")

# Skim the data
using DataSkimmer
skim(iris)

┌─────────────────────┬───────────┐
│                Type │ DataFrame │
│             N. rows │       150 │
│             N. cols │         5 │
│     N. numeric cols │         4 │
│ N. categorical cols │         1 │
│    N. datetime cols │         0 │
└─────────────────────┴───────────┘

4 numeric columns
┌─────────────┬─────────┬──────────┬──────────┬──────┬──────┬──────┬──────┬──────┬───────┐
│        Name │    Type │ Missings │ Complete │ Mean │ Std. │ Min. │ Med. │ Max. │ Hist. │
├─────────────┼─────────┼──────────┼──────────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ SepalLength │ Float64 │        0 │   100.0% │ 5.84 │ 0.83 │ 4.30 │ 5.80 │ 7.90 │ ▂▃▃▂▁ │
│  SepalWidth │ Float64 │        0 │   100.0% │ 3.06 │ 0.44 │ 2.00 │ 3.00 │ 4.40 │ ▁▃▄▂▁ │
│ PetalLength │ Float64 │        0 │   100.0% │ 3.76 │ 1.77 │ 1.00 │ 4.35 │ 6.90 │ ▃▁▂▃▁ │
│  PetalWidth │ Float64 │        0 │   100.0% │ 1.20 │ 0.76 │ 0.10 │ 1.30 │ 2.50 │ ▃▁▃▂▂ │
└─────────────┴─────────┴──────────┴──────────┴──────┴──────┴──────┴──────┴──────┴───────┘

1 categorical column
┌─────────┬─────────────────────────────────┬──────────┬──────────┐
│    Name │                            Type │ Missings │ Complete │
├─────────┼─────────────────────────────────┼──────────┼──────────┤
│ Species │ CategoricalValue{String, UInt8} │        0 │   100.0% │
└─────────┴─────────────────────────────────┴──────────┴──────────┘

dataskimmer.jl's Issues

Boolean columns are categorised as numeric

I think they feel more categorical, plus most of the statistics don't apply to booleans.

Columns of Type Time incorrectly file under Categorical Columns

Just caught this. A DataFrame with columns of Type Date and DateTime correctly file under the Datetime columns. However, a column of type Time files under categorical columns. See attached screenshot. Column names have been obscured on purpose.

Missing Data with Numerical Columns handled incorrectly

When a column is a Union with type missing and a number, skim incorrectly categorizes it as a categorical column. For example, starting with the example provided (using the iris dataset):

allowmissing!(iris, :SepalLength)
skim(iris)

makes SepalLength go to categorical instead of numeric. It seems the package is designed to keep this as a numeric column since there is "Missings" there as well.

Output:

┌─────────────────────┬───────────┐
│                Type │ DataFrame │
│             N. rows │       150 │
│             N. cols │         5 │
│     N. numeric cols │         3 │
│ N. categorical cols │         2 │
│    N. datetime cols │         0 │
└─────────────────────┴───────────┘

3 numeric columns
┌─────────────┬─────────┬──────────┬──────────┬──────┬──────┬──────┬──────┬──────┬───────┐
│        Name │    Type │ Missings │ Complete │ Mean │ Std. │ Min. │ Med. │ Max. │ Hist. │
├─────────────┼─────────┼──────────┼──────────┼──────┼──────┼──────┼──────┼──────┼───────┤
│  SepalWidth │ Float64 │        0 │   100.0% │ 3.06 │ 0.44 │  2.0 │  3.0 │  4.4 │ ▁▃▄▂▁ │
│ PetalLength │ Float64 │        0 │   100.0% │ 3.76 │ 1.77 │  1.0 │ 4.35 │  6.9 │ ▃▁▂▃▁ │
│  PetalWidth │ Float64 │        0 │   100.0% │  1.2 │ 0.76 │  0.1 │  1.3 │  2.5 │ ▃▁▃▂▂ │
└─────────────┴─────────┴──────────┴──────────┴──────┴──────┴──────┴──────┴──────┴───────┘

2 categorical columns
┌─────────────┬─────────────────────────────────┬──────────┬──────────┐
│        Name │                            Type │ Missings │ Complete │
├─────────────┼─────────────────────────────────┼──────────┼──────────┤
│ SepalLength │         Union{Missing, Float64} │        0 │   100.0% │
│     Species │ CategoricalValue{String, UInt8} │        0 │   100.0% │
└─────────────┴─────────────────────────────────┴──────────┴──────────┘

The output is cut off when there are many columns

Example:

using DataSkimmer
using DataFrames
df = DataFrame(Dict([Symbol(k) => 1:10 for k in 'a':'z'])...)
skim(df)

Output:

┌─────────────────────┬───────────┐
│                Type │ DataFrame │
│             N. rows │        10 │
│             N. cols │        26 │
│     N. numeric cols │        26 │
│ N. categorical cols │         0 │
│    N. datetime cols │         0 │
└─────────────────────┴───────────┘

26 numeric columns
┌──────┬───────┬──────────┬──────────┬──────┬──────┬──────┬──────┬───────┬───────┐
│ Name │  Type │ Missings │ Complete │ Mean │ Std. │ Min. │ Med. │  Max. │ Hist. │
├──────┼───────┼──────────┼──────────┼──────┼──────┼──────┼──────┼───────┼───────┤
│    o │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    b │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    p │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    n │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    j │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    e │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    c │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    h │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    l │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    w │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    x │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    d │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    k │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    s │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│    v │ Int64 │        0 │   100.0% │ 5.50 │ 3.03 │ 1.00 │ 5.50 │ 10.00 │ ▂▂▂▂▂ │
│  ⋮   │   ⋮   │    ⋮     │    ⋮     │  ⋮   │  ⋮   │  ⋮   │  ⋮   │   ⋮   │   ⋮   │
└──────┴───────┴──────────┴──────────┴──────┴──────┴──────┴──────┴───────┴───────┘
                                                                   11 rows omitted

Errors for NamedTuples of vectors or vectors of NamedTuples

Reported by @pdeffebach on the Julia Discourse who has kindly opened a PR to fix it.

Linking here so I have a record of it.

Min and Max columns should ignore missing

Recently ran skim on a dataset with lots of missing values in DateTime columns. Perhaps these should ignore missing in reporting the Min and Max values. The number of missing is already reported in the table. Having missing in min and max doesn't add much value.

20 datetime columns
┌───────────────────────────┬──────────────────────────┬──────────┬──────────┬─────────────────────┬─────────────────────┬───────┐
│                      Name │                     Type │ Missings │ Complete │                Min. │                Max. │ Hist. │
├───────────────────────────┼──────────────────────────┼──────────┼──────────┼─────────────────────┼─────────────────────┼───────┤
│                      Time │                 DateTime │        0 │   100.0% │ 2014-05-27T05:50:00 │ 2019-05-22T01:18:00 │ ▂▂▂▂▂ │
│                      Date │                 DateTime │        0 │   100.0% │ 1990-05-14T00:00:00 │ 2019-01-04T00:00:00 │ ▁▃▂▂▃ │
│                      Time │                 DateTime │        0 │   100.0% │ 2014-05-19T09:08:00 │ 2019-04-18T01:23:00 │ ▂▂▂▂▂ │
│                      Time │ Union{Missing, DateTime} │      116 │    18.9% │             missing │             missing │ ▃▃▃▁▁ │
│                      Time │                 DateTime │        0 │   100.0% │ 2014-04-12T15:31:00 │ 2019-03-09T02:10:00 │ ▂▂▂▂▂ │
│                      Time │ Union{Missing, DateTime} │       64 │    55.2% │             missing │             missing │ ▃▃▂▂▁ │
│                      Time │ Union{Missing, DateTime} │       94 │    34.3% │             missing │             missing │ ▃▄▁▁▁ │
│                      Time │ Union{Missing, DateTime} │      118 │    17.5% │             missing │             missing │ ▄▄▁▁▁ │
│                      Time │ Union{Missing, DateTime} │       61 │    57.3% │             missing │             missing │ ▂▃▂▃▂ │
│                      Time │ Union{Missing, DateTime} │       67 │    53.1% │             missing │             missing │ ▂▂▂▂▂ │
│                      Time │ Union{Missing, DateTime} │      111 │    22.4% │             missing │             missing │ ▂▂▂▃▂ │
│                      Time │ Union{Missing, DateTime} │      141 │     1.4% │             missing │             missing │ ▅   ▅ │
│                      Time │ Union{Missing, DateTime} │      116 │    18.9% │             missing │             missing │ ▃▂▂▂▂ │
│                      Time │ Union{Missing, DateTime} │      127 │    11.2% │             missing │             missing │ ▂▂▂▃▂ │
│                      Time │ Union{Missing, DateTime} │      128 │    10.5% │             missing │             missing │ ▄▁▂▂▂ │
│                      Time │ Union{Missing, DateTime} │      137 │     4.2% │             missing │             missing │ ▂▅  ▃ │
│                      Time │ Union{Missing, DateTime} │       91 │    36.4% │             missing │             missing │ ▂▂▃▂▂ │
│                      Time │ Union{Missing, DateTime} │      121 │    15.4% │             missing │             missing │ ▃▁▂▂▂ │
│                      Time │ Union{Missing, DateTime} │      121 │    15.4% │             missing │             missing │ ▂▃▂▂▁ │
│                      Date │ Union{Missing, DateTime} │        1 │    99.3% │             missing │             missing │ ▂▂▂▂▂ │
└───────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────────────────┴─────────────────────┴───────┘

Update version of dependency PrettyTables.jl

PrettyTables has been updated to version 2. The dependency in Project.toml compat for DataSkimmer should be updated toPrettyTables = "1.0, 2" which should then allow PrettyTable version [1.0, 3.0.0) as per this documentation.

Entirely missing columns are categorised as numeric

Should there be a new section for this?

Show the number of unique values for categorical columns

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Columns with data entirely missing: put into Numeric and DateTime columns

When a column has 100% of the rows missing, skim puts it in under both the Numeric and DateTime lists. I think this is because both numeric and datetime column structs have Union with Missing as part of their definition while Categorical does not and is defined as !is_numeric(x) && !is_datetime(x) in helpers.jl

There may need to be a is_missing(x) function in helpers.jl for columns with all rows missing. Additionally, a new struct in DataSkimmer.jl called AllMissingColumn. This would be similar to the Categorical Column struct.

I opened one pull request (just figuring all this out) to correct one other error with Missing in datetime. I will try to work on this more extensive change if you don't get to it.

Date and DateTime columns with missing values do not show up under 'datetime' category

Both the numeric and categorical columns buckets or categories tolerate types that are unions with missing. For example, Union{Missing, Int64} shows up under the numeric column category. Union{Missing, String} is under the categorical columns category. However, Union{Missing, Date} and Union{Missing, DateTime} continue to be listed as 'categorical' columns.