nshiab/simple-data-analysis

Easy-to-use and high-performance JavaScript library for data analysis. Works with tabular and geospatial data.

Simple data analysis (SDA)

SDA is an easy-to-use and high-performance JavaScript library for data analysis. You can use it with tabular and geospatial data.

The documentation is available here. Tests are run for Node.js and Bun.

To use the library in your browser, check out simple-data-analysis-flow. You might also find the journalism library and Code Like a Journalist interesting.

The library is maintained by Nael Shiab, computational journalist and senior data producer for CBC News.

To install the library:

npm i simple-data-analysis

To setup a basic repo, create a folder and run:

npx simple-data-analysis

The options are:

• --bun : To use Bun instead of Node.js.
• --js : To use JavaScript instead of TypeScript.
• --force : To overwrite files like package.json, .gitignore, etc.

Core principles

SDA is born out of the frustration of switching between Python, R, and JavaScript to produce data journalism projects. Usually, data crunching and analysis are done with Python or R, and interactive data visualizations are coded in JavaScript. However, being proficient in multiple programming languages is hard. Why can't we do everything in JS?

The missing piece in the JavaScript/TypeScript ecosystem was an easy-to-use and performant library for data analysis. This is why SDA was created.

The library is based on DuckDB, a fast in-process analytical database. Under the hood, SDA sends SQL queries to be executed by DuckDB. We use duckdb-node and duckdb-wasm. This means SDA can run in the browser and with Node.js and other runtimes. For geospatial computations, we rely on the duckdb_spatial extension.

The syntax and the available methods were inspired by Pandas (Python) and the Tidyverse (R).

You can also write your own SQL queries if you want to (check the customQuery method) or use JavaScript to process your data (check the updateWithJS method).

Feel free to start a conversation or open an issue. Check how you can contribute.

Performance

To test and compare the library's performance, we calculated the average temperature per decade and city with the daily temperatures from the Adjusted and Homogenized Canadian Climate Data. See this repository for the code.

We ran the same calculations with [email protected] (both Node.js and Bun), Pandas (Python), and the tidyverse (R).

In each script, we:

1. Loaded a CSV file (Importing)
2. Selected four columns, removed rows with missing temperature, converted date strings to date and temperature strings to float (Cleaning)
3. Added a new column decade and calculated the decade (Modifying)
4. Calculated the average temperature per decade and city (Summarizing)
5. Wrote the cleaned-up data that we computed the averages from in a new CSV file (Writing)

Each script has been run ten times on a MacBook Pro (Apple M1 Pro / 16 GB).

Small file

With ahccd-samples.csv:

• 74.7 MB
• 19 cities
• 20 columns
• 971,804 rows
• 19,436,080 data points

[email protected] is the fastest, but if we skip writing the CSV file, the tidyverse shows impressive speed with small files.

Big file

With ahccd.csv:

• 1.7 GB
• 773 cities
• 20 columns
• 22,051,025 rows
• 441,020,500 data points

Thanks to DuckDB, [email protected] really shines with big files. It's the fastest option.

We also tried the One Billion Row Challenge, which involves computing the min, mean, and max temperatures for hundreds of cities in a 1,000,000,000 rows CSV file. The library has been able to crunch the numbers in 1 minute 32 seconds on the same computer (Apple M1 Pro / 16 GB). For more, check this repo forked from this one. The JavaScript code is here.

Note that DuckDB, which powers SDA, can also be used with Python and R.

SDA in an Observable notebook

Observable notebooks are great for data analysis in JavaScript.

In this example, we calculate the average temperature per decade in three cities and check for trends. We will also join two tables to retrieve the names of the cities.

This other example focuses on geospatial analysis. We create point geometries from the latitude and longitude of 2023 wildfires in Canada, do a spatial join with provinces' boundaries, and then compute the number of fires and the total area burnt per province.

SDA in an HTML page

If you want to add the library directly to your webpage, you can use a npm-based CDN like jsDelivr.

Here's some code you can copy and paste into an HTML file.

In this example, we load a CSV file with the latitude and longitude of 2023 wildfires in Canada, create point geometries from it, do a spatial join with provinces boundaries, and then compute the number of fires and the total area burnt per province.

You could also use a framework or a bundler. Install the library with npm npm i simple-data-analysis and import it into your project import { SimpleWebDB } from 'simple-data-analysis'.

<script type="module">
    // We import the SimpleWebDB class from the esm bundle.
    import { SimpleWebDB } from "https://cdn.jsdelivr.net/npm/[email protected]/+esm"

    async function main() {
        // We start a SimpleWebDB instance.
        const sdb = new SimpleWebDB()

        // We create a new table.
        const provinces = sdb.newTable("provinces")
        // We fetch the provinces' boundaries. It's a geoJSON.
        await provinces.fetchGeoData(
            "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json"
        )
        // We log the provinces.
        await provinces.logTable()

        // We create a new table.
        const fires = sdb.newTable("fires")
        // We fetch the wildfires data. It's a CSV.
        await fires.fetchData(
            "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv"
        )
        // We create point geometries from the lat and lon columns
        // and we store the points in the new column geom.
        await fires.points("lat", "lon", "geom")
        // We log the fires
        await fires.logTable()

        // We match fires with provinces
        // and we output the results into a new table.
        // By default, joinGeo will automatically look
        // for columns storing geometries in the tables,
        // do a left join, and put the results
        // in the left table.
        const firesInsideProvinces = await fires.joinGeo(provinces, "inside", {
            outputTable: "firesInsideProvinces",
        })

        // We summarize to count the number of fires
        // and sum up the area burnt in each province.
        await firesInsideProvinces.summarize({
            values: "hectares",
            categories: "nameEnglish",
            summaries: ["count", "sum"],
            decimals: 0,
        })
        // We rename columns.
        await firesInsideProvinces.renameColumns({
            count: "nbFires",
            sum: "burntArea",
        })
        // We want the province with
        // the greatest burnt area first.
        await firesInsideProvinces.sort({ burntArea: "desc" })

        // We log the results. By default, the method
        // logs the first 10 rows, but there are 13
        // provinces and territories in Canada.
        await firesInsideProvinces.logTable(13)

        // We can also retrieve the data as an array of objects.
        const data = await firesInsideProvinces.getData()
        console.log(data)

        // We close everything.
        await sdb.done()
    }

    main()
</script>

And here's what you'll see in your browser's console tab.

SDA with Node.js and similar runtimes

First, ensure that you have Node.js v18 or higher installed.

Then, run this command in a new folder to install what you need.

npx simple-data-analysis

Then copy and paste the code below into the index.js or index.ts file and run it with npm run sda.

In this example, we load a CSV file with the latitude and longitude of 2023 wildfires in Canada, create point geometries from it, do a spatial join with provinces' boundaries, and then compute the number of fires and the total area burnt per province.

It's the same code as the one you would run in a browser, except we use the SimpleDB class instead of SimpleWebDB and loadData instead of fetchData.

With Node.js and other runtimes, more methods are available to load and write data from/to local files. Check the SimpleTable class documentation.

import { SimpleDB } from "simple-data-analysis"

// We start a SimpleDB instance.
const sdb = new SimpleDB()

// We create a new table.
const provinces = sdb.newTable("provinces")
// We fetch the provinces' boundaries. It's a geoJSON.
await provinces.loadGeoData(
    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json"
)
// We log the provinces.
await provinces.logTable()

// We create a new table.
const fires = sdb.newTable("fires")
// We fetch the wildfires data. It's a CSV.
await fires.loadData(
    "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv"
)
// We create point geometries from the lat and lon columns
// and we store the points in the new column geom.
await fires.points("lat", "lon", "geom")
// We log the fires.
await fires.logTable()

// We match fires with provinces
// and we output the results into a new table.
// By default, joinGeo will automatically look
// for columns storing geometries in the tables,
// do a left join, and put the results
// in the left table.
const firesInsideProvinces = await fires.joinGeo(provinces, "inside", {
    outputTable: "firesInsideProvinces",
})

// We summarize to count the number of fires
// and sum up the area burnt in each province.
await firesInsideProvinces.summarize({
    values: "hectares",
    categories: "nameEnglish",
    summaries: ["count", "sum"],
    decimals: 0,
})
// We rename columns.
await firesInsideProvinces.renameColumns({
    count: "nbFires",
    sum: "burntArea",
})
// We want the province with
// the greatest burnt area first.
await firesInsideProvinces.sort({ burntArea: "desc" })

// We log the results. By default, the method
// logs the first 10 rows, but there are 13
// provinces and territories in Canada.
await firesInsideProvinces.logTable(13)

// We close everything.
await sdb.done()

And here's what you should see in your console.

Caching fetched and computed data

Instead of running the same code over and over again, you can cache the results. This can speed up your workflow, especially when fetching data or performing computationally expensive operations.

Here's the previous example adapted to cache data. For more information, check the cache method documentation.

The data is cached in the hidden folder .sda-cache at the root of your code repository. Make sure to add it to your .gitignore. If you want to clean your cache, just delete the folder.

If you set up with npx simple-data-analysis, .sda-cache is automatically added to your .gitignore and you can use npm run clean to clear the cache.

import { SimpleDB } from "simple-data-analysis"

// We enable two options to make our lives easier.
// cacheVerbose will log information about the cached
// data, and logDuration will log the total duration between
// the creation of this SimpleDB instance and its last operation.
const sdb = new SimpleDB({ cacheVerbose: true, logDuration: true })

const fires = sdb.newTable("fires")

// We cache these steps with a ttl of 60 seconds.
// On the first run, the data will be fetched
// and stored in the hidden folder .sda-cache.
// If you rerun the script less than 60 seconds
// later, the data won't be fetched but loaded
// from the local cache. However, if you run the
// code after 60 seconds, the data will be
// considered outdated and fetched again.
// After another 60 seconds, the new data in the cache will
// expire again. This is useful when working with scraped data.
// If you update the code passed to the cache method,
// everything starts over.
await fires.cache(
    async () => {
        await fires.loadData(
            "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/firesCanada2023.csv"
        )
        await fires.points("lat", "lon", "geom")
    },
    { ttl: 60 }
)

const provinces = sdb.newTable("provinces")

// Same thing here, except there is no ttl option,
// so the cached data will never expire unless you delete
// the hidden folder .sda-cache. Again, if you update
// the code passed to the cache method, everything
// starts over.
await provinces.cache(async () => {
    await provinces.loadGeoData(
        "https://raw.githubusercontent.com/nshiab/simple-data-analysis/main/test/geodata/files/CanadianProvincesAndTerritories.json"
    )
})

const firesInsideProvinces = sdb.newTable("firesInsideProvinces")

// While caching is quite useful when fetching data,
// it's also handy for computationally expensive
// operations like joins and summaries.
// Since the fires table has a ttl of 60 seconds
// and we depend on it here, we need a ttl equal
// or lower. Otherwise, we won't work with
// up-to-date data.
await firesInsideProvinces.cache(
    async () => {
        await fires.joinGeo(provinces, "inside", {
            outputTable: "firesInsideProvinces",
        })
        await firesInsideProvinces.summarize({
            values: "hectares",
            categories: "nameEnglish",
            summaries: ["count", "sum"],
            decimals: 0,
        })
        await firesInsideProvinces.renameColumns({
            count: "nbFires",
            sum: "burntArea",
        })
        await firesInsideProvinces.sort({ burntArea: "desc" })
    },
    { ttl: 60 }
)

await firesInsideProvinces.logTable(13)

// It's important to call done() at the end.
// This method will remove the unused files
// in the cache. It will also log the total duration
// if the logDuration option was set to true.
await sdb.done()

After the first run, here's what you'll see in your terminal. For each cache(), a file storing the results has been written in .sda-cache.

The whole script took around a second to complete.

Nothing in cache. Running and storing in cache.
Duration: 311 ms. Wrote ./.sda-cache/fires.ff...68f.geojson.

Nothing in cache. Running and storing in cache.
Duration: 397 ms. Wrote ./.sda-cache/provinces.42...55.geojson.

Nothing in cache. Running and storing in cache.
Duration: 49 ms. Wrote ./.sda-cache/firesInsideProvinces.71...a8.parquet.

table firesInsideProvinces:
┌─────────┬────────────┬─────────────────────────────┬─────────┬───────────┐
│ (index) │ value      │ nameEnglish                 │ nbFires │ burntArea │
├─────────┼────────────┼─────────────────────────────┼─────────┼───────────┤
│ 0       │ 'hectares' │ 'Quebec'                    │ 706     │ 5024737   │
│ 1       │ 'hectares' │ 'Northwest Territories'     │ 314     │ 4253907   │
│ 2       │ 'hectares' │ 'Alberta'                   │ 1208    │ 3214444   │
│ 3       │ 'hectares' │ 'British Columbia'          │ 2496    │ 2856625   │
│ 4       │ 'hectares' │ 'Saskatchewan'              │ 560     │ 1801903   │
│ 5       │ 'hectares' │ 'Ontario'                   │ 741     │ 441581    │
│ 6       │ 'hectares' │ 'Yukon'                     │ 227     │ 395461    │
│ 7       │ 'hectares' │ 'Manitoba'                  │ 301     │ 199200    │
│ 8       │ 'hectares' │ 'Nova Scotia'               │ 208     │ 25017     │
│ 9       │ 'hectares' │ 'Newfoundland and Labrador' │ 85      │ 21833     │
│ 10      │ 'hectares' │ 'Nunavut'                   │ 1       │ 2700      │
│ 11      │ 'hectares' │ 'New Brunswick'             │ 202     │ 854       │
│ 12      │ 'hectares' │ null                        │ 124     │ 258       │
└─────────┴────────────┴─────────────────────────────┴─────────┴───────────┘
13 rows in total (nbRowsToLog: 13)

SimpleDB - Done in 891 ms

If you run the script less than 60 seconds after the first run, here's what you'll see.

Thanks to caching, the script ran five times faster!

Found ./.sda-cache/fires.ff...8f.geojson in cache.
ttl of 60 sec has not expired. The creation date is July 5, 2024, at 4:25 p.m.. There is 11 sec, 491 ms left.
Data loaded in 151 ms. Running the computations took 311 ms last time. You saved 160 ms.

Found ./.sda-cache/provinces.42...55.geojson in cache.
Data loaded in 8 ms. Running the computations took 397 ms last time. You saved 389 ms.

Found ./.sda-cache/firesInsideProvinces.71...a8.parquet in cache.
ttl of 60 sec has not expired. The creation date is July 5, 2024, at 4:25 p.m.. There is 11 sec, 792 ms left.
Data loaded in 1 ms. Running the computations took 49 ms last time. You saved 48 ms.

table firesInsideProvinces:
┌─────────┬────────────┬─────────────────────────────┬─────────┬───────────┐
│ (index) │ value      │ nameEnglish                 │ nbFires │ burntArea │
├─────────┼────────────┼─────────────────────────────┼─────────┼───────────┤
│ 0       │ 'hectares' │ 'Quebec'                    │ 706     │ 5024737   │
│ 1       │ 'hectares' │ 'Northwest Territories'     │ 314     │ 4253907   │
│ 2       │ 'hectares' │ 'Alberta'                   │ 1208    │ 3214444   │
│ 3       │ 'hectares' │ 'British Columbia'          │ 2496    │ 2856625   │
│ 4       │ 'hectares' │ 'Saskatchewan'              │ 560     │ 1801903   │
│ 5       │ 'hectares' │ 'Ontario'                   │ 741     │ 441581    │
│ 6       │ 'hectares' │ 'Yukon'                     │ 227     │ 395461    │
│ 7       │ 'hectares' │ 'Manitoba'                  │ 301     │ 199200    │
│ 8       │ 'hectares' │ 'Nova Scotia'               │ 208     │ 25017     │
│ 9       │ 'hectares' │ 'Newfoundland and Labrador' │ 85      │ 21833     │
│ 10      │ 'hectares' │ 'Nunavut'                   │ 1       │ 2700      │
│ 11      │ 'hectares' │ 'New Brunswick'             │ 202     │ 854       │
│ 12      │ 'hectares' │ null                        │ 124     │ 258       │
└─────────┴────────────┴─────────────────────────────┴─────────┴───────────┘
13 rows in total (nbRowsToLog: 13)

SimpleDB - Done in 184 ms

And if you run the script 60 seconds later, the fires and join/summary caches will have expired, but not the provinces one. Some of the code will have run, but not everything. The script still ran 1.5 times faster. This is quite handy in complex analysis with big datasets. The less you wait, the more fun you have!

Found ./.sda-cache/fires.ff...8f.geojson in cache
ttl of 60 sec has expired. The creation date is July 5, 2024, at 4:25 p.m.. It's is 4 min, 1 sec, 172 ms ago.
Running and storing in cache.
Duration: 424 ms. Wrote ./.sda-cache/fires.ff...8f.geojson.

Found ./.sda-cache/provinces.42...55.geojson in cache.
Data loaded in 10 ms. Running the computations took 397 ms last time. You saved 387 ms.

Fond ./.sda-cache/firesInsideProvinces.71...a8.parquet in cache
ttl of 60 sec has expired. The creation date is July 5, 2024, at 4:25 p.m.. It's is 4 min, 1 sec, 239 ms ago.
Running and storing in cache.
Duration: 42 ms. Wrote ./.sda-cache/firesInsideProvinces.71...a8.parquet.

table firesInsideProvinces:
┌─────────┬────────────┬─────────────────────────────┬─────────┬───────────┐
│ (index) │ value      │ nameEnglish                 │ nbFires │ burntArea │
├─────────┼────────────┼─────────────────────────────┼─────────┼───────────┤
│ 0       │ 'hectares' │ 'Quebec'                    │ 706     │ 5024737   │
│ 1       │ 'hectares' │ 'Northwest Territories'     │ 314     │ 4253907   │
│ 2       │ 'hectares' │ 'Alberta'                   │ 1208    │ 3214444   │
│ 3       │ 'hectares' │ 'British Columbia'          │ 2496    │ 2856625   │
│ 4       │ 'hectares' │ 'Saskatchewan'              │ 560     │ 1801903   │
│ 5       │ 'hectares' │ 'Ontario'                   │ 741     │ 441581    │
│ 6       │ 'hectares' │ 'Yukon'                     │ 227     │ 395461    │
│ 7       │ 'hectares' │ 'Manitoba'                  │ 301     │ 199200    │
│ 8       │ 'hectares' │ 'Nova Scotia'               │ 208     │ 25017     │
│ 9       │ 'hectares' │ 'Newfoundland and Labrador' │ 85      │ 21833     │
│ 10      │ 'hectares' │ 'Nunavut'                   │ 1       │ 2700      │
│ 11      │ 'hectares' │ 'New Brunswick'             │ 202     │ 854       │
│ 12      │ 'hectares' │ null                        │ 124     │ 258       │
└─────────┴────────────┴─────────────────────────────┴─────────┴───────────┘
13 rows in total (nbRowsToLog: 13)

SimpleDB - Done in 594 ms

Others

If you want to generate and save charts with Node.js and other runtimes, check the journalism library, more specifically the savePlotChart function.

simple-data-analysis's Issues

Support `fillMissingKeys` in `addItems` method

Adding tests for SimpleData class methods

Keeping _data private

It's possible to change the data inside SimpleData class like this:

simpleData._data = [{}]

It should not be allowed. Same for _keys.

Translate `integration/test.js` in TypeScript

correlation arguments check

At the beginning:
if (key1 === undefined && key2 === undefined) { ... }

It should also take into account when key2 = [].

getQuantile method

It would be useful to be able to call something like simpleData.getQuantile({key: "someKey", quantile: 0.9}).

plot legend

Make sure the legend is the same width as the chart.

removeDuplicates()

It would be a great method to add.

Create two classes, one for browser, one for NodeJS, inheriting from parent class SimpleData

showMissingValues

Add a method to show entries with missing values.

Cleaning methods should have newKey option

If you use valuesToString, the values for a specific key are replaced by their string version. But what if you want to put the output in a new key? An optional newKey should be available as an option. Same for all similar methods.

Finishing up documentation for v1.0.0

Fix `linter` warnings

We have 65 linter warnings:

warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any

Let's be explicit about typings and fix them all!

checkValues with overwrite false doesn't work

replaceStringValue

If not a string, it should just replace the value. So the method should be replaceValue and should handle any types.

Equivalent to pivot_longer() and pivot_wider() in R

These two methods to restructure could be:

• keysToValues()
• valuesToKeys()

Summarize should support strings and dates

Summarize should support string and dates values. If the summary function does not make sense, it should return NA (or undefined?).

Support multiple merging methods (left join, right join, etc).

sortValues and string

It would be great to use localeCompare for strings.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare

Add method getValue

add start and end index options for constructor

Adding method to generate Observable Plot charts

Scatter chart
Bar chart
Line chart
Box chart

getMean and others

It would be great to have these methods:

• getMin({key: "key1"}) => the minimum value for the key1
• getMax(key: "key1") => the maximum value for the key1
• getMean(key: "key1") => the average value for the key1
• getMedian(key: "key1") => the median value for the key1

They could be based on d3.min, d3.max, d3.mean, and d3.median. We already have https://github.com/d3/d3-array as a dependency.

The methods and the tests could be based on the existing src/methods/exporting/getArray.js and test/unit/methods/exporting/getArray.test.js.

add getLength()

Dates and getMin

getMin, getMax should work with dates.

Maybe getMean and getMedian too.

addRank / addIndex?

Could be useful to have a method to add a key with a rank for each item.

Problem with sortValues

When there are strings and numbers mixed in the data, the results are not what we expect. The method should probably check the type of the values and throw an error if mixed types.

add keepDuplicates

Implement parameters destructuring in `SimpleDocument`

In #22 we refactor parameters in SimpleClass. Once this PR is merged and we have agreed on a way to proceed, we should implement these changes in SimpleDocument too.

Save to CSV

As a user of the library, I would like to save the result to CSV.

color scale type

Add an option to change the colour scale. Documentation: https://observablehq.com/@observablehq/plot-scales?collection=@observablehq/plot

clone() and the returning class

The clone method should return an instance of the same class. For the moment, it always returns a SimpleData prototype.

Generate HTML document and/or React component with analysis

loadDataFromLocal file should support multiple delimiters for csv file

Fix logging when `overwrite` is false.

When overwrite is false, we log original data instead of processed data. We should fix it to log processed data instead.

When this is done, we should republish an npm version, update the Observable notebook import version (in the first notebook of the series) and make sure that it works (in the last notebook of the series).

Adding regression to the charts https://observablehq.com/@fil/plot-regression

json string in errors

When logging a JSON object in an error string, it would be great to have spaces in the string.

Check value before converting to float or integer

For example, "-9999.9M" is converted to -9999.9. But it should throw an error. And we should have an option to exclude it if we want to. Another method to check what is returning errors would be great too.

`addQuantiles`, `addBins` and `correlation` methods seem to be broken

See unit tests comment. I do not think that this is the expected behaviour. Let's fix it!

addQuantiles check type

Shouldn't addQuantiles throw an error when the values are not numbers?

Method to export complete dataset as arrays

autotype

When importing files, autoType should not be by default. It should be an option.

Adding a contributing file

Write Readme with basic documentation

Adding percentage method

Error when using checkValues on employees data with autoType=false

When using simple-data-analysis-flow

create test for helpers

addItems fillMissingKeys

addItems take two SimpleData instances (or array of objects) and append the second one to the first. When keys are missing from the second, fillMissingKeys adds undefined values. But it should work the other way around too. When keys are present in the second instance but not in the first one, it should also add undefined values.

apply private function (func, ...args) instead of decorator

Logging decorators

We repeat logging in every single function. Instead, we should create a function that's called as a decorator in SimpleData class.