frictionlessdata/frictionless-ci

Data management service that brings continuous data validation to tabular data in your repository via Github Action

frictionless-ci

Data management service that brings continuous data validation to tabular data in your repository via Github Action

Purpose

• Continuous Data Validation: With Frictionless Repository you can ensure the quality of your data. This Github Action will report any problems with your data like bad header or missing cells.

Features

• Open Source (MIT)
• Validation badges
• Visual quality reports
• Composable Github Action

Example

Take a look at the DEMO repository

Documentation

Please visit our documentation portal:

• https://repository.frictionlessdata.io

frictionless-ci's Issues

Duplicated primary key entries are reported as errors in framework but validate in repository

Overview

Some checks that fail validation in Frictionless Framework pass validation in Frictionless Repository.

Example

Recently I have added primary keys to a schema. Since then, Frictionless Framework v5.8.1 correctly points out that there are duplicated entries:

validation report

frictionless validate data/valid/datapackage.json 

# -------
# invalid: brazilian-transparency-and-open-data-portals.csv 
# -------

## Summary 

+------------------+--------------------------------------------------+
| Name             | Value                                            |
+==================+==================================================+
| File Place       | brazilian-transparency-and-open-data-portals.csv |
+------------------+--------------------------------------------------+
| File Size        | 20.0 kB                                          |
+------------------+--------------------------------------------------+
| Total Time       | 0.04 Seconds                                     |
+------------------+--------------------------------------------------+
| Rows Checked     | 183                                              |
+------------------+--------------------------------------------------+
| Total Errors     | 1                                                |
+------------------+--------------------------------------------------+
| PrimaryKey Error | 1                                                |
+------------------+--------------------------------------------------+

## Errors 

+-------+---------+-------------+------------------------------------------------------------------------------------+
|   Row | Field   | Type        | Message                                                                            |
+=======+=========+=============+====================================================================================+
|     3 |         | primary-key | Row at position "3" violates the primary key: the same as in the row at position 2 |
+-------+---------+-------------+------------------------------------------------------------------------------------+


# -------
# invalid: brazilian-municipality-and-state-websites.csv 
# -------

## Summary 

+------------------+-----------------------------------------------+
| Name             | Value                                         |
+==================+===============================================+
| File Place       | brazilian-municipality-and-state-websites.csv |
+------------------+-----------------------------------------------+
| File Size        | 292.2 kB                                      |
+------------------+-----------------------------------------------+
| Total Time       | 0.207 Seconds                                 |
+------------------+-----------------------------------------------+
| Rows Checked     | 2880                                          |
+------------------+-----------------------------------------------+
| Total Errors     | 40                                            |
+------------------+-----------------------------------------------+
| PrimaryKey Error | 40                                            |
+------------------+-----------------------------------------------+

## Errors 

+-------+---------+-------------+------------------------------------------------------------------------------------------+
|   Row | Field   | Type        | Message                                                                                  |
+=======+=========+=============+==========================================================================================+
|  1030 |         | primary-key | Row at position "1030" violates the primary key: the same as in the row at position 1029 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1182 |         | primary-key | Row at position "1182" violates the primary key: the same as in the row at position 1181 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1208 |         | primary-key | Row at position "1208" violates the primary key: the same as in the row at position 1207 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1256 |         | primary-key | Row at position "1256" violates the primary key: the same as in the row at position 1255 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1267 |         | primary-key | Row at position "1267" violates the primary key: the same as in the row at position 1266 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1310 |         | primary-key | Row at position "1310" violates the primary key: the same as in the row at position 1309 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1375 |         | primary-key | Row at position "1375" violates the primary key: the same as in the row at position 1374 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1479 |         | primary-key | Row at position "1479" violates the primary key: the same as in the row at position 1478 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1501 |         | primary-key | Row at position "1501" violates the primary key: the same as in the row at position 1500 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1519 |         | primary-key | Row at position "1519" violates the primary key: the same as in the row at position 1518 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1534 |         | primary-key | Row at position "1534" violates the primary key: the same as in the row at position 1533 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1607 |         | primary-key | Row at position "1607" violates the primary key: the same as in the row at position 1606 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1612 |         | primary-key | Row at position "1612" violates the primary key: the same as in the row at position 1611 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1622 |         | primary-key | Row at position "1622" violates the primary key: the same as in the row at position 1621 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1639 |         | primary-key | Row at position "1639" violates the primary key: the same as in the row at position 1638 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1702 |         | primary-key | Row at position "1702" violates the primary key: the same as in the row at position 1701 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1741 |         | primary-key | Row at position "1741" violates the primary key: the same as in the row at position 1740 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1750 |         | primary-key | Row at position "1750" violates the primary key: the same as in the row at position 1749 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1760 |         | primary-key | Row at position "1760" violates the primary key: the same as in the row at position 1759 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1787 |         | primary-key | Row at position "1787" violates the primary key: the same as in the row at position 1786 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1940 |         | primary-key | Row at position "1940" violates the primary key: the same as in the row at position 1939 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1941 |         | primary-key | Row at position "1941" violates the primary key: the same as in the row at position 1940 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  1963 |         | primary-key | Row at position "1963" violates the primary key: the same as in the row at position 1962 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2022 |         | primary-key | Row at position "2022" violates the primary key: the same as in the row at position 2021 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2023 |         | primary-key | Row at position "2023" violates the primary key: the same as in the row at position 2022 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2048 |         | primary-key | Row at position "2048" violates the primary key: the same as in the row at position 2047 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2081 |         | primary-key | Row at position "2081" violates the primary key: the same as in the row at position 2080 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2194 |         | primary-key | Row at position "2194" violates the primary key: the same as in the row at position 2193 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2224 |         | primary-key | Row at position "2224" violates the primary key: the same as in the row at position 2223 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2299 |         | primary-key | Row at position "2299" violates the primary key: the same as in the row at position 2298 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2339 |         | primary-key | Row at position "2339" violates the primary key: the same as in the row at position 2338 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2346 |         | primary-key | Row at position "2346" violates the primary key: the same as in the row at position 2345 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2347 |         | primary-key | Row at position "2347" violates the primary key: the same as in the row at position 2346 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2396 |         | primary-key | Row at position "2396" violates the primary key: the same as in the row at position 2395 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2491 |         | primary-key | Row at position "2491" violates the primary key: the same as in the row at position 2490 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2561 |         | primary-key | Row at position "2561" violates the primary key: the same as in the row at position 2560 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2672 |         | primary-key | Row at position "2672" violates the primary key: the same as in the row at position 2671 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2673 |         | primary-key | Row at position "2673" violates the primary key: the same as in the row at position 2672 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2723 |         | primary-key | Row at position "2723" violates the primary key: the same as in the row at position 2722 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+
|  2846 |         | primary-key | Row at position "2846" violates the primary key: the same as in the row at position 2845 |
+-------+---------+-------------+------------------------------------------------------------------------------------------+

However, Frictionless Repository v2 checks on this workflow do pass validation tests for some reason. Aren't they supposed to be the same?

Expected behaviour

Frictionless Repository was expected to fail validation due to the duplicated primary key violations in two of the csv files.

Broken links to resources in validation report

The names of the CSV files are clickable in the validation report. However, these links are broken. For example, the report for
the demo repository:

https://repository.frictionlessdata.io/pages/dashboard.html?user=frictionlessdata&repo=repository-demo&flow=frictionless

Links to data/valid.csv: https://repository.frictionlessdata.io/pages/data/valid.csv

Spaces in a workflow name breaks the link to the report

Overview

• https://github.com/augusto-herrmann/transparencia-dados-abertos-brasil/actions/runs/3998640675/jobs/6861546674
• https://github.com/augusto-herrmann/transparencia-dados-abertos-brasil/pull/82/files

Rebase on generating livemark-based site/report?

Overview

It's a long-shot and research but we can consider building a report site into a branch instead of using https://repository.frictionlessdata.io/pages/dashboard.html. It will solve the problems with artifacts not available for non-auth user

Links in main page lead to non-existing page

Overview

Open https://repository.frictionlessdata.io/. There are three links featured on the middle of the main page:

• Data Validation
• Visual Reports
• Markdown Badges

All of them lead to the very same broken link:

---

Please preserve this line to notify @roll (lead of this repository)

Handle the situation when report size over Github Action limits

Overview

See this workflow - https://github.com/vulekamali/budget-spending-data/actions/runs/2589987176 - it seems not to be able to save a report as it's too big

From #28

Validation report does not include info on resources with inline data

Overview

https://github.com/MaREI-EPMG/ireland-carbon-budget-scenarios-2021/ data package includes two resources consisting of inline data, series and scenarios. It seems that they are excluded from the validation report: https://repository.frictionlessdata.io/report/?user=MaREI-EPMG&repo=ireland-carbon-budget-scenarios-2021&flow=frictionless&run=1295757932

---

Please preserve this line to notify @roll (lead of this repository)

Release v2 (frictionless@5 compatible)

Overview

To follow - frictionlessdata/frictionless-py#1120

Cannot load a report

I am trying to create a Frictionless repo, but I have some problems.

1. I cannot see the report of a failed validation.
   

2. I have a question about the action YAML file.
   
   What is the meaning of these attributes?

I would appreciate any help. I appreciate any help you can provide.

Investigate a problem with dataportals.org

Overview

okfn/dataportals.org#238 (comment)

Goodtables migration guide missing from docs

Overview

I think there used to be a migration guide from Goodtables in Frictionless Repository's documentation. Right now, if you open the docs and use the search feature for "Goodtables" you even find a result there.

However, when you click on that result you get a "Page not found" error.

For why this is necessary, see #10.

Import error when running Github Action workflow for data validation

Overview

In our repository the validation workflow is failing with an import error when running frictionless validate inquiry.json --json. See the error report.

Our frictionless.yaml workflow for Frictionless Repository is pretty much the default one: https://github.com/okfn/publicbodies/blob/master/.github/workflows/frictionless.yaml

Create a demo video

Overview

E.g. using https://github.com/datasets/currency-codes/

Ability to add flags like `--schema-sync`?

Overview

Is there any way to add flags to the workflow such as --schema-sync ?

Use Github Dependency Caching to run validation faster

Overview

Frictionless Repository currently builds the container and installs all Python dependencies every time a validation check is triggered.

Proposal

This could probably be sped up quite a bit by using Github Dependency Caching to cache the image after all dependencies have been installed.

Download reports

It would be nice to have a report in JSON, CSV format post validation.

Provide a badge/markdown on the report page

Overview

A user needs to be able to copy a badge code from the report page - https://repository.frictionlessdata.io/report/?user=frictionlessdata&repo=repository&flow=experimental

GitHub action failing

Overview

I use the Frictionless GitHub action in a number of repositories to check datasets created updated on a weekly schedule. Today the action failed to run, generating the same error in all repositories:

 exec /bin/sh: no such file or directory
  The command '/bin/sh -c curl -fsSL https://deb.nodesource.com/setup_14.x | bash - &&   apt-get install -y nodejs &&   npm install --production &&   pip install -r requirements.txt' returned a non-zero code: 1

My workflow file contains:

name: frictionless

on:
  workflow_dispatch:
  workflow_run:
    workflows: ["Fetch latest data"]
    types:
      - completed

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          lfs: true # add this to download LFS files
          submodules: true # add this to download submodules
      - name: Validate data
        uses: frictionlessdata/repository@v1

I tried updating to repository@v2 but that didn't help.

Review action performance/caching

Overview

ATM, the Github Action builds a docker image every time it runs. We might try to find some optimizations.

Might be related - https://github.community/t/cache-a-docker-image-built-in-workflow/16260/9

Explore if we can support Gitlab/others

Overview

See https://frictionlessdata.slack.com/archives/C0369HZ2SLT/p1675530494127069?thread_ts=1675523690.362539&cid=C0369HZ2SLT

Noudesource.com deprecation warning makes build slow

Overview

The current build process uses a script hosted in noudesource.com to install Node.js, which is deprecated, generates a warning, and adds a whole minute of wait to the build process:

  #9 [4/4] RUN   curl -fsSL https://deb.nodesource.com/setup_16.x | bash - &&   apt-get install -y nodejs &&   npm install --omit dev &&   pip install -r requirements.txt
  #9 0.227 
  #9 0.227 ================================================================================
  #9 0.227 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  #9 0.227 ================================================================================
  #9 0.227 
  #9 0.227                            SCRIPT DEPRECATION WARNING                    
  #9 0.227 
  #9 0.227   
  #9 0.227   This script, located at https://deb.nodesource.com/setup_X, used to
  #9 0.227   install Node.js is deprecated now and will eventually be made inactive.
  #9 0.227 
  #9 0.227   Please visit the NodeSource distributions Github and follow the
  #9 0.227   instructions to migrate your repo.
  #9 0.227   https://github.com/nodesource/distributions
  #9 0.227 
  #9 0.227   The NodeSource Node.js Linux distributions GitHub repository contains
  #9 0.227   information about which versions of Node.js and which Linux distributions
  #9 0.227   are supported and how to install it.
  #9 0.227   https://github.com/nodesource/distributions
  #9 0.227 
  #9 0.227 
  #9 0.227                           SCRIPT DEPRECATION WARNING
  #9 0.227 
  #9 0.227 ================================================================================
  #9 0.227 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
  #9 0.227 ================================================================================
  #9 0.227 
  #9 0.227 TO AVOID THIS WAIT MIGRATE THE SCRIPT
  #9 0.227 Continuing in 60 seconds (press Ctrl-C to abort) ...
  #9 0.227 
  #9 60.23 
  #9 60.23 ## Installing the NodeSource Node.js 16.x repo...

This should be migrated to avoid breaking the build in the future and also to skip this 60 second wait.

It seems like a similar problem has happened before (#47), but new action is required.

Release stable v1

Overview

We need to gather feedback for 1-2 month and then release.

Improve code coverage

Overview

Currently, action.js is not tested. We need to mock github. See https://github.com/actions/checkout/blob/master/__test__/input-helper.test.ts

Plan/start migration from goodtables.io

Overview

@sapetti9 Once #6 is done we can start creating issues/PR for goodtables.io users with a recommendation to migrate

Detect and validate data packages?

Overview

Currently, it looks only for CSV and EXCEL files, although, it will be easy to support data packages lookup by default. At the same time, the logic needs to handle duplication among data package's files and independently found data files.

Setup the docs search

Overview

A request has been sent to Algolia Docsearch

Documentation: configuration page has spelling errors

Overview

• packge -> package
• "if not data packages are found" -> "if no data packages are found"

Improve homepage icons

Overview

Hi @sapetti9, can you please take a look - https://repository.frictionlessdata.io/

Error: Cannot validate inquiry: stdout maxBuffer length exceeded

Overview

Hi! I'm getting

Error: Cannot validate inquiry: stdout maxBuffer length exceeded

when running as a github action.

https://github.com/vulekamali/budget-spending-data/blob/987ce9308e316e6fe8570500473757a4ccf346a3/.github/workflows/frictionless.yml
https://github.com/vulekamali/budget-spending-data/blob/987ce9308e316e6fe8570500473757a4ccf346a3/.github/frictionless.yml

Seems to be coming from here https://github.com/frictionlessdata/repository/blob/5ad6a7b66627b76220872a47054d5c291db95913/lib/action.js#L77

https://frictionlessdata.slack.com/archives/C0362US1U3G/p1657011118812489

Empty field with required constraint passed the validation

Overview

The .csv file termos-parceria-contratos-gestao had pushed with an empty value marked with required constraint: true property in schema.

Job #76 on goodtables had warned about the error, but frictionless within github actions had not

Validation report is not being generated (`Cannot load a report`)

Overview

Validation report is not being generated (message: Cannot load a report) because of two errors shown in the 'report' artifact:

The task has an error: 'in <string>' requires string as left operand, not NoneType
The task has an error: 'fieldPosition'

The repo is using git LFS, and the yaml file is pointing to frictionlessdata/repository@v1 :

• yaml

• github actions summary

• report

Nevertheless, internal validation is showing results as expected. And in another repository of reproductible examples, also using git LFS, the report could be loaded:

• yaml

• github actions summary

• report

---

Please preserve this line to notify @roll (lead of this repository)

Rename this to Workflows

While Frictionless CI makes more sense to a developer than the old name ("Repository"), I would suggest renaming it Frictionless Workflow. Or creating something of an umbrella project which incorporates the techniques in this one.

Rationale: this project could also be very useful to people creating data pipelines, in some cases linked to ETL components, while others would be more interested in streaming data sources or other advanced scenarios. Continuous Integration is just a mechanism, mostly known as a term in software engineering, while Continuous Aggregation or Continuous Validation might be more appropriate when dealing with datasets.

Furthermore, the development of user-friendly tools related to the management of workflows using Frictionless Data standards could be well communicated and promoted through an easily recognizable term. One that is, nota bene, currently promoted by GitHub itself.

I've created a #workflows channel to discuss this idea, and added some thoughts in the CKAN camp at ckan/ideas#211

Make file names clickable in Report?

Overview

It's been disabled for #40 as, currently, it might be impossible to track full path for the file due to the current architecture

Custom input

Overview

In continuation conversation with @roll, replacing path
Wanting to give custom input for validating

Full S3 bug report

Failing validation by zipped resources

Overview

I am using the pattern suggested here to describe zipped CSVs.

However, the validation fails with an "Encoding Error" (see here for an example report) while the very same datapackage validate the very same files if they are unzipped and referenced one by one.

You can find my datapackages (both the zipped and unzipped resources version) in the attached zip
datapackages.zip

---

Please preserve this line to notify @roll (lead of this repository)

Enable `make release` for Github Marketplace when possible

Overview

Currently, we need to:

• update the version
• push vX.X.X commit
• manually release on Github (including Marketplace)
• git tag -fa v1 -m "Update v1 tag"
• git push origin v1 --force

Usually, for Frictionless Software, we just run make release to release a new version but it's not yet possible for Github Actions on Github Marketplace

Related - https://github.com/actions/toolkit/blob/master/docs/action-versioning.md

Pin major frictionless version

Overview

See Dockerfile