Comments (3)
thbar
commented on August 21, 2024
Preliminary notes, in brain dump mode for now:
I have first looked at the transport-site database, which contains history for validations, using:
SELECT
validation_json_size,
to_char(100 * validation_json_size::float / (SUM(validation_json_size) OVER()) , '999D99%') as ratio,
subquery.id,
resource_id,
url
FROM (SELECT *, pg_column_size(details) as validation_json_size from validations) as subquery
INNER JOIN resource r on r.id = resource_id
WHERE validation_json_size IS NOT NULL
ORDER by validation_json_size descOn a recent database, this gives (before optimisation - just the top extract):
I have then grabbed data from a large dataset:
- https://data.bretagne.bzh/explore/dataset/base-de-donnees-multimodale-transports-publics-en-bretagne-mobibreizh/information/
- https://data.bretagne.bzh/api/datasets/1.0/base-de-donnees-multimodale-transports-publics-en-bretagne-mobibreizh/attachments/12_2020_mobibreizhbret_gtfs_zip/
And run the validator locally with:
cargo run --release -- --input 12_2020_mobibreizhbret_gtfs.zip > 12_2020_mobibreizhbret_gtfs.validation.json
Finally, I filtered the JSON with jq to get an idea of where the big stuff is going:
cat 12_2020_mobibreizhbret_gtfs.validation.json | jq -c 'path(..)|[.[]|tostring]|join("/")' | sed -e 's/\([0-9]\)/X/g' | sort | uniq -c | sort -rn | grep "validations/CloseStops"
50998 "validations/CloseStops/XXX/related_objects/XX/object_type"
50998 "validations/CloseStops/XXX/related_objects/XX/name"
50998 "validations/CloseStops/XXX/related_objects/XX/id"
50998 "validations/CloseStops/XXX/related_objects/XX"
38169 "validations/CloseStops/XXX/related_objects/XXX/object_type"
38169 "validations/CloseStops/XXX/related_objects/XXX/name"
38169 "validations/CloseStops/XXX/related_objects/XXX/id"
38169 "validations/CloseStops/XXX/related_objects/XXX"
8675 "validations/CloseStops/XXX/related_objects/X/object_type"
8675 "validations/CloseStops/XXX/related_objects/X/name"
8675 "validations/CloseStops/XXX/related_objects/X/id"
8675 "validations/CloseStops/XXX/related_objects/X"
4702 "validations/CloseStops/XX/related_objects/XX/object_type"
4702 "validations/CloseStops/XX/related_objects/XX/name"
4702 "validations/CloseStops/XX/related_objects/XX/id"
4702 "validations/CloseStops/XX/related_objects/XX"
3861 "validations/CloseStops/XX/related_objects/XXX/object_type"
3861 "validations/CloseStops/XX/related_objects/XXX/name"
3861 "validations/CloseStops/XX/related_objects/XXX/id"
3861 "validations/CloseStops/XX/related_objects/XXX"
900 "validations/CloseStops/XXX/severity"
900 "validations/CloseStops/XXX/related_objects"
900 "validations/CloseStops/XXX/object_type"
900 "validations/CloseStops/XXX/object_name"
900 "validations/CloseStops/XXX/object_id"
I have also discussed with @antoine-de and indeed here:
- there is a large cardinality in the close stop related objects, because they refer to trips / stop times
- one optimisation to conduct will be to just store the route, rather than the trip; there is a loss of information, but probably good enough for validators
- this is preferable to trimming the list of errors for now (something we try to avoid at the moment)
I'll resume later to provide a change here.
from transport-validator.
thbar
commented on August 21, 2024
Also useful query by @antoine-de:
select
validations.resource_id, close_stops->>'object_id' as object_id, close_stops->>'details',
json_array_length(close_stops->'related_objects') as length from validations,
json_array_elements(validations.details->'CloseStops') as close_stops
where validations.resource_id is not null order by length desc;from transport-validator.
thbar
commented on August 21, 2024
Solved via #105 for now, this reduces the payload x14 for the largest file, and 3 to 4 times for more modest files, so a good improvement.
from transport-validator.
Related Issues (20)
- Extract feed_info.feed_contact_email as a metadata
- Metadata: add accessibility features
- Metadata: add stats for bikes_allowed
- routes.agency_id: add check HOT 3
- Wrong color format is not detected HOT 1
- IDs should be unique HOT 1
- transfer_type can be empty HOT 1
- Raise an error when files are in a subfolder HOT 5
- Add a warning when the calendar is empty
- agency_url is required: switch from Warning to Error
- Memory efficient validation HOT 16
- Review error levels for duplicates
- Move this repository to rust-transit? HOT 3
- Switch to github-actions instead of circle-ci
- New metadata: trips_with_trip_headsign_count
- New metadata: routes_with_route_short_name_count
- New metadata: routes_with_route_long_name_count
- New metadata: know if a file exist and is not empty HOT 2
- non-zero exit code if checking failed when using --input HOT 2
- stop_times.stop_id is now conditionally required
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transport-validator.