githubocto / flat-viewer Goto Github PK
View Code? Open in Web Editor NEWData viewer for Flat repos
Home Page: https://flatgithub.com
License: MIT License
Data viewer for Flat repos
Home Page: https://flatgithub.com
License: MIT License
@duckdb provides an in-process database query engine for common tabular file formats, including csv and JSON. A wasm distribution is available on npm, and can be used in JSONDetail to provide the user with an input field to run queries against a selected file and commit.
Richer support (eg, joins between multiple files and/or commits) could be addressed in subsequent issues.
Hi Team,
Hi, I have created Open In flat viewer Extension. when clicked on GitHub repo, will open that Flat Viewer. I will be publishing soon on the Chrome Web store. The extension simply provides a URL redirect: it reads the current Github repo URL and opens a new tab at https://flatgithub.com/ with the user, repository.
Link: https://github.com/misalraj/open_in_flatViewer.
Do check this. Feedback needed.
Thank you for Flat Viewer! I love this project.
This is a feature request, and I'm not sure how feasible it would be to implement. But we are using Flat Viewer to host and share datasets for the Post45 Data Collective, a repository for literary and cultural data from 1945 to the present. We would love to have the option of adding a title or perhaps some brief documentation to the top of the Flat Viewer page โ perhaps this could be customized by including information in a certain way in a README file in each folder in the repository?
Just to give a concrete example, here's what our current framework for hosting the datasets looks like:
vs
We would basically like to add the title, author, and contextual information.
Hi!
I'm trying to use flat viewer on a private repo (via local storage token).
Flat viewer can access files/commits, but I get an error in the data viewer pane: "Oh no, we couldn't load x/y.csv for some reason".
A 404 is logged to console because looking up the pattern:
https://raw.githubusercontent.com/${owner}/${name}/${sha}/${filename}
gives a 404 for my private repo.
Not sure whats wrong - should there be a token included in the URL (for private repos) for instance? (github links to raw data in a private repo using ?token=...
)
[cross-posting https://github.com/githubocto/flat/discussions/15 in case this is a better place to discuss this]
i.e. I'm using a Flat action to scrape the number of "things" from an API. I want to graph number of "things" over time ๐
Let me know if I can help!
Hi!
Great tool!
I know it would require new auth plumbing but I think it would be even better if I the viewer could work for private repos!
since Get repository content would use base64-encoded when size less than 1MB, but when the size exceed 1MB
, the encoded
payload became empty ''
{
content: ""
download_url: "https://raw.githubusercontent.com/zhang699/translations/b11706c2aff98a97988faf9d38a873565053d4c7/xxx.json?token=ABIL4IH7KXMQWJ2IS5LK7TTC4O3FG"
encoding: "none"
}
one of solution I think we can use download_url
to fetch the raw
Use case: It is nice to be able to visualise some files in a development branch
I think this is worth considering ๐
UI/homepage leaves lots of space for a sticker, and repo gives a lot of helpful context about the tool which the UI/homepage doesn't (being nice and concise themselves)
thanks for the great tool!
flatgithub.com is not recognizing a field that contains only integers and empty value as a number:
I've verified that there are no sneaky thousand separators or anything else.
The column in question is the "Number of employees" field
I have set up a repo to collect holdings of ARK funds using flat
When I sort by market value column, I notice that the sorting is done in pure alphabetical sort order and the data is not sorted by the actual value.
Going to https://flatgithub.com/smittyvb/smitop.com gives a 404, while going to the main page and entering Smittyvb
and smitop.com
works as expected.
I love this project! We used a flat-like data structure for @Karuna2020, our COVID-19 nonprofit, and you can for example see a list of all our donors and the amounts using Flat Viewer: https://flatgithub.com/Karuna2020/open-data?filename=donations.json
I'd also love to see YAML support, for example for this file: https://raw.githubusercontent.com/AnandChowdhary/life-data/master/heart-rate.yml (currently, YAML is not supported: https://flatgithub.com/AnandChowdhary/life-data?filename=heart-rate.yml).
Example: https://github.com/RandomFractals/vscode-data-preview/tree/master/data/arrow
Apache Arrow ecosystem page: https://arrow.apache.org/powered_by/
Sample preview of that arrow data file:
would be nice to view it via flatgithub url.
Thanks!
This would make many GIS devs and experts very happy.
See this data wrangling ticket for more context info: RandomFractals/vscode-leaflet#9
and sensible defaults and conventions that could be used for converting geo data to flat data files: https://github.com/keplergl/kepler.gl/blob/master/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.md#supported-file-formats
d3-dsv supports both CSV and TSV file formats. I'd love to see support for .tsv
files in addition to .csv
.
Using the flat data repo linked from the documentation (https://githubnext.com/projects/flat-data#part-iii-visualizing-our-data-for-easy-sharing-optional)
Sample filter:
https://flatgithub.com/the-pudding/data?filename=boybands%2Fboys.csv&filters=name%3DSlim&sha=9b4e89ed178e11bf70a9b3d8ffa57ac2ea181b96&sort=band%2Casc&stickyColumnName=band
The filter text is "slim": 3 records are shown but only the first record contains this value
Hi,
when I apply a filter I see something like:
?filename=dati%2Fsomministrazioni-vaccini-latest.csv
&filters=fornitore%3DJanssen
&sha=9e6a9f4a150e6b96ef52827e0e2692274d68701e
&sort=data_somministrazione%2Cdesc
&stickyColumnName=data_somministrazione
For some I can deduce the meaning. Is there a page with documentation?
What's a stickyColumnName, what is it for?
What's the date format I must use for date fields? etc.
Thank you
Files downloaded from Flat Viewer have the name flat-ui__data <timestamp>.extension. . Would it be possible to use flat-ui_<timestamp>_<filename> or similar naming schemes to make it easy to identify them?
It looks like we can't have \r\n
as a new line character in CSV and should use \n
.
Looking at data fetcher in this repo, I found https://github.com/githubocto/flat-viewer/blob/main/src/api/index.ts#L142 which parses CSV using https://github.com/githubocto/flat-viewer/blob/main/src/api/index.ts#L7
So it looks like that parser is here https://github.com/d3/d3-dsv/blob/main/src/dsv.js
The current flat viewer allows for easy filtering of single values.
However, I would like to know how to filter for multiple values in the same column.
For example, you have a column animals
with values: cat
, dog
, penguin
, etc.
How to filter for both cat
AND dog
?
Sorting by numbers seems to get confused if there are blank values as well.
Would it be possible to establish some way to configure the file location for dates and values?
It would be handy to extend this:
Lines 48 to 49 in 19477a0
Fantastic work on Flat, GitHub OCTO! ๐
I've used Flat Data to grab the 7-day asteroid forecast from NASA's Near Earth Object API.
Opening Flat Viewer on nasa-asteroids-postprocessed.csv
on desktop, everything looks great: https://flatgithub.com/nsgrantham/nasa-asteroids
However, when opened on mobile (iOS 14.4.2 using Safari and Chrome), there appears to be a problem with the datetime columns close_approach_datetime
and orbit_determination_datetime
.
These columns are of the form yyyy-MM-dd HH:mm
and yyyy-MM-dd HH:mm:ss
respectively.
There is no problem with the regular date columns of the form yyyy-MM-dd
.
I'm experimenting with a large CSV file (> 100 MB) so I had to use LFS. Everything went well during the Post Fetch Data step but the flat viewer is not able to visualize files stored with Git LFS.
url: https://flatgithub.com/rmaceissoft/flat-vaccination-peru-example?filename=registro_vacunacion.csv
Is there something I'm missing here? Thanks in advance
@Wattenberger can you please tag your releases and list what's in those release bumps?
https://github.com/githubocto/flat-viewer/releases
Thanks!
For non-technical users, Flat Viewer as hosted on flatgithub.com has been effective for exploring tabular data hosted on GitHub repositories. A team could create a GitHub organization containing Flat Data repositories, each one set with the corresponding flatgithub.com URL, and non-technical users can then be directed to explore these repositories on github.com/<org-name>
.
However, given how GitHub's UI is tailored mostly for developers, non-technical users may be unfamiliar (and possibly unwilling to be familiar) with reading through a list of repositories on a GitHub org page, and would hence be intimidated from exploring further.
To help such users, Flat Viewer could implement the listing of repositories that contain Flat Data (identified by eg being tagged with flat-data
) within a given GitHub organization. Users can then look at a GitHub org's flat-data
repos, through an interface specifically for exploring Flat Data, via a URL like flatgithub.com/<org-name>
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.