unicef / magicbox-kepler-app Goto Github PK
View Code? Open in Web Editor NEWVisualizing MagicBox data on country maps using KeplerGL
Visualizing MagicBox data on country maps using KeplerGL
In ./client/config.js, add item to indicate whether 'show data table' for all data sets should be included in UI.
Magic box currently serves
When user saves config, the entire config, with data, gets saved on server. Haven't decided what to do with files user drops into kepler, but above data sets shouldn't be copied.
Need to save pointer to dataset, and recreate when config is fetched on front-end.
User should be able to add description of visualization. This description should be visible to audience via modal or something.
Would think this would be something all kepler users want. Not sure if exists as issue here: https://github.com/uber/kepler.gl/issues
When saving a map, a user should be able to determine if the map should be auth protected or if it is available to the public. As a bonus, build a check or flag that prevents a user from saving a map as publicly accessible if it points to a dataset that is marked as sensitive.
The user should also be able to see a list of their saved maps. The list might the name of the map and/or include a search feature. From this list, it should be easy to share the URL.
Test datasets and sample maps will be made available on Azure storage or via a drive
Sometimes, we need to query external databases using only SQL. One of the common use cases when this occurs is calculating multi-hop mobility.
To better understand multi-hop, imagine 3 users (u1, u2, and u3) and 3 cellphone towers that they might ping off of (A, B, and C).
Over the course of a day, the first user may travel in such a way, that when using their phone, they ping the following towers:
AAAA BBB C A
Additional user trips might be:
u2: C A BB
u3: CC BBB A C
To calculate the mobility for each user user, we need to calculate all trips between areas, but not count "self-loops", or when the start and end towers are the same (i.e. A -> A).
First, we would simplify the trips to remove self-loops, finding their basic path in chronological order. For example, u1: A, B, C, A
Next, we need to calculate all permutations of trips among these locations:
If there were more trips, perhaps if the path was: A, B, C, A, C, A, B, we would continue until all hops are recorded.
SQL query should ultimately return either a table with only the hops, and no user information, such as:
The database could be hundreds of thousands to millions of lines, so the query should be efficient. It could be broken down into parts and use temporary tables, for example, so portions of the query can be run at a time to minimize chances of timing out. Variable names should be clear, so that this can be easily adapted to multiple different databases.
For testing purposes:
The columns are
sim_id : identifier of sim card
timestamp : datetime dd-mm-yyy hh:mm:ss, where hh is given in 24 hour format
event : the type of event that generated this row, can be sms, call, data
site_id : antenna identifier
The activity sequences are:
u1: AAAA BBB C A
u2: C A BB
u3: CC BBB A C
The simplified sequences are:
U1: A B C A
U2: C A B
U3: C B A C
The mobility results should be:
AB: 2
AC: 2
BA: 2
BC: 2
CA: 3
CB: 2
SQL online sandbox: http://rextester.com/l/sql_server_online_compiler
This might work: https://github.com/helmetjs/nocache#readme
When "display-share-button" is set to false in ./client/config.js, it should not be displayed in kepler app.
When working with region-based datasets, we usually encounter the situation where the polygons in one dataset do not geospatially align with the polygons in another.
Examples:
One feature we want for the MagicBox Platform is the ability to aggregate the values for polygons of one shapefile/GeoJSON/csv based on the polygons of another shapefile/GeoJSON/csv. Hence, corresponding with the examples above:
The pattern is spotted:
Here you see a Voronoi diagram over a country shapefile. The generating points are arbitrary, but we can assume they are cell tower locations (pairs of coordinates to be exact).
This Medium article discusses in-depth this use case, though it goes one step further with the output. Going beyond (origin_admin, destination_admin, mobility_value)
for each row/tuple in the resulting geofile, it asks for a mobility matrix. The latter is easily generated from the former, though.
When user selects borderfile, try to fetch healthsites for that country by default, and load as separate dataset.
Add this feature by creating a feature branch off dev.
Moved here
Clone https://github.com/unicef/magicbox-aad-app as new projectlet in root.
Right now, our app does not have an authorization layer yet. The authorization protocol we desire lives in a separate project (listed above). What we want is to integrate this auth service into our demo app, which, we think, will smooth out our deployment process.
We see two potential ways to integrate auth:
The ideal outcome would be to have a functioning, deployed version of the app (in its current iteration) with an auth layer. If you have an alternate suggestion for how this could be done, we are open to creative solutions different than the suggestions above that include a write up and code demo.
"Add data to Map" to should not display by default on load.
Questions:
Each user should have a personal directory where their map configs are stored.
@mikefab including here some sketch/diagram for our app architecture would be helpful.
Make "share" and "save config" buttons configurable in ./client/config.js
UNICEF uses Azure, so many static files are in Azure blob or file storage. Here is an npm module to interact with Azure.
When user clicks save, the data sets and layer configuration should be saved to the back end. When user returns to url in new browser, the updated config.json should load.
Use docker-compose to run Auth app, Kepler client, and backend.
When user logs in to AAD app, then the kepler app is beamed in through an iframe. Currently the back-end serves the config.json, but we might switch out backend for direct data exchange with Azure and external APIs. Let's discuss tomorrow.
In modal, make "Add Magic Box data" the first frame, and "Drop in data" the second.
Link in first modal to second might read, "Add your own datasets"
Deploy Magicbox Kepler app on Kubernetes. That includes both componentes Client and Server, and expose them for external consume.
Input:
Output:
We're creating datasets for all countries for school mapping and EPI.
Can we do something like add somecountry-school-mapping to fetch relevant map?
This needs to be fleshed out before being implemented.
Maps - kepler exports maps into a file called config.json. It has three nodes:
dataId
to be exact, which references the id
value inside the datasets node. See Kepler.gl docs here and here).UNICEF uses Azure, so many static files are in Azure blob or file storage. Here is an npm module to interact with Azure.
When a user logs in to create a map, a function should check to see if they have a storage folder already in Azure storage. If not, one should be created with their username. The user should be able to create a visualization and then name and save the map to this folder when it is completed.
As a bonus, create an autosave feature that saves an untitled map upon creation and periodically throughout (every x seconds or minutes, for example) and/or create an alert that would remind user to save before closing out the browser window.
Test datasets and sample maps will be made available on Azure storage or via a drive
Currently, the auth wrapper is the out-of-the-box create-react-app App.js. We would like to update this to make it part of the Magic Box Maps product. This would serve as the log in page for UNICEF staff and the public-facing interface of the product.
A clipped Voronoi diagram is one that is confined to a compact domain (i.e. a bounded and closed 2D region or a 3D volume). In the example above, a normally generated Voronoi diagram will have some cells infinite or partially outside of the country boundary. However, only the cells that lie within the country boundary are of our interests.
Admin 1 geojson: http://myjson.com/wvt2s
The output of this task could be used as input to #15.
@thoat started this.
Add most recent work to eslintignore and fix it gradually.
moved here
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.