Spread visualizes how viruses and other pathogens are spreading in time and space. It creates compellable shareable, interactive and time-animated visualization.
Home Page: https://spreadviz.org
License: MIT License
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Spread is a messaging application. We need to enable clients to subscribe to status changes.
Implement these wireframes:
DATA
https://app.zeplin.io/project/6075ecb45aa2eb47e1384d0b/screen/6075ed2e68de41453c244a13
https://app.zeplin.io/project/6075ecb45aa2eb47e1384d0b/screen/6075ed30375711443d473c6b
ERROR
https://app.zeplin.io/project/6075ecb45aa2eb47e1384d0b/screen/6075ed305a09c542e790702f
Regarding the arrow icons, use the svg icon we have for those instead of a font 👍
https://github.com/fbielejec/spread/blob/style-fixes/src/cljs/shared/components.cljs#L15
and
https://github.com/fbielejec/spread/blob/master/src/cljs/analysis_viewer/views.cljs#L450
IMHO it will look myriads better. See services/ui/icons/icn_dropdown.svg for the icons
You could use a css transformation to flip it 180 degrees on a class change, or just make a copy and use two icons, YMMV.
If a given analysis failed for a fixable reason we want to notify the user of that fact by surfacing the exact error message or the human-readable version of it.
Add a failure_reason column and persist error message when worker catches an exception.
These errors should be human readable. For runtime exceptions that are likely bugs we should use a general error message ("Your analysis with id could not be finished. Please contact us."
Failed analysis can be queried for their status.
Designs: https://app.zeplin.io/project/6075ecb45aa2eb47e1384d0b/screen/6075ed305a09c542e790702f
Implement the:
to support parsing discretely annotated data
search box can make it all dissapear:
GraphQL endpoints which return maps have to wrap their response to transform keys into camelCase that fits the graphql schema (example)
It would greatly improve readability and simplify the codebase to have a general mechanism that transparently transforms these responses.
The pedestal server allows for the use of interceptors.
And example is already used to inject custom context into the resolvers.
This is one possible solution.
Interceptor / middleware that transforms responses. Current endpoints cleaned of the clj->gql transformations.
Lacinia can be used separately from the pedestal server, and it is possible to replace the pedestal webserver with ring or yada if assignee deems it necessary.
Argument and response logging middleware/interceptors : #7
UI needs subscriptions to report ongoing analysis state and progress.
This PR is about delivering a POC/MVP of graphQL subscriptions with pure re-frame events/subs.
Potential libraries:
Home page with an arbitrarily subscription, demonstrated to update the db state and the components accordingly.
pressing on the "Completed..." makes all accordion buttons disappear / go out of view:
Likely caused by long names
Remove token from the localstore (for example from top header menu -> logout)
Login again using google
Initial request throws authorization error
Client should make the request with the token and obtain user data
See general/initialize event. It should probably wait for the token to be persisted in local-storage
Results page -> data tab should display more information about settings used to parse the content:
- name of the locations file
- name of the locations file
- name of the .trees file
None of this data is persisted at the moment, therefore some schema changes are needed (to DB and Gql schema).
Write a suitable migrations file and add alter the GraphQL schema accordingly. Display that additional data on the data tab of the results page.
This is a research ticket
Explore the use of the kepler library as a basis of the visualization component.
Component should take as input the json files generated by the libspread, examples can be found here:
https://github.com/fbielejec/spread/tree/master/docs/data_examples
It should overlay them on a map as lines, point and polygons (or circular polygons) and animat ethe over time (according to the timestamp fields of the JSON objects)
A Proof-of-concept implementation.
Time Slicer analysis should include branches and nodes form the MCC tree.
The corresponding MCC tree needs to be an additional input for this analysis.
Implement the:
to support parsing bssvs log files
decide on the output:
Research if certain buttons, (copy and paste-settings buttons from the wireframes) are even possible implement.
Remove the buttons if we won't be implementing the functionality.
Implement components for these wireframes:
https://xd.adobe.com/view/cab84bb6-15c6-44e3-9458-2ff4af17c238-9feb/screen/44bb1ba7-e9f8-4752-95da-942f04ea32d2/
Maintain a healthy and slim codebase.
Use clj-kondo.
Linter step added as part of CI build
When I started an analysis the "queued" item with the progress was not displayed.
Everything went fine with the analysis:
upsert-analysis {:is-new true, :of-type "CONTINUOUS_TREE", :created-on 1633087994796, :viewer-url-params "?output=8b28e9a0-f51e-44f3-af9a-d73f2eb223c3/64d692ec-9b3e-46a4-88f3-d179f5ef2b3a.json&maps=", :user-id "8b28e9a0-f51e-44f3-af9a-d73f2eb223c3", :status "SUCCEEDED", :id "64d692ec-9b3e-46a4-88f3-d179f5ef2b3a", :progress 1.0, :readable-name "WNV_relaxed_geo_gamma_MCC"}
I'll use this occasion to complain about the styling too :)
Scrollbar is in a weird place here, maybe we should rather put the viewport (the area between the header and the footer) in a scrollbar?
Also could the mrsd box be the same dimensions as other fields?
On another pages it somehow still can leave the bottom buttons impossible to scroll to:
Implement sorting the rows of BayefFactor results table by a selected column value,
Material UI has a table component with sorting implemented
https://material-ui.com/components/tables/#sorting-amp-selecting
Export (to CSV file) should be done in the order defined by that sorting.
Add support for notifying user of new completed analysis.
See this screen:
was-seen? boolean column that defaults to falsehttps://app.zeplin.io/project/6075ecb45aa2eb47e1384d0b/screen/6075ed23df1e283fbd6c71ff
Warning: Received `true` for a non-boolean attribute `button`.
If you want to write it to the DOM, pass a string instead: button="true" or button={value.toString()}.
at li
at div
at div
at div
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
at div
at div
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
at div
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
IMO this is coming from https://github.com/fbielejec/spread/blob/style-fixes/src/cljs/ui/component/app_container.cljs#L121
Change it to :button? maybe?
Another one is about a missing stylesheet, defined in the index.html file:
GET http://localhost:8020/css/material-design-iconic-font.min.css net::ERR_ABORTED 404 (Not Found)
GQL resolvers implement the same pattern of logging the arguments, binding the response in a let ninding, logging the response and returning.
Example
Existing endpoints would be greatly simplified by adding an interceptor / middleware that logs those for every request
Known steps towards feature implementation.
What needs further specifying and investigating.
It is possible to replace the webserver with ring or yada if assignee deems it necessary. In such a case it would be nice to keep a GraphQL ide, for example with:
Implement single continuous analysis.
If the user decides to upload the full posterior tree distribution together with an MCC tree he/she will have the option to choose between the MCC node based polygons or the time sliced polygons.
We want to render the files from the analysis using the rendering engine in analysis viewer service.
The viewer service accepts URL to the output file as URL parameter. It can use other url parameters if needed.
The href with the link should be here:
https://github.com/fbielejec/spread/blob/master/src/cljs/ui/analysis_results/page.cljs#L161
Use the config namespace to set the host part of the link to the URL where the analysis viewer is
User can click on the results link and is taken to the complete visualisation of his results in a separate browser tab.
He/she can use the Copy button next to the href element to share the URL.
When pasting the URL in a browser window the visualisation can be viewed.
Wireframes specify progress
Working progress reports in analysis status subscriptions.
DO NOT use just the error field to decide whether to render this tab, use the output instead. This is because a certain analysis might be a revisited analysis that has had an error before, but was parsed correctly with updated settings
As a developer I want to read the google access token as returned on the hash fragment of the redirect URI durin OAuth2 flow (see here )
The router component in html5 mode does not recognize url fragemnts/
Set {:html5? true} in router config
Enter in browser bar:
http://localhost:8020/auth#fu=bar
Hash part is removed by the router:
http://localhost:8020/auth
Router should recognize url query, argumetns and fragment parts.
Swap underlying router lib for a more modern implementation, like:
https://github.com/metosin/reitit
These two guys (or gals?) are not level:
sendgrid, emails + magic links
Right after starting the application:
Download the React DevTools for a better development experience: https://reactjs.org/link/react-devtools
main.cljs:34 Starting v1.0.3 ...
main.cljs:35 Config {:router {:routes [["/" :route/splash] ["/home" :route/home] ["/new-analysis" :route/new-analysis] ["/analysis-results" :route/analysis-results] ["/documentation" :route/documentation] ["/map" :route/map]], :default-route :route/splash, :scroll-top? true, :html5? true}, :analysis-viewer-url "http://localhost:8021", :logging {:level :debug, :console? true, :sentry false}, :graphql {:ws-url "ws://127.0.0.1:3001/ws", :url "http://127.0.0.1:3001/api"}, :environment "dev", :google {:client-id "806052757605-5sbubbk9ubj0tq95dp7b58v36tscqv1r.apps.googleusercontent.com", :redirect-uri "http://localhost:8020/?auth=google"}, :root-url "http://localhost:8020", :version "1.0.3", :public-key "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJliLjOIAqGbnjGBM1RJml/l0MHayaRH\ncgEg00O9wBYvoNXrstFSzKTCKtG5MayUKgdG7C/98nu/TEzhvRFjINcCAwEAAQ==\n-----END PUBLIC KEY-----"}
logging.cljs:28 ui.events.general:9 Active page changed cljs.core/PersistentArrayMap [count: 3]
browser.cljs:20 shadow-cljs: #3 ready!
logging.cljs:28 ui.splash.page:58 component-did-mount null
react-dom.development.js:68 Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in a useEffect cleanup function.
at f (http://localhost:8020/js/cljs-runtime/reagent.impl.component.js:756:31)
logging.cljs:28 ui.events.general:9 Active page changed cljs.core/PersistentArrayMap [count: 3]
Sometimes if the WS connection is interrupted during upload this analysis cannot be retrieved from the UI (stuck in ATTRIBUTE_PARSED status).
client will need paged queries to display analysis created by authenticated user.
Use SQL offdet and limit to create paged subscriptions
taoensso calls log to console twice, e.g.:
api_1 | 2020-12-11T10:01:07.544Z filip-Meerkat INFO [api.server:103] - Starting server {:env "dev", :logging {:level :debug}, :api {:port 3001}, :aws {:secret-access-key "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", :s3-port 9000, :s3-host "127.0.0.1", :sqs-host "localhost", :region nil, :sqs-port 9324, :bucket-name "spread-dev-uploads", :workers-queue-url "http://localhost:9324/queue/workers", :access-key-id "AKIAIOSFODNN7EXAMPLE"}, :db {:dbname "spread", :port "3306", :user "root", :password "Pa55w0rd", :host "127.0.0.1"}}
api_1 | INFO Starting server
api_1 | {:env "dev",
api_1 | :logging {:level :debug},
api_1 | :api {:port 3001},
api_1 | :aws
api_1 | {:secret-access-key "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
api_1 | :s3-port 9000,
api_1 | :s3-host "127.0.0.1",
api_1 | :sqs-host "localhost",
api_1 | :region nil,
api_1 | :sqs-port 9324,
api_1 | :bucket-name "spread-dev-uploads",
api_1 | :workers-queue-url "http://localhost:9324/queue/workers",
api_1 | :access-key-id "AKIAIOSFODNN7EXAMPLE"},
api_1 | :db
api_1 | {:dbname "spread",
api_1 | :port "3306",
api_1 | :user "root",
api_1 | :password "Pa55w0rd",
api_1 | :host "127.0.0.1"}}
api_1 | in api.server[api/server.clj:103] at Fri Dec 11 10:01:07 UTC 20202020-12-11T10:01:0
This greatly pollutes the logs.
Moreover it'd be great if a dev fflag could be added to the logging component that controls whether the output is formatted and on multiple lines (in dev) or on one line (qa/prod environment)
As a developer I sometimes want to build the artifacts and docker images locally.
pom.xml build configuration of libspread in the root directory is overwritten when generating uberjar artifacts of the services.
cd services/worker
./build.sh
pom.xml is overwritten, it is no longer possible to build libspread from it.
Implement the:
to support parsing continuously annotated tree distribution files
Each service should join flowstorm on start in dev environment
Add documentation to README/Development
Add user registration and authentication
Auth flow implemented and covered with tests.
... and one more, the buttons could use a on-hover effect from the frames, they look dead :)
It's silly to use Strings for time data. Store them as unix timestamps.
Pages have only been provisionally styled with the default Material-UI styles and are not pixel perfect.
The problem we ran into is that the default MUI styless (delivered with classnames) are hard to kill 😏
Part of this task is to research if there is some way to do it after all. I created a branch with some of the attempts:
#77
If not than either:
UI is as close to pixel-perfect as possible.
#81 should probably be finished first
Use material UI to style all the components
In certain places, to avoid data corruption SQL statements should be executed in a transaction and rolled back on error:
https://github.com/fbielejec/spread/blob/master/src/clj/worker/listener.clj#L306
Ideally a wrapper macro or a function that takes as arguments a database connection and a list of transactions, e.g.:
(with-tx db
(bayes-factor-model/insert-bayes-factors {:bayes-factor-analysis-id id
:bayes-factors (json/write-str bayesFactors)})
(bayes-factor-model/update! {:id id
:output-file-url url
:status :SUCCEEDED}))
Working and tested implementation of a transaction wrapper.
Scale no of workers when there are un-acked messages in the queue
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.