Comments (5)
yes, let me address in next PR, I'm doing some things with the tensor writer code anyways and this is easy to address
from ml4h.
Our current tensorization process involves 4 main steps:
Run Dataflow jobs to tensorize data from big query database. This means 6 separate data flow runs for each of the fields: ['categorical', 'continuous', 'icd', 'disease', 'death', 'phecode_disease']
Tensorize bulk data with tensor_writer_ukbb.py. This file has code to tensorize abdominal, cardiac, and brain MRI as well as resting and exercise ECG.
Merge tensors with merge_hd5s.py. Doing intersections or in-place merging as appropriate.
Append any CSV or TSV data with the `append[categorical,continuous]_[csv,tsv]' recipe modes.
Each step is currently run separately. We would also like to be able to run them all at once and store intermediate tensors in google cloud buckets.
from ml4h.
Partners data is tensorized with tensor_writer_partners.py. Currently only tensorizes Partners ECGs. Parallelization is achieved using joblib, should probably convert to strategy of tensor_writer_ukbb.py or rest of codebase
from ml4h.
Partners tensorize should switch from joblib
to multiprocess
to be consistent with the rest of the codebase.
@ndiamant any thoughts?
from ml4h.
@StevieSong is this still an issue?
from ml4h.
Related Issues (20)
- Type Checks in Conv Layer
- Tensorflow 2.2.0 errors HOT 1
- No OSX support for multithreading HOT 2
- p-values for bootstrapped performance comparison HOT 1
- Migrate to tf.data.Dataset and/or torch.utils.data.DataLoader HOT 1
- Support multi-dimensional TMAPs in output HOT 1
- Transformer Model Module
- Store MRI metadata in a DataFrame during hd5 production HOT 2
- ml4h install: vtk module is pinned to a version not supported in py3.8 HOT 4
- some tensormap key names need to be updated. HOT 1
- fix display of MRI images in plot_mri_tensor_as_animation HOT 2
- issues running the `mnist_survival_analysis_demo.ipynb` notebook HOT 3
- PCLR SavedModel different to PCLR.h5, unable to load PCLR.h5 HOT 2
- Jupyter, Welder failed to start after 10 minutes.
- missing function: slice
- Channels in mri_silhouettes
- PCLR_lead_I.h5
- droid docker image HOT 1
- Regarding the issue of DROID-LV prediction output values being too small HOT 1
- echo_supervised_inference_recipe.py wide_file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ml4h.