Assess handling of non-float values while writing tensor files

[kyuksel]

Assess handling of non-float values while writing tensor files

[carbocation]

This is a great issue. In general, my impression is that nearly every field can be missing, and missingness is usually not meaningful (because of how UK Biobank structures their data). However, when it is meaningful, it matters a lot.

The raw UK Biobank CSV -> BigQuery tools are opinionated about how this should be represented in the database - for example, the 39th medication field doesn't need to be duplicated and reflected as NULL for everyone, because the fact that there are 39+ medication fields is an artifact of how UK Biobank chose to represent the data and it has no biological meaning. Fields like LDL, blood pressure, etc., are all missing for some people. In the database, in general, missing values are not included. So, for now, when something is meaningfully missing, extra care has to be taken to assemble the missing value.

[gpbatra]

In general, I find logs are not read for ETL issues like this, and this can be a make or break for quality of data analysis. As you get more into the ETL code, please notice where handling of NULL or missing is handled in manners like this, instead of explicitly handled, and add tickets.