worldhealthorganization / mca-pretermbirths-analysis Goto Github PK
View Code? Open in Web Editor NEWUNICEF-WHO LBW and Preterm estimates (2020) - Input data, code and outputs
UNICEF-WHO LBW and Preterm estimates (2020) - Input data, code and outputs
I am sure that your computer is much more powerful than mine because when I had the line in the code below, it took a very long time to run even one iteration of the loop.
I could not possibly wait for the whole thing to execute so I reduced nreps=1 just so I could make it through.
So this code file is not fully tested in that I did not let it run to full execution.
How long did it take you to run?
Here is the code and the output.
#Runs validation on the final model
> modelValOutput<-runModelValidation2P(input.data=finalPreterm,
+ nchains = nchains, nburnin = nburnin, niter = niter, nthin = nthin,
+ model.file.path = model.file.path,
+ leave.out.method=leave.out.method,
+ leave.out.percent = leave.out.percent,
+ nreps = nreps,
+ alpha.level = alpha.level, alpha.n = alpha.n, seed = seed)
running Full Model!Running model.
Total iterations: 2e+05, Burn in: 1e+05, Thinning: 25
Something hasn't converged HAS IT.
Max Rhat is 1.10740493386024
saving to RDS.
returning the model
finished modelling!finished Full Model!Getting the left out samplesrunning model for iteration 1Running model.
Total iterations: 2e+05, Burn in: 1e+05, Thinning: 25
Something hasn't converged HAS IT.
Max Rhat is 1.63803950306297
saving to RDS.
returning the model
finished modelling!predicting for model for iteration 1doing calculations for iteration 1finished calculations for iteration 1
This error occurs because this code file is expecting a file to exist as output from one of the previous code files.
Unfortunately the file was not produced by the other code, but rather one with “800” in the middle.
It is not clear at all why my run of the code results in "pCCFullModel_800" whereas obviously previous runs of the code result in "pCCFullModel_16000" files. Or if there is a difference between these two.
Temporarily changing line #178 to be able to proceed. Also line #729.
Here is the first error:
#Read in the latest estimates
> rGLBW<-read.csv("output/pCCFullModel_16000_regionalAndGlobalEstimates.csv") %>%
+ filter(year==2020) %>% dplyr::select(regionName, estN, estNU,
+ estNL, est) %>%
+ rename(wpp_lb=estN)
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'output/pCCFullModel_16000_regionalAndGlobalEstimates.csv': No such file or directory
The 12 files are there, but how were they generated? They seem to an output of some code, but where is that code?
It seems like library(gridGraphics) is needed for 6.subgroupsMetaP.R to run, so I have added it to 0.loadPackages.R. I also copied all of the library calls from the other assorted files and made sure they were all present in 0.loadPackages.R.
Here is the error:
> grid.text("Preterm<28 weeks / Preterm<37 weeks", 0.5, 0.97, gp=gpar(cex=1))
Error in grid.text("Preterm<28 weeks / Preterm<37 weeks", 0.5, 0.97, gp = gpar(cex = 1)) :
could not find function "grid.text"
This one is likely fine as well, but addOn2 is empty with only 11 columns and no rows.
So adminAll2 has the columns now, but they contain nothing...which is probably what was desired, but it did throw a warning.
Here is the warning:
> adminAll2<-cbind(adminAll, addOn2)
Warning message:
In data.frame(..., check.names = FALSE) :
row names were found from a short variable and have been discarded
This is probably nothing but there are were quite a few warnings that seem to be involved in the code below. Looking at the columns in question in the raw data in excel it is most like because of a mix of numbers and text in the columns.
Here is the warning:
> #Data cleaning
>
> surveyP2<-surveyP %>%
+ mutate(Numberoftotalbirths22=as.numeric(Numberoftotalbirths22),
+ NumberofallLBs23=as.numeric(NumberofallLBs23),
+ NumberofbabieswithGA=as.numeric(NumberofbabieswithGA),
+ Pretermlt37weeks=as.numeric(Pretermlt37weeks),
+ Preterm32tolt37weeks=as.numeric(Preterm32tolt37weeks),
+ Preterm28tolt32weeks=as.numeric(Preterm28tolt32weeks),
+ Pretermlt28weeks=as.numeric(Pretermlt28weeks),
+ MissingGA=as.numeric(MissingGA),
+
+ y=Preterm_adjusted_point_est/100,
+ se=y*(1-y),
+ Preterm_adjusted_se=se,
+ hannahExcluded=ifelse(is.na(MidyearHB), 1, 0),
+ Year=ifelse(!is.na(MidyearHB), MidyearHB, YearEB),
+ isoYear=paste0(IsoCode, Year)) %>%
+ rename(ISO=IsoCode) %>%
+ dplyr::select(-c(YearEB, MidyearHB,Excluded,
+ paperAvailable)) %>%
+ filter(Year>=2010 & Year<=2020) %>%
+ filter(is.na(DQexclusion))
Warning message:
There were 8 warnings in `mutate()`.
The first warning was:
ℹ In argument: `Numberoftotalbirths22 = as.numeric(Numberoftotalbirths22)`.
Caused by warning:
! NAs introduced by coercion
ℹ Run dplyr::last_dplyr_warnings() to see the 7 remaining warnings.
>
> #-----------------------
> studyAll<-surveyP2
> dplyr::last_dplyr_warnings()
[[1]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Numberoftotalbirths22 = as.numeric(Numberoftotalbirths22)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[2]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofallLBs23 = as.numeric(NumberofallLBs23)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[3]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofbabieswithGA = as.numeric(NumberofbabieswithGA)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[4]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt37weeks = as.numeric(Pretermlt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[5]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm32tolt37weeks = as.numeric(Preterm32tolt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
... with 3 more warnings.
ℹ Run dplyr::last_dplyr_warnings(n = 10) to show more.
> dplyr::last_dplyr_warnings(n = 10)
[[1]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Numberoftotalbirths22 = as.numeric(Numberoftotalbirths22)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[2]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofallLBs23 = as.numeric(NumberofallLBs23)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[3]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofbabieswithGA = as.numeric(NumberofbabieswithGA)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[4]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt37weeks = as.numeric(Pretermlt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[5]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm32tolt37weeks = as.numeric(Preterm32tolt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─... %>% filter(is.na(DQexclusion))
2. ├─dplyr::filter(., is.na(DQexclusion))
3. ├─dplyr::filter(., Year >= 2010 & Year <= 2020)
4. ├─dplyr::select(., -c(YearEB, MidyearHB, Excluded, paperAvailable))
5. ├─dplyr::rename(., ISO = IsoCode)
6. ├─dplyr::mutate(...)
7. └─dplyr:::mutate.data.frame(...)
[[6]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm28tolt32weeks = as.numeric(Preterm28tolt32weeks)`.
Caused by warning:
! NAs introduced by coercion
[[7]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt28weeks = as.numeric(Pretermlt28weeks)`.
Caused by warning:
! NAs introduced by coercion
[[8]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingGA = as.numeric(MissingGA)`.
Caused by warning:
! NAs introduced by coercion
This is probably nothing but there are were quite a few warnings that seem to be involved in the code below. Looking at the columns in question in the raw data in excel it is most like because of a mix of numbers and text in the columns.
Here is the warning:
> #Remove LBW and stillbriths columms from the joint admin database
> adminBefore1<-adminBefore1 %>%
+ mutate (Numberoftotalbirths22=ifelse(is.na(as.numeric(Numberoftotalbirths22)),
+ as.numeric(NumberofallLBs23) + as.numeric(Numberofstillbirths),Numberoftotalbirths22))
Warning message:
There were 3 warnings in `mutate()`.
The first warning was:
ℹ In argument: `Numberoftotalbirths22 = ifelse(...)`.
Caused by warning in `ifelse()`:
! NAs introduced by coercion
This is a new one. And honestly I have no idea what it means. Is this ok?
Here is the warning:
> # Meta-anaylsis for each of the sub-envelopes, using random effects.
> metaPlot<-metaprop(event=Pretermlt28weeks, n=Preterm,
+ studlab = OfficialName,
+ subgroup=regionName, data=finalData4,
+ random=TRUE,
+ overall = TRUE#, method="Inverse"
+ )
> metaPlot2<-metaprop(event=Preterm28tolt32weeks, n=Preterm,
+ studlab = OfficialName,
+ subgroup=regionName, data=finalData4,
+ random=TRUE,
+ overall = TRUE)
> metaPlot3<-metaprop(event=Preterm32tolt37weeks, n=Preterm,
+ studlab = OfficialName,
+ subgroup=regionName, data=finalData4,
+ random=TRUE,
+ overall = TRUE)
Warning message:
Fisher scoring algorithm may have gotten stuck at a local maximum.
Setting tau^2 = 0. Check the profile likelihood plot with profile().
There were some packages missing from this file that appeared in the headings of the other files.
In the end, I updated 0.loadPackages.R to include every package in all of the files, and commented out the other library statements in the code files and everything seems to run ok.
But there is a special point in that we seem to need to have to install JAGS library from http://www.sourceforge.net/projects/mcmc-jags/files .
Maybe we need to add that to the documentation somewhere?
This is probably nothing but there are were quite a few warnings that seem to be involved in the code below. Looking at the columns in question in the raw data in excel it is most like because of a mix of numbers and text in the columns.
Here is the warning:
> adminBefore2<-adminBefore1 %>% mutate_at(vars(names(adminBefore1)[3:ncol(adminBefore1)]), as.numeric)
Warning message:
There were 13 warnings in `mutate()`.
The first warning was:
ℹ In argument: `Year = .Primitive("as.double")(Year)`.
Caused by warning:
! NAs introduced by coercion
ℹ Run dplyr::last_dplyr_warnings() to see the 12 remaining warnings.
> View(adminBefore1)
> View(adminBefore1)
> dplyr::last_dplyr_warnings()
[[1]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Year = .Primitive("as.double")(Year)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[2]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingBW = .Primitive("as.double")(MissingBW)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[3]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofallLBs23 = .Primitive("as.double")(NumberofallLBs23)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[4]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofbabieswithGA = .Primitive("as.double")(NumberofbabieswithGA)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[5]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt37weeks = .Primitive("as.double")(Pretermlt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
... with 8 more warnings.
ℹ Run dplyr::last_dplyr_warnings(n = 10) to show more.
> dplyr::last_dplyr_warnings(n = 10)
[[1]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Year = .Primitive("as.double")(Year)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[2]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingBW = .Primitive("as.double")(MissingBW)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[3]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofallLBs23 = .Primitive("as.double")(NumberofallLBs23)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[4]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofbabieswithGA = .Primitive("as.double")(NumberofbabieswithGA)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[5]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt37weeks = .Primitive("as.double")(Pretermlt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[6]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm32tolt37weeks = .Primitive("as.double")(Preterm32tolt37weeks)`.
Caused by warning:
! NAs introduced by coercion
[[7]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm28tolt32weeks = .Primitive("as.double")(Preterm28tolt32weeks)`.
Caused by warning:
! NAs introduced by coercion
[[8]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt28weeks = .Primitive("as.double")(Pretermlt28weeks)`.
Caused by warning:
! NAs introduced by coercion
[[9]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt25weeks = .Primitive("as.double")(Pretermlt25weeks)`.
Caused by warning:
! NAs introduced by coercion
[[10]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingGA = .Primitive("as.double")(MissingGA)`.
Caused by warning:
! NAs introduced by coercion
... with 3 more warnings.
ℹ Run dplyr::last_dplyr_warnings(n = 20) to show more.
> dplyr::last_dplyr_warnings(n = 20)
[[1]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Year = .Primitive("as.double")(Year)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[2]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingBW = .Primitive("as.double")(MissingBW)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[3]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofallLBs23 = .Primitive("as.double")(NumberofallLBs23)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[4]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NumberofbabieswithGA = .Primitive("as.double")(NumberofbabieswithGA)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[5]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt37weeks = .Primitive("as.double")(Pretermlt37weeks)`.
Caused by warning:
! NAs introduced by coercion
---
Backtrace:
▆
1. ├─adminBefore1 %>% ...
2. └─dplyr::mutate_at(...)
3. ├─dplyr::mutate(.tbl, !!!funs)
4. └─dplyr:::mutate.data.frame(.tbl, !!!funs)
[[6]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm32tolt37weeks = .Primitive("as.double")(Preterm32tolt37weeks)`.
Caused by warning:
! NAs introduced by coercion
[[7]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Preterm28tolt32weeks = .Primitive("as.double")(Preterm28tolt32weeks)`.
Caused by warning:
! NAs introduced by coercion
[[8]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt28weeks = .Primitive("as.double")(Pretermlt28weeks)`.
Caused by warning:
! NAs introduced by coercion
[[9]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Pretermlt25weeks = .Primitive("as.double")(Pretermlt25weeks)`.
Caused by warning:
! NAs introduced by coercion
[[10]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `MissingGA = .Primitive("as.double")(MissingGA)`.
Caused by warning:
! NAs introduced by coercion
[[11]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Reportedpretermrate = .Primitive("as.double")(Reportedpretermrate)`.
Caused by warning:
! NAs introduced by coercion
[[12]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `NewbornwithBWandGA = .Primitive("as.double")(NewbornwithBWandGA)`.
Caused by warning:
! NAs introduced by coercion
[[13]]
<warning/rlang_warning>
Warning in `mutate()`:
ℹ In argument: `Reasonforexclusion = .Primitive("as.double")(Reasonforexclusion)`.
Caused by warning:
! NAs introduced by coercion
The 12 files are there, but how were they generated? They seem to an output of some code, but where is that code?
It appears as if this is fine as NA is set where not a number
Here is the warning:
#Tidying variables
surveyBefore<-surveyBefore %>% rename(
+ Year=`MidtopointforLBWestimate`,
+ iso=`ISOCode`,
+ Countryname=Country, sourceSubTypeOther=Survey) %>%
+ dplyr::select(iso,Countryname,Year, LBW_adjusted_point_est,
+ LBW_adjusted_se, sourceSubTypeOther, Crude_LBW_point_est, Crude_LBW_se) %>%
+ mutate(Year=as.numeric(Year)) %>%
+ mutate(LBW_adjusted_point_est=LBW_adjusted_point_est*100,
+ Crude_LBW_point_est=as.numeric(Crude_LBW_point_est)*100,
+ LBW_adjusted_se=as.numeric(LBW_adjusted_se),
+ Crude_LBW_se=as.numeric(Crude_LBW_se)) %>%
+ mutate(Source="Survey", sourceSubType=ifelse(sourceSubTypeOther=="DHS", 5,
+ ifelse(sourceSubTypeOther=="MICS", 6,
+ 7))) %>%
+ mutate(isoYear=paste0(iso, Year))
Warning message:
There were 3 warnings in `mutate()`.
The first warning was:
ℹ In argument: `Crude_LBW_point_est = as.numeric(Crude_LBW_point_est) * 100`.
Caused by warning:
! NAs introduced by coercion
..\mca-pretermbirths-analysis\Preterm Bayesian Modelling Code\output\ already has files in the output folder before code is run when it feels like output folder should be empty. Also, they look like inputs rather than outputs based on code.
Can/should it be moved to inputs folder?
The files are:
Referenced in:
3.fitModelsP_singMultDQC.R
4.countryR&GratesandNumbers.R
6.subgroupsMetaP.R
functions\runModelValidationP.R
wpp2 already exists as a dataframe. and “../output/wpp2.rds” doesn’t exist.
Also in first mention wpp2.rds is filtered down to 2020 only.
To get code to run, and to not overwrite existing wpp2, I changed the offending line to use existing wpp2 as source and renamed output to wpp2020.
I then updated all subsequent references from wpp2 to wpp2020 as well.
One exception is lin3 #737 “wpp2<-readRDS("output/wpp2.RDS")” which I commented out because there are no further references to wpp2 in the file.
Here is the first error:
wpp2<-readRDS("output/wpp2.rds") %>%
+ filter(year==2020)
Error in gzfile(file, "rb") : cannot open the connection
In addition: Warning message:
In gzfile(file, "rb") :
cannot open compressed file 'output/wpp2.rds', probable reason 'No such file or directory'
Unfortunately it seems like the file "5. Plot countries P (1 model) 1 plot.R" is missing from the repo.
It is in 0.masterP.R. I have tried running the remaining files without having run "5. Plot countries P (1 model) 1 plot.R" first and there are several problems as documented in further issues.
Error:
> source("5. Plot countries P (1 model) 1 plot.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '5. Plot countries P (1 model) 1 plot.R': No such file or directory
I received the error below and corrected it by adding a folder called "models" in output folder.
Here is the error:
4: There was 1 warning in `mutate()`.
ℹ In argument: `literacy_femT = ifelse(literacy_fem > 100, log(100 - 99.99999), log(100 - literacy_fem))`.
Caused by warning in `log()`:
! NaNs produced
5: In gzfile(file, mode) :
cannot open compressed file 'output/models/pCCFullModel800.RDS', probable reason 'No such file or directory'
I received the error below and corrected it by adding a folder called "models" in output folder.
Here is the error:
9: In gzfile(file, mode) :
cannot open compressed file 'output/models/pCCFullModelP_dataType1DQC1_TEST16000.RDS', probable reason 'No such file or directory'
Called from: gzfile(file, mode)
Browse[1]>
LOPEZ-Gerard commented 4 hours ago
It seems like there is a problem in column V which was "Abstraction date for live birth LBW". It contains dates but clearly the R import from excel is not working properly as it is not picking them up as proper dates but rather excel dates which come in as numbers like this "44357".
Not clear if this is a real problem or not going forward.
Here is the warning:
> #Missing source sub type- replacing with the LBW one from the same database
> lbw<-readRDS("inputs/LBWfinalData.rds") %>% filter(Source=="Admin") %>%
+ dplyr::select(isoYear, sourceSubType)
>
> adminCodes=readxl::read_excel(paste0(adminName),
+ sheet = "VN Source database", skip=1)
There were 50 or more warnings (use warnings() to see the first 50)
> adminCodes<-tidyVariableNames(adminCodes) %>% mutate(isoYear=paste0(IsoCode, MidYear))
> warnings()
Warning messages:
1: Expecting numeric in V3 / R3C22: got a date
2: Expecting numeric in V4 / R4C22: got a date
3: Expecting numeric in V5 / R5C22: got a date
4: Expecting numeric in V6 / R6C22: got a date
5: Expecting numeric in V7 / R7C22: got a date
6: Expecting numeric in V8 / R8C22: got a date
7: Expecting numeric in V9 / R9C22: got a date
8: Expecting numeric in V10 / R10C22: got a date
9: Expecting numeric in V11 / R11C22: got a date
10: Expecting numeric in V12 / R12C22: got a date
11: Expecting numeric in V13 / R13C22: got a date
12: Expecting numeric in V14 / R14C22: got a date
13: Expecting numeric in V15 / R15C22: got a date
14: Expecting numeric in V16 / R16C22: got a date
15: Expecting numeric in V17 / R17C22: got a date
16: Expecting numeric in V18 / R18C22: got a date
17: Expecting numeric in V19 / R19C22: got a date
18: Expecting numeric in V20 / R20C22: got a date
19: Expecting numeric in V21 / R21C22: got a date
20: Expecting numeric in V22 / R22C22: got a date
21: Expecting numeric in V23 / R23C22: got a date
22: Expecting numeric in V24 / R24C22: got a date
23: Expecting numeric in V25 / R25C22: got a date
24: Expecting numeric in V26 / R26C22: got a date
25: Expecting numeric in V27 / R27C22: got a date
26: Expecting numeric in V28 / R28C22: got a date
27: Expecting numeric in V29 / R29C22: got a date
28: Expecting numeric in V30 / R30C22: got a date
29: Expecting numeric in V31 / R31C22: got a date
30: Expecting numeric in V32 / R32C22: got a date
31: Expecting numeric in V33 / R33C22: got a date
32: Expecting numeric in V34 / R34C22: got a date
33: Expecting numeric in V35 / R35C22: got a date
34: Expecting numeric in V36 / R36C22: got a date
35: Expecting numeric in V37 / R37C22: got a date
36: Expecting numeric in V38 / R38C22: got a date
37: Expecting numeric in V39 / R39C22: got a date
38: Expecting numeric in V40 / R40C22: got a date
39: Expecting numeric in V41 / R41C22: got a date
40: Expecting numeric in V42 / R42C22: got a date
41: Expecting numeric in V43 / R43C22: got a date
42: Expecting numeric in V44 / R44C22: got a date
43: Expecting numeric in V45 / R45C22: got a date
44: Expecting numeric in V46 / R46C22: got a date
45: Expecting numeric in V47 / R47C22: got a date
46: Expecting numeric in V48 / R48C22: got a date
47: Expecting numeric in V49 / R49C22: got a date
48: Expecting numeric in V50 / R50C22: got a date
49: Expecting numeric in V51 / R51C22: got a date
50: Expecting numeric in V52 / R52C22: got a date
Missing these R files?:
These files unused?:
It appears as if the ISO - 3 and ISO - 2 codes are not being picked up, most likely because the first rows are "Global" so don't have codes so R thinks those columns are boolean / logical data types and doesn't like it when the text for the ISO codes appears in the rows with countries.
Not a big deal I think because obviously the codes get picked up later in the code/input data.
Here are the warnings:
#WPP Livebirths estimates
wpp_2021 <- readxl::read_xlsx(paste0("input/", wpp2))
There were 50 or more warnings (use warnings() to see the first 50)
warnings()
Warning messages:
1: Expecting logical in E1730 / R1730C5: got 'BDI'
2: Expecting logical in F1730 / R1730C6: got 'BI'
3: Expecting logical in E1731 / R1731C5: got 'BDI'
4: Expecting logical in F1731 / R1731C6: got 'BI'
5: Expecting logical in E1732 / R1732C5: got 'BDI'
6: Expecting logical in F1732 / R1732C6: got 'BI'
7: Expecting logical in E1733 / R1733C5: got 'BDI'
8: Expecting logical in F1733 / R1733C6: got 'BI'
9: Expecting logical in E1734 / R1734C5: got 'BDI'
10: Expecting logical in F1734 / R1734C6: got 'BI'
11: Expecting logical in E1735 / R1735C5: got 'BDI'
12: Expecting logical in F1735 / R1735C6: got 'BI'
13: Expecting logical in E1736 / R1736C5: got 'BDI'
14: Expecting logical in F1736 / R1736C6: got 'BI'
15: Expecting logical in E1737 / R1737C5: got 'BDI'
16: Expecting logical in F1737 / R1737C6: got 'BI'
17: Expecting logical in E1738 / R1738C5: got 'BDI'
18: Expecting logical in F1738 / R1738C6: got 'BI'
19: Expecting logical in E1739 / R1739C5: got 'BDI'
20: Expecting logical in F1739 / R1739C6: got 'BI'
21: Expecting logical in E1740 / R1740C5: got 'BDI'
22: Expecting logical in F1740 / R1740C6: got 'BI'
23: Expecting logical in E1741 / R1741C5: got 'BDI'
24: Expecting logical in F1741 / R1741C6: got 'BI'
25: Expecting logical in E1742 / R1742C5: got 'BDI'
26: Expecting logical in F1742 / R1742C6: got 'BI'
27: Expecting logical in E1743 / R1743C5: got 'BDI'
28: Expecting logical in F1743 / R1743C6: got 'BI'
29: Expecting logical in E1744 / R1744C5: got 'BDI'
30: Expecting logical in F1744 / R1744C6: got 'BI'
31: Expecting logical in E1745 / R1745C5: got 'BDI'
32: Expecting logical in F1745 / R1745C6: got 'BI'
33: Expecting logical in E1746 / R1746C5: got 'BDI'
34: Expecting logical in F1746 / R1746C6: got 'BI'
35: Expecting logical in E1747 / R1747C5: got 'BDI'
36: Expecting logical in F1747 / R1747C6: got 'BI'
37: Expecting logical in E1748 / R1748C5: got 'BDI'
38: Expecting logical in F1748 / R1748C6: got 'BI'
39: Expecting logical in E1749 / R1749C5: got 'BDI'
40: Expecting logical in F1749 / R1749C6: got 'BI'
41: Expecting logical in E1750 / R1750C5: got 'BDI'
42: Expecting logical in F1750 / R1750C6: got 'BI'
43: Expecting logical in E1751 / R1751C5: got 'BDI'
44: Expecting logical in F1751 / R1751C6: got 'BI'
45: Expecting logical in E1752 / R1752C5: got 'BDI'
46: Expecting logical in F1752 / R1752C6: got 'BI'
47: Expecting logical in E1753 / R1753C5: got 'BDI'
48: Expecting logical in F1753 / R1753C6: got 'BI'
49: Expecting logical in E1754 / R1754C5: got 'BDI'
50: Expecting logical in F1754 / R1754C6: got 'BI'
One is that I can forsee is that similar to the files in the "Preterm Regional Groupings" and "LBW Regional Groupings" folders, there are multiple "Final Database" files in the main folder, but how were they generated? They seem to be output of some code, but where is that code?
The files in question:
..\mca-pretermbirths-analysis\LBWcovariates.csv
..\mca-pretermbirths-analysis\LBWcovariates.rds
..\mca-pretermbirths-analysis\LBWfinalDatabase.csv
..\mca-pretermbirths-analysis\LBWfinalDatabase.rds
..\mca-pretermbirths-analysis\PretermfinalDatabase.csv
..\mca-pretermbirths-analysis\PretermfinalDatabase.rds
Error:
pathdoes not exist: ‘C:/Users/EllenBradley/OneDrive – London School of Hygiene and Tropical Medicine/Documents/LBW SURVEY DATA/2022 12 01 LBW estimates combined database.xlsx’
I changed to surveyName<-“input/2022 12 01 LBW estimates combined database.xlsx”
When running this code I received the warning down below.
I think that it is fine as what is happening is that the excel sheet has a column where only 14 rows have comments in them. Because the first xxx number of rows are null, R is expecting a boolean logical data type so it doesn't like that on those 14 rows (which are thousands deep in the file I think) that there is text instead. But I don't think that this column is used anywhere.
One small suggestion though is that the excel file is a little off as the filter does not include the column where “'EB Updated 13/12…” text appears, so if someone sorts then the comment will be associated with the wrong row. So that excel file should be updated I think.
Here is the warning:
newAdmin<-readxl::read_excel(paste0(newAdminName),
+ sheet = "data", skip=1)
New names:
• `` -> `...22`
• `` -> `...35`
There were 14 warnings (use warnings() to see them)
warnings()
Warning messages:
1: Expecting logical in AI2103 / R2103C35: got 'EB Updated 13/12 so same as what went to CC'
2: Expecting logical in AI2104 / R2104C35: got 'EB Updated 13/12 so same as what went to CC'
3: Expecting logical in AI2105 / R2105C35: got 'EB Updated 13/12 so same as what went to CC'
4: Expecting logical in AI2106 / R2106C35: got 'EB Updated 13/12 so same as what went to CC'
5: Expecting logical in AI2107 / R2107C35: got 'EB Updated 13/12 so same as what went to CC'
6: Expecting logical in AI2108 / R2108C35: got 'EB Updated 13/12 so same as what went to CC'
7: Expecting logical in AI2109 / R2109C35: got 'EB Updated 13/12 so same as what went to CC'
8: Expecting logical in AI2110 / R2110C35: got 'EB Updated 13/12 so same as what went to CC'
9: Expecting logical in AI2111 / R2111C35: got 'EB Updated 13/12 so same as what went to CC'
10: Expecting logical in AI2112 / R2112C35: got 'EB Updated 13/12 so same as what went to CC'
11: Expecting logical in AI2113 / R2113C35: got 'EB Updated 13/12 so same as what went to CC'
12: Expecting logical in AI2114 / R2114C35: got 'EB Updated 13/12 so same as what went to CC'
13: Expecting logical in AI2115 / R2115C35: got 'EB Updated 13/12 so same as what went to CC'
14: Expecting logical in AI2116 / R2116C35: got 'EB Updated 13/12 so same as what went to CC'
There are two calls to source in this file to files that have already successfully run. Is this needed? "0.fileNames.R" and "1.cleanSurveyP2.R" are basic files that are setting all the subsequent files up, so it doesn't seem necessary to run them again at this point.
Here are the lines:
source("0.fileNames.R")
source("1.cleanSurveyP2.R")
It seems like there is a problem in column V which was "Abstraction date for live birth LBW". It contains dates but clearly the R import from excel is not working properly as it is not picking them up as proper dates but rather excel dates which come in as numbers like this "44357".
Not clear if this is a real problem or not going forward.
Here is the warning:
> finalPreterm1<-readRDS("output/finalPreterm.RDS")
>
> #-----------------------------
> #Missing source sub type- replacing with the LBW one from the same database
> lbw<-readRDS("inputs/LBWfinalData.rds") %>% filter(Source=="Admin") %>%
+ dplyr::select(isoYear, sourceSubType)
>
> adminCodes=readxl::read_excel(paste0(adminName),
+ sheet = "VN Source database", skip=1)
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: Expecting numeric in V3 / R3C22: got a date
2: Expecting numeric in V4 / R4C22: got a date
3: Expecting numeric in V5 / R5C22: got a date
4: Expecting numeric in V6 / R6C22: got a date
5: Expecting numeric in V7 / R7C22: got a date
6: Expecting numeric in V8 / R8C22: got a date
7: Expecting numeric in V9 / R9C22: got a date
8: Expecting numeric in V10 / R10C22: got a date
9: Expecting numeric in V11 / R11C22: got a date
10: Expecting numeric in V12 / R12C22: got a date
11: Expecting numeric in V13 / R13C22: got a date
12: Expecting numeric in V14 / R14C22: got a date
13: Expecting numeric in V15 / R15C22: got a date
14: Expecting numeric in V16 / R16C22: got a date
15: Expecting numeric in V17 / R17C22: got a date
16: Expecting numeric in V18 / R18C22: got a date
17: Expecting numeric in V19 / R19C22: got a date
18: Expecting numeric in V20 / R20C22: got a date
19: Expecting numeric in V21 / R21C22: got a date
20: Expecting numeric in V22 / R22C22: got a date
21: Expecting numeric in V23 / R23C22: got a date
22: Expecting numeric in V24 / R24C22: got a date
23: Expecting numeric in V25 / R25C22: got a date
24: Expecting numeric in V26 / R26C22: got a date
25: Expecting numeric in V27 / R27C22: got a date
26: Expecting numeric in V28 / R28C22: got a date
27: Expecting numeric in V29 / R29C22: got a date
28: Expecting numeric in V30 / R30C22: got a date
29: Expecting numeric in V31 / R31C22: got a date
30: Expecting numeric in V32 / R32C22: got a date
31: Expecting numeric in V33 / R33C22: got a date
32: Expecting numeric in V34 / R34C22: got a date
33: Expecting numeric in V35 / R35C22: got a date
34: Expecting numeric in V36 / R36C22: got a date
35: Expecting numeric in V37 / R37C22: got a date
36: Expecting numeric in V38 / R38C22: got a date
37: Expecting numeric in V39 / R39C22: got a date
38: Expecting numeric in V40 / R40C22: got a date
39: Expecting numeric in V41 / R41C22: got a date
40: Expecting numeric in V42 / R42C22: got a date
41: Expecting numeric in V43 / R43C22: got a date
42: Expecting numeric in V44 / R44C22: got a date
43: Expecting numeric in V45 / R45C22: got a date
44: Expecting numeric in V46 / R46C22: got a date
45: Expecting numeric in V47 / R47C22: got a date
46: Expecting numeric in V48 / R48C22: got a date
47: Expecting numeric in V49 / R49C22: got a date
48: Expecting numeric in V50 / R50C22: got a date
49: Expecting numeric in V51 / R51C22: got a date
50: Expecting numeric in V52 / R52C22: got a date
There were errors because the code was trying to remove data4, but data4 does not seem to exist and is not referenced anywhere I could find it.
I commented out that code so I could proceed.
Here is the error:
Error in `mutate()`:
ℹ In argument: `numbers = round(wpp_lb * inv.logit(mu), 0)`.
Caused by error:
! object 'wpp_lb' not found
Run `rlang::last_trace()` to see where the error occurred.
Warning messages:
1: In rm(data4) : object 'data4' not found
2: In rm(data4) : object 'data4' not found
3: In rm(data4) : object 'data4' not found
4: In rm(data4) : object 'data4' not found
5: In rm(data4) : object 'data4' not found
6: In rm(data4) : object 'data4' not found
..\mca-pretermbirths-analysis\LBW Bayesian Modelling Code\output\regionCodesOther.RDS is already in the output folder before code is run when it feels like output folder should be empty. Also, it looks like an input rather than an output based on code.
Can/should it be moved to inputs folder?
Referenced in:
It appears as if the ISO - 3 and ISO - 2 codes are not being picked up, most likely because the first rows are "Global" so don't have codes so R thinks those columns are boolean / logical data types and doesn't like it when the text for the ISO codes appears in the rows with countries.
Not a big deal I think because obviously the codes get picked up later in the code/input data.
Here are the warnings:
#wpp_2021 <- read_dta(paste0("input/", wpp))
wpp_2021 <- readxl::read_xlsx(paste0("input/", wpp2))
There were 50 or more warnings (use warnings() to see the first 50)
warnings()
Warning messages:
1: Expecting logical in E1730 / R1730C5: got 'BDI'
2: Expecting logical in F1730 / R1730C6: got 'BI'
3: Expecting logical in E1731 / R1731C5: got 'BDI'
4: Expecting logical in F1731 / R1731C6: got 'BI'
5: Expecting logical in E1732 / R1732C5: got 'BDI'
6: Expecting logical in F1732 / R1732C6: got 'BI'
7: Expecting logical in E1733 / R1733C5: got 'BDI'
8: Expecting logical in F1733 / R1733C6: got 'BI'
9: Expecting logical in E1734 / R1734C5: got 'BDI'
10: Expecting logical in F1734 / R1734C6: got 'BI'
11: Expecting logical in E1735 / R1735C5: got 'BDI'
12: Expecting logical in F1735 / R1735C6: got 'BI'
13: Expecting logical in E1736 / R1736C5: got 'BDI'
14: Expecting logical in F1736 / R1736C6: got 'BI'
15: Expecting logical in E1737 / R1737C5: got 'BDI'
16: Expecting logical in F1737 / R1737C6: got 'BI'
17: Expecting logical in E1738 / R1738C5: got 'BDI'
18: Expecting logical in F1738 / R1738C6: got 'BI'
19: Expecting logical in E1739 / R1739C5: got 'BDI'
20: Expecting logical in F1739 / R1739C6: got 'BI'
21: Expecting logical in E1740 / R1740C5: got 'BDI'
22: Expecting logical in F1740 / R1740C6: got 'BI'
23: Expecting logical in E1741 / R1741C5: got 'BDI'
24: Expecting logical in F1741 / R1741C6: got 'BI'
25: Expecting logical in E1742 / R1742C5: got 'BDI'
26: Expecting logical in F1742 / R1742C6: got 'BI'
27: Expecting logical in E1743 / R1743C5: got 'BDI'
28: Expecting logical in F1743 / R1743C6: got 'BI'
29: Expecting logical in E1744 / R1744C5: got 'BDI'
30: Expecting logical in F1744 / R1744C6: got 'BI'
31: Expecting logical in E1745 / R1745C5: got 'BDI'
32: Expecting logical in F1745 / R1745C6: got 'BI'
33: Expecting logical in E1746 / R1746C5: got 'BDI'
34: Expecting logical in F1746 / R1746C6: got 'BI'
35: Expecting logical in E1747 / R1747C5: got 'BDI'
36: Expecting logical in F1747 / R1747C6: got 'BI'
37: Expecting logical in E1748 / R1748C5: got 'BDI'
38: Expecting logical in F1748 / R1748C6: got 'BI'
39: Expecting logical in E1749 / R1749C5: got 'BDI'
40: Expecting logical in F1749 / R1749C6: got 'BI'
41: Expecting logical in E1750 / R1750C5: got 'BDI'
42: Expecting logical in F1750 / R1750C6: got 'BI'
43: Expecting logical in E1751 / R1751C5: got 'BDI'
44: Expecting logical in F1751 / R1751C6: got 'BI'
45: Expecting logical in E1752 / R1752C5: got 'BDI'
46: Expecting logical in F1752 / R1752C6: got 'BI'
47: Expecting logical in E1753 / R1753C5: got 'BDI'
48: Expecting logical in F1753 / R1753C6: got 'BI'
49: Expecting logical in E1754 / R1754C5: got 'BDI'
50: Expecting logical in F1754 / R1754C6: got 'BI'
There are multiple names in the 0.masterP.R file that do not match the files that are present.
I had thought to change the file names in 0.masterP.R file to match the folder contents, but it turns out that all of the source commands don't match with the exception of 0.fileNames.R and 0.loadPackages.R. So perhaps the 0.masterP.R file is out of date or incorrect in some other way?
Here are the errors:
> source("1.cleanAdminP2.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.cleanAdminP2.R': No such file or directory
> source("1.cleanSurveyP2.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.cleanSurveyP2.R': No such file or directory
> source("1.combineAdminandSurvey2.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.combineAdminandSurvey2.R': No such file or directory
> source("1.inclusionExclusion5.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.inclusionExclusion5.R': No such file or directory
> source("1.dqc4.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.dqc4.R': No such file or directory
> source("1.groupingMGA3.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '1.groupingMGA3.R': No such file or directory
> source("3.fitModelsP_singMultDQC.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '3.fitModelsP_singMultDQC.R': No such file or directory
> source("4.countryR&GratesandNumbers.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '4.countryR&GratesandNumbers.R': No such file or directory
> source("5. Plot countries P (1 model) 1 plot.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '5. Plot countries P (1 model) 1 plot.R': No such file or directory
> source("6.subgroupsMetaP.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '6.subgroupsMetaP.R': No such file or directory
> source("7. splineModelsValidationP.R")
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file '7. splineModelsValidationP.R': No such file or directory
I can’t see a problem in input data nor output…Not sure why NaNs are produced or if it has any impact downstream.
Here is the warning:
transform<-covarset %>% mutate(gniT=log(gni),
+ literacy_femT=ifelse(literacy_fem>100, log(100-99.99999), log(100-literacy_fem)),
+ contraception_metT=log(1-contraception_met),
+ bmi_f_underT=log(bmi_f_under),
+ urbanT=urban
+ #,nmrT=nmr-mean(covarset$nmr, na.rm=TRUE)
+ #underweightT=underweight
+ )
Warning message:
There was 1 warning in `mutate()`.
ℹ In argument: `literacy_femT = ifelse(literacy_fem > 100, log(100 - 99.99999), log(100 -
literacy_fem))`.
Caused by warning in `log()`:
! NaNs produced
I finished the main testing run for today/yesterday. And here are my final comments about it.
Main concern is that what seems like the final output produced doesn’t match the naming of the final output that I have seen. Namely that they have the prefix of pCCFullModel_800 rather than pCCFullModel_16000. [IGNORE THIS COMMENT IT COMES FROM LBW]
The output below was produced with the changes described in the various "Preterm Bayesian Modelling Code " issues and installation of JAGS-4.3.1.exe from http://www.sourceforge.net/projects/mcmc-jags/files. You will have to say whether this is what was expected or not from these inputs and code file.
The most obvious problem I have is that one of the source files that is referenced in 0.masterP.R, "5. Plot countries P (1 model) 1 plot.R", is not present in the Repo, so it is not clear how that impacts the final output.
I also only ran 7. splineModelsValidationP.R with nreps=1 so that I could finish review. So it is not fully tested.
FINAL OUTPUT FILES:
..\Preterm Bayesian Modelling Code\output\models\pCCFullModelP_dataType1DQC1_TEST_modelVal16000.RDS
..\Preterm Bayesian Modelling Code\output\models\pCCFullModelP_dataType1DQC1_TEST16000.RDS
..\Preterm Bayesian Modelling Code\output\DQC4_calculations.csv
..\Preterm Bayesian Modelling Code\output\DQC4_countries.csv
..\Preterm Bayesian Modelling Code\output\finalPreterm.RDS
..\Preterm Bayesian Modelling Code\output\finalPreterm_MGA.rds
..\Preterm Bayesian Modelling Code\output\methodGACountries.csv
..\Preterm Bayesian Modelling Code\output\missingSourceSubType.csv
..\Preterm Bayesian Modelling Code\output\modelVal.RDS
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_Estimates.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_Figure3data.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_Figure4editable.pptx
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_Figure4table-reorder.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_modelSummary.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_modelValidation1.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_modelValidation1.rds
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_pretermSubgroupForest.pdf
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_regionalAndGlobalEstimates.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_Table 10 - Preterm Meta Subgroup (applying total to all regions).csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_table2.csv
..\Preterm Bayesian Modelling Code\output\pCCFullModelP_dataType1DQC1_TEST_16000_tracePlots.pdf
..\Preterm Bayesian Modelling Code\output\Preterm subgroups.rds
..\Preterm Bayesian Modelling Code\output\pretermPreIncExc.rds
It seems like there is a problem in column V which was "Abstraction date for live birth LBW". It contains dates but clearly the R import from excel is not working properly as it is not picking them up as proper dates but rather excel dates which come in as numbers like this "44357".
Not clear if this is a real problem or not going forward.
Here is the warning:
> adminCodes=readxl::read_excel(paste0(adminName),
+ sheet = "VN Source database", skip=1)
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: Expecting numeric in V3 / R3C22: got a date
2: Expecting numeric in V4 / R4C22: got a date
3: Expecting numeric in V5 / R5C22: got a date
4: Expecting numeric in V6 / R6C22: got a date
5: Expecting numeric in V7 / R7C22: got a date
6: Expecting numeric in V8 / R8C22: got a date
7: Expecting numeric in V9 / R9C22: got a date
8: Expecting numeric in V10 / R10C22: got a date
9: Expecting numeric in V11 / R11C22: got a date
I can’t see a problem in input data nor output…Not sure why NaNs are produced or if it has any impact downstream.
Here is the warning:
source("3.fitModels.R")
Skipping install of 'distortr' from a github remote, the SHA1 (32474656) has not changed since last install.
Use `force = TRUE` to force installation
Running model.
Total iterations: 1200, Burn in: 1000, Thinning: 1
Something hasn't converged HAS IT.
Max Rhat is 17.8134681157874
saving to RDS.
returning the model
finished modelling!Warning message:
There was 1 warning in `mutate()`.
ℹ In argument: `literacy_femT = ifelse(literacy_fem > 100, log(100 - 99.99999), log(100 - literacy_fem))`.
Caused by warning in `log()`:
! NaNs produced
It appears as if the ISO - 3 and ISO - 2 codes are not being picked up, most likely because the first rows are "Global" so don't have codes so R thinks those columns are boolean / logical data types and doesn't like it when the text for the ISO codes appears in the rows with countries.
Not a big deal I think because obviously the codes get picked up later in the code/input data.
Here are the warnings:
> #INCLUSION AND EXCLUSION CRITERIA
>
> wpp_2021 <- readxl::read_xlsx(paste0("inputs/", wpp2))
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: Expecting logical in E1730 / R1730C5: got 'BDI'
2: Expecting logical in F1730 / R1730C6: got 'BI'
3: Expecting logical in E1731 / R1731C5: got 'BDI'
4: Expecting logical in F1731 / R1731C6: got 'BI'
5: Expecting logical in E1732 / R1732C5: got 'BDI'
6: Expecting logical in F1732 / R1732C6: got 'BI'
7: Expecting logical in E1733 / R1733C5: got 'BDI'
8: Expecting logical in F1733 / R1733C6: got 'BI'
9: Expecting logical in E1734 / R1734C5: got 'BDI'
10: Expecting logical in F1734 / R1734C6: got 'BI'
11: Expecting logical in E1735 / R1735C5: got 'BDI'
12: Expecting logical in F1735 / R1735C6: got 'BI'
13: Expecting logical in E1736 / R1736C5: got 'BDI'
14: Expecting logical in F1736 / R1736C6: got 'BI'
15: Expecting logical in E1737 / R1737C5: got 'BDI'
16: Expecting logical in F1737 / R1737C6: got 'BI'
17: Expecting logical in E1738 / R1738C5: got 'BDI'
18: Expecting logical in F1738 / R1738C6: got 'BI'
19: Expecting logical in E1739 / R1739C5: got 'BDI'
20: Expecting logical in F1739 / R1739C6: got 'BI'
21: Expecting logical in E1740 / R1740C5: got 'BDI'
22: Expecting logical in F1740 / R1740C6: got 'BI'
23: Expecting logical in E1741 / R1741C5: got 'BDI'
24: Expecting logical in F1741 / R1741C6: got 'BI'
25: Expecting logical in E1742 / R1742C5: got 'BDI'
26: Expecting logical in F1742 / R1742C6: got 'BI'
27: Expecting logical in E1743 / R1743C5: got 'BDI'
28: Expecting logical in F1743 / R1743C6: got 'BI'
29: Expecting logical in E1744 / R1744C5: got 'BDI'
30: Expecting logical in F1744 / R1744C6: got 'BI'
31: Expecting logical in E1745 / R1745C5: got 'BDI'
32: Expecting logical in F1745 / R1745C6: got 'BI'
33: Expecting logical in E1746 / R1746C5: got 'BDI'
34: Expecting logical in F1746 / R1746C6: got 'BI'
35: Expecting logical in E1747 / R1747C5: got 'BDI'
36: Expecting logical in F1747 / R1747C6: got 'BI'
37: Expecting logical in E1748 / R1748C5: got 'BDI'
38: Expecting logical in F1748 / R1748C6: got 'BI'
39: Expecting logical in E1749 / R1749C5: got 'BDI'
40: Expecting logical in F1749 / R1749C6: got 'BI'
41: Expecting logical in E1750 / R1750C5: got 'BDI'
42: Expecting logical in F1750 / R1750C6: got 'BI'
43: Expecting logical in E1751 / R1751C5: got 'BDI'
44: Expecting logical in F1751 / R1751C6: got 'BI'
45: Expecting logical in E1752 / R1752C5: got 'BDI'
46: Expecting logical in F1752 / R1752C6: got 'BI'
47: Expecting logical in E1753 / R1753C5: got 'BDI'
48: Expecting logical in F1753 / R1753C6: got 'BI'
49: Expecting logical in E1754 / R1754C5: got 'BDI'
50: Expecting logical in F1754 / R1754C6: got 'BI'
Not sure if this ran correctly. It failed the first time I did it, but think it had to do with my computer entering sleep mode.
So I changed all of those settings and let it run. It literally took all night but there is a resulting dataset and a report, pCCFullModel_800_modelValidation1000.csv.
There is a small warning about "covar_array", but successfully completed...I think.
Here is the last message:
running model for iteration 999Running model.
Total iterations: 1200, Burn in: 1000, Thinning: 1
Something hasn't converged HAS IT.
Max Rhat is 20.1721907892682
saving to RDS.
returning the model
finished modelling!predicting for model for iteration 999doing calculations for iteration 999finished calculations for iteration 999running model for iteration 1000Running model.
Total iterations: 1200, Burn in: 1000, Thinning: 1
Something hasn't converged HAS IT.
Max Rhat is 19.6129783671547
saving to RDS.
returning the model
finished modelling!predicting for model for iteration 1000doing calculations for iteration 1000finished calculations for iteration 1000Warning messages:
1: There was 1 warning in `mutate()`.
? In argument: `literacy_femT = ifelse(literacy_fem > 100, log(100 - 99.99999), log(100 -
literacy_fem))`.
Caused by warning in `log()`:
! NaNs produced
2: In rm(covar_array) : object 'covar_array' not found
>
> #Outputs comparison statistics between the full and left out model.
> table<-as.data.frame(list(reps=nreps,
+ RMSE=mean(modelValOutput$random$rmse),
+ MeanError=mean(modelValOutput$random$predE),
+ MeanAbsError=mean(modelValOutput$random$predEAbs),
+ desirableWithinPredCI=mean(modelValOutput$random$desinCI),
+ desirableAboveCI=mean(modelValOutput$random$desaboveCI),
+ desirableBelowCI=mean(modelValOutput$random$desbelowCI),
+ loWithinPredCI=mean(modelValOutput$random$inCI),
+ loAboveCI=mean(modelValOutput$random$aboveCI),
+ loBelowCI=mean(modelValOutput$random$belowCI),
+ predIn95=mean(modelValOutput$random$medInFCI),
+ predIn80=mean(modelValOutput$random$medInFCI80),
+ predIn50=mean(modelValOutput$random$medInFCI50),
+ DIC=mean(modelValOutput$random$dic)))
>
> write.csv(table,
+ paste0("output/" ,fileName,"_",
+ ((niter-nburnin)/nthin)*nchains,"_modelValidation", nreps, ".csv"))
>
> saveRDS(modelValOutput, paste0("output/" ,fileName,"_",
+ ((niter-nburnin)/nthin)*nchains,"_modelValidation", nreps, ".rds"))
>
I finished the main testing run for today/yesterday. And here are my final comments about it.
Main concern is that what seems like the final output produced doesn’t match the naming of the final output that I have seen. Namely that they have the prefix of pCCFullModel_800 rather than pCCFullModel_16000.
The output below was produced with the changes described in the various issues and installation of JAGS-4.3.1.exe from http://www.sourceforge.net/projects/mcmc-jags/files. You will have to say whether this is what was expected or not from these inputs and code file.
FINAL OUTPUT FILES:
..\output\adminWithoutBirthEstFB80HMIS.csv
..\output\allDataInc.RDS
..\output\allMetaEsts.csv
..\output\dqc5_CountryFlag.RDS
..\output\DQC5criteria_calculations.csv
..\output\DQC5Update_countries.csv
..\output\Figure4table-heightOrder.csv
..\output\finalData.RDS
..\output\latestMetaEsts.csv
..\output\LBW subgroup.pdf
..\output\modelVal.RDS
..\output\pCCFullModel_800_Coefficients.csv
..\output\pCCFullModel_800_Estimates.csv
..\output\pCCFullModel_800_modelSummary.csv
..\output\pCCFullModel_800_modelValidation1000.csv
..\output\pCCFullModel_800_modelValidation1000.RDS
..\output\pCCFullModel_800_regionalAndGlobalEstimates.csv
..\output\pCCFullModel_800_table2.csv
..\output\pCCFullModel_800_tracePlots.pdf
..\output\regionCodes.RDS
..\output\Table 10 - Meta Subgroup (applying total to LBW rates).csv
..\output\Table 10 - Meta Subgroup (NA applied to all).csv
..\output\models\pCCFullModel_modelVal800.RDS
..\output\models\pCCFullModel800.RDS
This concerns Line #541 mutate(numbers=round(wpp_lb*inv.logit(mu),0)) %>%
When I first saw this issue there wasn’t a “wpp_lb” but rather wpp_lb.x and wpp_lb.y . They look to be the same, but not clear.
So I tried to use wpp_lb.y because that comes from wpp2 which seems to be what is being added here.
But when I switched it that didn't work either because there are 7 regions/loops and one must have some difference in wpp_lb that is not allowing it to be merged….but other regions earlier in the loop are fine.
And it turned out the problematic loop is "i == 7"
Not sure what the deeper problem is so that needs to be addressed.
In the end, I added this so I could proceed with testing:
if (i==7){
data3 <- data3 %>%
rename("wpp_lb" = "wpp_lb.x")
data3 <- data3 %>%
dplyr::select(-wpp_lb.y)
}
Probably ok as NA seems to be set where value is not a number.
Here is the warning:
#Calculating the LBW subgroups from using the data that is already there
newAdmin3<-newAdmin2 %>% mutate(calculateBW=ifelse(NumberofbabieswithBW=="CL", 1,
+ ifelse(NumberofbabieswithBW=="NA_LBW%", 2, 0)),
+ calculateLBW=ifelse(LBWlt2500g=="CL", 1, 0),
+ calculate1000=ifelse(LBWlt1000g=="CL", 1, 0),
+ calculate1500=ifelse(LBWlt1500g=="CL", 1, 0),
+ calculate15002499=ifelse(LBW1500to2499g=="CL", 1, 0),
+ calculate10001999=ifelse(LBW1000to1999g=="CL", 1, 0)) %>%
+ mutate_at(c(names(newAdmin2)[4:22]), as.numeric)
Warning message:
There were 17 warnings inmutate()
.
The first warning was:
? In argument: `Numberoftotalbirths = .Primitive("as.double")(Numberoftotalbirths)`.
Caused by warning:
! NAs introduced by coercion
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.