pcrelate issue for large sample size

[uw-gac]

pcrelate issue for large sample size

[smgogarten]

We have run pcrelate on larger numbers of samples by calling its sub-functions separately, which also allows for running sample blocks in parallel on a compute cluster; see #38 for a description of how to do it. This allows filtering of the results for each block to only keep pairs above a certain kinship threshold before combining them.

We should really add this threshold option to the pcrelate function so it can be done more easily, but it might be a while before we have time to do that.

[GraceSheng]

Thank you Stephanie for suggesting the workaround. As I intend to keep all the pairwise kc rows, I tried an alternative approach: basically I use a list of data table to hold kinBtwn and write kinBtwn out to a file after the pcrelate call.

I browsed the post processing codes for kinBtwn (skipped the ones that are applicable to small sample size), both "correct k2 for HW departure" and "correctK0" appear to be doing row based corrections which does not need info from all the other pairs in the table. So I just run each of my kinBtwn data table through these corrections.

I tested this on a smaller dataset with both standard pcrelate and my custom pcrelate, their kinBtwn table looked identical after resorting by ID1 and ID2. Though my testing might not be sufficient.

Since my knowledge of pcrelate is very limited, is there any pitfall that I might have overlooked? I plan to only use mine when sample size is > ~65K.

Thank you for your time!

[mconomos]

The correctK0 function is certainly fine to be called by block as it operates only on each row.
The correctK2 function is also fine, as long as you also have the estimated kinSelf data loaded for each sample in the pair -- the f estimates from kinSelf are used as part of the correction.

[GraceSheng]

Thank you for confirming. yes, the full kinSelf data table is always passed to correctK2 which should cover each sample in the pair for each of my kinBtwn data tables.