gridsst-hackathon/data-pre-processing

data-pre-processing's Issues

Bias Correction Clarity

Starting this issue as thread on the topic for bias correction code. Quick status update of where things are, and where some clarity on next steps could be.

Currently it seems like the following pre-processing steps have been implemented & turned into functions:

1. Lookup and access s3 granules for L2P and L3 (CMC) resources
2. Pre-process L2P Sensor Data in preparation for bias-correction
3. Load CMC data to use for bias-correction

What is somewhat unclear, or in a pseudo-code form is what reshaping needs to happen, and what processes need to be performed to atually do the bias-correction. @andy123harris has sent me pseudo code for the process, and I have included it at the end of this notebook https://github.com/gridSST-hackathon/data-pre-processing/blob/main/bias_correct_pseudo.ipynb

Please respond here with any comments/feedback on that assesment or any details I've missed. HUGE thanks to all of y'all for letting me be a part of this group, cheers!