I first created new conda env and install the latest version of yadg using pip. The current version of the installed library is as follows:
yadg 5.0.2
pydantic 2.6.4
dgbowl-schemas 116
dgpost 2.1.1
I first try runing yadg /dgpost using the example files and command provided by you last summer.
yadg-5.0a5-pipeline.zip. The script works fine. The netcdf was created and dgpost work correctly.
Nevertheless, we nolonger use drycal's software to measure the flow to ease the issue with piston stucking during the measurement. I wrote a script to control the flow meter. We are now using the script exclusively during the measurement. The introduction of multiplex system (running 8 cells at the same time) also requires some pre-processing of the flow data/ pressure data and temperature data before processing using yadg/dgpost.
I have made the script that will pre-process the flow data, pressure data and temperature data before processing using yadg/dgpost. The pre-processed files will be called 'flow_for_yadg.csv', 'pressure_for_yadg.csv' and 'temperature_for_yadg.csv' respectively. You can find the data after pre-processing step
here I have tried to modify yadg yaml recipe file for these files yadg.preset.francesco_v5-EDLC_mod1.yaml.zip but after I tried to run yadg on this pre-process data using this modified yaml, I got the error below: It seems that extractor (which is a new feature introduced in yadg5) is required, but I am not quite sure how this work. I think the issue might stem from how I made the yaml file. Could you please have a look into this. Thank you.
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\Scripts\yadg.exe\__main__.py", line 7, in <module>
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\yadg\main.py", line 201, in run_with_arguments
args.func(args)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\yadg\subcommands.py", line 144, in preset
schema = to_dataschema(**preset)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\dgbowl_schemas\yadg\__init__.py", line 40, in to_dataschema
schema = Model(**kwargs)
File "C:\Users\plnu\Anaconda3\envs\test_yadg5\lib\site-packages\pydantic\main.py", line 171, in __init__
self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 56 validation errors for DataSchema
steps.10.Dummy.parser
Input should be 'dummy' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.BasicCSV.parser
Input should be 'basiccsv' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.BasicCSV.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.MeasCSV.parser
Input should be 'meascsv' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.MeasCSV.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.FlowData.parser
Input should be 'flowdata' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.FlowData.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.FlowData.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.ElectroChem.parser
Input should be 'electrochem' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.ElectroChem.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.ElectroChem.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.ChromTrace.parser
Input should be 'chromtrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.ChromTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.ChromTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.ChromData.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.ChromData.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.MassTrace.parser
Input should be 'masstrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.MassTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.MassTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.QFTrace.parser
Input should be 'qftrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.QFTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.QFTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.XPSTrace.parser
Input should be 'xpstrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.XPSTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.XPSTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.10.XRDTrace.parser
Input should be 'xrdtrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.10.XRDTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...sion.zip'}, 'tag': 'GC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.10.XRDTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='fusion.zip', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.Dummy.parser
Input should be 'dummy' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.BasicCSV.parser
Input should be 'basiccsv' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.BasicCSV.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.MeasCSV.parser
Input should be 'meascsv' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.MeasCSV.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.FlowData.parser
Input should be 'flowdata' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.FlowData.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.FlowData.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.ElectroChem.parser
Input should be 'electrochem' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.ElectroChem.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.ElectroChem.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.ChromTrace.parser
Input should be 'chromtrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.ChromTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.ChromTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.ChromData.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.ChromData.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.MassTrace.parser
Input should be 'masstrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.MassTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.MassTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.QFTrace.parser
Input should be 'qftrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.QFTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.QFTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.XPSTrace.parser
Input should be 'xpstrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.XPSTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.XPSTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
steps.11.XRDTrace.parser
Input should be 'xrdtrace' [type=literal_error, input_value='chromdata', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/literal_error
steps.11.XRDTrace.extractor
Field required [type=missing, input_value={'parser': 'chromdata', '...alc.xlsx'}, 'tag': 'LC'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.6/v/missing
steps.11.XRDTrace.parameters.filetype
Extra inputs are not permitted [type=extra_forbidden, input_value='empalc.xlsx', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden```