nrgpy/nrgpy

Python tools for NRG data files

nrgpy

nrgpy is the Python package for processing NRG Data Files

Website and source	https://github.com/nrgpy/nrgpy
Documentation	https://nrgpy.github.io/nrgpy/index.html
IEA Task 43 Parser	https://github.com/nrgpy/nrg-parser
Support	[email protected]

It provides tools for:

• Converting binary ".rld" and ".rwd" files to text format
  • using locally installed SymphoniePRO Desktop Software and Symphonie Data Retriever
  • using NRG Cloud API (compatible with Linux!) *(SymphoniePRO only at this time)
• Reading Symphonie text exports, LOGR dat, and Spidar zip/csv files into Pandas dataframes
• Timestamp adjustment (of text files)
• Simple quality checks on data

---

Contributing

See our Contributing guidelines for how to be a part of this project.

Installation

pip install nrgpy

Log and token files

Log files and NRG Cloud token files will be generated when using the app. These files can be found in your '''home''' directory:

Windows

C:\Users\{username}\.nrgpy\

Linux/Mac

/home/{username}/.nrgpy/

Examples

For more examples see the Documentation: https://nrgpy.github.io/nrgpy/index.html

RLD files

Convert a folder of RLD files to Text with SymphoniePRO Desktop Software

import nrgpy
date_filter = '2018-10' # filter on any text in the filenames
text_folder_name = 'text_outputs/'
converter = nrgpy.local_rld(rld_dir='', out_dir=text_folder_name, file_filter=date_filter)
converter.convert()

Convert a folder of RLD files to Text with NRG Cloud Convert API

import nrgpy
file_filter = "000110"
rld_directory = "rlds"
client_id = "https://cloud.nrgsystems.com/data-manager/api-setup"
client_secret = ""
converter = nrgpy.cloud_convert(
    file_filter=file_filter, 
    rld_dir=rld_directory, 
    client_id=client_id,
    client_secret=client_secret,
    start_date="2019-11-01",
    end_date="2019-11-30",
)
converter.process()

Convert a single RLD file to Text with NRG Cloud API

import nrgpy
filename = "path/to/rld"
txt_dir = "path/to/txt/"
client_id = "https://cloud.nrgsystems.com/data-manager/api-setup"
client_secret = ""
converter = nrgpy.CloudConvert(
    filename=filename, 
    client_id=client_id,
    client_secret=client_secret,
)

Read files

file_filter = "000110"
import nrgpy

reader = nrgpy.SymProTextRead()
reader.concat_txt(
    txt_dir=text_folder_name, 
    file_filter=file_filter, 
    start_date="2019-11-01",
    end_date="2019-11-30",
)

RWD files

Convert a folder of RWD files to Text with Symphonie Data Retriever

import nrgpy

file_filter = '0434201902' # for Feb 2019 files from site 0434
rwd_directory = 'C:/Users/[user]/rwd/'
out_directory = 'C:/Users/[user]/txt/'

converter = nrgpy.local_rwd(file_filter=file_filter, rwd_dir=rwd_directory, out_dir=out_directory)
converter.convert()

Convert a folder of RWD files to Text with Symphonie Data Retriever on Linux

import nrgpy

rwd_directory = '/home/user/datafiles/rwd'
out_directory = '/home/user/datafiles/txt'
wine_directory = '/home/user/prefix32/drive_c/' # path to wine's "C:\" drive


converter = nrgpy.local_rwd(
                file_filter=file_filter, 
                rwd_dir=rwd_directory, 
                out_dir=out_directory,
                wine_folder=wine_directory,
                use_site_file=False # set to True to use site files
            )
converter.convert()

You can also convert a single file with SDR, and save it in the same directory:

import nrgpy

filename = '/path/to/file'
converter = nrgpy.local_rwd(filename=filename)

Read TXT files exported from RWD files

import nrgpy

dt = 'rwd'
txt_file = '/path/to/file.txt'
reader = nrgpy.read_text_data(data_type=dt, filename=txt_file)

or concatenate a whole lot of files:

dt = 'rwd'
txt_dir = '/path/to/text/files'
file_filter = 'text_in_filenames_you_want'
reader = nrgpy.read_text_data(data_type=dt, txt_dir=txt_dir, file_filter=file_filter)
reader.concat()

Spidar files

Spidar Vertical Profiler remote sensors generate archived csv data files in a Zip format.

These can be read directly into the spidar_txt_read method. See the docstring in spidar_txt.py for more information.

Eg.

In [1]: import nrgpy

In [2]: fname = "/spidar/1922AG7777_CAG70-SPPP-LPPP_NRG1_AVGWND_2019-07-07_1.zip"                            

In [3]: reader = nrgpy.SpidarRead(filename=fname)                                                                              

In [4]: reader.heights                                                                                                         
Out[4]: [40, 60, 80, 90, 100, 120, 130, 160, 180, 200]

In [5]: reader.data                                                                                                            
Out[5]: 
                     pressure[mmHg]  temperature[C]  ...  dir_200_std[Deg]  wind_measure_200_quality[%]
Timestamp                                            ...                                               
2019-07-06 23:40:00          749.66           24.13  ...             28.77                           68
2019-07-06 23:50:00          749.63           24.08  ...             14.31                            0
2019-07-07 00:00:00          749.64           23.99  ...             20.59                            0
...

nrgpy's Issues

linux bash process rwd unable to copy C:\NRG\RawData\1238\20150812019.txt to root/txt/

import nrgpy

file_filter = '1238201508'
rwd_directory = '/root/raw'
out_directory = '/root/txt'
wine_directory = '/root/prefix32/drive_c' # path to wine's "C:\" drive


converter = nrgpy.local_rwd(
                file_filter=file_filter,
                rwd_dir=rwd_directory,
                out_dir=out_directory,
                wine_folder=wine_directory,
                use_site_file=False # set to True to use site files
            )
converter.convert()

Error while converting RWD to txt

SDR test OK!
Time elapsed: 0 s | 1 / 1 [=============================================] 100% Traceback (most recent call last):
File "C:\Users\kenneth\AppData\Local\Continuum\Anaconda3\lib\site-packages\nrgpy\convert_rwd.py", line 294, in _copy_txt_file
shutil.copy(txt_file_path, out_path)
File "C:\Users\kenneth\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 241, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\Users\kenneth\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\NRG\ScaledData\\018720120120200.txt'

Unable to copy 018720120120200.txt to ScaledData\

RWDs in : 1
TXTs out : 0
LOGs out : 0

Difference : 1

Cannot concatenate NRG Cloud exported txt data

To reproduce:

1. export two ranges of text data
2. attempt to concatenate
3. note exception in string_date_check function (nrgpy.utils.utilities.string_date_check)

file date of Cloud export has "." instead of "-" separating YYYYMMDD

Is there a way to get the sensor change date based on configuration change?

I can able to convert RLD files. Now I'm trying to find the sensor change date if configuration changes. Is there any trick to find the same.
I can able to view sensor details in each text file. then I need to run a loop to read each file and try to figure out the change manually.

With concat_txt function, it overwrites the sensor configuration with the latest text file.

Regards
Kenneth

Update check_intervals to handle duplicates

When duplicate timestamps exists, this exception is raised:

ValueError: cannot reindex on an axis with duplicate labels

Update to handle and return information when duplicate timestamps exist in a data set

sympro_txt.sympro_txt_read.concat_txt: file_type not implemented

   243     file_type : str                                                                             
   244         type of export (meas, event, comm, sample, etc...)

Update concatenate functions to remove dataframe appends

df.append will not longer be supported; change to

pd.concat([base.data, s.data], ignore_index=True, axis=0, join="outer")

Automate documentation updates

Use something like this methodology to automatically update and publish documentation on PR/push to master:

https://miguelvf.dev/blog/sphinx-github-actions-docs-guide/

output_txt_file - add feedback

add file path / name + 'OK!'

txt filtering for sympro_txt_read.concat_txt

if txt files are among rlds, the sympro_txt_read.concat_txt method will fail. Need to add a check for valid txt exports.

Interim fix would be to add exception handling for UnicodeDecodeError, which occurs when the original RLD file is in the same folder as the TXT export.

'convert_type' switch is missing from command line argument for convert_rld.local.directory()

Discovered when trying to process sample data using this method. Borrowing from single_file method, directory method's cmd could be updated to include "/type", '"'+self.convert_type+'"', as follows:

cmd = [self.sympro_path, 
                   "/cmd", "convert", 
                   "/file", '"'+file_filter+'"', 
                   encryption,  
                   "/type", '"'+self.convert_type+'"',
                   "/outputdir", '"'+self.out_dir[:-1]+'"'
            ]

Different behaviors for single files and directories

For both rld_local and rwd_local, if a single file is passed as filename in the init params, it is converted to txt and saved on disk. This behavior is confusing because as a user, I haven't called local_obj.convert() yet. On the other hand, if one passes in a directory instead of a single file through the rwd_dir or rld_dir init params, this requires calling .convert() on the local object for conversion.

Have you tried converting .rld files using nrgpy from RStudio?

I'm more an R & RStudio user. I've converted files using python but I've tried to convert files using Rmarkdown with the reticulate package but I can't.
I run my code

# R
library(reticulate)
# R 
use_condaenv("Anaconda3")
# Python chunk
import nrgpy
text_folder_name = '/archivos/txt' 
converter = nrgpy.local_rld(rld_dir="/Data/rld", out_dir=text_folder_name)
converter.convert()

but when I get to the converter.convert() I get the message

Converting 1446 files from \archivos\txt
Saving outputs to \Data\rld
Unable to process files in directory

To convert files python connects to SymphoniePro Desktop and I think RStudio can't get access to do it. (I'm just guessing)

Do you think it's possible to use python and nrgpy from RStudio? Or is it impossible?

Please some advice.

I use:
python 3.8.5 with anaconda3
R x64 4.0.3
RStudio v1.3.1093
Windows 10 Home and Pro (at work)

[Feature request] - add date filters to sympro_read_txt.concat_txt

Add ability to send date filters

1. as range
2. as list of specific dates
3. as before / after

convert_rld.local() method ONLY compatible with Windows OS.Please use convert_rld.nrg_convert_api() method instead.

#!/usr/bin/env python3

import argparse
import os
import nrgpy

parser = argparse.ArgumentParser(description='Use NRG SDR software to decode anemometer data.')
parser.add_argument('--pin', help='PIN code', required=True)
parser.add_argument('--rld-dir', help='RLD input data directory')
parser.add_argument('--out-dir', help='TXT output data directory')
parser.add_argument('--file-filter')
parser.add_argument('--tower', help='Assume rld_dir is raw and out_dir is txt.')
args = parser.parse_args()

if args.tower:
	args.rld_dir = f'/root/raw/{args.tower}/'
	args.out_dir = f'/root/txt/{args.tower}/'
	if not args.file_filter:
		args.file_filter = args.tower
else:
	if not args.rld_dir or not args.out_dir or not args.file_filter:
		print('[Error]: Option --rld-dir, --out-dir, --file-filter or --tower should be provided!')
		exit(1)
	args.rld_dir = f'/root/{args.rld_dir}/'
	args.out_dir = f'/root/{args.out_dir}/'

if not os.path.isdir(args.rld_dir):
	print(f'[Error]: RWD directory {args.rld_dir} does not exist!')
	exit(1)

if not os.path.isdir(args.out_dir):
	os.makedirs(args.out_dir)

wine_dir = '/root/prefix32/drive_c' # path to wine's "C:\" drive

converter = nrgpy.local_rld(
	encryption_pin=args.pin,
	file_filter=args.file_filter,
	rld_dir=args.rld_dir,
	out_dir=args.out_dir,
	sympro_path=wine_dir,
	use_site_file=False # set to True to use site files
)
converter.convert()

print('Done')

Invalid files are also processed "successfully"

If a file does not exist, the output to the screen is still the same as when a valid file is processed by nrgpy_rld.local. Example -

nrgpy_rld.local(filename="file_that_doesnt_exist.rld", out_dir="some_dir")

Will print this to the console -

file_that_doesnt_exist.rld ... 		[DONE]
Queue processed

Ideally, invalid paths or files should be reported differently to the user

add tags to repository in sync with pypi versions

LogrRead.output_txt fails

-> IN: 

reader.output_txt_file(out_file=f"{reader.site_number.zfill(6)}_{reader.site_description.replace(" ","_")}_meas.txt")

-> ERROR:

Outputting file: 000304_Crows_Nest_Counters_meas.txt   ...   [FAILED]
Traceback (most recent call last):
  File "/home/blackcap/gits/logr_testing/venv/lib/python3.12/site-packages/nrgpy/read/logr.py", line 647, in output_txt_file
    output_file.write(self.head)
                      ^^^^^^^^^
AttributeError: 'LogrRead' object has no attribute 'head'

sympro_txt.py:154 replace .append with .concat

nrgpy/read/sympro_txt.py:154: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.ch_info = self.ch_info.append(ch_list)