superdarn / pydarn Goto Github PK
View Code? Open in Web Editor NEWPython library for visualizing SuperDARN Data
License: GNU Lesser General Public License v3.0
Python library for visualizing SuperDARN Data
License: GNU Lesser General Public License v3.0
As pointed out in #24 lmfit2 data plots all the data in the range gates, unlike the conventional SuperDARN FITACF DMap. When filtering applied striping is more prevalent in the plot:
import pydarn
import matplotlib.pyplot as plt
fitacf_file = "20120614.0401.1200.sas.lmfit2"
darn_read = pydarn.SDarnRead(fitacf_file)
fitacf_data = darn_read.read_fitacf()
filts = {'min_scalar_filter':{'tfreq': 12500}}
fig = plt.figure(figsize=(10,8))
ax1 = fig.add_subplot(111)
pydarn.RTP.plot_range_time(fitacf_data, parameter='p_l',
beam_num=12, ax=ax1,
filter_settings=filts,
zmin=1,zmax=30)
plt.show()
When calculating the slant range in gate2slant()
, the center=0 and center=1 cases are flipped.
This bug was found by looking at the source code (see below), so I don't have any specific test code to demonstrate it.
pydarn/pydarn/utils/conversions.py
Lines 114 to 120 in 452f368
Compare to the equivalent calculation in RST:
/* If not calculating center position of range-beam cell then calculate
* position of near-left corner instead */
if (center==0) {
bm_edge=-pos->bmsep*0.5;
range_edge=-0.5*rsep*20/3;
}
Should be a simple swap:
if center:
range_offset = 0.0
else:
# 0.5 offset to the edge of the range gate
range_offset = -0.5 * record['rsep']
I am purposing a Major release due to the structural change with the move of the IO package.
Other new features:
import pydarn
Traceback (most recent call last):
File "", line 1, in
File "/home/k2/Documents/superdarn_github/borealis/pydarn/pydarn/init.py", line 38, in
from .io.borealis.borealis import BorealisRead
File "/home/k2/Documents/superdarn_github/borealis/pydarn/pydarn/io/borealis/borealis.py", line 49, in
from .borealis_site import BorealisSiteRead, BorealisSiteWrite
File "/home/k2/Documents/superdarn_github/borealis/pydarn/pydarn/io/borealis/borealis_site.py", line 47, in
from pathlib2 import Path
ModuleNotFoundError: No module named 'pathlib2'
When plotting multiple radars with overlapping FOVs, vectors are missing in the resultant plot.
The following code, plotting a handful of northern hemisphere radars:
site_file_dict = {'bks':'sddata/20160306.1200.02.bks.fitacf.bz2',
'fhe':'sddata/20160306.1200.02.fhe.fitacf.bz2',
'cly':'sddata/20160306.1200.02.cly.fitacf.bz2',
'fhw':'sddata/20160306.1200.02.fhw.fitacf.bz2',
'kap':'sddata/20160306.1200.03.kap.fitacf.bz2',
'wal':'sddata/20160306.1201.00.wal.fitacf.bz2',
'sto':'sddata/20160306.1201.00.sto.fitacf.bz2'}
fanplot = {}
for radar_site in site_file_dict.keys():
sdfpath = site_file_dict[radar_site]
with bz2.open(sdfpath) as grid_file:
fstream = grid_file.read()
SDarn_read = pydarnio.SDarnRead(fstream, True)
records = SDarn_read.read_fitacf()
kwargs = {'boundary':True, 'zmax':200, 'zmin':-200, 'groundscatter':True, 'lowlat':40, 'scan_index':89}
# only plot the colorbar once
if radar_site == 'bks':
fanplot[radar_site] = pydarn.Fan.plot_fan(records, colorbar=True, **kwargs)
else:
fanplot[radar_site] = pydarn.Fan.plot_fan(records, colorbar=False, **kwargs)
produces the following plot:
Hiding the overlapping radars (STO and KAP in this case) will show the missing vectors at CLY:
site_file_dict = {'bks':'sddata/20160306.1200.02.bks.fitacf.bz2',
'fhe':'sddata/20160306.1200.02.fhe.fitacf.bz2',
'cly':'sddata/20160306.1200.02.cly.fitacf.bz2',
'fhw':'sddata/20160306.1200.02.fhw.fitacf.bz2',
# 'kap':'sddata/20160306.1200.03.kap.fitacf.bz2',
# 'sto':'sddata/20160306.1201.00.sto.fitacf.bz2',
'wal':'sddata/20160306.1201.00.wal.fitacf.bz2'}
fanplot = {}
for radar_site in site_file_dict.keys():
sdfpath = site_file_dict[radar_site]
with bz2.open(sdfpath) as grid_file:
fstream = grid_file.read()
SDarn_read = pydarnio.SDarnRead(fstream, True)
records = SDarn_read.read_fitacf()
kwargs = {'boundary':True, 'zmax':200, 'zmin':-200, 'groundscatter':True, 'lowlat':40, 'scan_index':89}
# only plot the colorbar once
if radar_site == 'bks':
fanplot[radar_site] = pydarn.Fan.plot_fan(records, colorbar=True, **kwargs)
else:
fanplot[radar_site] = pydarn.Fan.plot_fan(records, colorbar=False, **kwargs)
yields
FITACFs from VT.
Line 204 in pydarn/fan.py: Fan.plot_fan() method
In above lines, set the alpha of the RGBA value to 0.
Update: @DannoPeters did some work around this last summer and will be continuing again this summer.
From @aburrell request on PR #19 to add altitude values to the list.
@DannoPeters can you list out some steps on how to do this or possible options?
With old dat files being converted to RAWACF files and higher data products, some of their issues are coming to light. As well, this could occur in current radar files as radar OS's are changing.
One issue is erroneous dates in the file. A warning can be raised in the case of the dates being consistently increasing with gaps something.
Then an error for some dates that jump around too much.
This will help DDWG spot these errors and potentially prevent errors in plotting methods as well.
While accessing some of the map2 files I'm getting this error
Error: The following fields in record 462 are missing: {'vector.mlat', 'vector.vel.median', 'vector.mlon', 'vector.kvect', 'vector.vel.sd', 'vector.channel', 'vector.index', 'vector.stid'}
Traceback (most recent call last):
File "convert_map_to_csv.py", line 22, in read_map_files
_md = reader.read_map()
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 493, in read_map
self._read_darn_records(file_struct_list)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 374, in _read_darn_records
self._read_darn_record(format_fields)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 350, in _read_darn_record
SDarnUtilities.missing_field_check(format_fields, record, self.rec_num)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 178, in missing_field_check
SuperDARNFieldMissingError(rec_num, missing_fields)
pydarnio.exceptions.superdarn_exceptions.SuperDARNFieldMissingError: Error: The following fields in record 462 are missing: {'vector.mlat', 'vector.vel.median', 'vector.mlon', 'vector.kvect', 'vector.vel.sd', 'vector.channel', 'vector.index', 'vector.stid'}
File - /sd-data/2012/map2/north/20120309.north.map2.bz2
Trying to install pyDARN on open suse 15.1 via:
pip3 install --user pydarn
pip3 install --user git+https://github.com/superdarn/pydarn
git clone https://github.com/superdarn/pydarn
cd pydarn
pip3 install --user .
python3 setup.py install --user
All resulting in the same error with:
Traceback (most recent call last):
File "setup.py", line 67, in <module>
'h5py', 'deepdish', 'pathlib2']
Last line in the error is:
File "/usr/lib64/python3.6/lib2to3/pgen2/grammar.py", line 108, in load
d = pickle.load(f)
EOFError: Ran out of input
This is just a housekeeping issue post. Make sure we don't lose track of things.
As mentioned in #4, we need to handle char arrays differently when reading dmap arrays. For more details see "Outstanding Issue" in this comment: #4 (comment)
We can fix this by adding:
if array_type_fmt == 'c':
array_value = np.array([ord(x) if len(x) == 1 else 0 for x in array_value])
edit: This fix only handles reading in the char arrays. Not sure what happens when we write them out.
Hi, I recently installed pydarn to work on SuperDARN data. I am able to read and do some plots shown on the tutorial for .fitacf files. My problem is I have other data with the old format, '.fit', which I need to read and create a data frame for further analysis (time series analysis). So I am wondering if it is possible to read .fit files using pydarn.
Thank you!
the pydarn.DarnRead
output doesn't have a user friendly object interface to let you know whether or not you successfully read in data.
import pydarn
fitacf_file = '20170120.0001.00.inv.fitacf'
sd_read = pydarn.DarnRead(fitacf_file)
print(sd_read)
<pydarn.io.superdarn.DarnRead object at 0x129385860>
This is addressing issue #59 where the exception should include:
Need to update the plotting documentation and put in tutorials for new plots:
I noticed that pydarn will incorrectly classify an input bz2 file as having an issue with the first record.
If it's not the intention of pydarn to handle compressed files, then it should produce an error message saying so. If it's the intention of pydarn to handle compressed files, it should uncompress it first.
import pydarn
in_fn = '/home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2'
sd = pydarn.SDarnRead(in_fn)
r = sd.read_rawacf()
Error: /home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2 contains an block size 643383601 > remaining bytes 13381653 at record = 0.
Traceback (most recent call last):
File "", line 1, in
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 421, in read_rawacf
self._read_darn_records(file_struct_list)
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 372, in _read_darn_records
self._read_darn_record(format_fields)
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 347, in _read_darn_record
record = self.read_record()
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/dmap.py", line 449, in read_record
remaining_bytes, "remaining bytes")
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/dmap.py", line 287, in bytes_check
self.rec_num)
pydarn.exceptions.dmap_exceptions.MismatchByteError: Error: /home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2 contains an block size 643383601 > remaining bytes 13381653 at record = 0.
When plotting range-time plots or summary plots using the slant=True
option the major ticks are shown at 200km starting at ~180km. Minor ticks are set to 100km, assuming that they are starting from the nearest '100' which in this case is at 200 km, +20km from the major ticks.
import matplotlib.pyplot as plt
import pydarn
#Load in dmap file
file = "/Users/carley/Documents/data/20161123.1601.00.mcm.a.fitacf"
SDarn_read = pydarn.SuperDARNRead(file)
fitacf_data = SDarn_read.read_fitacf()
#Plot and show
a = pydarn.RTP.plot_range_time(fitacf_data, beam_num=4, slant=True)
plt.show()
What have you tried already to fix it?
What have you tried to get around it?
Any fitacf file should recreate this bug in range-time plots.
Line 402 and 417 in rtp.py
if slant:
ax.yaxis.set_ticks(np.arange(ymin, ymax+1, yspacing))
else:
ax.yaxis.set_ticks(np.arange(ymin, ymax+1, (ymax)/5))
.......
if slant:
ax.yaxis.set_minor_locator(ticker.MultipleLocator(100))
else:
ax.yaxis.set_minor_locator(ticker.MultipleLocator(5))
Possibly need to set the base, or bottom tick for both to be the same?
No issue with range gates for the plots.
While we're playing with ticks, it might be nice to have them both round off at the hundreds?
This issue is addressing the potentially problem a user may rarely run into a missing hardware file.
Details are in #59 and discuss how one can fix it will be laid out in the pyDARN troubleshooting page.
Following the code of conduct template, I will list here additive documentation pyDARN will need.
Update: I have been working on developer guidelines and I am hoping to have out for review by the next meeting which should be in the next 3 weeks (after CEDAR conference).
Hey all, I'm largely a n00b here, so sorry if this has been covered elsewhere, though I couldn't find it anywhere. Has anyone looked at pydarn in the new Ubuntu20? I know it's not released yet...but will be soon.
Anywho, I was using a VM with that and think I didn't skimp on the software packages to be installed. But starting with the install instructions on the readthedocs I didn't have pip3 installed even though I have Python3.8.2. Anyone else run into that or is that maybe from using a pre-release of the OS?
EDIT: Also didn't come standard with git. These seem to both be needed to run:
pip3 install --user git+https://github.com/superdarn/pydarn@develop
module: fan.py
package: plotting
Add geographical coordinates to:
plot_fan
plot_fov
Currently plot_fov
and plot_fan
only plot in magnetic coordinates, they should also be able to provide geographic coordinates as well.
It's not very critical, but should the version badge be (or have been) updated to match the 1.0 release? Guess it's too late now to get into Zenodo, but something to watch for next time for a release.
As this is mentioned and shown how various installation methods may lead to errors example #59. Also, to someone not very familiar with github may not know how to switch branches easily.
When using plot_fan
or plot_fov
we get an UserWarning
from matplotlib using set_xtickslabels
but not setting the ticks beforehand. Because this is with a polar plot some thought needs to be given.
294: UserWarning: FixedFormatter should only be used together with FixedLocator
287: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance. In a future version, a new instance will always be created and returned. Meanwhile, this warning can be suppressed, and the future behavior ensured, bypassing a unique label to each axes instance.
Line 191 in cf8d83e
FOV only shows the first 75 range gates which maybe not be correct for radars with a large FOV.
make_grid
with FHW data or any known radar with more than 75 Rage gates.
Please note this is not required.
If you can link to some code in pyDARN where you think the bug is coming from this will help the developer fix it sooner.
Line 50 in a61a5e3
Change the default and get the range gate value from the hardware files.
When preparing to test PR #52 I encountered an unrelated error HardwareFileNotFoundError
. This also happens on the develop branch, so I believe is related to recent changes there.. The error message doesn't help me fix it: "Hardware file for dcn radar was not found. Please insure the abbreviation is correct and there exists a hardware file for it by checking: https://github.com/vtsuperdarn/hdw.dat".
I have the latest version of RST running and have a dcn hardware file.
Suggested changes:
Re-look at why matplotlibs deprecation warnings are showing up in plotting methods.
Name: plot_fan
module: fan.py
package: plotting/
plot_fan
scan_index
(don't think it's generalised enough to go in utils, or would be used anywhere else, so probably a few lines of code in plot_fan
?)scan_index
value for user to note/if it's needed elsewhere - possible need to be in utils?What will the feature do?
What will the feature not do?
New optional input for fan plots where a datetime object can be inputted which is used to find the scan index number at that time to plot the scan at the given time.
Do not intend to change how plot_fan
works (I.e using scan_index
to read in correct data) unless required.
Make as similar to plot_grid
datetime input functionality as possible.
Be aware scan_index
default is 1 - handle properly if optional input of datetime is given.
Be aware of previous conventions used in other libraries (I.e RST gets the next scan from the time given) and write documentation accordingly.
Discussion subject: function of converting datetime to scan index required in a utils package, or just as small addition to plot_fan
?
if datetime given as input
convert datetime object into scan_index in file
return scan_index
input new scan_index into remaining plot_fan code
Purposed fix for running into missing new radars is to have the raised exception attempt another git submodule pull to grab any new radars that are missing. Of course, this will come with the change to using radar.dat and the new hardware repo.
More details #59
As mentioned by @asreimer , @egthomas, and @billetd , users tend to plot velocity rang-time parameter plots most of the time, making the default colormap
Unsensible as it is not red-blue shifting colormap.
The suggestion is to change the default colormap to match for velocity to ensure conventions are met, and users are not confused. As well as my own suggestion to make the default parameter also velocity if it is the most commonly plotted parameter.
This is currently being under review in PR #25
@billetd we will also need to update the tutorials.
Side note: pyDARN does not follow exact conventions as DaVitpy, and should not be directly compared to. Unlike DaVitpy, pyDARN does not do parameter detection for colormaps and boundaries in plot_range_time
to ensure future-proofing if parameter names change. This lower future maintenance of the code.
As mentioned in #24 the scaling gets messed up when using lmfit2 data:
import pydarn
import matplotlib.pyplot as plt
fitacf_file = "20120614.0401.1200.sas.lmfit2"
darn_read = pydarn.SDarnRead(fitacf_file)
fitacf_data = darn_read.read_fitacf()
filts = {'min_scalar_filter':{'tfreq': 12500}}
fig = plt.figure(figsize=(10,8))
ax1 = fig.add_subplot(111)
pydarn.RTP.plot_range_time(fitacf_data, parameter='p_l',
beam_num=12, ax=ax1,
filter_settings=filts)
Fan plots are coming along with @DannoPeters and @billetd help.
I just wanted to let everyone it is happening and almost ready for PR.
Need to add documentation on GRID plotting
Will include how to use and how to process grid files from RST docs.
I'm using the develop
branch, specifically commit 9b5caa2 and I noticed several small issues. I might be able to help fix them, but figured they should be reported:
The default velocity colormap doesn't follow the redshift/blueshift convention. For example, I plotted up some Saskatoon radar data with this code:
import pydarn
from matplotlib import pyplot
fitted_file = "20120614.0401.1200.sas.lmfit2"
darn_read = pydarn.SDarnRead(fitted_file)
data = darn_read.read_fitacf()
filts = {'max_scalar_filter':{'tfreq': 12500}}
fig = pyplot.figure(figsize=(10,8))
ax1 = fig.add_subplot(111)
pydarn.RTP.plot_range_time(data, parameter='v',
beam_num=12, ax=ax1,
filter_settings=filts,
zmin=-1000, zmax=1000)
pyplot.show()
You can download the lmfit2 file here: https://github.com/asreimer/lmfit2/blob/master/example/20120614.0401.1200.sas.lmfit2.gz
But what I should have gotten is a plot where the colormap is reversed. See: https://github.com/asreimer/lmfit2/blob/master/example/no_filter.png or this RTI that was generated using the VT superdarn website:
Notice that red and blue are backwards.
If you look at the plot I generated using the code in 1, you'll see it has many white stripes. This is undesirable behaviour and will likely happen on all of the "twofsound" mode data.
The docstring for RTP says that one should use the plot_filter
keywork, which is incorrect. One needs to use filter_settings
instead.
The autoscaling for the parameter being plotted can go wacky. Running this code:
filts = {'min_scalar_filter':{'tfreq': 12500}}
fig = pyplot.figure(figsize=(10,8))
ax1 = fig.add_subplot(111)
pydarn.RTP.plot_range_time(data, parameter='p_l',
beam_num=12, ax=ax1,
filter_settings=filts)
pyplot.show()
but using the zmin
and zmax
fixes the problem:
filts = {'min_scalar_filter':{'tfreq': 12500}}
fig = pyplot.figure(figsize=(10,8))
ax1 = fig.add_subplot(111)
pydarn.RTP.plot_range_time(data, parameter='p_l',
beam_num=12, ax=ax1,
filter_settings=filts,
zmin=1,zmax=30)
pyplot.show()
Putting in a CODE_OF_CONNDUCT.md
to help understand:
Will be generated from a GitHub provided template to then be modified for the pyDARN/DAWG community
Name: plot_grid
module: grid.py
package: plotting
This feature will plot grid files on a polar plot in magnetic coordinates.
Will not plot on a geographic map or geo-coordinates.
We will use a grid file to plot data on the polar plot from magnetic coordinates.
feature request for native gzip or bzip decompression while reading. Most of our users have their data stored in compressed formats, and RST is capable of reading in compressed formats.
In #72 we noticed plt.figure
will override a user's plt.figure
, now is the time to investigate what is the best method to do this.
This is a low priority feature request.
It might be nice to include some cursor control methods in the DmapRead class so that one can jump to different records in a file without reading out all of the other records in the file.
For example:
import pydarn
filename='20171231.0001.00.sas.fitacf' #any dmap file will do
dmap_handler = pydarn.DmapRead(filename)
print(dmap_handler) # prints a 0
record = dmap_handler.read_record()
print(dmap_handler) # prints something larger than 0.
# begin new methods
dmap_handler.rewind() # sets cursor to beginning of the file
dmap_handler.next() # jumps to next record in the file
dmap_handler.jump(record_number) # sets the cursor to an offset where record "record_number" resides.
I've used this functionality in davitpy before, but it isn't critical to typical use cases.
title or suggestion of documentation
Adding workflow documentation similar to RST
An extra check for map file reading sees if the station ids are for the correct hemisphere.
Purposed solution:
Using the hardware files and SuperDARN.Radars class you can determine which stid
belongs to which hemisphere then cross-checking that will stid
present in the map file and the hemisphere field in the map file.
When trying to read a rawACF I get this error:
pydarn.exceptions.superdarn_exceptions.SuperDARNFieldMissingError: Error: The following fields in record 14 are missing: {'acfd'}
Example code (using python3.7)
in_fn = 'pydarn/raw_sas/20090101.1201.00.sas.rawacf'
sdf = pydarn.SDarnRead(in_fn)
rwr = sdf.read_rawacf()
Name: labels and dots to indicate where the radars are located
module: fan.py
package: /plotting
New option in plot_fov
to include Radar abbreviation and the radar site dots will be part of the FOV plot.
plot_fov(66, radar_label=True)
With the announcement of pyDARN splitting off the io, I am getting feedback on how users want the io to stay in to make pyDARN one complete package.
As I don't disagree with either separate or keep together, I thought an idea to achieve the best of both worlds.
We can make pyDARNio a dependency of pyDARN via install in setup file or sub-repo.
I am leaning to the sub-repo idea.
This way, we can keep changes separate but have io together with pyDARN for those that prefer it that way.
I will leave this up as an agenda topic for the next meeting.
Name: plot_map
module: map.py
package: /plottting
This feature will focus on the vectors and the other parts of the plotting will come.
plot_map(map_data)
currently have to use
from pydarn.utils.conversions import dict2dmap
in io/borealis.py
Feature request: make pydarn pip installable from git link.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.