Comments (16)
from clusterpyxt.
Thanks for the quick reply.
On the UI/ease of use, I find that the code is very easy to use and quite robust. Especially in the first times I used. I like the guiding text after each steps: very clear. After being familiar with the pipeline, I prefer running it with command lines rather than with GUI (so I add --continue when possible). It is more convenient to me. I understand this is really personal references. Other users might prefer GUI. But I think many users will like following additional features:
- the inputs for all steps are saved in a parset file by user at the beginning of run. This is very helpful when working on large number of clusters.
- automatic identification and removal of compact sources, in addition to compact sources defined by users (in case automatic pipeline does not work for some sources).
- automatic generation of regions files (acisI_region_0.reg and master_crop-ciaowcs.reg)
- automatic search for nH given the build-in databases.
- and probably comments on some important files generated by the pipeline and what corrections/processes have been done to them.
Sorry for too many requests. They are not crucial as I still can run the pipeline at its current stage. But with new features, much more can be done.
Thanks,
Duy
from clusterpyxt.
from clusterpyxt.
Thanks for the update! I have tried to run the branch ClusterPyXT-dev-CIAO-4.12. I got the error below. Do I miss something for the new version to work?
################
$ python clusterpyxt.py
File "clusterpyxt.py", line 130
region_file_label = QtWidgets.QLabel(f"A region file (e.g. {self.observations[0].acisI_region_0_filename}) containing\n"
^
SyntaxError: invalid syntax
################
from clusterpyxt.
from clusterpyxt.
I am using Python 3.5.4 that goes with Ciao-4.12 installation. The os is Ubuntu 18.04. After getting the new code, I still have the same error.
$ python clusterpyxt.py
File "clusterpyxt.py", line 133
While any size upto the full CCD may be used, a region larger than 60 arc seconds is generally not necessary.""")
^
SyntaxError: invalid syntax
Cheers,
Duy
from clusterpyxt.
Alright, the problem is that most of the strings are formatted for python 3.6 and up. I will rewrite these so it is compatible with python 3.5 again in ciao 4.12. I am at a conference the beginning of the week but should be able to complete the rewrite by the end of the week.
If that is too long as you are under a time crunch, let me know. Alternatively, upgrading to python 3.6+ in a different conda environment will also work (although may not be desirable).
Brian
from clusterpyxt.
The strings are updated and everything should work with python 3.5.
I've tested on a Mac with python 3.7.5 but the strings are formatted and should work with python 3.5.
I had some issues with the test VM today but I should have it tested on ubuntu with python 3.5 by Friday.
Brian
from clusterpyxt.
Thanks Brian for the fix. I have reinstalled Ciao 4.12 with Python 3.7.7 (installing via miniconda3, https://cxc.cfa.harvard.edu/ciao/download/conda.html) and updated ClusterPyXT-dev-CIAO-4.12. The installation of Ciao passed all smoke tests. The problem with the strings seems to be fixed.
But there is a new error (pget() Parameter not found). This seems to be related to Ciao. I follow this (https://cxc.cfa.harvard.edu/ciao/faq/pget_error.html) to remove files in ~/cxcds_param4/*. But the error still remains. Having checked the par file in the temporary directory (/tmp/tmp5i9h52qe.dmkeypar.par), but it is not there. It is not clear to me if this temporary file is created and then deleted when the pipeline crashed. Or the file was not created at all. Any ideas on how to fix this error?
#########
Found background at /home/duy/soft/miniconda3/envs/ciao-4.12/CALDB/data/chandra/acis/bkgrnd/acis1iD2000-12-01bkgrnd_ctiN0005.fits
pset: cannot convert parameter value : rval
/tmp/tmp5i9h52qe.dmkeypar.par: cannot convert parameter value : rval
Traceback (most recent call last):
File "clusterpyxt.py", line 496, in run_stage_1
ciao.run_stage_1(self._cluster_obj)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 1306, in run_stage_1
merge_observations(cluster)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 478, in merge_observations
ciao_back(cluster)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 184, in ciao_back
echo=True)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1810, in call
stackfiles = self._update_parfile(parfile)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1365, in _update_parfile
self._update_parfile_verify(parfile, stackfiles)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1294, in _update_parfile_verify
oval = _to_python(ptype, pio.pget(fp, oname))
ValueError: pget() Parameter not found
Aborted (core dumped)
########
from clusterpyxt.
update: if I run the command (on line 184 of ciao.py) in Python 3.7.7 separately (not with ClusterPyXT-dev-CIAO-4.12), no error occurs.
#######
from ciao_contrib import runtool as rt
acis_file='acisf04961_repro_evt2.fits'
rt.dmkeypar(infile=acis_file, keyword="GAINFILE", echo=True)
acisD2000-01-29gain_ctiN0008.fits
########
from clusterpyxt.
from clusterpyxt.
On my Mac OS, I can run the ClusterPyXT-dev-CIAO-4.12 version without any problem. This has Ciao-4.12 + Python 3.7.7 installed with miniconda. The only problem is that it takes lots of time when doing spectral fitting. So, it's not an ideal solution.
With the same installation on my Ubuntu 18 machine (Ciao-4.12+Python 3.7.7+miniconda), I had problems with parameter error (ValueError: pget() Parameter not found) when running dmkeypar (like in my earlier post). I added a punlearn command, rt.dmkeypar.punlearn(), before dmkeypar lines. This seems to fix the error for the dmkeypar lines that do not have keyword=TSTART or keyword=TSTOP. If I replace the keyword TSTART or TSTOP with other keywords (e.g. OBS_ID, TIMEZERO) found in the header, dmkeypar works just fine.
If I run this command, rt.dmkeypar(infile=data_hiE, keyword="TSTART", echo=True), outside of the pipeline (i.e. in python 3.7.7 with Ciao 4.12 in the environment), no error appears.
I also see the same behaviors for some other commands dmextract, deflare. They work outside of the ClusterPyXT-dev-CIAO-4.12, but do not work with the pipeline.
You have any idea why that is?
from clusterpyxt.
from clusterpyxt.
Yes, the the problem happens every run on Ubuntu. A quick trick I did was to create a python function that executes the function, but passing the arguments to bash shell. To call the function, simply replace, e.g., rt.dmextract by run_dmextract. The arguments are kept the same. Then, the pipeline works fine for these steps (without error on "Parameter not found"). Below are the python functions.
def run_dmextract(infile, outfile, bkg=None, error=None, bkgerror=None, bkgnorm=None, exp=None,bkgexp=None, sys_err=None, opt=None, defaults=None, wmap=None, clobber=None, verbose=None):
cmd = 'dmextract infile="%s" outfile="%s"' %(infile, outfile)
if bkg:
cmd += ' bkg=%s ' %bkg
if error:
cmd += ' error=%s ' %error
if bkgerror:
cmd += ' bkgerror=%s ' %bkgerror
if bkgnorm:
cmd += ' bkgnorm=%s' %bkgnorm
if exp:
cmd += ' exp=%s' %exp
if bkgexp:
cmd += ' bkgexp=%s' %bkgexp
if sys_err:
cmd += ' sys_err=%s ' %sys_err
if opt:
cmd += ' opt=%s ' %opt
if defaults:
cmd += ' defaults=%s ' %defaults
if wmap:
cmd += ' wmap=%s ' %wmap
if clobber:
cmd += ' clobber=yes '
if verbose:
cmd += ' verbose=%s ' %verbose
print ('Running %s' %cmd)
os.system(cmd)
return
def run_deflare(infile, outfile, method, save):
cmd = 'deflare infile="%s" outfile="%s" method=%s save=%s' %(infile, outfile, method, save)
print ('Running %s' %cmd)
os.system(cmd)
return
def run_dmhedit(infile, filelist, operation, key, value, datatype=None, unit=None, comment=None, verbose=None):
cmd = 'dmhedit infile="%s" filelist="%s" operation=%s key=%s value=%s ' %(infile, filelist, operation, key, value)
if datatype:
cmd += ' datatype=%s ' %datatype
if unit:
cmd += ' unit=%s ' %unit
if comment:
cmd += ' comment=%s ' %comment
if verbose:
cmd += ' verbose=%s ' %verbose
print ('Running %s' %cmd)
os.system(cmd)
return
##############
from clusterpyxt.
With the fix above, the code runs through all steps to Spectral Fitting. Here I encounter another error on instrument model not being found in the pi file. In the pi header there is keyword INSTRUME. Is this the info that the pipeline is reading? Any tips on how to fix this would be very helpful (sorry for borthering you so much on this)!
A copy of the pi header is here: https://www.dropbox.com/s/rtn7sl5f4fbgj3r/header.txt?dl=0
The error on the screen is copied below.
Duy
###########
Processing region number: 207
207: Working on observation id: 4215
Processing region number: 210
210: Working on observation id: 4215
210: Met S/N threshold for 4215
210: Extracting PI files for 4215
207: Met S/N threshold for 4215
207: Extracting PI files for 4215
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/acisI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_210_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/acisI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_207_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/backI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_207_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/backI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_210_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi" clobber=yes
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=EXPOSURE value=64220.940691801
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=EXPOSURE value=64220.940691801
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=RESPFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.rmf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=RESPFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.rmf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=ANCRFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.arf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=ANCRFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.arf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=BACKFILE value=/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=BACKFILE value=/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi" filelist="" operation=add key=EXPOSURE value=1717423.024726406
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi" filelist="" operation=add key=EXPOSURE value=1717423.024726406
207: Loading data pulse invariant files (PI files)
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[SPECTRUM]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[SPECTRUM]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI1]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI3]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934342,3214: 187001519,1142
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI2]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2214: 187001519,1552
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI0]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934336,1625: 187001519,2373
207: Loading background PI files
210: Loading data pulse invariant files (PI files)
Process Process-19: - WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[SPECTRUM]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[SPECTRUM]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI1]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2625: 187001519,1963
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI3]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934342,3214: 187001519,1142
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI2]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934339,2214: 187001519,1552
- WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI0]
- TSTART:TSTOP = 1,0000: 1,0000
- GTIs = 186934336,1625: 187001519,2373
Traceback (most recent call last):
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/hs/fs06/data2/AG_Brueggen/duy/soft/ClusterPyXT-dev-CIAO-4.12/cluster.py", line 1919, in fit_region_number
sherpa.set_analysis('energy')
File "", line 1, in set_analysis
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/ui/utils.py", line 6382, in set_analysis
self._get_pha_data(id).set_analysis(quantity, type, factor)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/data.py", line 836, in set_analysis
raise DataErr('noinstr', self.name)
sherpa.utils.err.DataErr: No instrument model found for dataset /home/duy/chandra_clusters/test/cc/acb/4215_207.pi
210: Loading background PI files
Process Process-20:
Traceback (most recent call last):
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/hs/fs06/data2/AG_Brueggen/duy/soft/ClusterPyXT-dev-CIAO-4.12/cluster.py", line 1919, in fit_region_number
sherpa.set_analysis('energy')
File "", line 1, in set_analysis
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/ui/utils.py", line 6382, in set_analysis
self._get_pha_data(id).set_analysis(quantity, type, factor)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/data.py", line 836, in set_analysis
raise DataErr('noinstr', self.name)
sherpa.utils.err.DataErr: No instrument model found for dataset /home/duy/chandra_clusters/test/cc/acb/4215_210.pi
###########
from clusterpyxt.
Related Issues (20)
- Refactor pipeline to fully integrate the Cluster obj class
- Refactor pipeline to better reflect the pipeline overview HOT 1
- Fully comment the code
- Implement unit tests with a known good cluster/obsids
- Errors occur at the beginning of preparing to merge the observations. HOT 7
- Add screenshots to README.md
- Clusters with many observations (~10 or more) causes significant slow down in the pipeline
- Error when creating temperature map. Variable referenced before assignment.
- Using the "low level" Sherpa API to fit PHA spectra HOT 2
- Error: pget() Parameter not found HOT 2
- AttributeError while running the spectral fitting portion of the pipeline in serial/parallel
- ModuleNotFoundError: No module named 'pychips' while running cluster.py
- Blank temperature and pressure map
- Unable to generate Temperature and pressure maps.
- CIAO4.14 HOT 2
- error in stage 2 HOT 3
- Pipeline freezes during Stage 1 download HOT 14
- Parallel fitting appears to have a memory leak HOT 1
- Allow for user selected model and which parameters are fixed/fitted
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clusterpyxt.