Giter Site home page Giter Site logo

norm map & free paremater about clusterpyxt HOT 16 CLOSED

bcalden avatar bcalden commented on June 14, 2024
norm map & free paremater

from clusterpyxt.

Comments (16)

bcalden avatar bcalden commented on June 14, 2024 1

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

Thanks for the quick reply.

On the UI/ease of use, I find that the code is very easy to use and quite robust. Especially in the first times I used. I like the guiding text after each steps: very clear. After being familiar with the pipeline, I prefer running it with command lines rather than with GUI (so I add --continue when possible). It is more convenient to me. I understand this is really personal references. Other users might prefer GUI. But I think many users will like following additional features:

  1. the inputs for all steps are saved in a parset file by user at the beginning of run. This is very helpful when working on large number of clusters.
  2. automatic identification and removal of compact sources, in addition to compact sources defined by users (in case automatic pipeline does not work for some sources).
  3. automatic generation of regions files (acisI_region_0.reg and master_crop-ciaowcs.reg)
  4. automatic search for nH given the build-in databases.
  5. and probably comments on some important files generated by the pipeline and what corrections/processes have been done to them.

Sorry for too many requests. They are not crucial as I still can run the pipeline at its current stage. But with new features, much more can be done.

Thanks,
Duy

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

Thanks for the update! I have tried to run the branch ClusterPyXT-dev-CIAO-4.12. I got the error below. Do I miss something for the new version to work?

################
$ python clusterpyxt.py
File "clusterpyxt.py", line 130
region_file_label = QtWidgets.QLabel(f"A region file (e.g. {self.observations[0].acisI_region_0_filename}) containing\n"
^
SyntaxError: invalid syntax
################

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

I am using Python 3.5.4 that goes with Ciao-4.12 installation. The os is Ubuntu 18.04. After getting the new code, I still have the same error.

$ python clusterpyxt.py
File "clusterpyxt.py", line 133
While any size upto the full CCD may be used, a region larger than 60 arc seconds is generally not necessary.""")
^
SyntaxError: invalid syntax

Cheers,
Duy

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

Alright, the problem is that most of the strings are formatted for python 3.6 and up. I will rewrite these so it is compatible with python 3.5 again in ciao 4.12. I am at a conference the beginning of the week but should be able to complete the rewrite by the end of the week.

If that is too long as you are under a time crunch, let me know. Alternatively, upgrading to python 3.6+ in a different conda environment will also work (although may not be desirable).

Brian

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

The strings are updated and everything should work with python 3.5.

I've tested on a Mac with python 3.7.5 but the strings are formatted and should work with python 3.5.

I had some issues with the test VM today but I should have it tested on ubuntu with python 3.5 by Friday.

Brian

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

Thanks Brian for the fix. I have reinstalled Ciao 4.12 with Python 3.7.7 (installing via miniconda3, https://cxc.cfa.harvard.edu/ciao/download/conda.html) and updated ClusterPyXT-dev-CIAO-4.12. The installation of Ciao passed all smoke tests. The problem with the strings seems to be fixed.

But there is a new error (pget() Parameter not found). This seems to be related to Ciao. I follow this (https://cxc.cfa.harvard.edu/ciao/faq/pget_error.html) to remove files in ~/cxcds_param4/*. But the error still remains. Having checked the par file in the temporary directory (/tmp/tmp5i9h52qe.dmkeypar.par), but it is not there. It is not clear to me if this temporary file is created and then deleted when the pipeline crashed. Or the file was not created at all. Any ideas on how to fix this error?

#########
Found background at /home/duy/soft/miniconda3/envs/ciao-4.12/CALDB/data/chandra/acis/bkgrnd/acis1iD2000-12-01bkgrnd_ctiN0005.fits
pset: cannot convert parameter value : rval
/tmp/tmp5i9h52qe.dmkeypar.par: cannot convert parameter value : rval
Traceback (most recent call last):
File "clusterpyxt.py", line 496, in run_stage_1
ciao.run_stage_1(self._cluster_obj)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 1306, in run_stage_1
merge_observations(cluster)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 478, in merge_observations
ciao_back(cluster)
File "/home/duy/soft/ClusterPyXT-dev-CIAO-4.12/ciao.py", line 184, in ciao_back
echo=True)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1810, in call
stackfiles = self._update_parfile(parfile)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1365, in _update_parfile
self._update_parfile_verify(parfile, stackfiles)
File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/ciao_contrib/runtool.py", line 1294, in _update_parfile_verify
oval = _to_python(ptype, pio.pget(fp, oname))
ValueError: pget() Parameter not found
Aborted (core dumped)
########

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

update: if I run the command (on line 184 of ciao.py) in Python 3.7.7 separately (not with ClusterPyXT-dev-CIAO-4.12), no error occurs.

#######

from ciao_contrib import runtool as rt
acis_file='acisf04961_repro_evt2.fits'
rt.dmkeypar(infile=acis_file, keyword="GAINFILE", echo=True)
acisD2000-01-29gain_ctiN0008.fits
########

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

On my Mac OS, I can run the ClusterPyXT-dev-CIAO-4.12 version without any problem. This has Ciao-4.12 + Python 3.7.7 installed with miniconda. The only problem is that it takes lots of time when doing spectral fitting. So, it's not an ideal solution.

With the same installation on my Ubuntu 18 machine (Ciao-4.12+Python 3.7.7+miniconda), I had problems with parameter error (ValueError: pget() Parameter not found) when running dmkeypar (like in my earlier post). I added a punlearn command, rt.dmkeypar.punlearn(), before dmkeypar lines. This seems to fix the error for the dmkeypar lines that do not have keyword=TSTART or keyword=TSTOP. If I replace the keyword TSTART or TSTOP with other keywords (e.g. OBS_ID, TIMEZERO) found in the header, dmkeypar works just fine.

If I run this command, rt.dmkeypar(infile=data_hiE, keyword="TSTART", echo=True), outside of the pipeline (i.e. in python 3.7.7 with Ciao 4.12 in the environment), no error appears.

I also see the same behaviors for some other commands dmextract, deflare. They work outside of the ClusterPyXT-dev-CIAO-4.12, but do not work with the pipeline.

You have any idea why that is?

from clusterpyxt.

bcalden avatar bcalden commented on June 14, 2024

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

Yes, the the problem happens every run on Ubuntu. A quick trick I did was to create a python function that executes the function, but passing the arguments to bash shell. To call the function, simply replace, e.g., rt.dmextract by run_dmextract. The arguments are kept the same. Then, the pipeline works fine for these steps (without error on "Parameter not found"). Below are the python functions.

def run_dmextract(infile, outfile, bkg=None, error=None, bkgerror=None, bkgnorm=None, exp=None,bkgexp=None, sys_err=None, opt=None, defaults=None, wmap=None, clobber=None, verbose=None):

cmd = 'dmextract infile="%s" outfile="%s"' %(infile, outfile)
if bkg:
    cmd += ' bkg=%s ' %bkg
if error:
    cmd += ' error=%s ' %error
if bkgerror:
    cmd += ' bkgerror=%s ' %bkgerror
if bkgnorm:
    cmd += ' bkgnorm=%s' %bkgnorm
if exp:
    cmd += ' exp=%s' %exp
if bkgexp:
    cmd += ' bkgexp=%s' %bkgexp
if sys_err:
    cmd += ' sys_err=%s ' %sys_err
if opt:
    cmd += ' opt=%s ' %opt
if defaults:
    cmd += ' defaults=%s ' %defaults
if wmap:
    cmd += ' wmap=%s ' %wmap
if clobber:
    cmd += ' clobber=yes '
if verbose:
    cmd += ' verbose=%s ' %verbose

print ('Running %s' %cmd)
os.system(cmd)

return

def run_deflare(infile, outfile, method, save):

cmd = 'deflare infile="%s" outfile="%s" method=%s save=%s' %(infile, outfile, method, save)

print ('Running %s' %cmd)
os.system(cmd)

return

def run_dmhedit(infile, filelist, operation, key, value, datatype=None, unit=None, comment=None, verbose=None):

cmd = 'dmhedit infile="%s" filelist="%s" operation=%s key=%s value=%s ' %(infile, filelist, operation, key, value)
if datatype:
    cmd += ' datatype=%s ' %datatype
if unit:
    cmd += ' unit=%s ' %unit
if comment:
    cmd += ' comment=%s ' %comment
if verbose:
    cmd += ' verbose=%s ' %verbose

print ('Running %s' %cmd)
os.system(cmd)

return

##############

from clusterpyxt.

duyhoang-astro avatar duyhoang-astro commented on June 14, 2024

With the fix above, the code runs through all steps to Spectral Fitting. Here I encounter another error on instrument model not being found in the pi file. In the pi header there is keyword INSTRUME. Is this the info that the pipeline is reading? Any tips on how to fix this would be very helpful (sorry for borthering you so much on this)!

A copy of the pi header is here: https://www.dropbox.com/s/rtn7sl5f4fbgj3r/header.txt?dl=0
The error on the screen is copied below.

Duy

###########
Processing region number: 207
207: Working on observation id: 4215
Processing region number: 210
210: Working on observation id: 4215
210: Met S/N threshold for 4215
210: Extracting PI files for 4215
207: Met S/N threshold for 4215
207: Extracting PI files for 4215
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/acisI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_210_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/acisI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_207_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/backI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_207_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi" clobber=yes
Running dmextract infile="/home/duy/chandra_clusters/test/cc/acb/backI_clean_4215.fits[sky=region(/home/duy/chandra_clusters/test/cc/acb/temp_210_4215.reg)][bin pi]" outfile="/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi" clobber=yes
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=EXPOSURE value=64220.940691801
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=EXPOSURE value=64220.940691801
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=RESPFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.rmf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=RESPFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.rmf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=ANCRFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.arf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=ANCRFILE value='/home/duy/chandra_clusters/test/cc/acb/cc_4215.arf'
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_207.pi" filelist="" operation=add key=BACKFILE value=/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_210.pi" filelist="" operation=add key=BACKFILE value=/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_back_207.pi" filelist="" operation=add key=EXPOSURE value=1717423.024726406
Running dmhedit infile="/home/duy/chandra_clusters/test/cc/acb/4215_back_210.pi" filelist="" operation=add key=EXPOSURE value=1717423.024726406
207: Loading data pulse invariant files (PI files)

  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[SPECTRUM]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[SPECTRUM]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI1]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI3]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934342,3214: 187001519,1142
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI2]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2214: 187001519,1552
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_207.pi[GTI0]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934336,1625: 187001519,2373
    207: Loading background PI files
    210: Loading data pulse invariant files (PI files)
    Process Process-19:
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[SPECTRUM]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[SPECTRUM]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI1]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2625: 187001519,1963
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI3]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934342,3214: 187001519,1142
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI2]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934339,2214: 187001519,1552
  • WARNING: TSTART and TSTOP do not match GTIs in /home/duy/chandra_clusters/test/cc/acb/4215_210.pi[GTI0]
  • TSTART:TSTOP = 1,0000: 1,0000
  • GTIs = 186934336,1625: 187001519,2373
    Traceback (most recent call last):
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/hs/fs06/data2/AG_Brueggen/duy/soft/ClusterPyXT-dev-CIAO-4.12/cluster.py", line 1919, in fit_region_number
    sherpa.set_analysis('energy')
    File "", line 1, in set_analysis
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/ui/utils.py", line 6382, in set_analysis
    self._get_pha_data(id).set_analysis(quantity, type, factor)
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/data.py", line 836, in set_analysis
    raise DataErr('noinstr', self.name)
    sherpa.utils.err.DataErr: No instrument model found for dataset /home/duy/chandra_clusters/test/cc/acb/4215_207.pi
    210: Loading background PI files
    Process Process-20:
    Traceback (most recent call last):
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/hs/fs06/data2/AG_Brueggen/duy/soft/ClusterPyXT-dev-CIAO-4.12/cluster.py", line 1919, in fit_region_number
    sherpa.set_analysis('energy')
    File "", line 1, in set_analysis
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/ui/utils.py", line 6382, in set_analysis
    self._get_pha_data(id).set_analysis(quantity, type, factor)
    File "/home/duy/soft/miniconda3/envs/ciao-4.12/lib/python3.7/site-packages/sherpa/astro/data.py", line 836, in set_analysis
    raise DataErr('noinstr', self.name)
    sherpa.utils.err.DataErr: No instrument model found for dataset /home/duy/chandra_clusters/test/cc/acb/4215_210.pi
    ###########

from clusterpyxt.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.