ossos/mop

The Moving Object Pipeline: discovery & tracking of trans-Neptunian objects

OSSOS

The public website for OSSOS.

mop's Issues

add a TAP query to populate_ossuary to populate iq_elixir

so the homebrew IQ and the elixir IQ can be compared if there's any concern over validation philosophies.

FOV of image too small

The fov should be twice as big as is currently used to allow for fast movers. Use the middle exposure for centroid would solve this and be better than current too.

Verify apcor parameters being passed to photometry calculations

@ijiraq I found a note to myself to check with you about the apcor parameters being passed to daophot.py. Maybe you can take a quick look, specifically at the swidth property of the ApcorData class in https://github.com/drusk/MOP/blob/master/src/ossos-pipeline/ossos/gui/image.py

I did it this way based on our e-mail conversations while you were away, in which you said apcor files have the following format:

ap_in ap_out apcor apcor_err
numbers like
4 10 0.4 0.05
are normal.

ap == ap_in
insky == ap_out+1
outsky == insky + ap_in

You then also said:

swidth = outsky - insky (if you look through the dophot.pl script there is likely a line someplace that says: outsky = ousky - insky ... poor choice of variable names).

Therefore I calculated that swidth = insky + ap_in - insky = ap_in.
But then swidth == aperature ❓ (both get passed to daophot)

Did I misinterpret something? If you can do a quick review of this it would be good.

Accept real button error

In the process reals task, accepting via the keyboard shortcut works fine, but using the button causes the following error:

ERROR: AttributeError: 'ProcessRealsController' object has no attribute 'on_initiate_accept' [pymop.gui.views]
ERROR:astropy:AttributeError: 'ProcessRealsController' object has no attribute 'on_initiate_accept'
Traceback (most recent call last):
File "/home/drusk/gitcadc/MOP/src/ossos-pipeline/pymop/pymop/gui/views.py", line 459, in _on_click_accept
self.controller.on_initiate_accept()
AttributeError: 'ProcessRealsController' object has no attribute 'on_initiate_accept'

return counter of candidates remaining

Just realised commit [ea62775] removed the counter of how many candidates are left to check. This was quite psychologically nice as it gave an indication of rate of progress/how long until field completion. It should be the total count of real candidates + fake candidates, to avoid any bias.

pymop doesn't understand fk images?

seams that the naming of image cutout is incorrect for fk images in cands.comb.astrom file as load by pymop

Handle empty workload gracefully

When constructing the workload of astrom files to process, if there are no astrom files in the selected directory or they are completed, a ValueError gets raised saying empty workload. The user should just get a nice prompt that there is nothing to process before exiting the program.

Blink backwards

Tab key blinks forward. Shift + Tab should blink backwards.

Lock working directory

Following up on #8, it will become necessary to restrict processing of a directory to one user at a time to avoid stomping on each other's work.

While a task is active, create a .pymop.lock file containing the user's name. The application can then check for the presence of this file on startup and abort if someone already has the lock.

Focus on note 1

For the process reals task, note 1 is the most likely field for the user to modify. It should obtain focus, and it should be possible to use the keyboard to enter the value as well.

Candidates from real list and from fake list are not being randomly interspersed

Candidates both real and fake need to be presented to the user in a random order to avoid obvious bias. At the moment it is pretty obvious when a file of fake candidates starts.

One colormap per source

Adjusting the bias/contrast should apply to the colormap for each image of a source, so that they are consistent as you blink through.

The users should be able to undo a previously accepted/rejected source.

Sometimes during candidate review you get excited and tap 'r' or 'a' before your brain stops you. Should be able to 'u' (undo) that step.. (perhaps 'crtr-u' so that we don't accidentally type 'u' too often).

Don't re-download already processed images

If you process half the images in a file, quite then restart, you will have to wait while the first images are downloaded again. Change the downloader to take into account partial progress so it doesn't do unnecessary downloads.

Phot error with old versions of IRAF

The magfile for output must not already exist when the phot task is run in older versions of IRAF (it will complain it can't overwrite it). Should test to make sure this works on the SL5.5 machine.

Process reals task: indicate already processed

For the process reals task you can keep blinking through a source's observations until you have processed all of them, and it is confusing which you have already accepted or rejected. Disabling the accept and reject buttons might be a simple way to address this.

Discovery asterisk when reloading

The mechanism for checking if we need the discovery asterisk currently doesn't take into account that the file may have already been partially processed and is being resumed. I.e. if someone processes 1 observation of a source, quits, and someone else loads it up, the discovery asterisk gets written a second time.

Option to launch task directly from command line without task setup UI

Users who know what they are doing might not want to go through the task setup UI. Provide command line options to make the selections.

For instance, something like:
python mop.py --task=reals --dir=./test/data

Record candidate rejection

Currently in the process reals task if an object is rejected then no MPC line is written. This should be changed to write out a line with the first character replaced with a '!'

crash while processing reals

When I selected 'a' to accept the object the dialogue appears but then for the next 'a' I get this message. and crash.

RROR:astropy:IndexError: list index out of range
Traceback (most recent call last):
File "/Users/jjk/gitRepos/MOP/src/ossos-pipeline/ossos/gui/views.py", line 281, in on_accept_src_keybind
self.controller.on_accept()
File "/Users/jjk/gitRepos/MOP/src/ossos-pipeline/ossos/gui/controllers.py", line 100, in on_accept
self.model.get_current_source_observed_magnitude(),
File "/Users/jjk/gitRepos/MOP/src/ossos-pipeline/ossos/gui/models.py", line 179, in get_current_source_observed_magnitude
return self.get_current_image().get_observed_magnitude(x, y, maxcount=maxcount)
File "/Users/jjk/gitRepos/MOP/src/ossos-pipeline/ossos/gui/image.py", line 116, in get_observed_magnitude
maxcount=maxcount)
File "/Users/jjk/gitRepos/MOP/src/ossos-pipeline/ossos/daophot.py", line 152, in phot_mag
return hdu["data"]["MAG"][0]
IndexError: list index out of range

COMMENT box in accept/reject dialog needs to be bigger.

Sometimes I have a lot to say.

Process reals task: accept/reject on last obs moves you to next source

It should instead wrap back to the first observation of that group, and only move to the next source when all observations of the current source have been accepted or rejected.

The UI draws multiple windows: all should move together when minimising

Minimising the main Moving Object Pipeline window with the blinking images displayed in it reveals another teeny and featureless window (OSX 10.7.5). It would be nice if their movement, ie. minimisation, was synched.

Pubsub compatibility

On the SL5.5 machine with an older version of wxpython, the setupv1 import causes an error. Need to settle on a robust way to provide consistent pubsub behaviour on both old and new versions of wxpython. Perhaps try/except blocks around the import can be used.

The FWHM etc. values should be vtag properties on the dbimages directory.

add fwhm/apcor/zeropoint as vtags is mkpsf.py

Exiting program via clicking on close window handle causes error...

(Canopy 64bit) mach87:pymop jjk$ python mop.py
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Applications/Canopy.app/appdata/canopy-1.0.0.1160.macosx-x86_64/Canopy.app/Contents/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/io/imgaccess.py", line 85, in run
fitsimage, reading, source_num, obs_num)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/io/imgaccess.py", line 50, in on_image_downloaded
self.image_loaded_callback(source_num, obs_num)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/gui/models.py", line 206, in _on_image_loaded
pub.sendMessage(MSG_IMG_LOADED, (source_num, obs_num))
File "/Users/jjk/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/wx/lib/pubsub/pubsub1/pub.py", line 750, in sendMessage
self.__topicTree.sendMessage(aTopic, message, onTopicNeverCreated)
File "/Users/jjk/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/wx/lib/pubsub/pubsub1/pub.py", line 423, in sendMessage
deliveryCount += node.sendMessage(message)
File "/Users/jjk/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/wx/lib/pubsub/pubsub1/pub.py", line 261, in sendMessage
listener(message)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/gui/controllers.py", line 55, in on_image_loaded
self.model.get_total_image_count())
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/gui/views.py", line 30, in new_guithread_function
if wx.Thread_IsMain():
File "/Users/jjk/Library/Enthought/Canopy_64bit/System/lib/python2.7/site-packages/wx/_misc.py", line 655, in Thread_IsMain
return misc.Thread_IsMain(*args)
PyNoAppError: The wx.App object must be created first!

Save progress

Implement a mechanism for saving the user's progress so they can resume analyzing the .astrom files in a directory without starting over.

Possible way to implement:
write to a log file (ex: .pymop.log) indicating which files have been completed, and which have been partially processed. For the partially processed ones note indices of items that were processed.
JSON might be a convenient way to implement the log file, as it is very easy to parse and write from a Python dictionary.

Jumps to second observation when going to next workunit

The application skips to looking at the second observation of the source when moving to a new workunit (file).

Reject doesn't flush MPC

For the reals task, rejecting doesn't check whether it should flush the MPC lines it has buffered (it should be checking and flushing when the source is finished).

Losing internet connection makes the application hang

If the connection is lost, validate.py hangs with the log (below):

Interestingly, the 'Readings' window still responds to tabbing, but the image does not change.

Exception in thread Thread-2:
Traceback (most recent call last):
File "/Applications/Canopy.app/appdata/canopy-1.0.0.1160.macosx-x86_64/Canopy.app/Contents/lib/python2.7/threading.py", line 551, in bootstrap_inner
self.run()
File "/Users/michele/Dropbox/ossos-pipeline/src/ossos-pipeline/ossos/gui/downloads.py", line 74, in run
downloaded_image = self.downloader.download(reading)
File "/Users/michele/Dropbox/ossos-pipeline/src/ossos-pipeline/ossos/gui/downloads.py", line 195, in download
apcor_str = self._download_apcor_file(apcor_uri)
File "/Users/michele/Dropbox/ossos-pipeline/src/ossos-pipeline/ossos/gui/downloads.py", line 173, in _download_apcor_file
vofile = self.vosclient.open(uri, view="data")
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 1227, in open
URL = self.getNodeURL(uri, method=method, view=view, limit=limit, nextURI=nextURI, cutout=cutout)
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 993, in getNodeURL
return self._get(uri)
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 1104, in _get
return self.transfer(uri, "pullFromVoSpace")
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 1125, in transfer
con = VOFile(transURL, self.conn, method="GET", followRedirect=True)
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 616, in __init
self.open(URL, method)
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 689, in open
self.httpCon = self.connector.getConnection(URL)
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/vos.py", line 131, in getConnection
raise ex
IOError: [Errno 61] _ssl.c:489: The handshake operation timed out: 'www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca'

VOSpace can be slow, we should use the 'short_cut' system built into vos

When making a vos client connection the option cadc_short_cut=True will make the connection speeds quick (skips some VOSpace standard overhead). This will make pulls happen at x2 the speed as currently.

vospace = vos.Client(cadc_short_cut=True)

Colormap should start out 'reversed' so that black is high and white is low.

Align cutouts to keep background steady

Currently cutouts are taken centered at X0, Y0. This results in the background moving, which is distracting. The background should stay constant, and the object should move.

All images should show all locations

The location of the source in all exposures should appear marked on all images. But the colour used should indicate if this is the in exposure position or the next/previous image.

AstromParser doesn't seam to read my .astrom file?

I tried to get AstromParse to read the file:

vos:OSSOS/measure3/fk1616682s00.measure3.cands.astrom

but it said:

IndexError: list index out of range

ast = astrom.AstromParser().parse('fk1616682s00.measure3.cands.astrom')
Traceback (most recent call last):
File "", line 1, in
File "/home/jkavelaars/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/ossos-0.1-py2.7.egg/pymop/io/astrom.py", line 182, in parse
self._parse_observation_headers(filestr, observations)
File "/home/jkavelaars/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/ossos-0.1-py2.7.egg/pymop/io/astrom.py", line 109, in _parse_observation_headers
obs = observations[obsnum]
IndexError: list index out of range

Don't download apcor for candidates task

Apcor files are for the photometry calculations, which are only done when processing reals. Downloading them when processing candidates just unnecessarily slows things down.

Testing on Mac

Test out installing and running the application on Macs.

Add capability to load full image

Sometimes seeing the full image would be ver diagnostic.

App doesn't start up unless in debug mode.

Running python mop.py just returns without an error if debug mode is set to false.

Candidate not always in field of view

Sometimes candidates are entirely off the cutout.
eg. one on images 1616682 1616693 1616704.

(This also showed that values can't be copied as normal text out of the Readings or Observation Header tables).

[MOP] Dialogue box when certificate expired

@drusk: If the user's VOSpace access certificate is expired, the Exception should be raised to a dialog box, or should trigger retrieval of a new certificate.

VOSpace working directory

Allow the selected working directory to be either on the local file system or in VOSpace.

URL parsing

init methods in views classes need to use urllib to parse rather than the quick-and-dirty string parsing currently in place.

Cutout's near edges not quite right..

Looking at detections listed in

vos:OSSOS/measure3/fk1616682s24.measure3.cands.astrom

one of the candidates is at 810,64 and the cutout doesn't include the source... looks like the bottom of the image has been cut-off?

opening for 'reals' doesn't work, some validation fails...

(Canopy 64bit) mach87:pymop jjk$ python mop.py
ERROR: AssertionError: Source doesn't have same number of observations (1) as in observations list (3). [pymop.io.astrom]
Traceback (most recent call last):
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/gui/taskselect.py", line 65, in _on_ok
self.setup_manager.set_task_info(self.browser.GetValue(), task)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/gui/taskselect.py", line 86, in set_task_info
self.app.set_task_info(working_directory, task)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/app.py", line 107, in set_task_info
self.launch(working_directory, task)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/app.py", line 117, in launch
self.start_task(working_directory, task)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/app.py", line 128, in start_task
self.task.start(working_directory)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/app.py", line 50, in start
workload = AstromWorkload(working_directory, workload_files)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/io/astrom.py", line 306, in init
self.astrom_data_list = [parser.parse(filename) for filename in self.full_paths]
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/io/astrom.py", line 183, in parse
sources = self._parse_source_data(filestr, observations)
File "/Users/jjk/Develop/MOP/src/ossos-pipeline/pymop/pymop/io/astrom.py", line 136, in _parse_source_data
len(source_obs), len(observations))
AssertionError: Source doesn't have same number of observations (1) as in observations list (3).

There should be a 'reset colormap' keystroke, setting the color map back to default levels

Freeze on exit reals processing

After processing the last source for the reals task and clicking yes on the prompt to exit the program, things lock up. This does not happen for the candidates task.

seams image loads are blocking until all images retrieved.

use vos:OSSOS/measure3.tgz file as input dir for validate.py to reproduce this problem...

Centre circle on click

If the user clicks without dragging in the matplotlib image viewer, the circle should be centered there. This should happen whether or not the click is inside or outside the circle.

Contextual comments are needed for each MPC line.

The real.mpc file should have an comment for each line. For various format reasons that comment should be the line before the line that is being commented on. Comments have format of

expnum_ccd xcen ycen other text

The other text part is user edited strings. the expnum_ccd xcen ycen part is take from the application. All line should have the default comment line:

1234567p00 334.56 884.22