so-artn/artn

Arizona Robotic Telescope Network

ARTN

Arizona Robotic Telescope Network

Documents, operating notes, requirements, observing scripts, etc for the Arizona Robotic Telescope Network - queue scheduling , remote observing, and automation of UAO telescopes.

ARTN Weekly Meeting Notes and Agenda

Observing Run Notes and Draft Documentation

Possibly old and deprecated information below this line. Ignore unless you are in trouble!

###Usefull links

Scott's RTS2 Forck

Useful links:

RTS2 homepage

AZCam Documentation

Where does RTS2 put data

The path to the image data is defined in the rts2.ini file found in /etc/rts2/ directory. Currently you will see

Currently you will see these definitions in the file:

; %b expansion
base_path = "/home/rts2obs/rts2images"

; Images are stored on this path before they are processed.
que_path = "%b/queue/%N/%c/%t/%f"

; The following paths specify where to put images after processing. Every image is first placed
; in queue, and then renamed to target location based on image type, result of astrometry,..
;
; If the following paths are empty/not set, then the images will be not be renamed.
;
; Path for acqusition images
acq_path = "%b/%N/acq/%f"a
; Path for good images - images that were matched with the catalogue
archive_path = "%b/%N/archive/%t/%f"
; Path for bad images - images without match with catalogue.
trash_path = "%b/%N/trash/%t/%f"
; Path for raw skyflats.
flat_path = "%b/%N/skyflats/%f"
; Path for raw darks
dark_path = "%b/%N/darks/%f"

For the most part this means that objects in a queue will be stored in /home/rts2obs/rts2images/queue//C0/<object_id>/<unique_name>

artn's Issues

Punch a hole in azcam firewall for bigpop

RTS2 webapp unavailable after gerard crashed

after rebooting gerard after its crash and restarting the RTS2 stuff there, we could not access bigartn:1080, the webapp server port. tried to telnet to port 1080 while logged into bigartn, but it said connection refused. something was either locked up or crashed and the rts2obs account did not have the privileges necessary to restart it.

Suggestions for AORP page

Monday, Sept 30th was a night full of potential host galaxies of a GW event. This meant that there were 90+ targets in addition to SN/LBV targets. Trying to manage this queue in combination with the non-infrequent system failures led to a hectic night, with subpar efficiency. The multiple kernel panics were a pain, but in this note I mention ideas to improve the utility of the AORP.

1. When a target is sent to the queue (the eye icon), the page returns to default settings. This means that if I have navigated to a page with 3 targets I want to load.....the webpage jumps back to page one/default sorting after each selection. When 90 targets had to be selected, with many on page 9 or 11 or 12.....this led to a LOT of additional clicking. It would be good to preserve the sorting and page number after selecting a target. There can be a default (or reset) button to get back to default.
2. When one uses the upper "next" button to page to the next page, going from page 1 -> 2 leads to a shift....which puts the mouse right over the "logout" button. It makes it too easy to accidentally logout when having to repeatedly (50 times) click from page 1 -> 9 . It would be good to move the logout. button to not be above the "next" button, or to prevent the page jumping when next is clicked.
3. It would be wonderful for the list of records to show if the target is currently in the queue, maybe also showing if it has been observed this session. With a very long list, it is rough to have to keep switching tabs to the web-app page to see if an individual target has been loaded already. Remember that there are 13 pages of targets now and 100 targets were to be loaded tonight. Perhaps there can be a column with a letter or icon showing whether the target is already loaded and/or has been observed.

Stuck taking darks at beginning of the night

Scott seems to know what is going on here.

Slewing to DEC=+60:11:34

SN 2017eaw is at DEC=+60:11:34.9
One plan attempted to slew to that position, but ended up at +58:30:11.0
The sequence of 8 images ran with the TCS monitor showing "DEC" and "S1"
blinking yellow (presumably because it didn't get to +60. The plan ran again
(probably a mix-up loading plans), and the second time the telescope moved to the
correct position. This probably just points out a grey zone between the various
limits, but as this showed...it can happen that we are right at the limits where RTS2
accepts the target, but the telescope doesn't quite get there.

Use wiki for how-to's. Scott will start with a 'start-up' wiki.

Scott will write a 'how to start rts2' wiki page that merges Ben's notes with some of his updated scripts, etc.

Object names in header

Communicate target names from the RTS2 queue to image header writing so the images get that name in the header field rather than "RTS2."

CRVAL1 and CRVAL2 are now being set to 0

The correct RA and Dec are being populated in the RA and DEC headers, but are not being populated into the WCS headers CRVAL1 and CRVAL2.

Moon Angle Constraints for Targets

Is it possible to add moon angle constraints for input targets? A default value should be something like >25 deg, but maybe this can be user specified as well.

More monitors for ARTN

There are many windows needed to monitor ARTN+RTS2, we need more monitors.

Escalate this!

Clean up FITS template

Take a close look at the currently used RTS2/ARTN fits template. What should go in there and be removed?

Kuiper did not move to requested target

BIG61 moved to 14:43:15.230 +056:03:24.30 requested 14:42:42.020 +056:08:10.24 target 09:39:10.670 +032:14:12.10

This happened for SN2019bie on night of 4/17/2019

Figure out the basis of the fits names spit out by rts2

Right now, RTS2 naming scheme for fits files is a bit of a mystery. here are some examples:

20171204001224-655-RA.fits
20171204005254-444-RA.fits

It is clear the first part is the date. Less clear what '001224' or '655' or even what 'RA' are. This will require interrogation of Petr, unless someone else knows?

Next level scheduling test -- Observe an object on a 3 hour cadence

We would like to push on the scheduling abilities of rts2 -- Dave Sand will look into observing objects with a cadence tonight. One example would be 'Observe this object for 2x60s in V and 2x60s in R roughly every 3 hours'.

Astroplan -- seems useful for planning observations

https://arxiv.org/pdf/1712.09631.pdf

This paper above came out on astroph tonight. Seems like it has some useful functions for planning observations (including things like figuring out when the next transit of an exoplanet is, etc). It may be good to incorporate in our 'brain' or higher-level scheduler.

Sorry, this is not an issue. More of an FYI.

Update documentation for automated dome and mirror cover buttons

Write Script that calculates focus from focus-sequence images

binning and region of interest

There seem to be some issues with the binning and ROI code implemented by Scott. Good thing we got on sky long enough to test it out. Example image screenshot attached.

Get Scott's astrometry.net code on scopenet

181219: repeating first target in queue

sn2018ilu was observed as the first target, but then accepted as the second target as well. Only in queue once.

Flat fielding script did not execute

[I'm copying an email I sent to Petr + Scott here]
1st email:
Hey guys --
RTS2 and the telescope started up like a dream tonight. It even went to try to do a set of flats (/etc/rts2/flat.py), but it is stuck, and not taking any data. Attached is a screenshot I took with my phone. You can zoom in and see what is going on. (see img_1311)

2nd email:
It sat there for about 10 min, and now it seems to be looking for targets. See new picture. (see img_1312)

Contact Emilio about RTS2 use with Kepler

Fits template file doesn't seem to work

According to rts2.template man page, you can create a fits template file and populate the fits headers with various rts2 values, basically anything in the rts2-mon left hand column can be put in the fits header. I tried this and it didn't seem to work.

Here is what I did:

• created a file called /etc/rts2/fitstemplate.ini with the lines:
  [PRIMARY]
FOCUS = @F0.FOC_POS / Focus postion provided by RTS2

• added the following line to the rts2.ini file:
  template = "/etc/rts2/fitstemplate.ini

• restarted rts2 and took a dark

FOCUS was not added to the fits header.

Easy creation of observing scripts

We need an interface that will allow users to easily specify an observation without them needing to know the RTS2 script format.

I made a step in this direction with the script make_newobs.py, which will read a simple text catalog that specifies 1 target per line with ra, dec, desired filters, exposures and repeats. It can also be run to type the numbers in interactively. They still need to give numbers for the filters rather than names.

To do: figure out how to do a dither pattern. and to translate filter names to numbers. If the script can read a file of filter wheel names/numbers somewhere on the observing computer, that would help.

Does CAMTEMP header keyword ever work?

After a quick look at random images from the Fall 2017 ARTN runs, it looks like the CAMTEMP header keyword is always set to -999.9.

Is this always the case? If so, we should get help from Lesser.

Redesign rts2 homepage

Current version does not resize well when window is changed. We need to make it functional even on the smallest window size because not all of us have two 32" monitors for observing.

Tech requirements / long term plan

We need a set of requirements / plan for how we want queue/remote/robotic observing at the 61" to work, both long term and semester by semester. For example:

By end of fall 2017:

• will be able to observe a full night of queue with RTS2 scheduling on sidereal targets
• Adding targets and desired exposures through a user-friendly interface, ideally both 1) web interface or similar for casual user; 2) scriptable for users who want to add many targets.
• autofocus / flats / standards / etc
• pipeline and data quality measurement
  ....

Long term:

• safety plan and audit
• robotic operation
• compatibility with a network manager / Big Brother
• non-sidereal targets
• etc

Disable RTS2 from going into database to observe targets not in queue

As Petr just mentioned at the meeting today, there is a way to make sure that RTS2 never goes into the database to observe objects when they are not in the queue. We should make this option a permanent part of our RTS2 initialization.

Get AzCam to start remotely without the need for VNC.

This would be very useful if we are to enable remote observing. Right now we need to vnc to bigccd and manually open AzCam. It should just startup with ARTN scripts.

Make Ben's m4k pipeline RTS2 friendly

I put oscan and merge as seperate modules in a python package. See here

Retire Issues prior to June 2020?

Wonder if we should retire all of our issues prior to June 2020, or archive them? Just to clean up our issue list a bit and refocus our efforts on the tasks at hand.

Thoughts, @danavner? I wonder if there is a way to archive issues.

Please confirm that the right filter is ALWAYS as commanded and in the header

When interacting with RTS2.

Out of sync 'Close Dome' and 'Close Mirror Cover' buttons

I arrived for night 2 of an ARTN run, with the 'Start Night' button already up and running. However, the 'Close Dome' and 'Close Mirror Cover' buttons were up in red, even though both were already closed. The fix was to close the 'Start Night' button, and restart by pressing the 'Safe Telescope Buttons' icon.

Scott said he can resolve this, easy peasy.

Rewrite 'Observing with RTS2'

Dan's request:

Rewrite the "Observing with RTS2" to have a numbered procedure. Reading paragraphs as part of the procedure was more difficult than it needed to be. Also should rearrange that guide to have a more consistent flow.

galil.error

galil server error. permissions seemed to change abruptly on a galil related file, not allowing us to do our rts2 thing.

3x3 binning loses a column in the center of the mosaiced image

The way the binning is currently implemented, 3x3 binning is causing a column at the center of the image to be dropped due to number of pixels not being even factor of 3. This is obvious if there's a star straddling the boundary of the readout regions. One workaround is to switch to 4x4 binning as the ARTN default. This would mean 0.6" pixels which is getting to be undersampled under the best of conditions, but the majority of the time has IQ > 1.2". Being able to switch between 2x2 and 4x4 would be another option...

Fill in the wiki for a 'normal shutdown'

Filters need to be read before starting rts2 on first night.

Filter wheel labels not read correctly by rts2, but is read correctly by mont4k gui to do so.

update camera driver for new AzCam API

Mike Lesser is changing the camera API this summer. The new API is defined here. This will need to be done before the Fall observing runs and will likely take a few hours.

Are we losing darks?

Where does Petr stash darks?

Get github repos in order

This is almost two issues.

1. Right now there are at least two github repos that have ARTN-related material in them: mo-ops and this one, ARTN. Once could imagine the need for a third -- a Mont4k pipeline. Should we consolidate these two or three? Or have multiple repos, but very clear definitions for each. Lets continue discussions, at least for a little while.

2. There are several active repos, at least for mo-ops, and it is not always clear which is the 'active' one at the telescope. It is also true that the main branch code on the ARTN repo does not always seem to be in use at the telescope. We should have clear 'development' and 'in use' repos, or something similar. And stay up to date at the telescope.

Computation of limits and failure to slew to objects that are up

We were observing objects that were very far west. RTS2 continued to follow objects to airmass ~ 3, I am not sure if this is beyond the actual software limit horizon, but it was outside the horizon plotted in xephem. We should re-implement airmass limits per target so it doesn't observe that low anyway.

At about 3:57 AM it was pointed about 5 hours west, and should have slewed to a new target that was only 2.5 hours west, but did not, emitting an error message that the target was below the horizon. (Either its computation was flawed, or it skipped the target for some reason and then went to the next in queue, which actually was below the horizon.) However, it kept taking images at the old target and putting them in the new target's directory. I set rts2-mon to off, stowed the telescope, added the target back into the queue, and set rts2-mon back on. It was then able to slew to the target.

Scott will get Sand & Wyatt Focus set with notes

Camera and telescope drivers dropping out

Need to figure out why the AzCam and BIG61 drivers sometimes lose communication, and 1) have a way of detecting when this happens / issuing a warning; 2) friendly way of restarting driver ideally without stopping RTS2 and the queue; 3) fix it so it doesn't happen.

AzCam initialization error

This error tends to occur at the very beginning of the night when the first bias frame is attempted.

Azcam writes out a 'initialization error'.

Autologging exposures

As images are taken, write a human-readable log summarizing useful fields: image name, target name, filter, exptime, UT, RA, Dec, airmass. Ideally have some kind of autologger window for a user to write comments.

Autofocus routine

We need a routine that will take a series of frames stepping through the focus, and then an analysis script that evaluates the images (eg with astropy photutils?) to find the best focus value.

RTS2 should be able to change the focus with "focpos=2300", "focpos+=50" etc, but when Sam and I tried this, it did not work as expected.

rsync to scopenet

I believe I have a cronjob that will rsync the data to the proper place on scopent. @pndaly could you make sure this is correct.

Not taking appropriate flats at end of night

Not sure why, but ARTN seems to be taking zillions of 1 second flats at the end of the night in I-band. My understanding is that it should be rescaling these exposure times until it hits 20000 counts. Will have to look into this.

AzCam + RTS2 interaction issue coming out of 2020 shutdown

To quote from a Dan Avner email to Lesser:

We ran into a serious issue with how AzCam/RTS2 interact. After taking an image, rts2 would wait indefinitely for a response from AzCam saying that the image is done. The telescope would not slew to the next target or continue to take the next image.

Our next run isn't until the end of October, but I can head up any time before to work with you resolving the issue. Either we need to change the azcam driver on our end or if you made changes on feedback from azcam, revert back to the previous feedback.