User's Guide to IRCAL


Table of Contents


Introduction
Summary Table
Quick start
Instrument Characteristics
Software Overview
ircalui
ircaldisplay
soundserver
lickstrehl
Data File Info
Filters, Grisms, Polarimeter
Nodding and Dithering Scripts
Observing Procedures and Tips
Trouble-Shooting
Technical Details
Logsheet

Mt. Hamilton Homepage

Trouble-Shooting

General Tip
No signal on array
Exposure Fail error message
Excess Noise
Exposure Palette is empty
Motor status is string of unusual characters
Image Browser won't display new images
Motor Communication TimeOut
Soundserver doesn't work or hangs
NO_DELIVERY errors


General Tip

Most software problems and hardware problems (if everything has been up and running already during the observing run) can be solved by exiting ircalui and restarting the software. Restarting the software not only reinitializes the software parameters, but also reinitializes much of the hardware.


No signal on array

There are a number of things that could have happened. Symptoms can range from not seeing the star or internal alignment source to only seeing noise from the amplifiers. The list below is a reasonable set of things to check and try.

  1. Make sure the dark cover is off of IRCAL.
  2. Make sure the filters are in the proper position. Home motors and reposition them if in doubt about where the filters are. N.B. Be sure to use the "Home" button on the filter widget "Config" pulldown (simply sending the filters to their home positions via a normal filter move does not ensure that the wheels are actually "homed").
  3. Check to make sure array is on. To do this, go to the ircalui window and click on Config - Show Reset DSPs Button, then press Reset DSPs button.
  4. If that fails, then cycle power to IRCAL's electronics and then the Reset DSPs button.
  5. If you still can't figure out what is going on, change to MCDS No Sub and see what sort of flux you get. To change to that mode, go to the ircalui window and click on Config - Show Advanced Config and choose MCDS No Sub from the CDS pulldown menu. Take a short exposure and see what the count levels are. Just read noise means that the array is probably not on for some reason or that the amplifiers are not powered on. If the array is saturated in a short exposure (over 10000 for a 57ms exposure or 20000 for a longer exposure) then the chip may have warmed up. In either case you probably have to call in expert assistance. But the data you have taken will help them diagnose the problem.


Exposure Fail error message

If when taking an exposure you get the Exposure Fail error message, you need to try the following steps to regain communication with the IRCAL readout electronics.

Note that anytime you cycle power to the IRCAL electronics you must then Reset the DSPs from the ircalui software (select Config - Show Reset DSPs Button, then click Reset DSPs).

  1. If AO and IRCAL have just been hooked up, you need to check to be sure the two fiber optic cables from the IRCAL readout electronics to the ircalbox2 computer are properly connected. If the fibers are properly connected and the IRCAL electronics are on, the green LED above the Rx and Tx fiber connections on ircalbox2 should be on. If it is not on, swap the two fibers. If the green LED didn't come on after swapping the fibers, cycle power to the IRCAL electronics box. If that fails swap the fibers back. If all else fails push the white reset button on the fiber optic interface card in ircalbox2 and repeat swapping fibers.
  2. If IRCAL has been running OK and now is giving and Exposure Fail error message, check to see if the green LED is on (above the Rx and Tx fiber connections on ircalbox2). If it is not on, cycle power to the IRCAL readout electronics.
  3. If the fiber connection is ok, the usual cause for an Exposure Fail is that software for controlling the readout electronics has gotten confused. Usually Resetting the DSPs from the ircalui software is sufficient to reset the software. If all else fails, cycle power to the IRCAL electronics and Reset DSPs.


Excess Noise

There are times when excess noise (as much as 300 e- rms) shows up in the IRCAL frames. The source of the noise is unknown, but once it appears it does not go away on its own. Noise in regular operations should be about 30 e- rms for a single CDS and 15 e- rms for 16 reads.

A quick way to measure the noise levels is to take two dark exposures and subtract them. Use the Stats button in the Image Browser to measure the standard deviation of the subtracted image. Noise will be:

Noise (e- rms) = std. dev * 10. / sqrt(2) per frame

The noise should be less than 3 DN RMS for subtracted 1 second, 16 read dark exposures.

If you see excess noise, cycle power to the IRCAL electronics box (usually leave it off for 30 to 60 seconds before turning it back on), then reset the DSPs. If this fails to reduce the noise to acceptable levels, repeat the procedure until it does (it may take 2 or 3 tries on the worst occasions of noise).

If the above procedure doesn't eliminate the noise, try the following:

  1. Check the power connections to the IRCAL electronics box, making sure all are firm.
  2. Change the new 3ft cable with the old 8ft cable to make sure the cable is ok (this will most likely make the noise worse because of miscellaneous RF interference that the longer cable picks up).
  3. If all else fails and you can't identify the source of the excess noise, carefully power off the AO racks (only AO operators should cycle power to any AO equipment). If the noise goes away, then it is caused by RF pickup in a ground loop from something in the AO system (though you may have to live with this noise because the AO electronics need to be on, but it will help us diagnose the problem and maybe correct it for future runs).


Exposure Palette is Empty

This is a problem that occurs when you try to Edit the Exposure Palette and you check none of the exposure times before hitting the OK button. You can add exposure modes to the Exposure Palette by taking an exposure and (if it doesn't get automatically added) clicking on Config - Rebuild in the Exposure Palette window.

If you want the entire selection of generic exposure modes to be listed in the Exposure Palette, click on Exposure Palette Config - Add Default All Biases. To add just the default long or short exposures, select Config - Add Default Long Exps or Short Exps, respectively.


Motor status is string of unusual characters

This is a problem that happens occasionally when trying to move motors because communication with the motor controller fails. Symptoms are an unusual string of characters (RA@Dec or something similar) instead of Ready or Busy for motor Status. To correct this problem, you should :

  1. Click on Config - Exit Status Loop. This should stop the status updating.
  2. Click on Config - Advanced - Reset Ctrl.
  3. Click on Config - Advanced - Init Ctrl.
  4. Click on Config - Advanced - Enable Move.
  5. Press the Move button to move the motors to the desired positions.
If there is any doubt as to the positions of the motors, you can press Config - Home to home the motors and then move the motors to the desired position.


Image Browser won't display new image

When the image browser won't display a new image, it is caused by one of two problems:

  1. auto display latest is not checked in ircalui or
  2. the image browser is busy (or stuck) doing another task (fhwm, or other function).
The first is easy to solve by checking the auto display latest checkbox. The second is solved by right clicking in the imagedisplay window a few times to exit the stuck task. If that doesn't work try pressing Reset IDL. If that is unsuccessful, click on Image Browser File - Full IDL Restart to fully restart the IDL GUI and processes. Finally, if that fails, exit the image browser and restart ircaldisplay.


Motor Communication TimeOut

If motor communication times out, it is usually because the motor controller didn't actually power on when you tried to move the motors. Check the motor controller power from the AO software (Power Control GUI) to be sure it is on. Next, try moving the motors again (you may have to click Config - Advanced - Move Enable first). If you still get a TimeOut for communication, then there is a cable problem between the motor controller and ircalbox2 (this requires a telescope technician or support astronomer to fix it).


Soundserver doesn't work or hangs

2008 Oct 9: Soundserver has been replaced by eventsounds ircal. Trouble-shooting is not yet up to date.


NO_DELIVERY Errors

On occasion ircalui will put up a popup window saying 'NO_DELIVERY to 5' or similar message. These typically indicate communications problems between various pieces of software. Communication between IRCAL software, AO software, and the telescope is governed by a program called traffic. No delivery errors generally occur when a piece of required software was not started or has died. A good way to indentify the problem is to look at the traffic log file (on ircalbox2 in /home/operator/kroot/music/log/trafficlog). This will usually identify which piece of software is not functioning so it can be restarted or otherwise repaired. Certain programs (such as leachd) will stop running due to hardware problems in ircalbox2.


Elinor Gates
Last modified: Thu Nov 13 16:41:02 PST 2008