ShaneAO - How to make a control matrix

UNIVERSITY OF CALIFORNIA OBSERVATORIES / LICK OBSERVATORY

Shane AO - How to make a control matrix

Making a new control matrix is not a common operation and is relatively complicated. If the alignment between the deformable mirror and the wave-front sensor (WFS) camera changes, it will be necessary to to create a new control matrix.

Also, it is useful to start with the 8x lenslet array because it is a little less sensitive to errors in alignment and useful for creating a "flat" shape for the deformable mirrors for use when making the 16x control matrix.

Procedure

First log into the user account on the real-time control (rtc) computer, real. Then start the rtc software by typing shaneao at the prompt.

You'll also want to look at the images from the WFS and TT cameras. From a new xterm from the user account on shimmy type the following:

  1. ssh -Y gavel@shade xterm -sb -T Peeko -e peeko &
  2. ssh -Y gavel@shade xterm -sb -T TTPeeko -e ttpeeko &
The directions for doing 8x and 16x control matrices are slightly different. So separate procedures are written for each case.

Creating 8x control matrices or Creating 16x control matrices or Creating Pyramid control matrices

Additionally, once the matrices in both 8x and 16x are created, you'll need to tune the system and install the control matrices in the proper directory.

Tuning and Installing

Creating 8x control matrices

Now we need to load the right procedures into shaneao. At the shaneao prompt type the following commands:
  1. init(8)
  2. import calibrate as cal
  3. import datetime
  4. mode('8x')
  5. bias0()
  6. zero()
  7. workDir = datetime.date.today().isoformat()
  8. workDir
The last command should give you output akin to '2020-10-15', which is the working directory into which the newly created files will be put. Hence, that directory now needs to be created. In a new xterm for the user account on real do the following:
  1. cd /data/parameterFiles/reconMatrix_8x
  2. mkdir 2020-10-15 (substituting the date that workDir output above)
Now go back to the shaneao software and setup the rtc with the proper centroider and wave-front sensor camera rate for high signal to noise measurements with the white light calibration source.
  1. cent('COG')
  2. rate(100) (enter the appropriate rate to deliver high signal to noise data, but not saturate any pixels)
  3. refcent(0)
  4. Turn off the white light calibration source (B8 in the ShaneAO Power Control GUI)
  5. dark()
  6. Turn the white light calibration source back on.
At this point make sure that everything is well aligned. This may involve small adjustments to the lenslet stage, WFS X and Y stages, or calibration source position, but be careful, as you don't want to introduce any misalignments between the WFS camera and deformable mirrors.

Once you have things aligned as well as possible, we can take the control matrix data for the tweeter (MEMS) and woofer deformable mirrors. This procedure will push actuators and measure the displacements of the Hartmann spots on the WFS camera and will take a few minutes to run. Type the following commands in the shaneao software:

  1. Make sure that calibration procedure is in right subaperture mode:
    cal.self.mode='8x'
  2. cal.calibrate(dm='tweeter',amp=5000,delay=1)
    Note that amp is the amplitude of the actuator motion during calibration. For the tweeter this ranges from 0 to 43,000 D/A, with the bias level set at 21500 (middle range). The delay is in seconds, and the longer the delay for more data is acquired for each mode.
  3. Wait for calibration procedure to complete. This will take a few minutes and you should see the Hartmann spots moving on the WFS camera as well as the tweeter actuator display updating with various patterns.
  4. Now save the tweeter data:
    cal.saveH(workDir=workDir)
  5. Next calibrate the woofer DM:
    cal.calibrate(dm='woofer',amp=0.5,delay=1)
    Note that the amp range for the woofer is -1 to 1.
  6. Again, wait for the command to run and it may take a few minutes. You should see the Hartmann spots moving on the WFS camera as well as the woofer actuator display updating with various patterns.
  7. Save the woofer data:
    cal.saveHw(workDir=workDir)
There should now be H.fits and Hw.fits files in the workDir directory on real. At this point you need to copy the data to shimmy to examine and process it, archiving previous files as necessary.
  1. First log into the user account on shimmy:
    ssh user@shimmy
  2. Change to the directory for working with control matrix data:
    cd observers/lgs/Calibration
  3. If an H.fits and/or Hw.fits file exists, move it to the archive, giving it a unique name (typically the date the file was created plus other unique identifiers as needed), e.g.:
    mv H.fits archive/H.2020-08-12.fits
    mv Hw.fits archive/Hw.2020-08-12.fits
  4. Now change to the parameterFiles directory:
    cd parameterFiles/reconMatrix_8x
  5. Next create a working directory with the same date as workDir on real, e.g.
    mkdir 2020-10-15
  6. In a user@real xterm, copy the file from real to shimmy:
    cd /data/parameterFiles/reconMatrix_8x/2020-10-15 (substitute current workDir directory)
    scp H*.fits shimmy:observers/lgs/Calibration/.
Now that the data are on shimmy, you can look at them with your favorite FITS file viewing utility, e.g. ds9 and make sure they look similar to these sample images.


Figure 1: H.fits for 8x


Figure 2: Hw.fits for 8x

If these files don't look reasonable, you'll have to try again, double checking that all paramters were set correctly, the WFS camera had a good background, and optical and mechanical alignment was correct. Sometimes adjusting the amp value in the cal.calibrate command is necessary (though the values in the example here were the best used for the alignment in Oct 2020, the most recent time this procedure was done).

If these files look nominally OK, you can then move on to creating the control matrices. The software to do this is on shimmy and runs in python. Do the following commands in a user@shimmy xterm:

  1. python
  2. import sys
  3. sys.path.append('/home/lick/lib/python')
  4. import param_gen as pg
  5. pg.home='/u/user/observers/lgs/Calibration/'
  6. workDir='2020-10-15' (substitute the appropriate workDir date)
  7. import os
  8. pg.paramsDir=os.path.join(pg.home,'parameterFiles')
  9. pg.gen_CM_8x(LGS_also=True, use_penalty=True, weighting= None, workDir=workDir)
  10. pg.gen_CM_8x(LGS_also=True, use_penalty=True, weighting= 'wls', workDir=workDir)
  11. pg.gen_CM_8x(LGS_also=True, use_penalty=True, weighting= 'dim', workDir=workDir)
This should create a number of files in the workDir directory on shimmy:
controlMatrix_8x.fits
controlMatrix_8xLGS.fits
controlMatrix_8x_dim.fits
controlMatrix_8xLGS_dim.fits
controlMatrix_8x_wls.fits
controlMatrix_8xLGS_wls.fits

These files now need to be copied to real for testing.

  1. In a user@shimmy xterm cd to the directory with the new control matrices:
    cd /u/user/observers/lgs/Calibration/parameterFiles/reconMatrix_8x/2020-10-15 (substitute the appropriate workDir date)
  2. scp control*.fits real:/data/parameterFiles/reconMatrix_8x/2020-10-15 (again, substitute the appropriate workDir date)
It is now necessary to test the new control matrix to make sure it works. Going back to the shaneao prompt on real do the following:
  1. set_CM('reconMatrix_8x/2020-10-15/controlMatrix_8x.fits')
  2. bias0()
  3. zero()
  4. refcent(0)
  5. bleed(.9,'tweeter')
  6. bleed(.995,'woofer')
  7. Turn off the white light calibration source.
  8. dark()
  9. Turn the white light calibration source back on.
  10. To test just the woofer loop: (remember that setting the gain also closes the loops)
    gain(0,.2)
  11. To test both loops:
    gain(.2,.2)
  12. Check to see how stable the performance is, optionally changing gain to see how stable performance is at higher gains.
  13. Optionally take exposures on ShARCS to measure performance. Strehl may not be great, but should be at some reasonable level that could be improved with image sharpening later.
If the performance is not OK (loops unstable, strehl not good, etc.), you'll want to go back to the beginning and check optical and mechanical alignment, that you have a good background (dark) measurement for the WFS camera, and/or using a different amp setting in the cal.calibrate command.

If the performance is OK, you can optionally make a DM flat using just the woofer to help with creating the 16x control matrix doing the following commands:

  1. Make sure all settings are as above for testing the control matrix, including taking a fresh background for the WFS camera.
  2. Close the loops with only the woofer by setting
    gain(0,.2)
  3. flat()

Creating 16x control matrices

Now we need to load the right procedures into shaneao. At the shaneao prompt type the following commands:
  1. init(16)
  2. import calibrate as cal
  3. import datetime
  4. mode('16x')
  5. bias0()
  6. zero()
  7. workDir = datetime.date.today().isoformat()
  8. workDir
The last command should give you output akin to '2020-10-15', which is the working directory into which the newly created files will be put. Hence, that directory now needs to be created. In a new xterm for the user account on real do the following:
  1. cd /data/parameterFiles/reconMatrix_16x
  2. mkdir 2020-10-15 (substituting the date that workDir output above)
Now go back to the shaneao software and setup the rtc with the proper centroider and wave-front sensor camera rate for high signal to noise measurements with the white light calibration source.
  1. cent('COG')
  2. rate(100) (enter the appropriate rate to deliver high signal to noise data, but not saturate any pixels)
  3. refcent(0)
  4. Turn off the white light calibration source (B8 in the ShaneAO Power Control GUI)
  5. dark()
  6. Turn the white light calibration source back on.
  7. [Optional] If you created a new flat mirror file during the 8x control matrix testing, you can apply it:
    flat()
At this point make sure that everything is well aligned. This may involve small adjustments to the lenslet stage, WFS X and Y stages, or calibration source position, but be careful, as you don't want to introduce any misalignments between the WFS camera and deformable mirrors.

Once you have things aligned as well as possible, we can take the control matrix data for the tweeter (MEMS) and woofer deformable mirrors. This procedure will push actuators and measure the displacements of the Hartmann spots on the WFS camera and will take a few minutes to run. Type the following commands in the shaneao software:

  1. Make sure that calibration procedure is in right subaperture mode:
    cal.self.mode='16x'
  2. cal.calibrate(dm='tweeter',amp=5000,delay=1)
    Note that amp is the amplitude of the actuator motion during calibration. For the tweeter this ranges from 0 to 43,000 D/A, with the bias level set at 21500 (middle range). The delay is in seconds, and the longer the delay for more data is acquired for each mode.
  3. Wait for calibration procedure to complete. This will take a few minutes and you should see the Hartmann spots moving on the WFS camera as well as the tweeter actuator display updating with various patterns.
  4. Now save the tweeter data:
    cal.saveH(workDir=workDir)
  5. Next calibrate the woofer DM:
    cal.calibrate(dm='woofer',amp=0.4,delay=1)
    Note that the amp range for the woofer is -1 to 1.
  6. Again, wait for the command to run and it may take a few minutes. You should see the Hartmann spots moving on the WFS camera as well as the woofer actuator display updating with various patterns.
  7. Save the woofer data:
    cal.saveHw(workDir=workDir)
There should now be H.fits and Hw.fits files in the workDir directory on real. At this point you need to copy the data to shimmy to examine and process it, archiving previous files as necessary.
  1. First log into the user account on shimmy:
    ssh user@shimmy
  2. Change to the directory for working with control matrix data:
    cd observers/lgs/Calibration
  3. If an H.fits and/or Hw.fits file exists, move it to the archive, giving it a unique name (typically the date the file was created plus other unique identifiers as needed), e.g.:
    mv H.fits archive/H.2020-08-12.fits
    mv Hw.fits archive/Hw.2020-08-12.fits
  4. Now change to the parameterFiles directory:
    cd parameterFiles/reconMatrix_16x
  5. Next create a working directory with the same date as workDir on real, e.g.
    mkdir 2020-10-15
  6. In a user@real xterm, copy the file from real to shimmy:
    cd /data/parameterFiles/reconMatrix_16x/2020-10-15 (substitute current workDir directory)
    scp H*.fits shimmy:observers/lgs/Calibration/.
Now that the data are on shimmy, you can look at them with your favorite FITS file viewing utility, e.g. ds9 and make sure they look similar to these sample images.


Figure 1: H.fits for 16x


Figure 2: Hw.fits for 16x

If these files don't look reasonable, you'll have to try again, double checking that all paramters were set correctly, the WFS camera had a good background, and optical and mechanical alignment was correct. Sometimes adjusting the amp value in the cal.calibrate command is necessary (though the values in the example here were the best used for the alignment in Oct 2020, the most recent time this procedure was done).

If these files look nominally OK, you can then move on to creating the control matrices. The software to do this is on shimmy and runs in python. Do the following commands in a user@shimmy xterm:

  1. python
  2. import sys
  3. sys.path.append('/home/lick/lib/python')
  4. import param_gen as pg
  5. pg.home='/u/user/observers/lgs/Calibration/'
  6. workDir='2020-10-15' (substitute the appropriate workDir date)
  7. import os
  8. pg.paramsDir=os.path.join(pg.home,'parameterFiles')
  9. pg.gen_CM_16x(LGS_also=True, use_penalty=True, weighting= None, workDir=workDir)
  10. pg.gen_CM_16x(LGS_also=True, use_penalty=True, weighting= 'wls', workDir=workDir)
  11. pg.gen_CM_16x(LGS_also=True, use_penalty=True, weighting= 'dim', workDir=workDir)
This should create a number of files in the workDir directory on shimmy:
controlMatrix_16x.fits
controlMatrix_16xLGS.fits
controlMatrix_16x_dim.fits
controlMatrix_16xLGS_dim.fits
controlMatrix_16x_wls.fits
controlMatrix_16xLGS_wls.fits

These files now need to be copied to real for testing.

  1. In a user@shimmy xterm cd to the directory with the new control matrices:
    cd /u/user/observers/lgs/Calibration/parameterFiles/reconMatrix_16x/2020-10-15 (substitute the appropriate workDir date)
  2. scp control*.fits real:/data/parameterFiles/reconMatrix_16x/2020-10-15 (again, substitute the appropriate workDir date)
It is now necessary to test the new control matrix to make sure it works. Going back to the shaneao prompt on real do the following:
  1. set_CM('reconMatrix_16x/2020-10-15/controlMatrix_16x.fits')
  2. bias0()
  3. zero()
  4. refcent(0)
  5. bleed(.9,'tweeter')
  6. bleed(.995,'woofer')
  7. Turn off the white light calibration source.
  8. dark()
  9. Turn the white light calibration source back on.
  10. To test just the woofer loop: (remember that setting the gain also closes the loops)
    gain(0,.2)
  11. To test both loops:
    gain(.2,.2)
  12. Check to see how stable the performance is, optionally changing gain to see how stable performance is at higher gains.
  13. Optionally take exposures on ShARCS to measure performance. Strehl may not be great, but should be at some reasonable level that could be improved with image sharpening later.
If the performance is not OK (loops unstable, strehl not good, etc.), you'll want to go back to the beginning and check optical and mechanical alignment, that you have a good background (dark) measurement for the WFS camera, and/or using a different amp setting in the cal.calibrate command.

Creating Pyramid control matrices

This procedure assumes that there are good DM flat shapes created with 16x or 8x closed loops that produce good Strehls on ShARCS images.

Now we need to load the right procedures into shaneao. At the shaneao prompt type the following commands:

  1. init(3)
  2. import calibrate as cal
  3. import datetime
  4. flat0()
  5. flat()
  6. workDir = datetime.date.today().isoformat()
  7. workDir
The last command should give you output akin to '2020-10-15', which is the working directory into which the newly created files will be put. Hence, that directory now needs to be created. In a new xterm for the user account on real do the following:
  1. cd /data/parameterFiles/reconMatrix_pyr
  2. mkdir 2020-10-15 (substituting the date that workDir output above)
Now go back to the shaneao software and setup the rtc with the proper centroider and wave-front sensor camera rate for high signal to noise measurements with the white light calibration source.
  1. cent('pyr')
  2. rate(100) (enter the appropriate rate to deliver high signal to noise data, but not saturate any pixels)
  3. refcent(0)
  4. Turn off the white light calibration source (B8 in the ShaneAO Power Control GUI)
  5. dark()
  6. Turn the white light calibration source back on.
At this point make sure that everything is well aligned. This may involve small adjustments to the lenslet stage, WFS X and Y stages, or calibration source position, but be careful, as you don't want to introduce any misalignments between the WFS camera and deformable mirrors.

Once you have things aligned as well as possible, we can take the control matrix data for the tweeter (MEMS) and woofer deformable mirrors. This procedure will push actuators and measure the displacements of the Hartmann spots on the WFS camera and will take a few minutes to run. Type the following commands in the shaneao software:

  1. Make sure that calibration procedure is in right subaperture mode:
    cal.self.mode='pyr'
  2. cal.calibrate(dm='tweeter',amp=500,delay=1)
    Note that amp is the amplitude of the actuator motion during calibration. For the tweeter this ranges from 0 to 43,000 D/A, with the bias level set at 21500 (middle range). The delay is in seconds, and the longer the delay for more data is acquired for each mode.
  3. Wait for calibration procedure to complete. This will take a few minutes and you should see the Hartmann spots moving on the WFS camera as well as the tweeter actuator display updating with various patterns.
  4. Now save the tweeter data:
    cal.saveH(workDir=workDir)
  5. Next calibrate the woofer DM:
    cal.calibrate(dm='woofer',amp=0.1,delay=1)
    Note that the amp range for the woofer is -1 to 1.
  6. Again, wait for the command to run and it may take a few minutes. You should see the Hartmann spots moving on the WFS camera as well as the woofer actuator display updating with various patterns.
  7. Save the woofer data:
    cal.saveHw(workDir=workDir)
There should now be H.fits and Hw.fits files in the workDir directory on real. At this point you need to copy the data to shimmy to examine and process it, archiving previous files as necessary.
  1. First log into the user account on shimmy:
    ssh user@shimmy
  2. Change to the directory for working with control matrix data:
    cd observers/lgs/Calibration
  3. If an H.fits and/or Hw.fits file exists, move it to the archive, giving it a unique name (typically the date the file was created plus other unique identifiers as needed), e.g.:
    mv H.fits archive/H.2020-08-12.fits
    mv Hw.fits archive/Hw.2020-08-12.fits
  4. Now change to the parameterFiles directory:
    cd parameterFiles/reconMatrix_pyr
  5. Next create a working directory with the same date as workDir on real, e.g.
    mkdir 2020-10-15
  6. If the At.fits and Aw.fits files haven't already been copied to shimmy from real do the following in a user@real xterm:
    cd /data/parameterFiles/reconMatrix_pyr/
    scp A*.fits shimmy:observers/lgs/Calibration/reconMatrix_pyr/.
  7. In a user@real xterm, copy the file from real to shimmy:
    cd /data/parameterFiles/reconMatrix_pyr/2020-10-15 (substitute current workDir directory)
    scp H*.fits shimmy:observers/lgs/Calibration/parameterFiles/reconMatrix_pyr/2020-10-15.
Now that the data are on shimmy, you can look at them with your favorite FITS file viewing utility, e.g. ds9 and make sure they look similar to these sample images (see 16x H.fits and Hw.fits above, as they will look similar).

If these files don't look reasonable, you'll have to try again, double checking that all paramters were set correctly, the WFS camera had a good background, and optical and mechanical alignment was correct. Sometimes adjusting the amp value in the cal.calibrate command is necessary (though the values in the example here were the best used for the alignment in Oct 2020, the most recent time this procedure was done).

If these files look nominally OK, you can then move on to creating the control matrices. The software to do this is on shimmy and runs in python. Do the following commands in a user@shimmy xterm:

  1. python
  2. import sys
  3. sys.path.append('/home/lick/lib/python')
  4. import param_gen as pg
  5. pg.home='/u/user/observers/lgs/Calibration/'
  6. workDir='2020-10-15' (substitute the appropriate workDir date)
  7. import os
  8. pg.paramsDir=os.path.join(pg.home,'parameterFiles')
  9. pg.gen_CM_pyr(LGS_also=True, use_penalty=True, weighting= None, workDir=workDir)
  10. pg.gen_CM_pyr(LGS_also=True, use_penalty=True, weighting= 'wls', workDir=workDir)
  11. pg.gen_CM_pyr(LGS_also=True, use_penalty=True, weighting= 'dim', workDir=workDir)
This should create a number of files in the workDir directory on shimmy:
controlMatrix_pyr.fits
controlMatrix_pyrLGS.fits
controlMatrix_pyr_dim.fits
controlMatrix_pyrLGS_dim.fits
controlMatrix_pyr_wls.fits
controlMatrix_pyrLGS_wls.fits

These files now need to be copied to real for testing.

  1. In a user@shimmy xterm cd to the directory with the new control matrices:
    cd /u/user/observers/lgs/Calibration/parameterFiles/reconMatrix_pyr/2020-10-15 (substitute the appropriate workDir date)
  2. scp control*.fits real:/data/parameterFiles/reconMatrix_pyr/2020-10-15 (again, substitute the appropriate workDir date)
It is now necessary to test the new control matrix to make sure it works. Going back to the shaneao prompt on real do the following:
  1. set_CM('reconMatrix_pyr/2020-10-15/controlMatrix_pyr.fits')
  2. flat0()
  3. flat()
  4. refcent(0)
  5. bleed(.9,'tweeter')
  6. bleed(.995,'woofer')
  7. Turn off the white light calibration source.
  8. dark()
  9. Turn the white light calibration source back on.
  10. To test just the woofer loop: (remember that setting the gain also closes the loops)
    gain(0,.2)
  11. To test both loops:
    gain(.2,.2)
  12. Check to see how stable the performance is, optionally changing gain to see how stable performance is at higher gains.
  13. Optionally take exposures on ShARCS to measure performance. Strehl may not be great, but should be at some reasonable level that could be improved with image sharpening later.
If the performance is not OK (loops unstable, strehl not good, etc.), you'll want to go back to the beginning and check optical and mechanical alignment, that you have a good background (dark) measurement for the WFS camera, and/or using a different amp setting in the cal.calibrate command.

Tuning

Once you've determined the new control matrices are nominally acceptable, you can move forward with image sharpening, tuning WFS focus, etc. to get the best possible strehl on ShARCS. With luck this will converge to a suitably high strehl for science operations.

If suitable strehl can not be obtained, you need to examine the optical alignment of the system, mechanical stability, make sure the procedure for creating the control matrices was properly followed, adjust the amp parameter, etc. Note that it is not unusual to have to run through the procedure a few times to get good control matrices.

Installing Control Matrices for routine operations

Once you have achieved a good strehl on ShARCS, you'll need to install the new control matrices in the default directory on real for the real-time code to find them for routine operations.

The default location for the control matrix on real is /data/parameterFiles/. There will already be many control matrix FITS files in the directory, so first you'll have to save them to a back up directory before installing the new control matrix files. In a user@real xterm do the following:

  1. cd /data/parameterFiles
  2. mkdir controlMatrix_backup-2020-11-10 (substituting the current date)
  3. mv controlMatrix*.fits controlMatrix_backup-2020-11-10/. (again, substitute the current date)
  4. cp /data/parameterFiles/reconMatrix_16x/2020-10-15/controlMatrix*.fits . (substitute the workDir date)
  5. cp /data/parameterFiles/reconMatrix_8x/2020-10-15/controlMatrix*.fits . (substitute the workDir date)
  6. cp /data/parameterFiles/reconMatrix_pyr/2020-10-15/controlMatrix*.fits . (substitute the workDir date)
Unfortunately the shaneao code wants some of the files named differently than the default names. Hence, we now need to copy a few of the files to new file names. Below are the relevant copy commands to run:
  1. cp controlMatrix_16xLGS_dim.fits controlMatrix_16x_dimLGS.fits
  2. cp controlMatrix_16xLGS_wls.fits controlMatrix_16x_wlsLGS.fits
  3. cp controlMatrix_8xLGS_dim.fits controlMatrix_8x_dimLGS.fits
  4. cp controlMatrix_8xLGS_wls.fits controlMatrix_8x_wlsLGS.fits
  5. cp controlMatrix_pyrLGS_dim.fits controMatrix_pyr_dimLGS.fits
  6. cp controlMatrix_pyrLGS_wls.fits controlMatrix_pyr_wlsLGS.fits
At this point the newly installed control matrices should be loaded during routine operations instead of the old matrices that have been archived.
Elinor Gates