To just start typing commands and see some images, skip down to Jump Starting Photo.
Disclaimer: As the title suggests, this is not "Photo for Dummies." The author of this page is responsible for any errors herein. If the instructions below don't work, it is not necessarily because photo is broken. File a bug report only when you're fairly certain that the problem is in photo rather than in this documentation. The commands below have been tested only in Peyton Hall. This documentation will require revision before it can be used at other SDSS sites.
Driftscanning at sidereal rate yields an exposure time in each filter of approximately 55 seconds. During a scan, an object traverses a single column from top to bottom and thus is registered in each of the 5 filters nearly simultaneously.
For each run (a continuous readout of the camera), the data stream from each CCD is cut into frames of 2048 columns and 1361 rows. Before processing by photo, the 128 rows from the next frame are added to the top of each frame, so that photo actually works on a 2048x1489 image at one time. The resulting overlap between reduced frames (128 rows) is roughly the same as the number of columns that overlap with the other strip of a stripe. Remember this overlap when you look at photo's outputs; some objects will be detected in multiple frames!
Photo analyzes the data one frame at a time. Because information about each object is contained in five separate frames, one for each filter, the five frames for each field are processed together.
Basic operation of the photometric pipeline is also described in The Photometric Pipeline document (very large Postscript file), which is a status report as of 2/98; much of this information is also contained in the Black Book.
Photo is coded in ANSI-C and Tcl (Tool command language) and uses a framework called Dervish, which was developed at Fermilab. Image display from photo is done in FSAOimage , which is Fermilab's version of the SAOimage display familiar to most astronomers.
In short, photo's outputs are FITS binary tables of the parameters and Atlas Images that are stored in compressed form in a binary heap structure of a FITS file. As we describe below, the parameter files may be unpacked and examined within photo itself (which this document describes), through macros run within SM (parameters only),or by any program (see the FITS/IO library) that can read FITS binary tables. The atlas image files may be examined in photo (again, described in this document) or by using a standalone reader. Don't even think of trying to get an atlas image out of an fpAtlas file without this code.
We anticipate that the SX will be available for use in Spring 1999. In the near term we will use photo to directly access these data. Even when the SX is on-line, we will continue to use photo to directly examine and test its outputs.
From the May dark run:
MJD Run Column Good frames ----- --- ------ ----------- 50961 Early data not worth looking at again 50962 ditto
From the June dark run:
MJD Run Column Good frames
----- --- ------ -----------
50990 75 1
50990 76 2, 3, 4 NOT ON DISK ANY MORE at Princeton
77 Reduced and on-line at Fermilab
85 " " " "
(see sdss-obs/msg.85.html)
From the September dark run:
MJD Run Column Good frames ----- --- ------ ----------- 51075 94 1-6 beware of 0.2-0.3 mag zeropoint variation 51078 109 TBD? 51081 125 TBD? (see sdss-obs/msg.75.html)
From the October dark run:
MJD Run Column Good frames ----- --- ------ ----------- 51115 211 tapes on their way to Fermilab (see sdss-obs/msg.82.html)
To check for yourself what data are ready to roll, you can look in the
SDSS data directories and see if the necessary files are there.
Directories in Peyton Hall look like
/u/dss/data/(mjd)/(run)/output/(column).
The necessary files have names like (these for run 75 column 1 frame 100)
fpObjc-000075-1-0100.fit,
fpObjc-000075-1-0100-TST.fit,
and fpAtlas-000075-1-0100.fit
Note that the commands in photo automatically configure the pathnames, so you don't need to remember these or the file names. The directory structure and file name might change but the code will take this into account (n.b. this is a Peyton Hall-ocentric statement).
fpObjc files are binary FITS tables of the measured
object parameters. fpAtlas files are FITS files with all
the compressed atlas images from a frame stored in binary heap.
The Data Model describes all the files that are used in taking and processing the data. Dig down in the imaging stuff to see what the FITS headers for the files look like.
set table [openit $frame].
export DISPLAY="myworkstation:0" (and likewise for the other
setenv commands).
First, to get the display from photo to appear on your workstation you may
have to say
setenv DISPLAY myworkstation:0To add the startup command to your path, enter
setenv PATH /host/hermes/home/trq/bin.sdss:${PATH}
By default, photo will start by reading some procedure
definitions from /host/hermes/home/trq/photo/dervish.tcl, among others.
If you have rolled some of your own Tcl procedures and
want photo to add these definitions,
setenv PHOTO_USER $HOME/myphotodirectory/dervish.tclNow start photo using
start_photoIf you've done this right, you'll now see the
photo> prompt.
Congratulations, you're in!
If you want to run this at another site, or just see what's happening,
look at start_photo
to see
what things get set up and which files get sourced to get going.
If start_photo doesn't exist, then
you will probably need to consult your local data guru to find out how to get
things going.
The command start_photo assumes that you have a complete
SDSS environment installed. It runs setup photo, specifies
the
DERVISH_USER file that is sourced, and starts photo.
mjd 50990 75 1Note that previous documentation and some version of Tcl files describe a procedure
MJD (upper case). Don't use MJD.
If the path to the data directories is not /u/dss/data, you can specify the root directory by invoking mjd as, e.g.,
mjd -rootname /scr/trq/sdss 50990 75 1
set_mtv g rNow set the current frame number to one less than the one you want to examine, e.g.,
set frame 100if you want to see frame 101. To display all of the detected objects in this frame in the r' band, along with their id numbers, type
next_frame(Patience...this may take a moment.) The left-hand fsaoimage display now shows all detected objects in frame 101. What
next_frame
has done is to reconstruct an image of the sky from
the individual atlas images (but without putting the sky values in
between them).
This uses the procedure recon,
which is not identical to
reconstruct_frame.
See More About Using Photo below to learn how to
use next_frame to insert binned sky in between the atlas
images or to show only certain objects (e.g., only galaxies).
By default, next_frame shows the atlas images, id numbers
of the objects, and properties of the mask.
Note the variation in the color of the id numbers. Most id's are in
yellow; these are objects that photo has classified as stars. The
objects whose id's are in red within a white box are "not stars," thus
most of these are galaxies.
Note that the meaning of these colors is different from what is
used by, e.g., mtv_blend below.
To toggle these id's on and off, click Cursor -> region
-> view on the fsaoimage display buttons.
The most common mask features are green vertical lines that indicate interpolated columns and red objects. The meaning of all the mask colors is as follows:
INTERP green
SATUR purple
NOTCHECKED yellow
BRIGHTOBJECT blue
OBJECT red
BINOBJECT magenta
CATOBJECT cyan
SUBTRACTED orange
GHOST orange
CR green
To toggle the mask display on and off, click on "Scale" in the fsaoimage
window and right-click the display.
To step through a sequence of frames just keeping typing
next_frame
next_frame binnedNote that the sky that is used in binned down 4x4 and so looks smoother than the atlas images themselves. Because the object finder was run on the original pixel data (1x1), a 2x2 binned image, and a 4x4 binned image, most of the statistically significant information about the sky is contained in this binned data. Let us know if you find any LSB objects lurking in the sky!
set table [openit $frame]Note that
$frame is a variable that was already set by
set frame and
then incremented along by next_frame.
You can issue more than one command at a time in Tcl by separating the
commands with a semi-colon. A useful combination is, for example,
next_frame; set table [openit $frame]To examine the atlas image of one single object in one filter, type
mtv_objc table 123which displays object number 123 from the current frame in the right-hand fsaoimage window
To look at an atlas image of one object in all five colors and see its deblended children, read its id number from the display that next_frame created and type, e.g.,
mtv_blend table 123which displays object 123 in the right-hand fsaoimage display window. If more than one set of images appears, that means that this "parent" object was deblended into several "children." To include display of the mask bits, use
mtv_blend table 123 1The meaning of the mask colors and procedure for toggling the mask on/off are as described above for
next_frame.
Note the label in the upper right-hand corner: red type indicates that this is a deblended child, yellow that this is a parent.
The children
always have id numbers sequentially higher than their parent, so you
can find the parent by using mtv_blend to step back and
examine objects with smaller id numbers. If it's a parent, then the
parent object is shown in the lower left panel and the children are in
separate rectangles on the display.
For objects that were first detected as "BRIGHT,"
or that are children of
such an object, stepping back through the sequence of object numbers
will lead you back to a display of the parent with all children, but
this mosaic also has a red id number; the object before this one is the
true parent and this "mosaic" has the peculiar property of having one
display region blank.
The center of each child is marked with a yellow symbol. Other centers are marked in red. A cross "X" indicates that photo thinks it's a PSF, otherwise the center is marked with a plus "+" symbol.
The ID number of a child's parent may be found using
set objcIo [objcIoRead table childsID] puts [exprGet $objcIo.parent]where you just substitute for
childsID.
p_objc table 123The columns of this list, from left to right, are the name of the parameter and then the parameter value in u'g'r'i'z'. If you leave off the arguments table and id number,
p_objc will list
parameters of the last object that you examined. To see specific parameters
of a specific object, say, the exponential scale size and fiber
surface brightness in all five colors of object 123,
p_objc table 123 all {r_exp fiberSB}
See
this file for an explanation of the meaning of the parameters.
Related Tcl procedures (see the list below) include
read_objc and show_objc.
Among the possible outputs of p_objc are the calibrated
magnitudes and the (roughly) calibrated RA and DEC (J2000) of the object,
p_objc table 123 all {ra dec psfMag} filename.dat
In this example, the argument "filename.dat" specifies that the output
should be sent to this file rather than the screen. Prepending a plus sign
to the filename, as in "+filename.dat" will append this output to an
existing file.
After using p_objc, the ID number of a child's parent may be
found by typing
puts [exprGet $objcIo.parent]
A list of all the flags that photo set for this object may be seen by typing
pflagsIn this case, the flags for each color are on a separate line, beginning with OBJC (flags for the blended object, comprised of all five colors), followed by 0-4 (u' through z' bands). This file also contains an explanation of these flags.
help -facil=phOutputswhich yields the list
apertures_objc classify_obj1 classify_objc display_mask display_mtv_objc_blend dump_test_info_list flags get_mask mosaic_id mtv_blend mtv_blend_mosaic mtv_objc mtv_objc_blend mtv_objc_list mtv_objc_mosaic objcIoRead objc_intro objfileClose objfileOpen objtable2chain offsetsRead p_objc p_profs read_objc reconstruct_frame regUnbin select show_annuli show_ids show_obj1 show_objc show_peaks show_petro write_atlas_images_as_fitsUsing help with the procedure name will extract more detail (but not much) about a specific procedure, for example
help mtv_objc_mosaicOptional parameters are denoted by delimiting them with [ and ].
To load your own definitions, enter
source mydefinitions.tclIn the future, you can load these by adding them to your own dervish.tcl file and setting the PHOTO_USER environment variable (see above) to point to this file.
proc sel_gal {o {band 2}} {
global OBJECT1
set flgs [exprGet $o.flags<$band>]
if {[exprGet -enum $o.objc_type]=="OBJ_SKY"} {
return 0
}
if {$flgs & ($OBJECT1(EDGE)|$OBJECT1(BLENDED))} {
return 0;
}
return [expr [sel_star $o]==0?1:0]
}
Look for this procedure in
startup.rhl
to see this and other examples of such a filter. This procedure is
not loaded automatically.
Some other examples live in Vogeley's
select.tcl
file. The latter includes examples of how to select on magnitude
and color.
You should write your own (copy and modify
the one above) in a file called, say, select.tcl and
load it into photo by saying
source select.tcl
Earlier we showed how to display all the detected objects in a frame
using next_frame. You may also display only a subset of
the objects, also positioned in the display where they were found, by
invoking, e.g..,
next_frame "" 1 r sel_galwhich displays all the objects that survived
sel_gal in
r' band. Replace "" with binned to get the binned sky. Replace 1 with 0
to not show the mask colors.
To display a mosaic of detected objects in one or all of the filters,
use mtv_blend_mosaic. This procedure takes several
arguments, for example
mtv_blend_mosaic table 200 300 sel_gal all 1 300displays a mosaic in all (one could also specify only 'r' to get the r band) filters of the objects 1 through 300 that passed the sel_gal procedure. Each sub-image is 200x300 pixels in size. The argument 'table' refers to the binary table loaded for this frame. In the following example, 50x50 pixel i-band images are shown of the objects 1 through 1000 that met the sel_test criteria:
mtv_blend_mosaic table 50 50 sel_test i 1 1000
mosaic_id sao#will add object ID numbers to the objects displayed in a mosaic in fsaoimage window number sao#. Clicking on Cursor->region -> view will toggle the id's and peak symbols on/off.
p_objc, mtv_objc, or any
other procedure that specifes a specific object, you can print out the
values of its radial profile using
p_profs...How to get nicer format for this output?
The command make_ppm_file, which is defined in
/u/rhl/photo/etc/wallpaper.tcl, makes a nice color PPM (this format
can be converted later to TIFF, GIF or whatever) files from a specified
series of frames in one column. You can choose which filters are used
to form the color image and what stretch is applied. In addition, you
can indicate that xv should be launched to display the result. For
example,
make_ppm_file r75-c1 21 21will make a large (c. 9MByte) PPM file from frame 21 of the current run and column (set by mjd) using (by default) the full frame and the g', r', and i' filters. The argument 'r75-c1' provides the base name for the file, not the run and column numbers themselves. Thus, the name of this file is r75-c1-21.ppm and it will live in the current directory.
make_ppm_file r75-c1 21 21 i r g 1 0:10s 500 500 0 0 1makes a smaller image of frame 21, using i', r', and g', binning 1x1, stretching from 0 to 10 sigma, and using only a 500x500 pixel region. The ending '1' indicates that xv should automatically display the image.
Dervish has a built in command called "regPrint" that saves a region to
a postscript file. After displaying a frame with next_frame,
try
regPrint $frame_reg junk.psto save the frame to filename junk.ps in the current directory. There are lots of parameters to play with to determine how it translates into postscript (use
help regPrint to see these options).
Also see Zeljko's procedure "fit2ps" that reads FITS into region and calls regPrint: Doesn't this assume that we've already made a FITS image?
# make a PS file from a region stored in a fit file
proc fit2ps {file {pathout ""} {title ""}} {
set reg [regReadAsFits [regNew] $file.fit]
if {$title == "name"} {
set title $file
}
regPrint $reg $pathout$file.ps -title $title -noreverse -nodither -sqrt
regDel $reg
}
write_atlas_images_as_fits table "" "" 1 100which writes the atlas images of objects 1-100. The last four arguments are optional; by default it will write atlas images of all the objects to FITS files in the current directory with names that begin "AI."
p_objc to get calibrated
magnitudes and (roughly, not taking into account chromatic effects)
calibrated positions of one object at a time.
Exercise for the reader:
Look at /host/hermes/home/trq/photo/etc/read_objc.tcl to see how this procedure gets
the calibrated quantities. With this example in hand, you should be able
to write your own procedure that uses calibrated quantities.
Fermilab will eventually produce calibrated files that have names like
tsObjc... that include magnitudes and equatorial coordinates.
Measured counts may be converted to AB magnitudes by using the
flux20 values - the number of counts in a 20th magnitude object -
from the CB files. The SM macros referred to below
read these files and produce calibrated photometry.
fpC-.... These
are FITS images. However, these are quite large and are not always
saved on disk, because (in principle) they can be easily reconstructed
from their predecessor files. Instructions on how to reconstruct
a corrected frames live in a short
make_corrected_frames primer.
source /u/ivezic/photo/etc/sanityCheck.tcl(Soon this will be automatically loaded.)
After loading those procedures, you can see a simple demonstration of PGPLOT by typing
demo_plot1This command displays a simple x vs. y plot and writes the displayed vectors to a file in the current directory called plot_xy.dat. Type RETURN on the command line to clear the PGPLOT display window.
To plot photo outputs, you must first invoke some commands that
create the appropriate data structure. Assuming that you have already
opened a binary table using set table [openit $frame],
type
set chain [objtable2chain $table]To make a simple plot of one object parameter versus another, enter
plot_chains /XWINDOW $chain fiberCounts<1> $chain fiberCounts<2>which plots the counts within a 3 arcsecond aperture in g' vs. r' bands.
See the
measured objects document for the names of the parameters that
you could substitute for fiberCounts.
Here and throughout, the filters u', g', r', i', z' correspond to the
the ordinal values 0, 1, 2, 3, and 4, respectively. Thus, the z'
Petrosian radius is petroRad<4>.
Replacing /XWINDOW with /PS will produce a postscript
file in the current directory with the name plot_xy.ps.
plot_chains
you should be able to write your own procedure to make plots.
Look at the Tcl procedures in
sanityCheck.tcl
to learn more.
The command
set table [openit $frame}reads the files from disk for this frame and creates a binary table in memory. Then the command
set chain [objtable2chain $table]makes a "linked list" in memory of all the object parameters for this frame. The plotting routines work on vectors, so the next step is to load the parameter values that you want into a pair of vectors. For example,
set fc1 [vFromChain $chain fiberCounts<1>] set fc2 [vFromChain $chain fiberCounts<2>]loads the 3 arcsecond fiber counts in g' and r' into the vectors
fc1 and fc2.
To make a plot, first start up a new PGPLOT process,
set pgs [pgstateNew] pgstateOpen $pgsTo plot the vectors defined above, type
vPlot $pgs $fc1 $fc2A (very slight) shortcut to this plot is
vPlot $pgs [vFromChain $chain fiberCounts<1>] [vFromChain $chain fiberCounts<2>]If you ask for another plot, it will appear in the same window after hitting RETURN on the command line.
To plot a restricted range of the parameters, give vPlot
more parameters, e.g.,
vPlot $pgs $fc1 $fc2 -xmin 0 -xmax 1e4 -ymin 0 -ymax 1e4
If you associate a name with each vector, then that name will appear as the axis label. For example,
vNameSet $fc1 "g' fiber counts" vNameSet $fc2 "r' fiber counts" vPlot $pgs $fc1 $fc2will appropriately label the axes in the example above. Add a title to this plot with
titlePlot "Fiber Counts g' vs. r'" 40To annotate a plot,
pgText 1000 1000 "HERE"will place the text "HERE" at parameter location x=1000, y=1000.
Several additional properties of the
PGPLOT may be controlled by pgstateSet.
Use help pgstateSet to see all the options.
(Also look in this
Dervish documentation
)
For example,
pgstateSet $pgs -nxwindow 2 -nywindow 2will cause the the PGPLOT window to have four available windows. If you execute four successive plot calls, each will be sent to a different window. In this example, the fifth plot will appear on the next "page" in the PGPLOT window.
To close a PGPLOT without producing a postscrip file,
endplot $pgs ""To get postscript output of a plot, set
pgstateSet -device /PS(the default is device=/XWINDOW) before creating the plot, and use
endplot $pgs "mynewplotname.ps"when you're all done. If you leave off the filename, the plot will appear in pgplot.ps. Among other things,
endplot executes
a pgstateClose and pgstateDel.
vectorsWriteToFile $x $y plot_xy.datwill write the vectors x and y to an ascii file called plot_xy.dat in the current directory.
Use a select procedure to make a mask vector, so only good data get plotted. Need to call vPlot including a vector passed with -vectorMask
Or make a chain out of only selected objects. Need a new objtable2chain command. See sanityCheck.tcl for examples of how to make a chain.
set psf2 [vFromChain $chain psfCounts<2>]next get the zeropoint for the current frame
set zero2 [exprGet $fieldparams.frame<2>.flux20]and, finally, calibrate the counts
set psf2 [vectorExprEval 20-2.5*lg($psf2/$zero2)]Note that the logarithm operation fails for counts less than or equal to zero, so you'll get lots of 20th mag objects this way. This is an example of why we use inverse hyperbolic sine instead of magnitude. (ref for this?).
Similarly, one can make vector g'-r' and r'-i' colors after creating the appropriate magnitude vectors by using
set gr [vectorExprEval $f1-$f2] set ri [vectorExprEval $f2-$f3]
vectorGetFromRegion
to make a vector of the the mean (or, if you choose, the median) of
the pixels in the orthogonal direction.
For example, after using next_frame
to display a frame in an fsaoimage window, typing
set x [vectorExprEval (1,100)] set y [vectorGetFromRegion $frame_reg 1 100 15 20 1]puts the row number values 1 through 100 in the x vector and puts the mean of the pixels in columns 15 through 20 in each row into the y vector. The final argument of vectorGetFromRegion specifies the direction of the cut, 1 for vertical, 0 for horizontal (the averaging is done in the orthogonal direction). As usual see
help
vectorGetFromRegion for all options.
plot_histogram /XWINDOW $ywill make a simple histogram of values in the current y vector.
fpstats 100 110which makes a plot for each color (notes in upper right corner of the PGPLOT window). Step through the series of plots by hitting RETURN on the command line.
Other useful plots to look at are saved along with the data files in the /u/dss/data/(MJD)/(RUN)/output directories. For example, look at
/u/dss/data/50990/75/output/psCB-000075-psf_width.psto see a plot of how the PSF width varies during run 75 of the June dark run, or
/u/dss/data/50990/75/output/psCB-000075-skymag.psto see how the sky brightness varied during this run. Back to the Table of Contents
For the grand finale of this primer, we'll use fsaoimage and pgplot to display a mosaic and a color-magnitude diagram of galaxies selected by a Tcl procedure.
You must have SM v2.4.1 or later to do this. At Princeton, please note that only certain machines can see the SDSS data directories on stribor (the LINUX box that has the disks). Try from another machine if you have trouble.
Extracting the atlas images from the fpAtlas files is trickier, because they are stored in heap, not as FITS images. As an alternative to using photo to examine Atlas Images, there are a couple of utilities in photo/readAtlasImages which can be built independently of photo or dervish; alternatively or additionally they could be built into e.g. astrotools. They are not especially friendly, but they are supposed to be easily modifiable when you decide to write your own special purpose code. The code lives in the directory /u/rhl/photo/readAtlasImages
To read an atlas image, say the r' band image from object 12, say
read_atlas_image fpAtlas.fit -c 2 12 outfile.fitTo read a set of masks, say the SATUR mask, say
read_mask fpM.fit 1 outfile.fitIn each case the output is a FITS file, in the former with BITPIX==16, in the latter with BITPIX==8. In the respective main programs, the atlas image is returned as the type ATLAS_IMAGE, and the different "objects" making up the mask are returned as OBJMASKs.
The mask planes in successive HDUs of the fpM files are:
1 S_MASK_INTERP, /* pixel's value has been interpolated */ 2 S_MASK_SATUR, /* pixel is/was saturated */ 3 S_MASK_NOTCHECKED, /* pixel was NOT examined for an object */ 4 S_MASK_OBJECT, /* pixel is part of some object */ 5 S_MASK_BRIGHTOBJECT, /* pixel is part of bright object */ 6 S_MASK_BINOBJECT, /* pixel is part of binned object */ 7 S_MASK_CATOBJECT, /* pixel is part of a catalogued object */ 8 S_MASK_SUBTRACTED, /* model has been subtracted from pixel */ 9 S_MASK_GHOST, /* pixel is part of a ghost */ 10 S_MASK_CR, /* pixel is part of a cosmic ray */ (See photo/include/phSpanUtil.h)You can get a reminder of the arguments with the -h flag
Thanks to David Schlegel.
One can most easily navigate among the source code files by using the TAGS feature of emacs. To learn more about TAGS, read emacs info. Primer: 'M-.' (that's a period while holding down the Meta key) with the point (cursor) on the tag takes you to the file that defines the tag. 'C-x 4.' will open the file with the tag definition in another emacs window. Watch the mini-buffer to see what tag emacs is looking for and check that it is referencing the /host/hermes/home/trq/photo/TAGS for this information. Use 'C-h a tag' (apropos command) to see info on other tags-related commands.
help some_command
Good books from which to learn Tcl include
Look at the SAOimage home page for documentation about the original software product.
http://www.astro.princeton.edu:81/gnats-SOFTWARE/YOUR_NAME/username/,
which keeps you from repeatedly having to enter this info.
In your report, please include information about the version of photo
that you are running, by quoting the result of typing
photoVersion
Note that bugs may be reported for both the software and its documentation. THIS document is not part of photo, so please report all errors herein to the author.
TBD: How to
dump object parameters to an ascii file (or binary FITS?)
cf. use of SM to read, plot parameters
look at a color image of frame that combines colors ("true color?")
see wallpaper.tcl (note that we have already described
how to use make_ppm)
information about frames
where on the sky (asTrans files)
PSF characteristics
calibration info (psCB files)
combine objects from different frames into one table
Needed to fix before exporting to other sites:
machines that photo will run on
start_photo
can't see RHL files, code unless local
Please send any comments or suggestions to Tom Quinn