LDAC - Catalogue Matching¶
Note
The concrete examples on this page use the LDAC catalogue W3m0m2_photoz.cat, the ASCII tables W3m0m2_specz_1.asc, W3m0m2_specz_2.asc and the three configuration files asctoldac_specz.conf, associate_photoz_specz.conf and make_ssc_photoz_specz.conf
A tar archive with the complete set of LDAC Tools example files is here
We often need to merge astronomical catalogues from different sources to compare object properties or to merge quantities distributed in different object catalogues. The LDAC tools provide the program pair associate and make_ssc to perform these tasks with a large amount of options. We cover here only the most basic application by associating three catalogues and creating a merged one. For a more complete description of the available options consult Chapter 5 of Erik Deuls Pipeline Documentation
LDAC Catalogue Matching - associate¶
The object matching process implemented within the LDAC tools is a two-stage process. First, the program associate matches sources in different catalogues (we neglect here the possibility of matches within the same catalogue). The extent of each source is characterised by an (error) ellipse in a two-dimensional position space, for instance sky-coordinates with Ra and Dec or pixel coordinates with X and Y. To successfully run associate a minimum set of parameters needs to be present for each object in each catalogue that participates in the matching process. The naming conventions in the following table list parameter names of SExtractor. However, the actually used names can differ and are subject to settings in a configuration file. We list possible parameters for a matching in Sky / Pixel coordinates:
Parameter SKY / PIXEL Meaning for Object Association SeqNr Unique Object Identification Number (ALPHA_J2000, DELTA_J2000) / (X_IMAGE, Y_IMAGE) Object Ellipse Center (A_WORLD, B_WORLD) / (A_IMAGE, B_IMAGE) Semi major and Semi minor ellipse axis THETA_WORLD / THETA_IMAGE position angle of semi major axis Flag Extraction - Quality Flags of the object
The associate program establishes an association between objects in different catalogues if the center position of one object lies within the ellipse spanned by the other. The following figure illustrates several association possibilities - it is taken from Erik Deuls Pipeline Documentation
In the following examples we have an LDAC catalogue W3m0m2_photoz.cat which lists galaxy properties, especially photometric redshifts, in a one square degree area of sky. In addition, we have spectroscopic redshifts from the same sky-area from two different sources. Those are listed in ASCII tables W3m0m2_specz_1.asc and W3m0m2_specz_2.asc. Our final goal is to compare the estimated photometric redshifts with their spectroscopic cousins and we need to merge the three catalogues. The first step is to convert the ASCII tables into the LDAC format and then to perform object association. Note that we need to give error ellipses, sequence numbers and a flag key to the obejcts in the ASCII tables. For the ellipses we consider circles with a radius of half an arcsecond and the quality flags are set to zero (object is o.k.) everywhere. We perform this initial modification of the ASCII catalogues with a simple awk command:
user$ less W3m0m2_specz_1.asc
213.8106384 51.94233322 0.2906414866
213.8327332 51.95418167 0.9577051401
213.8175049 51.95326233 0.9090785980
.
.
user$ awk '{printf("%d %15.8f %15.8f %15.8f %15.8f %f %d %f\n",
NR, $1, $2, 0.5 / 3600., 0.5 / 3600., 0., 0, $3)}' W3m0m2_specz_1.asc >\
tmp.asc
user$ asctoldac -i tmp.asc -o W3m0m2_zspec_1.cat -t OBJECTS -c asctoldac_specz.conf
user$ awk '{printf("%d %15.8f %15.8f %15.8f %15.8f %f %d %f\n",
NR, $1, $2, 0.5 / 3600., 0.5 / 3600., 0., 0, $3)}' W3m0m2_specz_2.asc >\
tmp.asc
user$ asctoldac -i tmp.asc -o W3m0m2_zspec_2.cat -t OBJECTS -c asctoldac_specz.conf
user$ rm tmp.asc
user$ associate
Object Association Program Version: 1.3.4 (Aug 24 2010)
(Pipeline Global Version: 1.2.0)
Usage: associate
-i incat1 [incat2 [incat3 [...]]] (input catalogs)
-o outcat1 [outcat2 [outcat3 [...]]] (output catalogs)
[-t table_name] (table name of objects)
[-c conf_file]
[-OPTION1 OPTVAL1 [-OPTION OPTVAL2] ...]
user$ associate -i W3m0m2_photoz.cat W3m0m2_zspec_1.cat W3m0m2_zspec_2.cat \
-o assoc_1.cat assoc_2.cat assoc_3.cat \
-t OBJECTS -c associate_photoz_specz.conf
Object Association Program Version: 1.3.4 (Aug 24 2010)
(Pipeline Global Version: 1.2.0)
Reading file W3m0m2_zspec_1.cat
Reading file W3m0m2_zspec_2.cat
Done
Note
- All catalogues that participate in an association process must carry the object properties in tables with the same names and in keys with the same names and the the same keytypes
- The configuration file associate_photoz_specz.conf lists the keys which hold the ellipse parameters. In addition, there is the possibility to exclude problematic sources, characterised by certain values of Flag, from the association process. The parameter INTER_COLOR_TOL is a scaling factor. The object ellipse parameters representing the semi-minor and semi-major axis are scaled with that value before the association takes place. Hence, it is easy to redo the association with different positional uncertainties.
- You can simultaneously associate up to 255 catalogues.
Warning
- If association should happen in pixel coordinates, the parameter -p must be given to the associate program. associate will NOT issue warnings or errors if you forget this parameter!
- You must ensure that all involved association parameters (ALPHA_J2000 etc.) are of the same keytype in all involved catalogues. This is avery common source for difficult to spot errors!
The associate program creates for each input catalogue a corresponding output catalogue. Note that the order is important, i.e. the first input catalogue is linked to the first output catalogue and so on. :
user$ ldactoasc -i assoc_1.cat -t OBJECTS -k SeqNr Pair_0 Pair_1 Pair_2
# 1 SeqNr Running object number
# 2 Pair_0 Pairing with catalog dummy
# 3 Pair_1 Pairing with catalog Unknown
# 4 Pair_2 Pairing with catalog Unknown
35 0 0 0
.
61 0 487 0
.
2775 0 0 114
.
84164 0 4773 72
associate adds Pair keys to the objects tables which carry the information on the matching. Note that the counting starts at zero for the first catalogue. The above outputs reads as follows:
- Object 35 in the first catalogue has no matches with objects in other catalogues
- Object 61 in the first catalogue has a match with object 487 from the second catalogue
- Object 2775 from the first catalogue has a match with object 114 from the third catalogue
- Object 84164 from the first catalogue has a match with obejct 4773 from the second catalogue and a match with object 72 from the third catalogue
Note
A number unequal to zero in the Pair_0 column of assoc_1.cat would mean that an object of assoc_1.cat has a match within that catalogue (close pairs)!
Correspondingly for the assoc_2.cat and assoc_3.cat:
user$ ldactoasc -i assoc_2.cat -t OBJECTS -k SeqNr Pair_0 Pair_1 Pair_2
# 1 SeqNr Object Sequence Number [""]
# 2 Pair_0 Pairing with catalog dummy
# 3 Pair_1 Pairing with catalog Unknown
# 4 Pair_2 Pairing with catalog Unknown
1 0 0 0
.
526 587 0 0
527 0 0 0
528 0 0 0
529 642 0 0
.
- Object 526 in the second catalogue has a match with object 587 from the first catalogue
- Objects 527 and 528 from the second catalogue have no matches with obejcts in other catalogues
- Object 529 in the second catalogue has a match with object 642 from the first catalogue
The information of the Pair keys can be used with the program make_ssc to create a catalogue with merged object information.
LDAC Catalogue Matching - make_ssc¶
The make_ssc uses the Pairs information from associate to create a merged catalogue:
user$ less make_ssc_photoz_specz.conf
COL_NAME = ALPHA_J2000
COL_INPUT = ALPHA_J2000
COL_MERGE = AVE_REG
COL_CHAN = 0
#
.
COL_NAME = Z_SPEC_1
COL_INPUT = Z_SPEC
COL_MERGE = AVE_REG
COL_CHAN = 1
#
.
user$ make_ssc
Source Catalog Production Program Version: 2.0.4 (Aug 24 2010)
Usage: make_ssc
-i incat1 [incat2 [..]] (input catalogs)
-o outcat (output catalog)
[-t table_name] (input catalog tablename to merge)
[-p table_name catalog_nr [table_name catalog_nr ...]]
(tables to copy from input to output)
[-c config_file]
[-OPTION1 OPTVAL1 [-OPTION OPTVAL2] ...]
user$ make_ssc -i assoc_1.cat assoc_2.cat assoc_3.cat -o merged.cat \
-c make_ssc_photoz_specz.conf
Source Catalog Production Program Version: 2.0.4 (Aug 24 2010)
Indexing channel 2
WARNING(ssc_put_row): No key Ra in table PSSC
Objects read: 53248, written: 49195
user$ ldacdesc -i merged.cat
.
.
****** Table #2
Extension type:.........BINTABLE
Extension name:.........PSSC
.
.
user$ ldacdesc -i merged.cat -t PSSC
Key name:...............ALPHA_J2000
.
Key name:...............DELTA_J2000
.
Key name:...............Z_B
.
Key name:...............Z_SPEC_1
.
Key name:...............Z_SPEC_2
.
Key name:...............N_00
.
Key name:...............N_01
.
Key name:...............N_02
.
Key name:...............SeqNr
make_ssc creates a table PSSC with the merged information. The configuration file make_ssc_photoz_specz.conf for make_ssc consists of blocks that determine which quantities from the initial catalogue are transfered how to the merged one:
COL_NAME = Z_SPEC_1 There will be a key Z_SPEC_1 in PSSC COL_INPUT = Z_SPEC The original key has the name Z_SPEC COL_MERGE = AVE_REG This line is always AVE_REG for the purpose discussed here COL_CHAN = 1 The input key comes from the second input catalogue (counting starts with 0 for the first catalogue)
Note
make_ssc offers a rich set of further options in the merging process. Please consult Erik Deuls Pipeline Documentation for thses possibilities.
Warning
As associate, make_ssc often produces difficult to find errors if keys with the same name in different input catalogues DO NOT have the same keytype!
The keys N_XX in the PSSC table give the number of obejcts that participate from the corresponding input catalogue (counting starts at zero!) in that entry:
user$ ldactoasc -i merged.cat -t PSSC
# 1 ALPHA_J2000 Right ascension of barycenter (J2000) [deg]
# 2 DELTA_J2000 Declination of barycenter (J2000) [deg]
# 3 Z_B "" [""]
# 4 Z_SPEC_1 spectroscopic redshift [""]
# 5 Z_SPEC_2 spectroscopic redshift [""]
# 6 N_00 Number of objects merged for channel 0
# 7 N_01 Number of objects merged for channel 1
# 8 N_02 Number of objects merged for channel 2
# 9 SeqNr Object sequence number (from make_ssc)
214.2449074 52.12472016 0.7200000000 0.00000 0.00000 1 0 0 1
214.2613235 52.12476932 0.4800000000 0.00000 0.00000 1 0 0 2
214.5122864 52.12514428 0.8800000000 0.00000 0.00000 1 0 0 3
214.2435491 52.12493391 0.5200000000 0.00000 0.00000 1 0 0 4
.
.
213.8923482 52.12469253 0.8000000000 0.765736 0.00000 1 1 0 15
.
.
0.000000000 0.000000000 0.000000000 0.00000 0.132505 0 0 1 49747
0.000000000 0.000000000 0.000000000 0.00000 0.132391 0 0 1 49748
In the first lines, only the first object catalogue contributes. Hence, there are entries fro the coordinates and the photometric redshift Z_B. As no object from the second and third catalogue contributes the columns for Z_SPEC_1 and Z_SPEC_2 are filled with zero. Correspondingly, if no object from the first catalogue contrbutes, the coordinates and Z_B entries are set to zero.
The N_XX keys allows us now an easy filtering of important information: We plot comparisons of photometric redshifts with their spectroscopic counterparts:
user$ ldacfilter -i merged.cat -o comp_specz_1.cat -t PSSC -c "(N_00=1)AND(N_01=1);"
user$ ldacfilter -i merged.cat -o comp_specz_2.cat -t PSSC -c "(N_00=1)AND(N_02=1);"
user$ ldacplot.py -i comp_specz_1.cat -t PSSC -k Z_B Z_SPEC_1 -l 0 1.5 0 1.5 \
-ti "Z_B vs. Z_SPEC_1" -o z_b_z_spec_1.png
Saving plot to z_b_z_spec_1.png
user$ ldacplot.py -i comp_specz_2.cat -t PSSC -k Z_B Z_SPEC_1 -l 0 0.5 0 0.5 \
-ti "Z_B vs. Z_SPEC_2" -o z_b_z_spec_2.png
Saving plot to z_b_z_spec_2.png
