.. -*- coding: utf-8 -*- .. _LDAC_matching: =========================== LDAC - Catalogue Matching =========================== .. note:: The concrete examples on this page use the LDAC catalogue `W3m0m2_photoz.cat <../_static/W3m0m2_photoz.cat>`_, the ASCII tables `W3m0m2_specz_1.asc <../_static/W3m0m2_specz_1.asc>`_, `W3m0m2_specz_2.asc <../_static/W3m0m2_specz_2.asc>`_ and the three configuration files `asctoldac_specz.conf <../_static/asctoldac_specz.conf>`_, `associate_photoz_specz.conf <../_static/associate_photoz_specz.conf>`_ and `make_ssc_photoz_specz.conf <../_static/make_ssc_photoz_specz.conf>`_ A tar archive with the complete set of LDAC Tools example files is `here <../_static/LDAC_Tools_Examples.tgz>`_ We often need to merge astronomical catalogues from different sources to compare object properties or to merge quantities distributed in different object catalogues. The LDAC tools provide the program pair :command:`associate` and :command:`make_ssc` to perform these tasks with a large amount of options. We cover here only the most basic application by associating three catalogues and creating a merged one. For a more complete description of the available options consult Chapter 5 of `Erik Deuls Pipeline Documentation`__ __ LDAC_docu_ .. _LDAC_associate: LDAC Catalogue Matching - :command:`associate` ============================================== The object matching process implemented within the LDAC tools is a two-stage process. First, the program :command:`associate` matches sources in different catalogues *(we neglect here the possibility of matches within the same catalogue)*. The extent of each source is characterised by an (error) ellipse in a two-dimensional position space, for instance sky-coordinates with `Ra` and `Dec` or pixel coordinates with `X` and `Y`. To successfully run :command:`associate` a minimum set of parameters needs to be present for each object in each catalogue that participates in the matching process. The naming conventions in the following table list parameter names of `SExtractor`_. However, the actually used names can differ and are subject to settings in a configuration file. We list possible parameters for a matching in Sky / Pixel coordinates: +---------------------------------------------------+------------------------------------------+ | **Parameter SKY / PIXEL** | **Meaning for Object Association** | +===================================================+==========================================+ | SeqNr | *Unique* Object Identification Number | +---------------------------------------------------+------------------------------------------+ | (ALPHA_J2000, DELTA_J2000) / (X_IMAGE, Y_IMAGE) | Object Ellipse Center | +---------------------------------------------------+------------------------------------------+ | (A_WORLD, B_WORLD) / (A_IMAGE, B_IMAGE) | Semi major and Semi minor ellipse axis | +---------------------------------------------------+------------------------------------------+ | THETA_WORLD / THETA_IMAGE | position angle of semi major axis | +---------------------------------------------------+------------------------------------------+ | Flag | Extraction - Quality Flags of the object | +---------------------------------------------------+------------------------------------------+ The :command:`associate` program establishes an association between objects in different catalogues if the center position of one object lies within the ellipse spanned by the other. The following figure illustrates several association possibilities - it is taken from `Erik Deuls Pipeline Documentation`__ __ LDAC_docu_ .. image:: ../_images/LDAC_associate.png :align: center :alt: Object merging with associate In the following examples we have an LDAC catalogue `W3m0m2_photoz.cat <../_static/W3m0m2_photoz.cat>`_ which lists galaxy properties, especially photometric redshifts, in a one square degree area of sky. In addition, we have spectroscopic redshifts from the same sky-area from two different sources. Those are listed in ASCII tables `W3m0m2_specz_1.asc <../_static/W3m0m2_specz_1.asc>`_ and `W3m0m2_specz_2.asc <../_static/W3m0m2_specz_2.asc>`_. Our final goal is to compare the estimated photometric redshifts with their spectroscopic cousins and we need to merge the three catalogues. The first step is to convert the ASCII tables into the LDAC format and then to perform object association. Note that we need to give error ellipses, sequence numbers and a flag key to the obejcts in the ASCII tables. For the ellipses we consider circles with a radius of half an arcsecond and the quality flags are set to zero (object is o.k.) everywhere. We perform this initial modification of the ASCII catalogues with a simple :command:`awk` command: :: user$ less W3m0m2_specz_1.asc 213.8106384 51.94233322 0.2906414866 213.8327332 51.95418167 0.9577051401 213.8175049 51.95326233 0.9090785980 . . user$ awk '{printf("%d %15.8f %15.8f %15.8f %15.8f %f %d %f\n", NR, $1, $2, 0.5 / 3600., 0.5 / 3600., 0., 0, $3)}' W3m0m2_specz_1.asc >\ tmp.asc user$ asctoldac -i tmp.asc -o W3m0m2_zspec_1.cat -t OBJECTS -c asctoldac_specz.conf user$ awk '{printf("%d %15.8f %15.8f %15.8f %15.8f %f %d %f\n", NR, $1, $2, 0.5 / 3600., 0.5 / 3600., 0., 0, $3)}' W3m0m2_specz_2.asc >\ tmp.asc user$ asctoldac -i tmp.asc -o W3m0m2_zspec_2.cat -t OBJECTS -c asctoldac_specz.conf user$ rm tmp.asc user$ associate Object Association Program Version: 1.3.4 (Aug 24 2010) (Pipeline Global Version: 1.2.0) Usage: associate -i incat1 [incat2 [incat3 [...]]] (input catalogs) -o outcat1 [outcat2 [outcat3 [...]]] (output catalogs) [-t table_name] (table name of objects) [-c conf_file] [-OPTION1 OPTVAL1 [-OPTION OPTVAL2] ...] user$ associate -i W3m0m2_photoz.cat W3m0m2_zspec_1.cat W3m0m2_zspec_2.cat \ -o assoc_1.cat assoc_2.cat assoc_3.cat \ -t OBJECTS -c associate_photoz_specz.conf Object Association Program Version: 1.3.4 (Aug 24 2010) (Pipeline Global Version: 1.2.0) Reading file W3m0m2_zspec_1.cat Reading file W3m0m2_zspec_2.cat Done .. note:: #. All catalogues that participate in an association process must carry the object properties in tables with the same names and in keys with the same names and the :ref:`the same keytypes ` #. The configuration file `associate_photoz_specz.conf <../_static/associate_photoz_specz.conf>`_ lists the keys which hold the ellipse parameters. In addition, there is the possibility to exclude *problematic* sources, characterised by certain values of `Flag`, from the association process. The parameter `INTER_COLOR_TOL` is a scaling factor. The object ellipse parameters representing the semi-minor and semi-major axis are scaled with that value before the association takes place. Hence, it is easy to redo the association with different positional uncertainties. #. You can simultaneously associate up to 255 catalogues. .. warning:: #. If association should happen in pixel coordinates, the parameter `-p` **must** be given to the :command:`associate` program. :command:`associate` will **NOT** issue warnings or errors if you forget this parameter! #. You must ensure that all involved association parameters (`ALPHA_J2000` etc.) are of :ref:`the same keytype ` in all involved catalogues. This is avery common source for difficult to spot errors! The :command:`associate` program creates for each input catalogue a corresponding output catalogue. Note that the *order is important*, i.e. the first input catalogue is linked to the first output catalogue and so on. : :: user$ ldactoasc -i assoc_1.cat -t OBJECTS -k SeqNr Pair_0 Pair_1 Pair_2 # 1 SeqNr Running object number # 2 Pair_0 Pairing with catalog dummy # 3 Pair_1 Pairing with catalog Unknown # 4 Pair_2 Pairing with catalog Unknown 35 0 0 0 . 61 0 487 0 . 2775 0 0 114 . 84164 0 4773 72 :command:`associate` adds `Pair` keys to the objects tables which carry the information on the matching. Note that the counting starts at zero for the first catalogue. The above outputs reads as follows: #. Object 35 in the first catalogue has no matches with objects in other catalogues #. Object 61 in the first catalogue has a match with object 487 from the second catalogue #. Object 2775 from the first catalogue has a match with object 114 from the third catalogue #. Object 84164 from the first catalogue has a match with obejct 4773 from the second catalogue and a match with object 72 from the third catalogue .. note:: A number unequal to zero in the `Pair_0` column of `assoc_1.cat` would mean that an object of `assoc_1.cat` has a match within that catalogue (close pairs)! Correspondingly for the `assoc_2.cat` and `assoc_3.cat`: :: user$ ldactoasc -i assoc_2.cat -t OBJECTS -k SeqNr Pair_0 Pair_1 Pair_2 # 1 SeqNr Object Sequence Number [""] # 2 Pair_0 Pairing with catalog dummy # 3 Pair_1 Pairing with catalog Unknown # 4 Pair_2 Pairing with catalog Unknown 1 0 0 0 . 526 587 0 0 527 0 0 0 528 0 0 0 529 642 0 0 . #. Object 526 in the second catalogue has a match with object 587 from the first catalogue #. Objects 527 and 528 from the second catalogue have no matches with obejcts in other catalogues #. Object 529 in the second catalogue has a match with object 642 from the first catalogue The information of the `Pair` keys can be used with the program :command:`make_ssc` to create a catalogue with merged object information. .. _LDAC_make_ssc: LDAC Catalogue Matching - :command:`make_ssc` ============================================= The :command:`make_ssc` uses the `Pairs` information from :command:`associate` to create a merged catalogue: :: user$ less make_ssc_photoz_specz.conf COL_NAME = ALPHA_J2000 COL_INPUT = ALPHA_J2000 COL_MERGE = AVE_REG COL_CHAN = 0 # . COL_NAME = Z_SPEC_1 COL_INPUT = Z_SPEC COL_MERGE = AVE_REG COL_CHAN = 1 # . user$ make_ssc Source Catalog Production Program Version: 2.0.4 (Aug 24 2010) Usage: make_ssc -i incat1 [incat2 [..]] (input catalogs) -o outcat (output catalog) [-t table_name] (input catalog tablename to merge) [-p table_name catalog_nr [table_name catalog_nr ...]] (tables to copy from input to output) [-c config_file] [-OPTION1 OPTVAL1 [-OPTION OPTVAL2] ...] user$ make_ssc -i assoc_1.cat assoc_2.cat assoc_3.cat -o merged.cat \ -c make_ssc_photoz_specz.conf Source Catalog Production Program Version: 2.0.4 (Aug 24 2010) Indexing channel 2 WARNING(ssc_put_row): No key Ra in table PSSC Objects read: 53248, written: 49195 user$ ldacdesc -i merged.cat . . ****** Table #2 Extension type:.........BINTABLE Extension name:.........PSSC . . user$ ldacdesc -i merged.cat -t PSSC Key name:...............ALPHA_J2000 . Key name:...............DELTA_J2000 . Key name:...............Z_B . Key name:...............Z_SPEC_1 . Key name:...............Z_SPEC_2 . Key name:...............N_00 . Key name:...............N_01 . Key name:...............N_02 . Key name:...............SeqNr :command:`make_ssc` creates a table `PSSC` with the merged information. The configuration file `make_ssc_photoz_specz.conf <../_static/make_ssc_photoz_specz.conf>`_ for :command:`make_ssc` consists of blocks that determine which quantities from the initial catalogue are transfered how to the merged one: +---------------------+---------------------------------------------------------------+ | COL_NAME = Z_SPEC_1 | There will be a key `Z_SPEC_1` in `PSSC` | +---------------------+---------------------------------------------------------------+ | COL_INPUT = Z_SPEC | The original key has the name `Z_SPEC` | +---------------------+---------------------------------------------------------------+ | COL_MERGE = AVE_REG | This line is always `AVE_REG` for the purpose discussed here | +---------------------+---------------------------------------------------------------+ | COL_CHAN = 1 | The input key comes from the *second* input catalogue | | | (counting starts with 0 for the first catalogue) | +---------------------+---------------------------------------------------------------+ .. note:: :command:`make_ssc` offers a rich set of further options in the merging process. Please consult `Erik Deuls Pipeline Documentation`__ for thses possibilities. __ LDAC_docu_ .. warning:: As :command:`associate`, :command:`make_ssc` often produces difficult to find errors if keys with the same name in different input catalogues **DO NOT** have :ref:`the same keytype `! The keys `N_XX` in the `PSSC` table give the number of obejcts that participate from the corresponding input catalogue (counting starts at zero!) in that entry: :: user$ ldactoasc -i merged.cat -t PSSC # 1 ALPHA_J2000 Right ascension of barycenter (J2000) [deg] # 2 DELTA_J2000 Declination of barycenter (J2000) [deg] # 3 Z_B "" [""] # 4 Z_SPEC_1 spectroscopic redshift [""] # 5 Z_SPEC_2 spectroscopic redshift [""] # 6 N_00 Number of objects merged for channel 0 # 7 N_01 Number of objects merged for channel 1 # 8 N_02 Number of objects merged for channel 2 # 9 SeqNr Object sequence number (from make_ssc) 214.2449074 52.12472016 0.7200000000 0.00000 0.00000 1 0 0 1 214.2613235 52.12476932 0.4800000000 0.00000 0.00000 1 0 0 2 214.5122864 52.12514428 0.8800000000 0.00000 0.00000 1 0 0 3 214.2435491 52.12493391 0.5200000000 0.00000 0.00000 1 0 0 4 . . 213.8923482 52.12469253 0.8000000000 0.765736 0.00000 1 1 0 15 . . 0.000000000 0.000000000 0.000000000 0.00000 0.132505 0 0 1 49747 0.000000000 0.000000000 0.000000000 0.00000 0.132391 0 0 1 49748 In the first lines, only the first object catalogue contributes. Hence, there are entries fro the coordinates and the photometric redshift `Z_B`. As no object from the second and third catalogue contributes the columns for `Z_SPEC_1` and `Z_SPEC_2` are filled with zero. Correspondingly, if no object from the first catalogue contrbutes, the coordinates and `Z_B` entries are set to zero. The `N_XX` keys allows us now an easy filtering of important information: We plot comparisons of photometric redshifts with their spectroscopic counterparts: :: user$ ldacfilter -i merged.cat -o comp_specz_1.cat -t PSSC -c "(N_00=1)AND(N_01=1);" user$ ldacfilter -i merged.cat -o comp_specz_2.cat -t PSSC -c "(N_00=1)AND(N_02=1);" user$ ldacplot.py -i comp_specz_1.cat -t PSSC -k Z_B Z_SPEC_1 -l 0 1.5 0 1.5 \ -ti "Z_B vs. Z_SPEC_1" -o z_b_z_spec_1.png Saving plot to z_b_z_spec_1.png user$ ldacplot.py -i comp_specz_2.cat -t PSSC -k Z_B Z_SPEC_1 -l 0 0.5 0 0.5 \ -ti "Z_B vs. Z_SPEC_2" -o z_b_z_spec_2.png Saving plot to z_b_z_spec_2.png .. image:: ../_images/z_b_z_spec_1.png :scale: 30% :align: left :alt: Z_B Z_SPEC_1 comparison .. image:: ../_images/z_b_z_spec_2.png :scale: 30% :align: right :alt: Z_B Z_SPEC_2 comparison .. include:: links.txt