.. -*- coding: utf-8 -*- .. _LDAC_concepts: ============================================================ The LDAC Catalogue Format - Description and General Concepts ============================================================ .. note:: The concrete examples on this page use the LDAC catalogue `D1_r_sex.cat <../_static/D1_r_sex.cat>`_. A tar archive with the complete set of LDAC Tools example files is `here <../_static/LDAC_Tools_Examples.tgz>`_ Overview ======== The LDAC (Leiden Data Analysis Center) catalogue format was developed by Erik Deul and Emmanuel Bertin in the late 90's. Their main intention was to provide astronomical object catalogues and the necessary access tools from *Large* Imaging Surveys. Amongst others, it is the primary catalogue format for the `Deep Near Infrared Survey `_ , the first parts of the `ESO Imaging Survey `_, the `GaBoDS Survey `_ and the `CARS project `_. Most importantly, it is the primary binary catalogue option ``(-CATALOGUE_TYPE FITS_LDAC)`` of Emmanuel Bertins `SExtractor`_ program. As such it is also one of the pillars of the :ref:`THELI` pipeline. Within that project we added new tools and improvements to the existing ones. Main features of the format are: * The format and the access tools scale well up to **VERY LARGE** catalogues. Within the `CFHTLenS project `_ we successfully created and manipulated a 50GB large catalogue containing data from a five-band, 72 square degrees patch of the sky. * Catalogue entries are accessed with meaningful *table* and *key* names. There is no need to fiddle with the correspondence of column numbers and column meaning as is the case with traditional ASCII data. * It is a *binary* format which allows quick access to data within programs. * The format is sufficiently close to the `FITS`_ standard so that LDAC catalogues can be accessed with standard FITS binary reader tools such as `pyfits`_, `IDL `_ or `fv `_. Structure of an LDAC catalogue ============================== An LDAC catalogue typically consists of several *tables*. Think of a *table* as a traditional ASCII table with the data organised in columns. The columns are termed *keys* in the LDAC language. A single LDAC catalogue can contain an arbitrary number of tables. To find out about the principal contents of an LDAC catalogue, we use the tool :command:`ldacdesc` (LDAC description). For the following we use the catalogue `D1_r_sex.cat <../_static/D1_r_sex.cat>`_ which is a small part of a `SExtractor`_ output with the ``(-CATALOGUE_TYPE FITS_LDAC)`` option.:: user$ ldacdesc -i D1_r_sex.cat > Reading catalog(s) ------------------Catalog information---------------- Filename:..............D1_r_sex.cat Number of segments:....3 ****** Table #1 Extension type:.........(Primary HDU) Extension name:......... ****** Table #2 Extension type:.........BINTABLE Extension name:.........LDAC_IMHEAD Number of dimensions:...2 Number of elements:.....1 Number of data fields...1 Body size:..............9760 bytes ****** Table #3 Extension type:.........BINTABLE Extension name:.........LDAC_OBJECTS Number of dimensions:...2 Number of elements:.....100 Number of data fields...24 Body size:..............10200 bytes > All done We note that we have two tables with contents and the names *LDAC_IMHEAD* containing meta-data such as the FITS image header and the chosen SExtractor parameters, and *LDAC_OBJECTS* which holds extracted parameters from our sources. The first table just is the typical header present in every FITS file. For *LDAC_OBJECTS* we read: #. Number of dimensions: 2 The LDAC format only support two-dimensional data #. Number of elements: 100 The table contains data for 100 entries (100 objects) #. Number of data fields: 24 Each entry has 24 keys (24 columns) So, *LDAC_OBJECTS* contains 24 parameters for 100 objects. .. _LDAC_keyinformation: The :command:`ldacdesc` command is also used to learn about available keys in a particular table: :: user$ ldacdesc -i D1_r_sex.cat -t LDAC_OBJECTS ****** Key #1 Key name:...............NUMBER Key comment:............Running object number Key type:...............Long Int Key dimension:..........0 ****** Key #2 Key name:...............X_IMAGE Key comment:............Object position along x Key type:...............Float Key dimension:..........0 Key unit:...............pixel ****** Key #3 Key name:...............Y_IMAGE Key comment:............Object position along y Key type:...............Float Key dimension:..........0 Key unit:...............pixel . . ****** Key #15 Key name:...............MAG_APER Key comment:............Fixed aperture magnitude vector Key type:...............Float Key dimension:..........1 (4) Key unit:...............mag . . ****** Key #24 Key name:...............NIMAFLAGS_ISO Key comment:............Number of flagged pixels entering IMAFLAGS_ISO Key type:...............Long Int Key dimension:..........0 > All done The names of the keys are self-explanatory. We read: #. Key name: MAG_APER (The name of the key) #. Key comment: Fixed aperture magnitude vector (self explanatory) #. Key type: Float This gives the data type of the key. Possible values are: * Floating point numbers: Possible are *Float* (four-byte floating-point numbers) and *Double* (eight-byte floating point numbers) * Integer Number: Possible are *Short Int* (two-byte signed integers) and *Long Int* (four-byte signed integers) * *String* for character strings #. Key dimension: 1 (4) This key is: * 0 for *simple* values. * The number gives the maximum number of characters in case of keytype *String*. * A value unequal to zero for other keytypes than *String* means that the quantity is a *vector*. Here, SExtractor has been asked to evaluate an aperture magnitude in four different apertures (the value in parentheses). #. Key unit: mag (self explanatory) History of an LDAC Catalogue ============================ The primary header of each LDAC catalogue (listed as *Primary HDU* by :command:`ldacdesc`) contains the complete history of catalogue creation and manipulation. It can be accessed with any standard tool, such as `dfits`_, to examine FITS headers: :: user$ dfits D1_r_sex.cat ====> file D1_r_sex.cat (main) <==== SIMPLE = T / LETS STAY SIMPLE BITPIX = 8 / NAXIS = 0 / EXTEND = T / MORE STUFF MAY FOLLOW HISTORY HISTORY sex: THELI Pipeline Version: 0.12.36 HISTORY sex: Catalog created at 2010-08-25T17:29:56 HISTORY HISTORY ldacfilter: THELI Pipeline Version: 0.12.36 HISTORY ldacfilter called at 2010-08-25T15:37:08 HISTORY ldacfilter calling sequence: HISTORY ldacfilter -i D1_r_sex.cat -t LDAC_OBJECTS -o D1_r_sex_filt.cat -c (NU HISTORY MBER<101); END user$ fold D1_r_sex.cat | less # the poor mans way to get a FITS header SIMPLE = T / LETS STAY SIMPLE BITPIX = 8 / NAXIS = 0 / . . END Working with the LDAC tools =========================== The LDAC tools are a small toolbox of UNIX command line programs to manipulate LDAC catalogues. The calling sequence and the most common command line options are: :: user$ ldac.... -i (input catalogue) -o (output catalogue) -t (table name) -k (key name) -c (config file or condition) -p (proto catalogue) All programs give information on their accepted command-line arguments when invoking them. Tools that require a configuration file explain its parameters by using the option `-@`: :: user$ ldactoasc ldactoasc V1.3 Purpose: convert a FITS table to ASCII > SYNTAX: ldactoasc -i Catalog1 Catalog2 [-t ] [-k keys] [options] Options are: -b (suppress printing of the banner) -r (Print the row number) -s (Print strings and arrays) -q (Quiet flag: defaulted to verbose!) user$ asctoldac -@ # Configuration parameter file for: # asctoldac # # The separator characters used to separate elements from the ascii file # Keyword type is TEXT allowing any string SEPARATOR = # # The name of the output column # Keyword type is TEXTVEC allowing any string # which could be a repetative keyword with others COL_NAME = # . . It clearly needs to be stated that most of the tools have been written many years ago and having a high-end userinterface never was a priority! As Doug Applegate stated it: "The LDAC tools are not the most elegant programs but they get the job done!" The typical work-flow is the consecutive manipulation and creation of derived catalogues with the LDAC tools: :: user$ sex_theli D1_r.V1.6A_Paris.cut.fits -CATALOG_NAME D1_r_sex.cat .. user$ ldacconv -i D1_r_sex.cat -o tmp.cat1 -b 1 -c MEGAPRIME -f r user$ ldacrenkey -i tmp.cat1 -o D1_r.cat -t OBJECTS -k MAG_APER MAG_APER_r user$ rm tmp.cat1 A sequence of LDAC commands often leads to the creation of intermediate catalogues that are handed to another LDAC program. Hence, for large catalogues, most of the processing time is spent in reading/writing data from/to disk. .. include:: links.txt