The LDAC Catalogue Format - Description and General Concepts

Note

The concrete examples on this page use the LDAC catalogue D1_r_sex.cat.

A tar archive with the complete set of LDAC Tools example files is here

Overview

The LDAC (Leiden Data Analysis Center) catalogue format was developed by Erik Deul and Emmanuel Bertin in the late 90’s. Their main intention was to provide astronomical object catalogues and the necessary access tools from Large Imaging Surveys. Amongst others, it is the primary catalogue format for the Deep Near Infrared Survey , the first parts of the ESO Imaging Survey, the GaBoDS Survey and the CARS project. Most importantly, it is the primary binary catalogue option (-CATALOGUE_TYPE FITS_LDAC) of Emmanuel Bertins SExtractor program. As such it is also one of the pillars of the THELI pipeline. Within that project we added new tools and improvements to the existing ones.

Main features of the format are:

• The format and the access tools scale well up to VERY LARGE catalogues. Within the CFHTLenS project we successfully created and manipulated a 50GB large catalogue containing data from a five-band, 72 square degrees patch of the sky.
• Catalogue entries are accessed with meaningful table and key names. There is no need to fiddle with the correspondence of column numbers and column meaning as is the case with traditional ASCII data.
• It is a binary format which allows quick access to data within programs.
• The format is sufficiently close to the FITS standard so that LDAC catalogues can be accessed with standard FITS binary reader tools such as pyfits, IDL or fv.

Structure of an LDAC catalogue

An LDAC catalogue typically consists of several tables. Think of a table as a traditional ASCII table with the data organised in columns. The columns are termed keys in the LDAC language. A single LDAC catalogue can contain an arbitrary number of tables. To find out about the principal contents of an LDAC catalogue, we use the tool ldacdesc (LDAC description). For the following we use the catalogue D1_r_sex.cat which is a small part of a SExtractor output with the (-CATALOGUE_TYPE FITS_LDAC) option.:

user$ ldacdesc -i D1_r_sex.cat

> Reading catalog(s)
------------------Catalog information----------------
Filename:..............D1_r_sex.cat
Number of segments:....3

******  Table #1
        Extension type:.........(Primary HDU)
        Extension name:.........

******  Table #2
        Extension type:.........BINTABLE
        Extension name:.........LDAC_IMHEAD
        Number of dimensions:...2
        Number of elements:.....1
        Number of data fields...1
        Body size:..............9760 bytes

******  Table #3
        Extension type:.........BINTABLE
        Extension name:.........LDAC_OBJECTS
        Number of dimensions:...2
        Number of elements:.....100
        Number of data fields...24
        Body size:..............10200 bytes


> All done

We note that we have two tables with contents and the names LDAC_IMHEAD containing meta-data such as the FITS image header and the chosen SExtractor parameters, and LDAC_OBJECTS which holds extracted parameters from our sources. The first table just is the typical header present in every FITS file. For LDAC_OBJECTS we read:

1. Number of dimensions: 2 The LDAC format only support two-dimensional data
2. Number of elements: 100 The table contains data for 100 entries (100 objects)
3. Number of data fields: 24 Each entry has 24 keys (24 columns)

So, LDAC_OBJECTS contains 24 parameters for 100 objects.

The ldacdesc command is also used to learn about available keys in a particular table:

user$ ldacdesc -i D1_r_sex.cat -t LDAC_OBJECTS

****** Key #1
       Key name:...............NUMBER
       Key comment:............Running object number
       Key type:...............Long Int
       Key dimension:..........0
****** Key #2
       Key name:...............X_IMAGE
       Key comment:............Object position along x
       Key type:...............Float
       Key dimension:..........0
       Key unit:...............pixel
****** Key #3
       Key name:...............Y_IMAGE
       Key comment:............Object position along y
       Key type:...............Float
       Key dimension:..........0
       Key unit:...............pixel

.
.

****** Key #15
       Key name:...............MAG_APER
       Key comment:............Fixed aperture magnitude vector
       Key type:...............Float
       Key dimension:..........1 (4)
       Key unit:...............mag

.
.

****** Key #24
       Key name:...............NIMAFLAGS_ISO
       Key comment:............Number of flagged pixels entering IMAFLAGS_ISO
       Key type:...............Long Int
       Key dimension:..........0

> All done

The names of the keys are self-explanatory. We read:

1. Key name: MAG_APER (The name of the key)

2. Key comment: Fixed aperture magnitude vector (self explanatory)

3. Key type: Float

   This gives the data type of the key. Possible values are:

   • Floating point numbers: Possible are Float (four-byte floating-point numbers) and Double (eight-byte floating point numbers)
   • Integer Number: Possible are Short Int (two-byte signed integers) and Long Int (four-byte signed integers)
   • String for character strings

4. Key dimension: 1 (4)

   This key is:

   • 0 for simple values.
   • The number gives the maximum number of characters in case of keytype String.
   • A value unequal to zero for other keytypes than String means that the quantity is a vector. Here, SExtractor has been asked to evaluate an aperture magnitude in four different apertures (the value in parentheses).

5. Key unit: mag (self explanatory)

History of an LDAC Catalogue

The primary header of each LDAC catalogue (listed as Primary HDU by ldacdesc) contains the complete history of catalogue creation and manipulation. It can be accessed with any standard tool, such as dfits, to examine FITS headers:

user$ dfits D1_r_sex.cat
====> file D1_r_sex.cat (main) <====
SIMPLE  =                    T / LETS STAY SIMPLE
BITPIX  =                    8 /
NAXIS   =                    0 /
EXTEND  =                    T / MORE STUFF MAY FOLLOW
HISTORY
HISTORY sex: THELI Pipeline Version: 0.12.36
HISTORY sex: Catalog created at 2010-08-25T17:29:56
HISTORY
HISTORY ldacfilter: THELI Pipeline Version: 0.12.36
HISTORY ldacfilter called at 2010-08-25T15:37:08
HISTORY ldacfilter calling sequence:
HISTORY ldacfilter -i D1_r_sex.cat -t LDAC_OBJECTS -o D1_r_sex_filt.cat -c (NU
HISTORY MBER<101);
END

user$ fold D1_r_sex.cat | less # the poor mans way to get a FITS header
SIMPLE  =                    T / LETS STAY SIMPLE
BITPIX  =                    8 /
NAXIS   =                    0 /

.
.

END

Working with the LDAC tools

The LDAC tools are a small toolbox of UNIX command line programs to manipulate LDAC catalogues. The calling sequence and the most common command line options are:

user$ ldac.... -i (input catalogue)
               -o (output catalogue)
               -t (table name)
               -k (key name)
               -c (config file or condition)
               -p (proto catalogue)

All programs give information on their accepted command-line arguments when invoking them. Tools that require a configuration file explain its parameters by using the option -@:

user$ ldactoasc
                                ldactoasc V1.3

Purpose: convert a FITS table to ASCII

> SYNTAX: ldactoasc -i Catalog1 Catalog2 [-t <Table_name>] [-k keys] [options]

Options are:    -b (suppress printing of the banner)
                -r (Print the row number)
                -s (Print strings and arrays)
                -q (Quiet flag: defaulted to verbose!)


user$ asctoldac -@

# Configuration parameter file for:
# asctoldac
#
# The separator characters used to separate elements from the ascii file
# Keyword type is TEXT allowing any string
SEPARATOR =
#
# The name of the output column
# Keyword type is TEXTVEC allowing any string
# which could be a repetative keyword with others
COL_NAME =
#

.
.

It clearly needs to be stated that most of the tools have been written many years ago and having a high-end userinterface never was a priority! As Doug Applegate stated it: “The LDAC tools are not the most elegant programs but they get the job done!”

The typical work-flow is the consecutive manipulation and creation of derived catalogues with the LDAC tools:

user$ sex_theli D1_r.V1.6A_Paris.cut.fits -CATALOG_NAME D1_r_sex.cat ..

user$ ldacconv -i D1_r_sex.cat -o tmp.cat1 -b 1 -c MEGAPRIME -f r

user$ ldacrenkey -i tmp.cat1 -o D1_r.cat -t OBJECTS -k MAG_APER MAG_APER_r

user$ rm tmp.cat1

A sequence of LDAC commands often leads to the creation of intermediate catalogues that are handed to another LDAC program. Hence, for large catalogues, most of the processing time is spent in reading/writing data from/to disk.