THELI - General Concepts

This section describes the most important concepts for the use of THELI. It covers aspects of running individual processing tasks and the initial organisation of your raw imaging data. We assume use of THELI on ONE (possibly multi-processor and/or multi-core) machine with all data-processing happening on a single disk. Please email_me if you intend to use THELI on distributed computer clusters with the data split over different disks.

THELI The reduce directory

All THELI reductions and script calls are launched within a reduce directory (we refer to the complete absolute path of this directory as REDUCEDIR). It can be any directory with an arbitrary name. When being created, it has to contain all the scripts and configuration files that were created during THELI install in dir/theli-x.x.x/scripts/MACHINE.

THELI Instrument Specification

The THELI scripts are called without any instrument specific arguments. Most instrument interna are stored in configuration files with the name INSTRUMENT.ini. The THELI standrad distribution currently contains the necessary setup for the instruments:

• WFI@MPG/ESO2.2m (WFI.ini)
• MEGAPRIME@CFHT (MEGAPRIME.ini).

The instrument for which you want to process data is specified in the UNIX environment variable INSTRUMENT. For instance, within the bash shell you would issue the command:

export INSTRUMENT=WFI

for a reduction of WFI@MPG/ESO2.2m data. THELI processing tasks then read instrument specific information from the corresponding files. If you want to setup a new instrument you need to provide the following:

• The THELI instrument configuration file. The WFI configuration file is discussed in more detail in the THELI - Instrument Setup section.
• An instrument specific preparation script which transforms raw data to the THELI structure:
  • THELI entirely works on single-chip data. Most telescopes deliver exposures from multi-chip instruments in a so-called Multi-Extension-FITS (MEF) file. Hence, we need to split these data up first.
  • For many instruments, the raw MEF files contain the single-chip data in various orientations with respect to the sky; this mostly concerns flips in the x-coordinate or rotations by multiple of 90 degrees. To make a general processing as smooth as possible, THELI requires all single, splitted data with the same orientation. The setup script is responsible for the necessary image pixel transformations.
  • THELI uses a fixed chip-labelling system to identify the position of a given chip within the mosaic. If the THELI scheme differs from the instrument specific one, the chip-numbering needs to be modified.
  • To process data from different instruments in a coherent way, we need to agree on a set of required FITS image header information. The instrument preparation script also needs to set this up properly. See also the THELI - Image Header section.
• The astrometric tool scamp requires a reasonable zeroth order astrometric solution for good results. This needs to be created and stored in a file called INSTRUMENT.ahead

THELI work-flow

A THELI reduction session consists of the successive call of processing scripts that perform each a small and well defined part of data reduction. After having organised your data and having setup a REDUCEDIR, the beginning of a WFI@MPG/ESO2.2m data set reduction could look like:

# specify instrument:
export INSTRUMENT=WFI

# split up WFI Multi-Extension FITS Files
./process_split_WFI_eclipse.sh <arguments to script>
./process_split_WFI_eclipse.sh <arguments to script>

.
.

# process BIAS frames:
./parallel_manager.sh ./process_bias_eclipse_para.sh <arguments to script>


# process FLAT fields:
./parallel_manager.sh ./process_flat_eclipse_para.sh <arguments to script>

The typical THELI work-flow is to create superscripts that collect several indivdual steps and that allow an unproblematic rerun of old processings if necessary. These superscripts also serve as templates for the processing of data sets with similar characteristica; for instance in a long-term survey operation.

THELI parallel processing

Many THELI scripts operate on individual chips from a multi-CCD camera. Typically the processing on different chips is completely independent from each other, and hence such tasks can be parallelised in a natural way. Scripts that allow parallelisation are indicated by the ending para.sh, e.g. process_flat_eclipse_para.sh (process flat-field images), create_skysub_para.sh (subtract the sky-background). THELI implements a direct and simple parallelisation scheme. Scripts that can be parallelised are not called directly, but via a scripted called parallel_manager.sh. A typical call to such a script looks like:

./parallel_manager.sh ./create_skysub_para.sh <arguments to create_skysub_para.sh>

The parallel_manager.sh would divide the chips of your camera and launch as many instances of create_skysub_para.sh as processors/cores are available on your machine.

Note

You have to manually provide THELI with the available CPUs/cores of your machine. To this end, edit the configuration file progs.ini in REDUCEDIR and change the line NPARA=1 accordingly. A simple way to find out the number to put on a Linux machine is the command grep process /proc/cpuinfo | wc -l

Note

It can be tedious to interrupt a THELI parallel task with the system kill command because the parallel_manager spawns subprocesses that need to be killed separately. To cleanly get rid of these jobs use the parakill.sh THELI script.