.. -*- coding: utf-8 -*-
.. _THELI_concepts:
************************
THELI - General Concepts
************************
This section describes the most important concepts for the use of
THELI. It covers aspects of running individual processing tasks and
the initial organisation of your raw imaging data. We assume use
of THELI on **ONE** (possibly multi-processor and/or multi-core)
machine with all data-processing happening on a single disk. Please
`email_me`_ if you intend to use THELI on distributed computer clusters
with the data split over different disks.
.. _THELI_reducedir:
THELI The reduce directory
==========================
**All** THELI reductions and script calls are launched within a *reduce*
directory (we refer to the complete absolute path of this directory
as REDUCEDIR). It can be any directory with an arbitrary
name. When being created, it has to contain all the scripts
and configuration files that were created during THELI install in
:file:`{dir}/theli-{x}.{x}.{x}/scripts/{MACHINE}`.
THELI Instrument Specification
==============================
The THELI scripts are called without any instrument specific arguments.
Most instrument interna are stored in configuration files with the name
*INSTRUMENT*.ini. The THELI standrad distribution currently contains
the necessary setup for the instruments:
* `WFI\@MPG/ESO2.2m`__ (WFI.ini)
* `MEGAPRIME\@CFHT `_
(MEGAPRIME.ini).
__ ESO_WFI_
The instrument for which you want to process data is specified in the
UNIX environment variable *INSTRUMENT*. For instance, within the *bash* shell
you would issue the command::
export INSTRUMENT=WFI
for a reduction of WFI\@MPG/ESO2.2m data. THELI processing tasks then
read instrument specific information from the corresponding files. If
you want to setup a new instrument you need to provide the following:
* The THELI instrument configuration file. The WFI configuration file
is discussed in more detail in the :ref:`THELI_instrument_setup` section.
* An instrument specific preparation script which transforms raw data to the
THELI structure:
* THELI entirely works on single-chip data. Most telescopes deliver
exposures from multi-chip instruments in a so-called
Multi-Extension-FITS (MEF) file. Hence, we need to split these
data up first.
* For many instruments, the raw MEF files contain the single-chip
data in various orientations with respect to the sky; this mostly
concerns flips in the x-coordinate or rotations by multiple of 90
degrees. To make a general processing as smooth as possible, THELI
requires all single, splitted data with the same orientation. The
setup script is responsible for the necessary image pixel
transformations.
* THELI uses a fixed chip-labelling system to identify the position
of a given chip within the mosaic. If the THELI scheme differs
from the instrument specific one, the chip-numbering needs to be
modified.
* To process data from different instruments in a coherent way, we need to
agree on a set of required FITS image header information. The instrument
preparation script also needs to set this up properly. See also
the :ref:`THELI_image_header` section.
* The astrometric tool `scamp`_ requires a reasonable zeroth order
astrometric solution for good results. This needs to be created
and stored in a file called *INSTRUMENT*.ahead
THELI work-flow
===============
A THELI reduction session consists of the successive call of processing
scripts that perform each a small and well defined part of data reduction.
After having organised your data and having setup a :term:`REDUCEDIR`, the
beginning of a WFI\@MPG/ESO2.2m data set reduction could look like:
.. code-block:: bash
# specify instrument:
export INSTRUMENT=WFI
# split up WFI Multi-Extension FITS Files
./process_split_WFI_eclipse.sh
./process_split_WFI_eclipse.sh
.
.
# process BIAS frames:
./parallel_manager.sh ./process_bias_eclipse_para.sh
# process FLAT fields:
./parallel_manager.sh ./process_flat_eclipse_para.sh
The typical THELI work-flow is to create *superscripts* that collect several
indivdual steps and that allow an unproblematic rerun of old processings if
necessary. These *superscripts* also serve as templates for the processing of
data sets with similar characteristica; for instance in a long-term survey
operation.
THELI parallel processing
=========================
Many THELI scripts operate on individual chips from a multi-CCD camera.
Typically the processing on different chips is completely independent from
each other, and hence such tasks can be parallelised in a natural way.
Scripts that allow parallelisation are indicated by the ending ``para.sh``,
e.g. ``process_flat_eclipse_para.sh`` (process flat-field images),
``create_skysub_para.sh`` (subtract the sky-background). THELI implements
a direct and simple parallelisation scheme. Scripts that can be parallelised
are not called directly, but via a scripted called
``parallel_manager.sh``. A typical call to such a script looks like:
.. code-block:: bash
./parallel_manager.sh ./create_skysub_para.sh
The ``parallel_manager.sh`` would divide the chips of your camera and
launch as many instances of ``create_skysub_para.sh`` as processors/cores are
available on your machine.
.. note::
You have to manually provide THELI with the available CPUs/cores of
your machine. To this end, edit the configuration file
``progs.ini`` in :term:`REDUCEDIR` and change the line ``NPARA=1``
accordingly. A simple way to find out the number to put on
a **Linux** machine is the command ``grep process /proc/cpuinfo | wc
-l``
.. note::
It can be tedious to interrupt a THELI parallel task with the system
:command:`kill` command because the ``parallel_manager`` spawns subprocesses
that need to be killed separately.
To cleanly get rid of these jobs use the ``parakill.sh`` THELI script.
.. include:: links.txt