.. -*- coding: utf-8 -*- .. _THELI_concepts: ************************ THELI - General Concepts ************************ This section describes the most important concepts for the use of THELI. It covers aspects of running individual processing tasks and the initial organisation of your raw imaging data. We assume use of THELI on **ONE** (possibly multi-processor and/or multi-core) machine with all data-processing happening on a single disk. Please `email_me`_ if you intend to use THELI on distributed computer clusters with the data split over different disks. .. _THELI_reducedir: THELI The reduce directory ========================== **All** THELI reductions and script calls are launched within a *reduce* directory (we refer to the complete absolute path of this directory as REDUCEDIR). It can be any directory with an arbitrary name. When being created, it has to contain all the scripts and configuration files that were created during THELI install in :file:`{dir}/theli-{x}.{x}.{x}/scripts/{MACHINE}`. THELI Instrument Specification ============================== The THELI scripts are called without any instrument specific arguments. Most instrument interna are stored in configuration files with the name *INSTRUMENT*.ini. The THELI standrad distribution currently contains the necessary setup for the instruments: * `WFI\@MPG/ESO2.2m`__ (WFI.ini) * `MEGAPRIME\@CFHT `_ (MEGAPRIME.ini). __ ESO_WFI_ The instrument for which you want to process data is specified in the UNIX environment variable *INSTRUMENT*. For instance, within the *bash* shell you would issue the command:: export INSTRUMENT=WFI for a reduction of WFI\@MPG/ESO2.2m data. THELI processing tasks then read instrument specific information from the corresponding files. If you want to setup a new instrument you need to provide the following: * The THELI instrument configuration file. The WFI configuration file is discussed in more detail in the :ref:`THELI_instrument_setup` section. * An instrument specific preparation script which transforms raw data to the THELI structure: * THELI entirely works on single-chip data. Most telescopes deliver exposures from multi-chip instruments in a so-called Multi-Extension-FITS (MEF) file. Hence, we need to split these data up first. * For many instruments, the raw MEF files contain the single-chip data in various orientations with respect to the sky; this mostly concerns flips in the x-coordinate or rotations by multiple of 90 degrees. To make a general processing as smooth as possible, THELI requires all single, splitted data with the same orientation. The setup script is responsible for the necessary image pixel transformations. * THELI uses a fixed chip-labelling system to identify the position of a given chip within the mosaic. If the THELI scheme differs from the instrument specific one, the chip-numbering needs to be modified. * To process data from different instruments in a coherent way, we need to agree on a set of required FITS image header information. The instrument preparation script also needs to set this up properly. See also the :ref:`THELI_image_header` section. * The astrometric tool `scamp`_ requires a reasonable zeroth order astrometric solution for good results. This needs to be created and stored in a file called *INSTRUMENT*.ahead THELI work-flow =============== A THELI reduction session consists of the successive call of processing scripts that perform each a small and well defined part of data reduction. After having organised your data and having setup a :term:`REDUCEDIR`, the beginning of a WFI\@MPG/ESO2.2m data set reduction could look like: .. code-block:: bash # specify instrument: export INSTRUMENT=WFI # split up WFI Multi-Extension FITS Files ./process_split_WFI_eclipse.sh ./process_split_WFI_eclipse.sh . . # process BIAS frames: ./parallel_manager.sh ./process_bias_eclipse_para.sh # process FLAT fields: ./parallel_manager.sh ./process_flat_eclipse_para.sh The typical THELI work-flow is to create *superscripts* that collect several indivdual steps and that allow an unproblematic rerun of old processings if necessary. These *superscripts* also serve as templates for the processing of data sets with similar characteristica; for instance in a long-term survey operation. THELI parallel processing ========================= Many THELI scripts operate on individual chips from a multi-CCD camera. Typically the processing on different chips is completely independent from each other, and hence such tasks can be parallelised in a natural way. Scripts that allow parallelisation are indicated by the ending ``para.sh``, e.g. ``process_flat_eclipse_para.sh`` (process flat-field images), ``create_skysub_para.sh`` (subtract the sky-background). THELI implements a direct and simple parallelisation scheme. Scripts that can be parallelised are not called directly, but via a scripted called ``parallel_manager.sh``. A typical call to such a script looks like: .. code-block:: bash ./parallel_manager.sh ./create_skysub_para.sh The ``parallel_manager.sh`` would divide the chips of your camera and launch as many instances of ``create_skysub_para.sh`` as processors/cores are available on your machine. .. note:: You have to manually provide THELI with the available CPUs/cores of your machine. To this end, edit the configuration file ``progs.ini`` in :term:`REDUCEDIR` and change the line ``NPARA=1`` accordingly. A simple way to find out the number to put on a **Linux** machine is the command ``grep process /proc/cpuinfo | wc -l`` .. note:: It can be tedious to interrupt a THELI parallel task with the system :command:`kill` command because the ``parallel_manager`` spawns subprocesses that need to be killed separately. To cleanly get rid of these jobs use the ``parakill.sh`` THELI script. .. include:: links.txt