SHORT SUMMARY ============= This HOWTO describes how to setup a machine to process multi-colour CFHT data. This especially includes the surveys CFHTLS/CARS and NGVS (Next Generation Virgo Survey). File version: 26/03/2014 (Thomas Erben) I. DATA PROCESSING ================== The following sections cover all software/installations which are necessary for data processing. This step ends with having co-added images and associated products. Installation procedures for software to perform some science analysis are given below in section II. NEEDED FILES ============ - THELI INTERNAL FILES: ===================== ldacpipeline-X.XX.XX.tar.gz : main sources for LDAC pipeline (non-public version required!) WIFIX_Distrib_X.X_theli.tgz : sources for Astrometrix; (we do not use Astrometrix currently but the package contains software needed by other modules; hence this needs to be installed) NOTE: Currently (July 2012) only 'stracktools' accesses the cfitsio library installed by WIFIX. Furthermore we need the cdsclient package. stracktools-X.X.X.tar.gz : sources for the THELI satellite detection module pipelog-XXXXXXXX.tgz : J. Dietrichs pipelog package to generate WWW pages; only necessary if check WWW pages should be created CFHTLS-X.XX.XX.tar.gz : a collection of supplementary scripts, source codes, satellite track region files etc. - EXTERNAL FILES/SOFTWARE MODULES =============================== The following files are necessary only if not yet installed on your machine: scamp : an executable of Emmanuel Bertin's scamp program (currently version 1.7.0; http://terapix.iap.fr/rubrique.php?id_rubrique=105); installation from source is a pain! Just download an appropriate executable for your machine. sm2_X_XX.tar.gz : sources for R. Luptin's 'sm' program (any version; not free software but your institute probably has a copy) - PYTHON MODULES ============== You need a recent version of the Python 2.X series installed (V2.6 and higher). In addition you need the Python module 'astropy' and necessary dependencies. See http://www.astropy.org/ for further information. INSTALLATION INSTRUCTIONS ========================= We assume that you want to install THELI in a directory ../somedir/THELI; we call this directory henceforth. We further assume that your machine is 'reasonably' setup for scientific use, i.e. you have the GNU compiler family installed (gcc, g++, g77 or gfortran), perl, python etc. - Other, preliminary machine setup: Make also sure that you have the Linux tools 'gawk' (it must be 'gawk'; other versions of awk are not sufficient), 'find' (GNU version), 'sort' and the script languages perl and python installed. Furthermore you will need the image conversion tool 'convert' installed on your machine (it is part of ImageMagick). Please ensure that your installation of convert can deal with the image formats TIFF, JPG and PS. - If necessary install 'SuperMongo' (sm) and scamp and make sure that the executables 'sm' and 'scamp' are in your PATH, i.e. they are found if you simply call 'sm' and 'scamp' in an arbitrary directory. - Install the WIFIX (Astrometrix/Photometrix) package Unpack in tar xvfz WIFIX_Distrib_X.X_theli.tar.gz cd WIFIX_Distrib_X.X_theli/Install From therein run perl mkStart -i=/PipeLibs This creates the files /PipeLibs/WIFIX_pipe.csh /PipeLibs/WIFIX_pipe.bash Depending on your shell, do either source /home//THELI/PipeLibs/WIFIX_pipe.csh rehash or, if using bash . /home//THELI/PipeLibs/WIFIX_pipe.bash and put corresponding lines into your .bashrc (or .cshrc) shell init files. Then, install WIFIX, i.e. within WIFIX_Distib_X.X_theli/Install call: perl mkLibs We currently only need the products of the 'mkLibs' stage to compile the 'stracktools' later. Hence, you do NOT need to execute the following two commands perl mkPerl perl mkPipe Successful installation or errors are acknowledged with corresponding messages! - Install the ldacpipeline package Please make sure that files WIFIX_pipe.csh (see above) are properly sourced before continuing! Unpack in tar xvfz ldacpipeline-X.XX.XX.tar.gz cd ldacpipeline-X.XX.XX/scripts/pipe_install Edit the file 'install.sh' Modify the line 'PD=.......' to point to your 'ldacpipeline-X.XX.XX' directory. Just IGNORE all other directory lines; most of them are obsolete and no longer used! Run './install.sh -m "TEST"' and record the machine type (MTYPE) in the phrase 'Your machine has type MTYPE'. Correct any problems the script might report. Run './install.sh -m "ALL"' This should compile all necessary programs Source the script pipe_env.csh.MTYPE (or pipe_env.bash.MTYPE) and put corresponding lines into your .cshrc (or .bashrc) configuration files. This is important before you continue! - Test the installation you have done so far Open a NEW console/shell and type 'ldactoasc', 'imcombflat', 'sex_theli' If you get program start messages your 'ldacpipeline' installation seems to be fine. Please do this really in a 'new' shell to see whether your configuration files are sourced correctly. - Create a 'reduce' directory () All THELI reductions and script calls are done in a 'reduce' directory (the absolute path to this directory is called henceforth). Please create one somewhere on your hard disks. Copy everything from /ldacpipeline-X.XX.XX/scripts/MTYPE to (lots of scripts and some config files). This has to be done before installing the 'stracktools' (see below)! - Install the stracktools Unpack in tar xvfz stracktools-X.X.tar.gz cd stracktools Call './to_update.pl ' This should compile all sources and modify the file 'progs.ini' in (that is why you should have created it before this step). - Install WWW page creation tools (ONLY required if you want check WWW pages to be created; but install the python package 'numpy' in any case!) Install 'pyfits' and/or 'numpy' if not yet on your machine; you can test whether it is on your machine by typing 'python' and then, within the Python interpreter, do a 'import pyfits' AND 'import numpy'. If you get error messages you need to setup the corresponding packages. Unpack in tar xvfz pipelog-XXXXXXXX.tgz Add to your environment (.cshrc or .bashrc file) the variable setenv PIPEWWW /pipelog-XXXXXXXX (C-shell) or export PIPEWWW=/pipelog-XXXXXXXX (bash) setenv PYTHONPATH /pipelog-XXXXXXXX (C-shell) export PYTHONPATH=/pipelog-XXXXXXXX (bash-shell) or setenv PYTHONPATH ${PYTHONPATH}:/pipelog-XXXXXXXX (C-shell) export PYTHONPATH=${PYTHONPATH}:/pipelog-XXXXXXXX (bash-shell) if the variable PYTHONPATH was defined before. Copy the script 'pipelog.py' to and call it therein with './pipelog.py' If you get error messages you probably need to setup other Python modules. - You can also skip this step! Test the functionality of the photometric redshift code BPZ With the installation of numpy (done at the last step) the photo-z code BPZ which is packaged with THELI should be functional: Go to /ldacpipeline-X.XX.XX/scripts/BPZ Define the shell environment variable BPZPATH to point to /ldacpipeline-X.XX.XX/scripts/BPZ Run the command 'python useful.py' If you get no error message the code works fine. - Install and setup supplementary codes/files - Unpack in tar xvfz CFHTLS-X.XX.XX.tgz - If you do not yet have installed on your machine the FITSIO tools 'imcopy' and 'fitsverify': Go to /CFHTLS/C-sources and compile the two programs as described in the README. Make sure to copy the two programs to a directory within your PATH! - copy all files *sh, *ret and *ini from /CFHTLS to your 'reduce' directory. - Copy the tarfile GLOBALWEIGHTS.tar.gz from /CFHTLS/GLOBALWEIGHTS to your reduce directory. II. SCIENCE ANALYSIS ==================== Note: The following is very specific to CFHTLS processing. You probably cannot use the science analysis module without modification in another context! This section describes the installation of codes necessary for basic science analysis of THELI processed images. This includes the creation of image masks, of multi-colour catalogues and the estimation of photometric redshifts. NEEDED FILES ============ - THELI INTERNAL FILES ==================== automask-X.X.tgz : sources for the THELI masking module - EXTERNAL FILES ============== GSC.tar.gz : The GSC 1 catalogue needed for mask creation around bright stars (CodeC.tar.gz, README.C, SFD_4096.tar.gz, filter.txt) (see also http://www.astro.princeton.edu/~schlegel/dust/) : sources for the Schlegel dust maps; needed to correct magnitudes for Milky Way dust (photo-z determination) INSTALLATION INSTRUCTIONS ========================= - Install the automask tools Unpack in tar xvfz automask-X.X.tgz cd automask Call './install.sh -m ALL' This should compile the automask codes. See whether a call to .../automask/bin/MTYPE/maskvoids gives you a welcome message. Copy all files from .../automask/scripts/MTYPE to . Include the directory ..../automask/ext_libs/MTYPE/bin into your Unix search PATH for executable programs. - Install the Guide Star Catalogue 1 on your disk: Unpack in tar xvfz GSC.tgz Go to /GSC/src and type 'make'. This should give you the executable 'gsc.exe' (You can ignore errors as long as this file is produced). Move it to /GSC/bin Define the SHELL environment variable GSCDAT to point to /GSC, i.e. include setenv GSCDAT /GSC (C-shell) or export GSCDAT=/GSC (bash) in your shell setup files .cshrc or .bashrc respectively. To test the installation: - Make sure the GSCDAT environment variable is set - Call the executable 'gsc.exe' with 'gsc.exe -c 210.0 +36.0'. You should get a list of seven catalogue entries. - Install the Schlegel dust maps: Copy the 'dust' subdirectory to Unpack the Schlegel dust maps in /dust: tar xvfz SFD_4096.tar.gz Create a subdirectory 'maps' and move the maps (FITS files) therein. Unpack the C code package CodeC.tar.gz tar xvfz CodeC.tar.gz Go to /dust/CodeC, edit the Makefile for correct settings of your machine. Type 'make all' to compile sources. The only important file to get is 'dust_getval' To test the installation define the environment variable DUST_DIR to point to /dust and call 'dust_getval'. Type 210.0 as galactic longitude and 21.0 as galactic latitude. You should get: "210.000 21.000 0.02721" In your reduce directory modify the file 'progs.ini'. You find lines reading 'P_DUSTGET=' and 'DUSTMAPS='. Fill in the location of the dust_getval program and the location of the dust map FITS files.