U/mrabus/bulge exploration

Projects/Bulge_exploration/Explore_Bulge_Part1A_IngestDataSet.ipynb

@@ -0,0 +1,213 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Bulge data ingestion\n",
+    "\n",
+    "In this notebook we show how to ingest the raw DECam images. We are using one of the five fields of the bulge data set from [Saha et al. (2019)](https://arxiv.org/abs/1902.05637). The field has been observed from Apr. 1 - 3, 2015, with 43 visits and 62 detectors. The raw data is available on `/project/stack-club/course_data/decam-bulge-rawdata`. \n",
+    "\n",
+    "The raw images occupy around 50 GB of disk space.\n",
+    "\n",
+    "**Ingesting and processing this one field takes several days and around 300 GB of disk space (42 visits x 62 CCDs).**\n",
+    "\n",
+    "We have created another Jupyter notebook `Explore_Bulge_singleVisit.ipynb`, which only ingest a single visit and a single CCD speeding up the process of ingestion to a few minutes. The whole ingested data set can be found `/project/stack-club/course_data/DECAM_BULGE`.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import numpy as np\n",
+    "import sqlite3\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setting up the directories."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Raw data directory\n",
+    "DATA_DIR = '/project/stack-club/course_data/decam-bulge-rawdata/'\n",
+    "\n",
+    "# Repo directory\n",
+    "REPO_DIR = '/project/stack-club/course_data/DECAM_BULGE/' \n",
+    "CALIB_DIR = REPO_DIR + \"CALIB\"\n",
+    "RERUN_DIR = REPO_DIR + \"rerun\"\n",
+    "\n",
+    "# Create the REPO_DIR and add the mapper.\n",
+    "! mkdir -p {REPO_DIR}\n",
+    "! echo \"lsst.obs.decam.DecamMapper\" > {REPO_DIR+\"_mapper\"}\n",
+    "! mkdir -p {CALIB_DIR}\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Following the standard procedure, we start by linking the reference catalogs to the respository directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "args = REPO_DIR+'/ref_cats'\n",
+    "!mkdir {args}\n",
+    "\n",
+    "args = '/datasets/refcats/htm/v1/sdss-dr9-fink-v5b/ '+REPO_DIR+'/ref_cats/sdss'\n",
+    "! ln -s {args}\n",
+    "\n",
+    "args = '/datasets/refcats/htm/v1/ps1_pv3_3pi_20170110/ '+REPO_DIR+'/ref_cats/ps1_pv3_3pi_20170110'\n",
+    "! ln -s {args}\n",
+    "\n",
+    "args = '/datasets/refcats/htm/v1/gaia_DR1_v1/ '+REPO_DIR+'/ref_cats/gaia'\n",
+    "! ln -s {args}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## We have to ingest several calibration images like crosstalk, defects, flats, biases and the images itself:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ingesting crosstalk\n",
+    "args = REPO_DIR + ' --calib ' + CALIB_DIR + ' ' + DATA_DIR + 'DECam/crosstalk/'\n",
+    "! ingestCuratedCalibs.py {args}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ingesting defects\n",
+    "args = REPO_DIR + ' --calib ' + CALIB_DIR + ' ' + DATA_DIR + 'DECam/defects/'\n",
+    "! ingestCuratedCalibs.py {args}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ingesting the raw images\n",
+    "args = REPO_DIR + ' ' + DATA_DIR + \"raw/c4d_*.fits.fz --mode=link\"\n",
+    "! ingestImages.py {args} "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ingesting flat and bias images\n",
+    "args = REPO_DIR + ' --calib ' + CALIB_DIR + ' ' + DATA_DIR + \\\n",
+    "        \"cal/c4d_150403*.fits.fz --mode=link --validity 999 \"\n",
+    "! ingestCalibs.py {args}\n",
+    "\n",
+    "#Works only if I ingest one night of cal data????"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run `processCcd.py` in command line!!\n",
+    "\n",
+    "`processCcd.py` will several image calibration and analsis steps, including astrometric solution, source detection and photometry.\n",
+    "\n",
+    "\n",
+    "This will take a lot of time and space. It will also output the log and fill-up the notebook with data, therefore we run this command in a terminal.\n"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "'''\n",
+    "#args = REPO_DIR + \" --calib \" + CALIB_DIR + \" --rerun \" + RERUN_DIR + \\\n",
+    "#        \" --id \" \n",
+    "#! processCcd.py {args}\n",
+    "#print(args)\n",
+    "'''"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "processCcd.py /project/stack-club/course_data/DECAM_BULGE/ --calib /project/stack-club/course_data/DECAM_BULGE/CALIB --rerun /project/stack-club/course_data/DECAM_BULGE/rerun --id > /home/mrabus/process_log.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## After ingesting and processing the images we can continue with Part 2."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "LSST",
+   "language": "python",
+   "name": "lsst"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}