20190721_practical_pandas_tutorial_for_aspiring_data_scientists.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- author: Lee Meng\n",
    "- date: 2019-07-27 21:00\n",
    "- title: 資料科學家的 pandas 實戰手冊：掌握 40 個實用數據技巧\n",
    "- slug: practical-pandas-tutorial-for-aspiring-data-scientists\n",
    "- tags: 資料科學, Python, pandas\n",
    "- description: 熟練地使用 pandas 是資料科學家處理數據與分析時不可或缺的重要技能之一。透過 40 個 pandas 實用技巧，這篇文章將帶你由淺入深地掌握最基礎且重要的 pandas 能力。文中也將介紹多個適合與 pandas 一起使用的強大函式庫，提升你的數據處理能力。\n",
    "- summary: 熟練地使用 pandas 是資料科學家處理數據與分析時不可或缺的重要技能之一。透過 40 個 pandas 實用技巧，這篇文章將帶你由淺入深地掌握最基礎且重要的 pandas 能力。文中也將介紹多個適合與 pandas 一起使用的強大函式庫，提升你的數據處理能力。\n",
    "- image: nick-hillier-yD5rv8_WzxA-unsplash.jpg\n",
    "- image_credit_url: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!quote\n",
    "- 故學然後知不足，教然後知困。知不足，然後能自反也；知困，然後能自強也，故曰：教學相長也。\n",
    "- 《禮記．學記》"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[pandas](https://pandas.pydata.org/) 是 [Python](https://www.python.org/) 的一個資料分析函式庫，提供如 [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) 等十分容易操作的資料結構，是近年做數據分析時不可或需的工具之一。\n",
    "\n",
    "雖然已經有滿坑滿谷的教學文章、影片或是線上課程，正是因為 pandas 學習資源之多，導致初學者常常不知如何踏出第一步。在這篇文章裡頭，我以自身作為資料科學家（**D**ata **S**cientist, DS）的工作經驗，將接近 40 個實用的 pandas 技巧由淺入深地分成 6 大類別：\n",
    "1. [建立 DataFrame](#建立-DataFrame)\n",
    "2. [客製化 DataFrame 顯示設定](#客製化-DataFrame-顯示設定_1)\n",
    "3. [數據清理 & 整理](#數據清理-&-整理_1)\n",
    "4. [取得想要關注的數據](#取得想要關注的數據_1)\n",
    "5. [基本數據處理與轉換](#基本數據處理與轉換_1)\n",
    "6. [簡單匯總 & 分析數據](#簡單匯總-&-分析數據_1)\n",
    "\n",
    "透過有系統地呈現這些 pandas 技巧，我希望能讓更多想要利用 Python 做資料分析或是想成為 DS 的你，能用最有效率的方式掌握核心 pandas 能力；同時也希望你能將自己認為實用但本文沒有提到的技巧與我分享，達到開頭引文所說的**教學相長**：）\n",
    "\n",
    "如果你是使用寬螢幕瀏覽本文，隨時可以點擊左側傳送門瀏覽各章內容："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/toc_overview.mp4\n",
    "- images/pandas/toc_overview.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了 pandas 的操作技巧以外，我在最後一節：[與 pandas 相得益彰的實用工具](#與-pandas-相得益彰的實用工具_1)裡也介紹了幾個實用函式庫。就算你是 pandas 老手，或許也能從中得到些收穫。當然也非常歡迎與我分享其他厲害工具，我會更新到文章裡頭讓更多人知道。\n",
    "\n",
    "前言已盡，讓我們開始這趟 pandas 旅程吧！當然，首先你得 `import pandas`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.24.2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建立 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas 裡有非常多種可以初始化一個 [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) 的技巧。以下列出一些我覺得實用的初始化方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用 Python dict 建立 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 Python 的 `dict` 來初始化 DataFrame 十分直覺。基本上 `dict` 裡頭的每一個鍵值（key）都對應到一個欄位名稱，而其值（value）則是一個 iterable，代表該欄位裡頭所有的數值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col 1</th>\n",
       "      <th>col 2</th>\n",
       "      <th>col 3</th>\n",
       "      <th>col 4</th>\n",
       "      <th>col 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>x</td>\n",
       "      <td>a</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>y</td>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>z</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col 1  col 2 col 3 col 4  col 5\n",
       "0      1     10     x     a      0\n",
       "1      2     20     y     b      1\n",
       "2      3     30     z     c      2"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dic = {\n",
    "    \"col 1\": [1, 2, 3], \n",
    "    \"col 2\": [10, 20, 30],\n",
    "    \"col 3\": list('xyz'),\n",
    "    \"col 4\": ['a', 'b', 'c'],\n",
    "    \"col 5\": pd.Series(range(3))\n",
    "}\n",
    "df = pd.DataFrame(dic)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在需要管理多個 DataFrames 時你會想要用更有意義的名字來代表它們，但在資料科學領域裡只要看到 `df`，每個人都會預期它是一個 **D**ata**F**rame，不論是 Python 或是 R 語言的使用者。\n",
    "\n",
    "很多時候你也會需要改變 DataFrame 裡的欄位名稱："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>10x</th>\n",
       "      <th>col 3</th>\n",
       "      <th>col 4</th>\n",
       "      <th>col 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>x</td>\n",
       "      <td>a</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>y</td>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>z</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x  10x col 3 col 4  col 5\n",
       "0  1   10     x     a      0\n",
       "1  2   20     y     b      1\n",
       "2  3   30     z     c      2"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rename_dic = {\"col 1\": \"x\", \"col 2\": \"10x\"}\n",
    "df.rename(rename_dic, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊也很直覺，就是給一個將舊欄位名對應到新欄位名的 Python `dict`。值得注意的是參數 `axis=1`：在 pandas 裡大部分函式預設處理的軸為列（row）：以 `axis=0` 表示；而將 `axis` 設置為 `1` 則代表你想以行（column）為單位套用該函式。\n",
    "\n",
    "你也可以用 `df.columns` 的方式改欄位名稱："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x(new)</th>\n",
       "      <th>10x(new)</th>\n",
       "      <th>col 3</th>\n",
       "      <th>col 4</th>\n",
       "      <th>col 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>x</td>\n",
       "      <td>a</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>y</td>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>z</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x(new)  10x(new) col 3 col 4  col 5\n",
       "0       1        10     x     a      0\n",
       "1       2        20     y     b      1\n",
       "2       3        30     z     c      2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = ['x(new)', '10x(new)'] + list(df.columns[2:])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用 pd.util.testing 隨機建立 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "當你想要隨意初始化一個 DataFrame 並測試 pandas 功能時，`pd.util.testing` 就顯得十分好用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>xPlIXbQqGU</th>\n",
       "      <td>1.086581</td>\n",
       "      <td>-0.002484</td>\n",
       "      <td>-0.335693</td>\n",
       "      <td>0.226988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IAFe6K8mpA</th>\n",
       "      <td>-0.547556</td>\n",
       "      <td>-0.290935</td>\n",
       "      <td>-0.014313</td>\n",
       "      <td>-0.301007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OoATGY0k2M</th>\n",
       "      <td>1.017636</td>\n",
       "      <td>0.568835</td>\n",
       "      <td>-0.272382</td>\n",
       "      <td>0.659657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>uRN2yGacDw</th>\n",
       "      <td>-0.662390</td>\n",
       "      <td>1.929820</td>\n",
       "      <td>-1.206670</td>\n",
       "      <td>0.250626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ElphZli9nK</th>\n",
       "      <td>-0.697235</td>\n",
       "      <td>0.942415</td>\n",
       "      <td>-0.894887</td>\n",
       "      <td>0.701790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oiEoCPCXK8</th>\n",
       "      <td>-1.049284</td>\n",
       "      <td>-1.019107</td>\n",
       "      <td>-0.640271</td>\n",
       "      <td>-0.613056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NUrQFrYQw1</th>\n",
       "      <td>0.759355</td>\n",
       "      <td>0.717367</td>\n",
       "      <td>-0.449368</td>\n",
       "      <td>1.889321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oC9iiEBneW</th>\n",
       "      <td>0.665412</td>\n",
       "      <td>-0.391204</td>\n",
       "      <td>-0.974010</td>\n",
       "      <td>0.248326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4hD6Eea7yF</th>\n",
       "      <td>-0.862819</td>\n",
       "      <td>2.092149</td>\n",
       "      <td>0.976645</td>\n",
       "      <td>-0.388735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3QD5mMfstw</th>\n",
       "      <td>-0.312762</td>\n",
       "      <td>-0.110278</td>\n",
       "      <td>1.162421</td>\n",
       "      <td>-0.335144</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "xPlIXbQqGU  1.086581 -0.002484 -0.335693  0.226988\n",
       "IAFe6K8mpA -0.547556 -0.290935 -0.014313 -0.301007\n",
       "OoATGY0k2M  1.017636  0.568835 -0.272382  0.659657\n",
       "uRN2yGacDw -0.662390  1.929820 -1.206670  0.250626\n",
       "ElphZli9nK -0.697235  0.942415 -0.894887  0.701790\n",
       "oiEoCPCXK8 -1.049284 -1.019107 -0.640271 -0.613056\n",
       "NUrQFrYQw1  0.759355  0.717367 -0.449368  1.889321\n",
       "oC9iiEBneW  0.665412 -0.391204 -0.974010  0.248326\n",
       "4hD6Eea7yF -0.862819  2.092149  0.976645 -0.388735\n",
       "3QD5mMfstw -0.312762 -0.110278  1.162421 -0.335144"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.util.testing.makeDataFrame().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`head` 函式預設用來顯示 DataFrame 中前 5 筆數據。要顯示後面數據則可以使用 `tail` 函式。\n",
    "\n",
    "你也可以用 `makeMixedDataFrame` 建立一個有各種資料型態的 DataFrame 方便測試："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo1</td>\n",
       "      <td>2009-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>foo2</td>\n",
       "      <td>2009-01-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo3</td>\n",
       "      <td>2009-01-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>foo4</td>\n",
       "      <td>2009-01-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo5</td>\n",
       "      <td>2009-01-07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B     C          D\n",
       "0  0.0  0.0  foo1 2009-01-01\n",
       "1  1.0  1.0  foo2 2009-01-02\n",
       "2  2.0  0.0  foo3 2009-01-05\n",
       "3  3.0  1.0  foo4 2009-01-06\n",
       "4  4.0  0.0  foo5 2009-01-07"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.util.testing.makeMixedDataFrame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其他函式如 `makeMissingDataframe` 及 `makeTimeDataFrame` 在後面的章節都還會看到。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將剪貼簿內容轉換成 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以從 Excel、Google Sheet 或是網頁上複製表格並將其轉成 DataFrame。\n",
    "\n",
    "簡單 2 步驟：\n",
    "- 複製其他來源的表格\n",
    "- 執行 `pd.read_clipboard`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/pandas_clipboard.mp4\n",
    "- images/pandas/pandas_clipboard.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個技巧在你想要快速將一些數據轉成 DataFrame 時非常方便。當然，你得考量重現性（reproducibility）。\n",
    "\n",
    "為了讓未來的自己以及他人可以重現你當下的結果，必要時記得另存新檔以供後人使用：\n",
    "\n",
    "```python\n",
    "df.to_csv(\"some_data.csv\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 讀取線上 CSV 檔"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不限於本地檔案，只要有正確的 URL 以及網路連線就可以將網路上的任意 CSV 檔案轉成 DataFrame。\n",
    "\n",
    "比方說你可以將 Kaggle 著名的[鐵達尼號競賽](https://www.kaggle.com/c/titanic)的 CSV 檔案從網路上下載下來並轉成 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('http://bit.ly/kaggletrain')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下則是另個使用 pandas 爬取網路上數據並作分析的真實案例。\n",
    "\n",
    "我在之前的[ Chartify 教學文](https://leemeng.tw/chartify-a-simple-yet-powerful-python-data-visualization-tool-which-boost-your-productivity-as-a-data-scientist.html)中從臺北市的資料開放平台爬取[ A1 及 A2 類交通事故數據](https://data.taipei/#/dataset/detail?id=2f238b4f-1b27-4085-93e9-d684ef0e2735)後做了簡單匯總："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>發生年</th>\n",
       "      <th>發生月</th>\n",
       "      <th>發生日</th>\n",
       "      <th>發生時</th>\n",
       "      <th>發生分</th>\n",
       "      <th>處理別</th>\n",
       "      <th>區序</th>\n",
       "      <th>肇事地點</th>\n",
       "      <th>死亡人數</th>\n",
       "      <th>受傷人數</th>\n",
       "      <th>當事人序</th>\n",
       "      <th>車種</th>\n",
       "      <th>性別</th>\n",
       "      <th>年齡</th>\n",
       "      <th>受傷程度</th>\n",
       "      <th>4天候</th>\n",
       "      <th>7速限</th>\n",
       "      <th>8道路型態</th>\n",
       "      <th>9事故位置</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>107</td>\n",
       "      <td>3</td>\n",
       "      <td>29</td>\n",
       "      <td>15</td>\n",
       "      <td>54</td>\n",
       "      <td>2</td>\n",
       "      <td>01大同區</td>\n",
       "      <td>大同區民權西路108號</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>B03</td>\n",
       "      <td>2</td>\n",
       "      <td>41.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "      <td>50</td>\n",
       "      <td>14.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "      <td>3</td>\n",
       "      <td>29</td>\n",
       "      <td>15</td>\n",
       "      <td>54</td>\n",
       "      <td>2</td>\n",
       "      <td>01大同區</td>\n",
       "      <td>大同區民權西路108號</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>C03</td>\n",
       "      <td>2</td>\n",
       "      <td>58.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>50</td>\n",
       "      <td>14.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>107</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "      <td>01大同區</td>\n",
       "      <td>大同區重慶北路2段與南京西路口</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>B01</td>\n",
       "      <td>1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>107</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "      <td>01大同區</td>\n",
       "      <td>大同區重慶北路3段與南京西路口</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>C03</td>\n",
       "      <td>1</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>107</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>9</td>\n",
       "      <td>56</td>\n",
       "      <td>2</td>\n",
       "      <td>01大同區</td>\n",
       "      <td>大同區承德路3段與民族西路口</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>C03</td>\n",
       "      <td>1</td>\n",
       "      <td>20.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>50</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   發生年  發生月  發生日  發生時  發生分  處理別     區序             肇事地點  死亡人數  受傷人數  當事人序  \\\n",
       "0  107    3   29   15   54    2  01大同區      大同區民權西路108號     0     1     1   \n",
       "1  107    3   29   15   54    2  01大同區      大同區民權西路108號     0     1     2   \n",
       "2  107    1    7   17   42    2  01大同區  大同區重慶北路2段與南京西路口     0     1     1   \n",
       "3  107    1    7   17   42    2  01大同區  大同區重慶北路3段與南京西路口     0     1     2   \n",
       "4  107    1   14    9   56    2  01大同區   大同區承德路3段與民族西路口     0     1     1   \n",
       "\n",
       "    車種  性別    年齡  受傷程度  4天候  7速限  8道路型態  9事故位置  \n",
       "0  B03   2  41.0   3.0    8   50   14.0    9.0  \n",
       "1  C03   2  58.0   2.0    8   50   14.0    9.0  \n",
       "2  B01   1  59.0   3.0    6   40    4.0    2.0  \n",
       "3  C03   1  18.0   2.0    6   40    4.0    2.0  \n",
       "4  C03   1  20.0   2.0    8   50    4.0    1.0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>區序</th>\n",
       "      <th>性別</th>\n",
       "      <th>死亡人數</th>\n",
       "      <th>受傷人數</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>南港區</td>\n",
       "      <td>女性</td>\n",
       "      <td>2</td>\n",
       "      <td>799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>萬華區</td>\n",
       "      <td>女性</td>\n",
       "      <td>2</td>\n",
       "      <td>1084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>松山區</td>\n",
       "      <td>女性</td>\n",
       "      <td>30</td>\n",
       "      <td>1146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>大同區</td>\n",
       "      <td>女性</td>\n",
       "      <td>3</td>\n",
       "      <td>1336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>士林區</td>\n",
       "      <td>女性</td>\n",
       "      <td>3</td>\n",
       "      <td>1660</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    區序  性別  死亡人數  受傷人數\n",
       "0  南港區  女性     2   799\n",
       "1  萬華區  女性     2  1084\n",
       "2  松山區  女性    30  1146\n",
       "3  大同區  女性     3  1336\n",
       "4  士林區  女性     3  1660"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 你可以用類似的方式爬取任何網路上的公開數據集\n",
    "base_url = \"https://data.taipei/api/getDatasetInfo/downloadResource?id={}&rid={}\"\n",
    "_id = \"2f238b4f-1b27-4085-93e9-d684ef0e2735\"\n",
    "rid = \"ea731a84-e4a1-4523-b981-b733beddbc1f\"\n",
    "csv_url = base_url.format(_id, rid)\n",
    "df_raw = pd.read_csv(csv_url, encoding='big5')\n",
    "\n",
    "# 複製一份做處理\n",
    "df = df_raw.copy()\n",
    "\n",
    "# 計算不同區不同性別的死亡、受傷人數\n",
    "df['區序'] = df['區序'].apply(lambda x: ''.join([i for i in x if not i.isdigit()]))\n",
    "df = (df[df['性別'].isin([1, 2])]\n",
    "      .groupby(['區序', '性別'])[['死亡人數', '受傷人數']]\n",
    "      .sum()\n",
    "      .reset_index()\n",
    "      .sort_values('受傷人數'))\n",
    "\n",
    "df['性別'] = df['性別'].apply(lambda x: '男性' if x == 1 else '女性')\n",
    "df = df.reset_index().drop('index', axis=1)\n",
    "\n",
    "# 顯示結果\n",
    "display(df_raw.head())\n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了匯總過後的 DataFrame，你可以用後面[簡易繪圖並修改預設樣式](#簡易繪圖並修改預設樣式)章節提到的 pandas `plot` 函式繪圖。但在這邊讓我用 [Chartify](https://leemeng.tw/chartify-a-simple-yet-powerful-python-data-visualization-tool-which-boost-your-productivity-as-a-data-scientist.html) 展示結果：\n",
    "\n",
    "```python\n",
    "import chartify\n",
    "ch = chartify.Chart(\n",
    "    x_axis_type='categorical', \n",
    "    y_axis_type='categorical')\n",
    "\n",
    "ch.plot.heatmap(\n",
    "    data_frame=df, \n",
    "    y_column='性別', \n",
    "    x_column='區序', \n",
    "    color_column='受傷人數', \n",
    "    text_column='受傷人數', \n",
    "    color_palette='Reds', \n",
    "    text_format='{:,.0f}')\n",
    "\n",
    "(ch.set_title('2017 年台北市各行政區交通意外受傷人數')\n",
    " .set_subtitle('性別分計')\n",
    " .set_source_label(\"Data.Taipei\")\n",
    " .axes.set_xaxis_label('性別')\n",
    " .axes.set_yaxis_label('行政區')\n",
    " .show('png'))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- pandas/read_csv_and_render_using_chartify.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "過來人經驗。雖然像這樣利用 pandas 直接從網路上下載並分析數據很方便，有時 host 數據的網頁與機構（尤其是政府機關）會無預期地修改他們網站，導致數據集的 URL 失效（苦主）。為了最大化重現性，我還是會建議將數據載到本地備份之後，再做分析比較實在。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 優化記憶體使用量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以透過 `df.info` 查看 DataFrame 當前的記憶體用量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 12 columns):\n",
      "PassengerId    891 non-null int64\n",
      "Survived       891 non-null int64\n",
      "Pclass         891 non-null int64\n",
      "Name           891 non-null object\n",
      "Sex            891 non-null object\n",
      "Age            714 non-null float64\n",
      "SibSp          891 non-null int64\n",
      "Parch          891 non-null int64\n",
      "Ticket         891 non-null object\n",
      "Fare           891 non-null float64\n",
      "Cabin          204 non-null object\n",
      "Embarked       889 non-null object\n",
      "dtypes: float64(2), int64(5), object(5)\n",
      "memory usage: 322.0 KB\n"
     ]
    }
   ],
   "source": [
    "df.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "從最後一列可以看出鐵達尼號這個小 DataFrame 只佔了 322 KB。如果你是透過 [Jupyter 筆記本](https://jupyter.org/)來操作 pandas，也可以考慮用 [Variable Inspector](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/varInspector/README.html) 插件來觀察包含 DataFrame 等變數的大小："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/variable_inspector.mp4\n",
    "- Variable Inspector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊使用的 `df` 不佔什麼記憶體，但如果你想讀入的 DataFrame 很大，可以只讀入特定的欄位並將已知的分類型（categorical）欄位轉成 `category` 型態以節省記憶體（在分類數目較數據量小時有效）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 4 columns):\n",
      "PassengerId    891 non-null int64\n",
      "Name           891 non-null object\n",
      "Sex            891 non-null object\n",
      "Embarked       889 non-null category\n",
      "dtypes: category(1), int64(1), object(2)\n",
      "memory usage: 134.9 KB\n"
     ]
    }
   ],
   "source": [
    "dtypes = {\"Embarked\": \"category\"}\n",
    "cols = ['PassengerId', 'Name', 'Sex', 'Embarked']\n",
    "df = pd.read_csv('http://bit.ly/kaggletrain', \n",
    "                 dtype=dtypes, usecols=cols)\n",
    "df.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過減少讀入的欄位數並將 `object` 轉換成 `category` 欄位，讀入的 `df` 只剩 135 KB。只需剛剛的 40 % 記憶體用量。\n",
    "\n",
    "另外如果你想在有限的記憶體內處理巨大 CSV 檔案，也可以透過 `chunksize` 參數來限制一次讀入的列數（rows）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId                                               Name     Sex  \\\n",
       "0            1                            Braund, Mr. Owen Harris    male   \n",
       "1            2  Cumings, Mrs. John Bradley (Florence Briggs Th...  female   \n",
       "2            3                             Heikkinen, Miss. Laina  female   \n",
       "3            4       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   \n",
       "\n",
       "  Embarked  \n",
       "0        S  \n",
       "1        C  \n",
       "2        S  \n",
       "3        S  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId                            Name   Sex Embarked\n",
       "4            5        Allen, Mr. William Henry  male        S\n",
       "5            6                Moran, Mr. James  male        Q\n",
       "6            7         McCarthy, Mr. Timothy J  male        S\n",
       "7            8  Palsson, Master. Gosta Leonard  male        S"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# chunksize=4 表示一次讀入 4 筆樣本\n",
    "reader = pd.read_csv('dataset/titanic-train.csv', \n",
    "                     chunksize=4, usecols=cols)\n",
    "# 秀出前兩個 chunks\n",
    "for _, df_partial in zip(range(2), reader):\n",
    "    display(df_partial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 讀入並合併多個 CSV 檔案成單一 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候因為企業內部 ETL 或是數據處理的方式（比方說[利用 Airflow 處理批次數據](https://leemeng.tw/a-story-about-airflow-and-data-engineering-using-how-to-use-python-to-catch-up-with-latest-comics-as-an-example.html)），相同類型的數據可能會被分成多個不同的 CSV 檔案儲存。\n",
    "\n",
    "假設在本地端 `dataset` 資料夾內有 2 個 CSV 檔案，分別儲存鐵達尼號上不同乘客的數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "df = pd.read_csv(\"dataset/titanic-train.csv\")\n",
    "columns = list(df.columns)\n",
    "# columns = [c for c in columns if c != 'Name']\n",
    "df.Name = df.Name.apply(lambda x: x.split()[0].replace(',', ''))\n",
    "\n",
    "df.loc[4:6, columns].to_csv(\"dataset/passenger1.csv\", index=False)\n",
    "df.loc[12:14, columns].to_csv(\"dataset/passenger2.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass      Name   Sex   Age  SibSp  Parch  Ticket  \\\n",
       "0            5         0       3     Allen  male  35.0      0      0  373450   \n",
       "1            6         0       3     Moran  male   NaN      0      0  330877   \n",
       "2            7         0       1  McCarthy  male  54.0      0      0   17463   \n",
       "\n",
       "      Fare Cabin Embarked  \n",
       "0   8.0500   NaN        S  \n",
       "1   8.4583   NaN        Q  \n",
       "2  51.8625   E46        S  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"dataset/passenger1.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外一個 CSV 內容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Saundercock</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5. 2151</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Andersson</td>\n",
       "      <td>male</td>\n",
       "      <td>39.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>347082</td>\n",
       "      <td>31.2750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Vestrom</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350406</td>\n",
       "      <td>7.8542</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass         Name     Sex   Age  SibSp  Parch  \\\n",
       "0           13         0       3  Saundercock    male  20.0      0      0   \n",
       "1           14         0       3    Andersson    male  39.0      1      5   \n",
       "2           15         0       3      Vestrom  female  14.0      0      0   \n",
       "\n",
       "      Ticket     Fare  Cabin Embarked  \n",
       "0  A/5. 2151   8.0500    NaN        S  \n",
       "1     347082  31.2750    NaN        S  \n",
       "2     350406   7.8542    NaN        S  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"dataset/passenger2.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意上面 2 個 DataFrames 的內容雖然分別代表不同乘客，其格式卻是一模一樣。這種時候你可以使用 `pd.concat` 將分散在不同 CSV 的乘客數據合併成單一 DataFrame，方便之後處理："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Saundercock</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5. 2151</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Andersson</td>\n",
       "      <td>male</td>\n",
       "      <td>39.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>347082</td>\n",
       "      <td>31.2750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Vestrom</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350406</td>\n",
       "      <td>7.8542</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass         Name     Sex   Age  SibSp  Parch  \\\n",
       "0            5         0       3        Allen    male  35.0      0      0   \n",
       "1            6         0       3        Moran    male   NaN      0      0   \n",
       "2            7         0       1     McCarthy    male  54.0      0      0   \n",
       "3           13         0       3  Saundercock    male  20.0      0      0   \n",
       "4           14         0       3    Andersson    male  39.0      1      5   \n",
       "5           15         0       3      Vestrom  female  14.0      0      0   \n",
       "\n",
       "      Ticket     Fare Cabin Embarked  \n",
       "0     373450   8.0500   NaN        S  \n",
       "1     330877   8.4583   NaN        Q  \n",
       "2      17463  51.8625   E46        S  \n",
       "3  A/5. 2151   8.0500   NaN        S  \n",
       "4     347082  31.2750   NaN        S  \n",
       "5     350406   7.8542   NaN        S  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob\n",
    "files = glob(\"dataset/passenger*.csv\")\n",
    "\n",
    "df = pd.concat([pd.read_csv(f) for f in files])\n",
    "df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你還可以使用 `reset_index` 函式來重置串接後的 DataFrame 索引。\n",
    "\n",
    "前面說過很多 pandas 函式**預設**的 `axis` 參數為 `0`，代表著以**列（row）**為單位做特定的操作。在 `pd.concat` 的例子中則是將 2 個同樣格式的 DataFrames 依照**列**串接起來。\n",
    "\n",
    "有時候同一筆數據的不同特徵值（features）會被存在不同檔案裡頭。以鐵達尼號的數據集舉例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "df = pd.read_csv(\"dataset/titanic-train.csv\")\n",
    "df.Name = df.Name.apply(lambda x: x.split()[0].replace(',', ''))\n",
    "df.iloc[:, :4].head().to_csv(\"dataset/feature_set1.csv\", index=False)\n",
    "df.iloc[:, 4:].head().to_csv(\"dataset/feature_set2.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass       Name\n",
       "0            1         0       3     Braund\n",
       "1            2         1       1    Cumings\n",
       "2            3         1       3  Heikkinen\n",
       "3            4         1       1   Futrelle\n",
       "4            5         0       3      Allen"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"dataset/feature_set1.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了乘客名稱以外，其他如年齡以及性別等特徵值則被存在另個 CSV 裡頭："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Sex   Age  SibSp  Parch            Ticket     Fare Cabin Embarked\n",
       "0    male  22.0      1      0         A/5 21171   7.2500   NaN        S\n",
       "1  female  38.0      1      0          PC 17599  71.2833   C85        C\n",
       "2  female  26.0      0      0  STON/O2. 3101282   7.9250   NaN        S\n",
       "3  female  35.0      1      0            113803  53.1000  C123        S\n",
       "4    male  35.0      0      0            373450   8.0500   NaN        S"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"dataset/feature_set2.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假設這 2 個 CSV 檔案裡頭**同列**對應到同個乘客，則你可以很輕鬆地用 `pd.concat` 函式搭配 `axis=1` 將不同 DataFrames 依照**行（column）**串接："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass       Name     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3     Braund    male  22.0      1      0   \n",
       "1            2         1       1    Cumings  female  38.0      1      0   \n",
       "2            3         1       3  Heikkinen  female  26.0      0      0   \n",
       "3            4         1       1   Futrelle  female  35.0      1      0   \n",
       "4            5         0       3      Allen    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = glob(\"dataset/feature_set*.csv\")\n",
    "pd.concat([pd.read_csv(f) for f in files], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 客製化 DataFrame 顯示設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "雖然 pandas 會盡可能地將一個 DataFrame 完整且漂亮地呈現出來，有時候你還是會想要改變預設的顯示方式。這節列出一些常見的使用情境。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 完整顯示所有欄位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時候一個 DataFrame 裡頭的欄位太多， pandas 會自動省略某些中間欄位以保持頁面整潔：\n",
    "\n",
    "```python\n",
    "df = pd.util.testing.makeCustomDataframe(5, 25)\n",
    "df\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/long_dataframe.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但如果你無論如何都想要顯示所有欄位以方便一次查看，可以透過 `pd.set_option` 函式來改變 `display.max_columns` 設定：\n",
    "\n",
    "```python\n",
    "pd.set_option(\"display.max_columns\", None)\n",
    "df\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/long_full_dataframe.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "pd.reset_option(\"display.max_columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意 `...` 消失了。另外你也可以使用 `T` 來轉置（transpose）當前 DataFrame，垂直顯示所有欄位：\n",
    "\n",
    "```python\n",
    "# 注意轉置後 `head(15)` 代表選擇前 15 個欄位\n",
    "df.T.head(15)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/transposed_dataframe.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個測試用的 DataFrame 欄位裡頭的 `C` 正代表著 column。你可以在 [pandas 官方文件裡查看其他常用的顯示設定](https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html#frequently-used-options)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 減少顯示的欄位長度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊你一樣可以透過 `pd.set_option` 函式來限制鐵達尼號資料集裡頭 `Name` 欄位的顯示長度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "display.max_colwidth 預設值： 50\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "注意 Name 欄位的長度被改變了：\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund...</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cuming...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikki...</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O...</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass       Name     Sex   Age  SibSp  Parch  \\\n",
       "0          1           0       3  Braund...    male  22.0      1      0   \n",
       "1          2           1       1  Cuming...  female  38.0      1      0   \n",
       "2          3           1       3  Heikki...  female  26.0      0      0   \n",
       "\n",
       "      Ticket     Fare Cabin Embarked  \n",
       "0  A/5 21171   7.2500   NaN        S  \n",
       "1   PC 17599  71.2833   C85        C  \n",
       "2  STON/O...   7.9250   NaN        S  "
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "print(\"display.max_colwidth 預設值：\", \n",
    "      pd.get_option(\"display.max_colwidth\"))\n",
    "\n",
    "# 使用預設設定來顯示 DataFrame\n",
    "df = pd.read_csv('http://bit.ly/kaggletrain')\n",
    "display(df.head(3))\n",
    "\n",
    "print(\"注意 Name 欄位的長度被改變了：\")\n",
    "# 客製化顯示（global）\n",
    "pd.set_option(\"display.max_colwidth\", 10)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 改變浮點數顯示位數"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了欄位長度以外，你常常會想要改變浮點數（float）顯示的小數點位數："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund...</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cuming...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.3</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikki...</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O...</td>\n",
       "      <td>7.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass       Name     Sex   Age  SibSp  Parch  \\\n",
       "0          1           0       3  Braund...    male  22.0      1      0   \n",
       "1          2           1       1  Cuming...  female  38.0      1      0   \n",
       "2          3           1       3  Heikki...  female  26.0      0      0   \n",
       "\n",
       "      Ticket  Fare Cabin Embarked  \n",
       "0  A/5 21171   7.2   NaN        S  \n",
       "1   PC 17599  71.3   C85        C  \n",
       "2  STON/O...   7.9   NaN        S  "
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option(\"display.precision\", 1)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你會發現 `Fare` 欄位現在只顯示小數點**後一位**的數值了。另外注意剛剛設定的 `max_colwidth` 是會被套用到所有 DataFrame 的。因此這個 DataFrame 的 `Name` 欄位顯示的寬度還跟上個 DataFrame 相同：都被縮減了。\n",
    "\n",
    "想要將所有調整過的設定初始化，可以執行：\n",
    "\n",
    "```python\n",
    "pd.reset_option(\"all\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "from IPython.display import clear_output\n",
    "pd.reset_option(\"all\")\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其他常用的 options 包含：\n",
    "- `max_rows`\n",
    "- `max_columns`\n",
    "- `date_yearfirst` \n",
    "\n",
    "等等。執行 `pd.describe_option()` 可以顯示所有可供使用的 options，但如果你是在 Jupyter 筆記本內使用 pandas 的話，我推薦直接在 `set_option` 函式的括號裡輸入 `Shift` + `tab` 顯示所有選項："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/pd_set_option.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 為特定 DataFrame 加點樣式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pd.set_option` 函式在你想要把某些顯示設定套用到**所有** DataFrames 時很好用。不過很多時候你會想要讓不同 DataFrame 有不同的顯示設定或樣式（styling）。\n",
    "\n",
    "比方說針對下面這個只有 10 筆數據的 DataFrame，你想要跟上一節一樣把 `Fare` 欄位弄成只有小數點後一位，但又不想影響到其他 DataFrame 或是其他欄位："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>183</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>347077</td>\n",
       "      <td>31.3875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>C 7077</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>659</th>\n",
       "      <td>660</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>35273</td>\n",
       "      <td>113.2750</td>\n",
       "      <td>D48</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431</th>\n",
       "      <td>432</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>376564</td>\n",
       "      <td>16.1000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>539</th>\n",
       "      <td>540</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>13568</td>\n",
       "      <td>49.5000</td>\n",
       "      <td>B39</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>478</th>\n",
       "      <td>479</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350060</td>\n",
       "      <td>7.5208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>791</th>\n",
       "      <td>792</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>239865</td>\n",
       "      <td>26.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>304</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>226593</td>\n",
       "      <td>12.3500</td>\n",
       "      <td>E101</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>71</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>C.A. 33111</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch      Ticket  \\\n",
       "4              5         0       3    male  35.0      0      0      373450   \n",
       "182          183         0       3    male   9.0      4      2      347077   \n",
       "376          377         1       3  female  22.0      0      0      C 7077   \n",
       "659          660         0       1    male  58.0      0      2       35273   \n",
       "431          432         1       3  female  29.0      1      0      376564   \n",
       "539          540         1       1  female  22.0      0      2       13568   \n",
       "478          479         0       3    male  22.0      0      0      350060   \n",
       "791          792         0       2    male  16.0      0      0      239865   \n",
       "303          304         1       2  female  29.0      0      0      226593   \n",
       "70            71         0       2    male  32.0      0      0  C.A. 33111   \n",
       "\n",
       "         Fare Cabin Embarked  \n",
       "4      8.0500   NaN        S  \n",
       "182   31.3875   NaN        S  \n",
       "376    7.2500   NaN        S  \n",
       "659  113.2750   D48        C  \n",
       "431   16.1000   NaN        S  \n",
       "539   49.5000   B39        C  \n",
       "478    7.5208   NaN        S  \n",
       "791   26.0000   NaN        S  \n",
       "303   12.3500  E101        Q  \n",
       "70    10.5000   NaN        S  "
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 隨機抽樣 10 筆數據來做 styling\n",
    "df_sample = df.sample(n=10, random_state=9527).drop('Name', axis=1)\n",
    "df_sample.Age.fillna(int(df.Age.mean()), inplace=True)\n",
    "df_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這時候你可以使用 [pandas Styler](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html) 底下的 `format` 函式來做到這件事情：\n",
    "\n",
    "```python\n",
    "# 一個典型 chain pandas 函式的例子\n",
    "(df_sample.style\n",
    "     .format('{:.1f}', subset='Fare')\n",
    "     .set_caption('★五顏六色の鐵達尼號數據集☆')\n",
    "     .hide_index()\n",
    "     .bar('Age', vmin=0)\n",
    "     .highlight_max('Survived')\n",
    "     .background_gradient('Greens', \n",
    "                          subset='Fare')\n",
    "     .highlight_null()\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- pandas/pandas_styling.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你從來沒有用過 [df.style](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html)，這應該是你這輩子看過最繽紛的 DataFrame。\n",
    "\n",
    "從上而下，上述程式碼對此 DataFrame 做了以下 styling：\n",
    "- 將 `Fare` 欄位的數值顯示限制到小數後第一位\n",
    "- 添加一個標題輔助說明\n",
    "- 隱藏索引（注意最左邊！）\n",
    "- 將 `Age` 欄位依數值大小畫條狀圖\n",
    "- 將 `Survived` 最大的值 highlight\n",
    "- 將 `Fare` 欄位依數值畫綠色的 colormap\n",
    "- 將整個 DataFrame 的空值顯示為紅色\n",
    "\n",
    "`pd.DataFrame.style` 會回傳一個 Styler。你已經看到除了 `format` 函式以外，還有很多其他函式可以讓你為 DataFrame 添加樣式。使用 `format` 函式的最大好處是你不需要用像是 `round` 等函式去修改**實際**數值，而只是改變**呈現結果**而已。\n",
    "\n",
    "熟悉 styling 技巧能讓你不需畫圖就能輕鬆與他人分享簡單的分析結果，也能凸顯你想讓他們關注的事物。小提醒：為了讓你能一次掌握常用函式，我把能加的樣式都加了。實務上你應該思考[什麼視覺變數是必要的](https://leemeng.tw/data-visualization-from-matplotlib-to-ggplot2.html)，而不是盲目地添加樣式。\n",
    "\n",
    "另外值得一提的是 pandas 函式都會回傳處理後的結果，而不是直接修改原始 DataFrame。這讓你可以輕鬆地把多個函式串（chain）成一個複雜的數據處理 pipeline，但又不會影響到最原始的數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>183</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>347077</td>\n",
       "      <td>31.3875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>C 7077</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>659</th>\n",
       "      <td>660</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>35273</td>\n",
       "      <td>113.2750</td>\n",
       "      <td>D48</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431</th>\n",
       "      <td>432</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>376564</td>\n",
       "      <td>16.1000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>539</th>\n",
       "      <td>540</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>13568</td>\n",
       "      <td>49.5000</td>\n",
       "      <td>B39</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>478</th>\n",
       "      <td>479</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350060</td>\n",
       "      <td>7.5208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>791</th>\n",
       "      <td>792</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>239865</td>\n",
       "      <td>26.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>304</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>226593</td>\n",
       "      <td>12.3500</td>\n",
       "      <td>E101</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>71</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>C.A. 33111</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch      Ticket  \\\n",
       "4              5         0       3    male  35.0      0      0      373450   \n",
       "182          183         0       3    male   9.0      4      2      347077   \n",
       "376          377         1       3  female  22.0      0      0      C 7077   \n",
       "659          660         0       1    male  58.0      0      2       35273   \n",
       "431          432         1       3  female  29.0      1      0      376564   \n",
       "539          540         1       1  female  22.0      0      2       13568   \n",
       "478          479         0       3    male  22.0      0      0      350060   \n",
       "791          792         0       2    male  16.0      0      0      239865   \n",
       "303          304         1       2  female  29.0      0      0      226593   \n",
       "70            71         0       2    male  32.0      0      0  C.A. 33111   \n",
       "\n",
       "         Fare Cabin Embarked  \n",
       "4      8.0500   NaN        S  \n",
       "182   31.3875   NaN        S  \n",
       "376    7.2500   NaN        S  \n",
       "659  113.2750   D48        C  \n",
       "431   16.1000   NaN        S  \n",
       "539   49.5000   B39        C  \n",
       "478    7.5208   NaN        S  \n",
       "791   26.0000   NaN        S  \n",
       "303   12.3500  E101        Q  \n",
       "70    10.5000   NaN        S  "
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "瞧！原來的 DataFrame 還是挺淳樸的。注意 `Fare` 欄位裡的小數點並沒有因為剛剛的 styling 而變少，而這讓你在呈現 DataFrame 時有最大的彈性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 數據清理 & 整理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這節列出一些十分常用的數據清理與整理技巧，如處理空值（null value）以及切割欄位。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 處理空值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "世界總是殘酷，很多時候手上的 DataFrame 裡頭會有不存在的值，如底下一格格額外顯眼的 `NaN`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6bAs9RrtPf</th>\n",
       "      <td>-1.864693</td>\n",
       "      <td>0.395997</td>\n",
       "      <td>-1.057175</td>\n",
       "      <td>1.261987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4pLVw54w1</th>\n",
       "      <td>0.817813</td>\n",
       "      <td>-0.893529</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KkhqmjKghu</th>\n",
       "      <td>0.182929</td>\n",
       "      <td>0.892841</td>\n",
       "      <td>1.487175</td>\n",
       "      <td>-1.218580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S4ppq42BKt</th>\n",
       "      <td>1.113693</td>\n",
       "      <td>0.979332</td>\n",
       "      <td>-0.761886</td>\n",
       "      <td>0.026489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mu2ryPVR0x</th>\n",
       "      <td>0.387857</td>\n",
       "      <td>0.334798</td>\n",
       "      <td>1.321586</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "6bAs9RrtPf -1.864693  0.395997 -1.057175  1.261987\n",
       "P4pLVw54w1  0.817813 -0.893529       NaN       NaN\n",
       "KkhqmjKghu  0.182929  0.892841  1.487175 -1.218580\n",
       "S4ppq42BKt  1.113693  0.979332 -0.761886  0.026489\n",
       "Mu2ryPVR0x  0.387857  0.334798  1.321586       NaN"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.util.testing.makeMissingDataframe().head()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以利用 `fillna` 函式將 DataFrame 裡頭所有不存在的值設為 `0`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6bAs9RrtPf</th>\n",
       "      <td>-1.864693</td>\n",
       "      <td>0.395997</td>\n",
       "      <td>-1.057175</td>\n",
       "      <td>1.261987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4pLVw54w1</th>\n",
       "      <td>0.817813</td>\n",
       "      <td>-0.893529</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KkhqmjKghu</th>\n",
       "      <td>0.182929</td>\n",
       "      <td>0.892841</td>\n",
       "      <td>1.487175</td>\n",
       "      <td>-1.218580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S4ppq42BKt</th>\n",
       "      <td>1.113693</td>\n",
       "      <td>0.979332</td>\n",
       "      <td>-0.761886</td>\n",
       "      <td>0.026489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mu2ryPVR0x</th>\n",
       "      <td>0.387857</td>\n",
       "      <td>0.334798</td>\n",
       "      <td>1.321586</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "6bAs9RrtPf -1.864693  0.395997 -1.057175  1.261987\n",
       "P4pLVw54w1  0.817813 -0.893529  0.000000  0.000000\n",
       "KkhqmjKghu  0.182929  0.892841  1.487175 -1.218580\n",
       "S4ppq42BKt  1.113693  0.979332 -0.761886  0.026489\n",
       "Mu2ryPVR0x  0.387857  0.334798  1.321586  0.000000"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.fillna(0) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "當然，這個操作的前提是你確定在當前分析的情境下，將不存在的值視為 `0` 這件事情是沒有問題的。\n",
    "\n",
    "針對字串欄位，你也可以將空值設定成任何容易識別的值，讓自己及他人明確了解此 DataFrame 的數據品質："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>C0</th>\n",
       "      <th>C_l0_g0</th>\n",
       "      <th>C_l0_g1</th>\n",
       "      <th>C_l0_g2</th>\n",
       "      <th>C_l0_g3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>R_l0_g0</th>\n",
       "      <td>R0C0</td>\n",
       "      <td>R0C1</td>\n",
       "      <td>R0C2</td>\n",
       "      <td>R0C3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R_l0_g1</th>\n",
       "      <td>R1C0</td>\n",
       "      <td>R1C1</td>\n",
       "      <td>R1C2</td>\n",
       "      <td>R1C3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R_l0_g2</th>\n",
       "      <td>R2C0</td>\n",
       "      <td>R2C1</td>\n",
       "      <td>R2C2</td>\n",
       "      <td>R2C3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R_l0_g3</th>\n",
       "      <td>R3C0</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>R3C2</td>\n",
       "      <td>R3C3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R_l0_g4</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>R4C1</td>\n",
       "      <td>R4C2</td>\n",
       "      <td>R4C3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "C0       C_l0_g0  C_l0_g1 C_l0_g2 C_l0_g3\n",
       "R0                                       \n",
       "R_l0_g0     R0C0     R0C1    R0C2    R0C3\n",
       "R_l0_g1     R1C0     R1C1    R1C2    R1C3\n",
       "R_l0_g2     R2C0     R2C1    R2C2    R2C3\n",
       "R_l0_g3     R3C0  Unknown    R3C2    R3C3\n",
       "R_l0_g4  Unknown     R4C1    R4C2    R4C3"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.util.testing.makeMissingCustomDataframe(nrows=5, \n",
    "                                                ncols=4, \n",
    "                                                dtype=str)\n",
    "df.fillna(\"Unknown\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 捨棄不需要的行列"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "給定一個初始 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>87dl3rTYXR</th>\n",
       "      <td>0.048544</td>\n",
       "      <td>0.133999</td>\n",
       "      <td>0.885555</td>\n",
       "      <td>1.164883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DE0ptDHz2Q</th>\n",
       "      <td>0.551378</td>\n",
       "      <td>0.012621</td>\n",
       "      <td>0.256241</td>\n",
       "      <td>0.197140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jiWMAjniDH</th>\n",
       "      <td>-1.275694</td>\n",
       "      <td>-1.473950</td>\n",
       "      <td>-1.602433</td>\n",
       "      <td>-0.286107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45lFb2QT9g</th>\n",
       "      <td>1.348614</td>\n",
       "      <td>2.235074</td>\n",
       "      <td>1.471781</td>\n",
       "      <td>0.449909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OOOtaUhOOp</th>\n",
       "      <td>-0.297613</td>\n",
       "      <td>-1.332934</td>\n",
       "      <td>0.194135</td>\n",
       "      <td>0.654267</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "87dl3rTYXR  0.048544  0.133999  0.885555  1.164883\n",
       "DE0ptDHz2Q  0.551378  0.012621  0.256241  0.197140\n",
       "jiWMAjniDH -1.275694 -1.473950 -1.602433 -0.286107\n",
       "45lFb2QT9g  1.348614  2.235074  1.471781  0.449909\n",
       "OOOtaUhOOp -0.297613 -1.332934  0.194135  0.654267"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.util.testing.makeDataFrame().head()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以使用 `drop` 函式來捨棄不需要的欄位。記得將 `axis` 設為 1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>87dl3rTYXR</th>\n",
       "      <td>0.048544</td>\n",
       "      <td>0.885555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DE0ptDHz2Q</th>\n",
       "      <td>0.551378</td>\n",
       "      <td>0.256241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jiWMAjniDH</th>\n",
       "      <td>-1.275694</td>\n",
       "      <td>-1.602433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45lFb2QT9g</th>\n",
       "      <td>1.348614</td>\n",
       "      <td>1.471781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OOOtaUhOOp</th>\n",
       "      <td>-0.297613</td>\n",
       "      <td>0.194135</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         C\n",
       "87dl3rTYXR  0.048544  0.885555\n",
       "DE0ptDHz2Q  0.551378  0.256241\n",
       "jiWMAjniDH -1.275694 -1.602433\n",
       "45lFb2QT9g  1.348614  1.471781\n",
       "OOOtaUhOOp -0.297613  0.194135"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['B', 'D']\n",
    "df.drop(columns, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同理，你也可以捨棄特定列（row）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>87dl3rTYXR</th>\n",
       "      <td>0.048544</td>\n",
       "      <td>0.133999</td>\n",
       "      <td>0.885555</td>\n",
       "      <td>1.164883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DE0ptDHz2Q</th>\n",
       "      <td>0.551378</td>\n",
       "      <td>0.012621</td>\n",
       "      <td>0.256241</td>\n",
       "      <td>0.197140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jiWMAjniDH</th>\n",
       "      <td>-1.275694</td>\n",
       "      <td>-1.473950</td>\n",
       "      <td>-1.602433</td>\n",
       "      <td>-0.286107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45lFb2QT9g</th>\n",
       "      <td>1.348614</td>\n",
       "      <td>2.235074</td>\n",
       "      <td>1.471781</td>\n",
       "      <td>0.449909</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "87dl3rTYXR  0.048544  0.133999  0.885555  1.164883\n",
       "DE0ptDHz2Q  0.551378  0.012621  0.256241  0.197140\n",
       "jiWMAjniDH -1.275694 -1.473950 -1.602433 -0.286107\n",
       "45lFb2QT9g  1.348614  2.235074  1.471781  0.449909"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop('OOOtaUhOOp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 重置並捨棄索引"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候你會想要重置一個 DataFrame 的索引，以方便使用 `loc` 或 `iloc` **屬性**來存取想要的數據。\n",
    "\n",
    "給定一個 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LI5Y0d3Ygk</th>\n",
       "      <td>0.704950</td>\n",
       "      <td>0.610508</td>\n",
       "      <td>-1.686467</td>\n",
       "      <td>0.165393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9EkrfcGaKh</th>\n",
       "      <td>1.088294</td>\n",
       "      <td>-0.947646</td>\n",
       "      <td>1.382226</td>\n",
       "      <td>-1.410468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jf891pQjSh</th>\n",
       "      <td>0.137240</td>\n",
       "      <td>1.490714</td>\n",
       "      <td>1.236335</td>\n",
       "      <td>-0.270623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Q9O2vu4Mg1</th>\n",
       "      <td>-0.375816</td>\n",
       "      <td>2.308924</td>\n",
       "      <td>-1.735557</td>\n",
       "      <td>1.734371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>z0xoUxOAWW</th>\n",
       "      <td>1.067996</td>\n",
       "      <td>0.474090</td>\n",
       "      <td>-0.492757</td>\n",
       "      <td>-0.042121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "LI5Y0d3Ygk  0.704950  0.610508 -1.686467  0.165393\n",
       "9EkrfcGaKh  1.088294 -0.947646  1.382226 -1.410468\n",
       "Jf891pQjSh  0.137240  1.490714  1.236335 -0.270623\n",
       "Q9O2vu4Mg1 -0.375816  2.308924 -1.735557  1.734371\n",
       "z0xoUxOAWW  1.067996  0.474090 -0.492757 -0.042121"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.util.testing.makeDataFrame().head()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以使用 `reset_index` 函式來重置此 DataFrame 的索引並輕鬆存取想要的部分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LI5Y0d3Ygk</td>\n",
       "      <td>0.704950</td>\n",
       "      <td>0.610508</td>\n",
       "      <td>-1.686467</td>\n",
       "      <td>0.165393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9EkrfcGaKh</td>\n",
       "      <td>1.088294</td>\n",
       "      <td>-0.947646</td>\n",
       "      <td>1.382226</td>\n",
       "      <td>-1.410468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jf891pQjSh</td>\n",
       "      <td>0.137240</td>\n",
       "      <td>1.490714</td>\n",
       "      <td>1.236335</td>\n",
       "      <td>-0.270623</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        index         A         B         C         D\n",
       "0  LI5Y0d3Ygk  0.704950  0.610508 -1.686467  0.165393\n",
       "1  9EkrfcGaKh  1.088294 -0.947646  1.382226 -1.410468\n",
       "2  Jf891pQjSh  0.137240  1.490714  1.236335 -0.270623"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reset_index(inplace=True)\n",
    "df.iloc[:3, :]\n",
    "# 豆知識：因為 iloc 是屬性而非函式，\n",
    "# 因此你得使用 [] 而非 () 存取數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "將函式的 `inplace` 參數設為 `True` 會讓 pandas 直接修改 `df`。一般來說 pandas 裡的函式並不會修改原始 DataFrame，這樣可以保證原始數據不會受到任何函式的影響。\n",
    "\n",
    "當你不想要原來的 DataFrame `df` 受到 `reset_index` 函式的影響，則可以將處理後的結果交給一個新 DataFrame（比方說 `df1`）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>fNMRqfPnw1</th>\n",
       "      <td>0.458255</td>\n",
       "      <td>1.736118</td>\n",
       "      <td>-1.659602</td>\n",
       "      <td>2.325742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>xeIJ4tJuxW</th>\n",
       "      <td>-1.102030</td>\n",
       "      <td>1.321306</td>\n",
       "      <td>-0.262345</td>\n",
       "      <td>0.864090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RDAJlNsfDS</th>\n",
       "      <td>1.047750</td>\n",
       "      <td>-0.420285</td>\n",
       "      <td>0.757464</td>\n",
       "      <td>0.384514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kIOVU4EU79</th>\n",
       "      <td>-0.843373</td>\n",
       "      <td>0.235789</td>\n",
       "      <td>-1.211142</td>\n",
       "      <td>0.656130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OKuRXq4eEK</th>\n",
       "      <td>-1.340040</td>\n",
       "      <td>0.674713</td>\n",
       "      <td>0.768126</td>\n",
       "      <td>-0.500718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "fNMRqfPnw1  0.458255  1.736118 -1.659602  2.325742\n",
       "xeIJ4tJuxW -1.102030  1.321306 -0.262345  0.864090\n",
       "RDAJlNsfDS  1.047750 -0.420285  0.757464  0.384514\n",
       "kIOVU4EU79 -0.843373  0.235789 -1.211142  0.656130\n",
       "OKuRXq4eEK -1.340040  0.674713  0.768126 -0.500718"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.458255</td>\n",
       "      <td>1.736118</td>\n",
       "      <td>-1.659602</td>\n",
       "      <td>2.325742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.102030</td>\n",
       "      <td>1.321306</td>\n",
       "      <td>-0.262345</td>\n",
       "      <td>0.864090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.047750</td>\n",
       "      <td>-0.420285</td>\n",
       "      <td>0.757464</td>\n",
       "      <td>0.384514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.843373</td>\n",
       "      <td>0.235789</td>\n",
       "      <td>-1.211142</td>\n",
       "      <td>0.656130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.340040</td>\n",
       "      <td>0.674713</td>\n",
       "      <td>0.768126</td>\n",
       "      <td>-0.500718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C         D\n",
       "0  0.458255  1.736118 -1.659602  2.325742\n",
       "1 -1.102030  1.321306 -0.262345  0.864090\n",
       "2  1.047750 -0.420285  0.757464  0.384514\n",
       "3 -0.843373  0.235789 -1.211142  0.656130\n",
       "4 -1.340040  0.674713  0.768126 -0.500718"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.util.testing.makeDataFrame().head()\n",
    "df1 = df.reset_index(drop=True)\n",
    "display(df)\n",
    "display(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過這樣的方式，pandas 讓你可以放心地對原始數據做任何壞壞的事情而不會產生任何不好的影響。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將字串切割成多個欄位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在處理文本數據時，很多時候你會想要把一個字串欄位拆成多個欄位以方便後續處理。\n",
    "\n",
    "給定一個簡單 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>feature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大雄</td>\n",
       "      <td>膽小, 翻花繩</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>胖虎</td>\n",
       "      <td>粗魯, 演唱會</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  name  feature\n",
       "0   大雄  膽小, 翻花繩\n",
       "1   胖虎  粗魯, 演唱會"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\n",
    "    \"name\": [\"大雄\", \"胖虎\"], \n",
    "    \"feature\": [\"膽小, 翻花繩\", \"粗魯, 演唱會\"]\n",
    "})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可能會想把這個 DataFrame 的 `feature` 欄位分成不同欄位，這時候利用 `str` 將字串取出，並透過 `expand=True` 將字串切割的結果擴大成（expand）成一個 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>feature</th>\n",
       "      <th>性格</th>\n",
       "      <th>特技</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大雄</td>\n",
       "      <td>膽小, 翻花繩</td>\n",
       "      <td>膽小</td>\n",
       "      <td>翻花繩</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>胖虎</td>\n",
       "      <td>粗魯, 演唱會</td>\n",
       "      <td>粗魯</td>\n",
       "      <td>演唱會</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  name  feature  性格    特技\n",
       "0   大雄  膽小, 翻花繩  膽小   翻花繩\n",
       "1   胖虎  粗魯, 演唱會  粗魯   演唱會"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['性格', '特技']] = df.feature.str.split(',', expand=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意我們使用 `df[columns] = ...` 的形式將字串切割出來的 2 個新欄位分別指定成 `性格` 與 `特技`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將 list 分成多個欄位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時候一個欄位裡頭的值為 Python `list`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>feature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大雄</td>\n",
       "      <td>[膽小, 翻花繩]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>胖虎</td>\n",
       "      <td>[粗魯, 演唱會]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  name    feature\n",
       "0   大雄  [膽小, 翻花繩]\n",
       "1   胖虎  [粗魯, 演唱會]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\n",
    "    \"name\": [\"大雄\", \"胖虎\"], \n",
    "    \"feature\": [[\"膽小\", \"翻花繩\"], [\"粗魯\", \"演唱會\"]]\n",
    "})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這時則可以使用 `tolist` 函式做到跟剛剛字串切割相同的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>性格</th>\n",
       "      <th>特技</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>膽小</td>\n",
       "      <td>翻花繩</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粗魯</td>\n",
       "      <td>演唱會</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   性格   特技\n",
       "0  膽小  翻花繩\n",
       "1  粗魯  演唱會"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['性格', '特技']\n",
    "pd.DataFrame(df.feature.tolist(), columns=cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以使用 `apply(pd.Series)` 的方式達到一樣的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>膽小</td>\n",
       "      <td>翻花繩</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粗魯</td>\n",
       "      <td>演唱會</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0    1\n",
       "0  膽小  翻花繩\n",
       "1  粗魯  演唱會"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.feature.apply(pd.Series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "遇到以 Python `list` 呈現欄位數據的情境不少，這些函式能讓你少抓點頭。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 取得想要關注的數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通常你會需要依照各種不同的分析情境，將整個 DataFrame 裡頭的一部份數據取出並進一步分析。這節內容讓你能夠輕鬆取得想要關注的數據。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基本數據切割"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 pandas 裡頭，切割（Slice）DataFrame 裡頭一部份數據出來做分析是稀鬆平常的事情。讓我們再次以鐵達尼號數據集為例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3    male  22.0      1      0   \n",
       "1            2         1       1  female  38.0      1      0   \n",
       "2            3         1       3  female  26.0      0      0   \n",
       "3            4         1       1  female  35.0      1      0   \n",
       "4            5         0       3    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('http://bit.ly/kaggletrain')\n",
    "df = df.drop(\"Name\", axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以透過 `loc` 以及 `:` 的方式輕鬆選取從某個起始欄位 C1 到結束欄位 C2 的所有欄位，而無需將中間的欄位一一列出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pclass     Sex   Age  SibSp  Parch            Ticket\n",
       "0       3    male  22.0      1      0         A/5 21171\n",
       "1       1  female  38.0      1      0          PC 17599\n",
       "2       3  female  26.0      0      0  STON/O2. 3101282\n",
       "3       1  female  35.0      1      0            113803"
      ]
     },
     "execution_count": 273,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:3, 'Pclass':'Ticket']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 反向選取行列"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過 Python 常見的 `[::-1]` 語法，你可以輕易地改變 DataFrame 裡頭所有欄位的排列順序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Parch</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Survived</th>\n",
       "      <th>PassengerId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C</td>\n",
       "      <td>C85</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>S</td>\n",
       "      <td>C123</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>113803</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Embarked Cabin     Fare            Ticket  Parch  SibSp   Age     Sex  \\\n",
       "0        S   NaN   7.2500         A/5 21171      0      1  22.0    male   \n",
       "1        C   C85  71.2833          PC 17599      0      1  38.0  female   \n",
       "2        S   NaN   7.9250  STON/O2. 3101282      0      0  26.0  female   \n",
       "3        S  C123  53.1000            113803      0      1  35.0  female   \n",
       "\n",
       "   Pclass  Survived  PassengerId  \n",
       "0       3         0            1  \n",
       "1       1         1            2  \n",
       "2       3         1            3  \n",
       "3       1         1            4  "
      ]
     },
     "execution_count": 274,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:3, ::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同樣概念也可以運用到列（row）上面。你可以將所有樣本（samples）排序顛倒並選取其中 N 列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>891</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>887</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age\n",
       "890          891         0       3    male  32.0\n",
       "889          890         1       1    male  26.0\n",
       "888          889         0       3  female   NaN\n",
       "887          888         1       1  female  19.0\n",
       "886          887         0       2    male  27.0"
      ]
     },
     "execution_count": 275,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[::-1, :5].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意我們同時使用 `:5` 來選出前 5 個欄位。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 條件選取數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 pandas 裡頭最實用的選取技巧大概非遮罩（masking）莫屬了。遮罩讓 pandas 將符合特定條件的樣本回傳："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col3 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col4 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col3 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col4 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col3 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col4 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col3 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col4 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col3 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col4 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }</style><table id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >PassengerId</th>        <th class=\"col_heading level0 col1\" >Survived</th>        <th class=\"col_heading level0 col2\" >Pclass</th>        <th class=\"col_heading level0 col3\" >Sex</th>        <th class=\"col_heading level0 col4\" >Age</th>        <th class=\"col_heading level0 col5\" >SibSp</th>        <th class=\"col_heading level0 col6\" >Parch</th>        <th class=\"col_heading level0 col7\" >Ticket</th>        <th class=\"col_heading level0 col8\" >Fare</th>        <th class=\"col_heading level0 col9\" >Cabin</th>        <th class=\"col_heading level0 col10\" >Embarked</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2level0_row0\" class=\"row_heading level0 row0\" >96</th>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col0\" class=\"data row0 col0\" >97</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col3\" class=\"data row0 col3\" >male</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col4\" class=\"data row0 col4\" >71</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col5\" class=\"data row0 col5\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col6\" class=\"data row0 col6\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col7\" class=\"data row0 col7\" >PC 17754</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col8\" class=\"data row0 col8\" >34.6542</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col9\" class=\"data row0 col9\" >A5</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row0_col10\" class=\"data row0 col10\" >C</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2level0_row1\" class=\"row_heading level0 row1\" >116</th>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col0\" class=\"data row1 col0\" >117</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col2\" class=\"data row1 col2\" >3</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col3\" class=\"data row1 col3\" >male</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col4\" class=\"data row1 col4\" >70.5</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col5\" class=\"data row1 col5\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col6\" class=\"data row1 col6\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col7\" class=\"data row1 col7\" >370369</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col8\" class=\"data row1 col8\" >7.75</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col9\" class=\"data row1 col9\" >nan</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row1_col10\" class=\"data row1 col10\" >Q</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2level0_row2\" class=\"row_heading level0 row2\" >493</th>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col0\" class=\"data row2 col0\" >494</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col2\" class=\"data row2 col2\" >1</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col3\" class=\"data row2 col3\" >male</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col4\" class=\"data row2 col4\" >71</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col5\" class=\"data row2 col5\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col6\" class=\"data row2 col6\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col7\" class=\"data row2 col7\" >PC 17609</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col8\" class=\"data row2 col8\" >49.5042</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col9\" class=\"data row2 col9\" >nan</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row2_col10\" class=\"data row2 col10\" >C</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2level0_row3\" class=\"row_heading level0 row3\" >630</th>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col0\" class=\"data row3 col0\" >631</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col1\" class=\"data row3 col1\" >1</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col2\" class=\"data row3 col2\" >1</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col3\" class=\"data row3 col3\" >male</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col4\" class=\"data row3 col4\" >80</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col5\" class=\"data row3 col5\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col6\" class=\"data row3 col6\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col7\" class=\"data row3 col7\" >27042</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col8\" class=\"data row3 col8\" >30</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col9\" class=\"data row3 col9\" >A23</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row3_col10\" class=\"data row3 col10\" >S</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2level0_row4\" class=\"row_heading level0 row4\" >851</th>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col0\" class=\"data row4 col0\" >852</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col1\" class=\"data row4 col1\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col2\" class=\"data row4 col2\" >3</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col3\" class=\"data row4 col3\" >male</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col4\" class=\"data row4 col4\" >74</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col5\" class=\"data row4 col5\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col6\" class=\"data row4 col6\" >0</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col7\" class=\"data row4 col7\" >347060</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col8\" class=\"data row4 col8\" >7.775</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col9\" class=\"data row4 col9\" >nan</td>\n",
       "                        <td id=\"T_68f75918_b060_11e9_8b6c_8c8590794fe2row4_col10\" class=\"data row4 col10\" >S</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1a34f720b8>"
      ]
     },
     "execution_count": 277,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male_and_age_over_70 = (df.Sex == 'male') & (df.Age > 70)\n",
    "(df[male_and_age_over_70]\n",
    "    .style\n",
    "    .applymap(lambda x: 'background-color: rgb(153, 255, 51)', \n",
    "              subset=pd.IndexSlice[:, 'Sex':'Age']))\n",
    "# 跟 df[(df.Sex == 'male') & (df.Age > 70)] 結果相同"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`male_and_age_over_70` 是我們定義的一個遮罩，可以把同時符合兩個布林判斷式（大於 70 歲、男性）的樣本選取出來。上面註解有相同效果，但當存在多個判斷式時，有個準確說明遮罩意義的變數（上例的 `male_and_age_over_70`）會讓你的程式碼好懂一點。\n",
    "\n",
    "另外你也可以使用 `query` 函式來達到跟遮罩一樣的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>97</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17754</td>\n",
       "      <td>34.6542</td>\n",
       "      <td>A5</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>70.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370369</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>493</th>\n",
       "      <td>494</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17609</td>\n",
       "      <td>49.5042</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>630</th>\n",
       "      <td>631</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27042</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>A23</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>852</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>74.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>347060</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass   Sex   Age  SibSp  Parch    Ticket  \\\n",
       "96            97         0       1  male  71.0      0      0  PC 17754   \n",
       "116          117         0       3  male  70.5      0      0    370369   \n",
       "493          494         0       1  male  71.0      0      0  PC 17609   \n",
       "630          631         1       1  male  80.0      0      0     27042   \n",
       "851          852         0       3  male  74.0      0      0    347060   \n",
       "\n",
       "        Fare Cabin Embarked  \n",
       "96   34.6542    A5        C  \n",
       "116   7.7500   NaN        Q  \n",
       "493  49.5042   NaN        C  \n",
       "630  30.0000   A23        S  \n",
       "851   7.7750   NaN        S  "
      ]
     },
     "execution_count": 278,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "age = 70\n",
    "df.query(\"Age > @age & Sex == 'male'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這個例子裡頭，你可以使用 `@` 來存取已經定義的 Python 變數 `age` 的值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 選擇任一欄有空值的樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一個 DataFrame 裡常會有多個欄位（column），而每個欄位裡頭都有可能包含空值。\n",
    "\n",
    "有時候你會想把在**任一**欄位（column）出現過空值的樣本（row）全部取出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col9 {\n",
       "            background-color:  red;\n",
       "        }    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col9 {\n",
       "            background-color:  red;\n",
       "        }    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col9 {\n",
       "            background-color:  red;\n",
       "        }    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col4 {\n",
       "            background-color:  red;\n",
       "        }    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col9 {\n",
       "            background-color:  red;\n",
       "        }    #T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col9 {\n",
       "            background-color:  red;\n",
       "        }</style><table id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >PassengerId</th>        <th class=\"col_heading level0 col1\" >Survived</th>        <th class=\"col_heading level0 col2\" >Pclass</th>        <th class=\"col_heading level0 col3\" >Sex</th>        <th class=\"col_heading level0 col4\" >Age</th>        <th class=\"col_heading level0 col5\" >SibSp</th>        <th class=\"col_heading level0 col6\" >Parch</th>        <th class=\"col_heading level0 col7\" >Ticket</th>        <th class=\"col_heading level0 col8\" >Fare</th>        <th class=\"col_heading level0 col9\" >Cabin</th>        <th class=\"col_heading level0 col10\" >Embarked</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col0\" class=\"data row0 col0\" >1</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col2\" class=\"data row0 col2\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col3\" class=\"data row0 col3\" >male</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col4\" class=\"data row0 col4\" >22</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col5\" class=\"data row0 col5\" >1</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col6\" class=\"data row0 col6\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col7\" class=\"data row0 col7\" >A/5 21171</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col8\" class=\"data row0 col8\" >7.25</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col9\" class=\"data row0 col9\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row0_col10\" class=\"data row0 col10\" >S</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2level0_row1\" class=\"row_heading level0 row1\" >2</th>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col0\" class=\"data row1 col0\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col1\" class=\"data row1 col1\" >1</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col2\" class=\"data row1 col2\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col3\" class=\"data row1 col3\" >female</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col4\" class=\"data row1 col4\" >26</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col5\" class=\"data row1 col5\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col6\" class=\"data row1 col6\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col7\" class=\"data row1 col7\" >STON/O2. 3101282</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col8\" class=\"data row1 col8\" >7.925</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col9\" class=\"data row1 col9\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row1_col10\" class=\"data row1 col10\" >S</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2level0_row2\" class=\"row_heading level0 row2\" >4</th>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col0\" class=\"data row2 col0\" >5</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col2\" class=\"data row2 col2\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col3\" class=\"data row2 col3\" >male</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col4\" class=\"data row2 col4\" >35</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col5\" class=\"data row2 col5\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col6\" class=\"data row2 col6\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col7\" class=\"data row2 col7\" >373450</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col8\" class=\"data row2 col8\" >8.05</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col9\" class=\"data row2 col9\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row2_col10\" class=\"data row2 col10\" >S</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2level0_row3\" class=\"row_heading level0 row3\" >5</th>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col0\" class=\"data row3 col0\" >6</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col1\" class=\"data row3 col1\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col2\" class=\"data row3 col2\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col3\" class=\"data row3 col3\" >male</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col4\" class=\"data row3 col4\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col5\" class=\"data row3 col5\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col6\" class=\"data row3 col6\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col7\" class=\"data row3 col7\" >330877</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col8\" class=\"data row3 col8\" >8.4583</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col9\" class=\"data row3 col9\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row3_col10\" class=\"data row3 col10\" >Q</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2level0_row4\" class=\"row_heading level0 row4\" >7</th>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col0\" class=\"data row4 col0\" >8</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col1\" class=\"data row4 col1\" >0</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col2\" class=\"data row4 col2\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col3\" class=\"data row4 col3\" >male</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col4\" class=\"data row4 col4\" >2</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col5\" class=\"data row4 col5\" >3</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col6\" class=\"data row4 col6\" >1</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col7\" class=\"data row4 col7\" >349909</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col8\" class=\"data row4 col8\" >21.075</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col9\" class=\"data row4 col9\" >nan</td>\n",
       "                        <td id=\"T_31d6affa_b061_11e9_8b6c_8c8590794fe2row4_col10\" class=\"data row4 col10\" >S</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1a34f72be0>"
      ]
     },
     "execution_count": 283,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.isnull().any(axis=1)].head() \\\n",
    "    .style.highlight_null()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊剛好所有樣本的 `Cabin` 欄位皆為空值。但倒數第 2 個樣本就算其 `Cabin` 欄不為空值，也會因為 `Age` 欄為空而被選出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 選取或排除特定類型欄位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時候你會想選取 DataFrame 裡特定數據類型（字串、數值、時間型態等）的欄位，這時你可以使用 `select_dtypes` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass   Age  SibSp  Parch     Fare\n",
       "0            1         0       3  22.0      1      0   7.2500\n",
       "1            2         1       1  38.0      1      0  71.2833\n",
       "2            3         1       3  26.0      0      0   7.9250\n",
       "3            4         1       1  35.0      1      0  53.1000\n",
       "4            5         0       3  35.0      0      0   8.0500"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.select_dtypes(include='number').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面我們用一行程式碼就把所有數值欄位取出，儘管我們根本不知道有什麼欄位。而你當然也可以利用 `exclude` 參數來排除特定類型的欄位："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo1</td>\n",
       "      <td>2009-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>foo2</td>\n",
       "      <td>2009-01-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo3</td>\n",
       "      <td>2009-01-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>foo4</td>\n",
       "      <td>2009-01-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>foo5</td>\n",
       "      <td>2009-01-07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B     C          D\n",
       "0  0.0  0.0  foo1 2009-01-01\n",
       "1  1.0  1.0  foo2 2009-01-02\n",
       "2  2.0  0.0  foo3 2009-01-05\n",
       "3  3.0  1.0  foo4 2009-01-06\n",
       "4  4.0  0.0  foo5 2009-01-07"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "A           float64\n",
       "B           float64\n",
       "C            object\n",
       "D    datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B\n",
       "0  0.0  0.0\n",
       "1  1.0  1.0\n",
       "2  2.0  0.0\n",
       "3  3.0  1.0\n",
       "4  4.0  0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 建立一個有多種數據形態的 DataFrame\n",
    "df_mix = pd.util.testing.makeMixedDataFrame()\n",
    "display(df_mix)\n",
    "display(df_mix.dtypes)\n",
    "display(df_mix.select_dtypes(exclude=['datetime64', 'object']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas 裡的函式使用上都很直覺，你可以丟入 1 個包含多個元素的 Python `list` 或是單一 `str` 作為參數輸入。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 選取所有出現在 list 內的樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候針對某一個特定欄位，你會想要取出所有出現在一個 `list` 的樣本。這時候你可以使用 `isin` 函式來做到這件事情："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>SC/Paris 2123</td>\n",
       "      <td>41.5792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>608</th>\n",
       "      <td>609</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>SC/Paris 2123</td>\n",
       "      <td>41.5792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>686</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>SC/Paris 2123</td>\n",
       "      <td>41.5792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>702</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17475</td>\n",
       "      <td>26.2875</td>\n",
       "      <td>E24</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch         Ticket  \\\n",
       "43            44         1       2  female   3.0      1      2  SC/Paris 2123   \n",
       "608          609         1       2  female  22.0      1      2  SC/Paris 2123   \n",
       "685          686         0       2    male  25.0      1      2  SC/Paris 2123   \n",
       "701          702         1       1    male  35.0      0      0       PC 17475   \n",
       "\n",
       "        Fare Cabin Embarked  \n",
       "43   41.5792   NaN        C  \n",
       "608  41.5792   NaN        C  \n",
       "685  41.5792   NaN        C  \n",
       "701  26.2875   E24        S  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tickets = [\"SC/Paris 2123\", \"PC 17475\"]\n",
    "df[df.Ticket.isin(tickets)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 選取某欄位為 top-k 值的樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候你會想選取在某個欄位中前 k 大的所有樣本。這時你可以先利用 `value_counts` 函式找出該欄位前 k 多的值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['347082', '1601', 'CA. 2343'], dtype='object')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_k = 3\n",
    "top_tickets = df.Ticket.value_counts()[:top_k]\n",
    "top_tickets.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊我們以欄位 `Ticket` 為例。另外你也可以使用 [pandas.Series](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html) 裡的 `nlargest` 函式取得相同結果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['347082', '1601', 'CA. 2343'], dtype='object')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Ticket.value_counts().nlargest(top_k).index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接著利用上小節看過的 `isin` 函式就能輕鬆取得 `Ticket`欄位值為前 k 大值的樣本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>39.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>347082</td>\n",
       "      <td>31.2750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>75</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1601</td>\n",
       "      <td>56.4958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>120</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>347082</td>\n",
       "      <td>31.2750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>160</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>69.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>170</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1601</td>\n",
       "      <td>56.4958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch    Ticket  \\\n",
       "13            14         0       3    male  39.0      1      5    347082   \n",
       "74            75         1       3    male  32.0      0      0      1601   \n",
       "119          120         0       3  female   2.0      4      2    347082   \n",
       "159          160         0       3    male   NaN      8      2  CA. 2343   \n",
       "169          170         0       3    male  28.0      0      0      1601   \n",
       "\n",
       "        Fare Cabin Embarked  \n",
       "13   31.2750   NaN        S  \n",
       "74   56.4958   NaN        S  \n",
       "119  31.2750   NaN        S  \n",
       "159  69.5500   NaN        S  \n",
       "169  56.4958   NaN        S  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.Ticket.isin(top_tickets.index)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 找出符合特定字串的樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時你會想要對一個字串欄位做正規表示式（regular expression），取出符合某個 pattern 的所有樣本。\n",
    "\n",
    "這時你可以使用 `str` 底下的 `contains` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Saundercock, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5. 2151</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    PassengerId  Survived  Pclass                            Name   Sex   Age  \\\n",
       "0             1         0       3         Braund, Mr. Owen Harris  male  22.0   \n",
       "4             5         0       3        Allen, Mr. William Henry  male  35.0   \n",
       "5             6         0       3                Moran, Mr. James  male   NaN   \n",
       "6             7         0       1         McCarthy, Mr. Timothy J  male  54.0   \n",
       "12           13         0       3  Saundercock, Mr. William Henry  male  20.0   \n",
       "\n",
       "    SibSp  Parch     Ticket     Fare Cabin Embarked  \n",
       "0       1      0  A/5 21171   7.2500   NaN        S  \n",
       "4       0      0     373450   8.0500   NaN        S  \n",
       "5       0      0     330877   8.4583   NaN        Q  \n",
       "6       0      0      17463  51.8625   E46        S  \n",
       "12      0      0  A/5. 2151   8.0500   NaN        S  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('http://bit.ly/kaggletrain')\n",
    "df[df.Name.str.contains(\"Mr\\.\")].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊我們將所有 `Name` 欄位值裡包含 `Mr.` 的樣本取出。注意 `contains` 函式接受的是正規表示式，因此需要將 `.` 轉換成 `\\.`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用正規表示式選取數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時候你會想要依照一些規則來選取 DataFrame 裡頭的值、索引或是欄位，尤其是在處理跟時間序列相關的數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-01</th>\n",
       "      <td>-0.143945</td>\n",
       "      <td>-0.020783</td>\n",
       "      <td>1.434651</td>\n",
       "      <td>1.044208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-08</th>\n",
       "      <td>-0.901508</td>\n",
       "      <td>-1.082932</td>\n",
       "      <td>-1.035743</td>\n",
       "      <td>0.334431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-15</th>\n",
       "      <td>-1.544868</td>\n",
       "      <td>1.183136</td>\n",
       "      <td>-0.913117</td>\n",
       "      <td>-1.199199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-22</th>\n",
       "      <td>0.604091</td>\n",
       "      <td>-0.233628</td>\n",
       "      <td>-0.348109</td>\n",
       "      <td>1.082138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-29</th>\n",
       "      <td>0.062112</td>\n",
       "      <td>1.565870</td>\n",
       "      <td>-0.791369</td>\n",
       "      <td>1.017766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-05</th>\n",
       "      <td>-0.013661</td>\n",
       "      <td>1.015528</td>\n",
       "      <td>-0.420123</td>\n",
       "      <td>-0.513559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-12</th>\n",
       "      <td>0.404094</td>\n",
       "      <td>0.486909</td>\n",
       "      <td>-0.815937</td>\n",
       "      <td>0.743381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-19</th>\n",
       "      <td>-0.288886</td>\n",
       "      <td>2.560776</td>\n",
       "      <td>-0.864528</td>\n",
       "      <td>0.727740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-26</th>\n",
       "      <td>0.975203</td>\n",
       "      <td>-0.551452</td>\n",
       "      <td>0.531635</td>\n",
       "      <td>-0.595716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-03-04</th>\n",
       "      <td>-0.045714</td>\n",
       "      <td>0.137413</td>\n",
       "      <td>2.187056</td>\n",
       "      <td>1.164371</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "2000-01-01 -0.143945 -0.020783  1.434651  1.044208\n",
       "2000-01-08 -0.901508 -1.082932 -1.035743  0.334431\n",
       "2000-01-15 -1.544868  1.183136 -0.913117 -1.199199\n",
       "2000-01-22  0.604091 -0.233628 -0.348109  1.082138\n",
       "2000-01-29  0.062112  1.565870 -0.791369  1.017766\n",
       "2000-02-05 -0.013661  1.015528 -0.420123 -0.513559\n",
       "2000-02-12  0.404094  0.486909 -0.815937  0.743381\n",
       "2000-02-19 -0.288886  2.560776 -0.864528  0.727740\n",
       "2000-02-26  0.975203 -0.551452  0.531635 -0.595716\n",
       "2000-03-04 -0.045714  0.137413  2.187056  1.164371"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_date = pd.util.testing.makeTimeDataFrame(freq='7D')\n",
    "df_date.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假設你想將所有索引在 2000 年 2 月內的樣本取出，則可以透過 `filter` 函式達成這個目的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-02-05</th>\n",
       "      <td>-0.013661</td>\n",
       "      <td>1.015528</td>\n",
       "      <td>-0.420123</td>\n",
       "      <td>-0.513559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-12</th>\n",
       "      <td>0.404094</td>\n",
       "      <td>0.486909</td>\n",
       "      <td>-0.815937</td>\n",
       "      <td>0.743381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-19</th>\n",
       "      <td>-0.288886</td>\n",
       "      <td>2.560776</td>\n",
       "      <td>-0.864528</td>\n",
       "      <td>0.727740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-26</th>\n",
       "      <td>0.975203</td>\n",
       "      <td>-0.551452</td>\n",
       "      <td>0.531635</td>\n",
       "      <td>-0.595716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "2000-02-05 -0.013661  1.015528 -0.420123 -0.513559\n",
       "2000-02-12  0.404094  0.486909 -0.815937  0.743381\n",
       "2000-02-19 -0.288886  2.560776 -0.864528  0.727740\n",
       "2000-02-26  0.975203 -0.551452  0.531635 -0.595716"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_date.filter(regex=\"2000-02.*\", axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`filter` 函式本身功能十分強大，有興趣的讀者可以[閱讀官方文件進一步了解其用法](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.filter.html)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 選取從某時間點開始的區間樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在處理時間數據時，很多時候你會想要針對某個起始時間挑出前 t 個時間點的樣本。\n",
    "\n",
    "讓我們再看一次剛剛建立的 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-01</th>\n",
       "      <td>-0.143945</td>\n",
       "      <td>-0.020783</td>\n",
       "      <td>1.434651</td>\n",
       "      <td>1.044208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-08</th>\n",
       "      <td>-0.901508</td>\n",
       "      <td>-1.082932</td>\n",
       "      <td>-1.035743</td>\n",
       "      <td>0.334431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-15</th>\n",
       "      <td>-1.544868</td>\n",
       "      <td>1.183136</td>\n",
       "      <td>-0.913117</td>\n",
       "      <td>-1.199199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-22</th>\n",
       "      <td>0.604091</td>\n",
       "      <td>-0.233628</td>\n",
       "      <td>-0.348109</td>\n",
       "      <td>1.082138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-29</th>\n",
       "      <td>0.062112</td>\n",
       "      <td>1.565870</td>\n",
       "      <td>-0.791369</td>\n",
       "      <td>1.017766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-05</th>\n",
       "      <td>-0.013661</td>\n",
       "      <td>1.015528</td>\n",
       "      <td>-0.420123</td>\n",
       "      <td>-0.513559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-12</th>\n",
       "      <td>0.404094</td>\n",
       "      <td>0.486909</td>\n",
       "      <td>-0.815937</td>\n",
       "      <td>0.743381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-02-19</th>\n",
       "      <td>-0.288886</td>\n",
       "      <td>2.560776</td>\n",
       "      <td>-0.864528</td>\n",
       "      <td>0.727740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "2000-01-01 -0.143945 -0.020783  1.434651  1.044208\n",
       "2000-01-08 -0.901508 -1.082932 -1.035743  0.334431\n",
       "2000-01-15 -1.544868  1.183136 -0.913117 -1.199199\n",
       "2000-01-22  0.604091 -0.233628 -0.348109  1.082138\n",
       "2000-01-29  0.062112  1.565870 -0.791369  1.017766\n",
       "2000-02-05 -0.013661  1.015528 -0.420123 -0.513559\n",
       "2000-02-12  0.404094  0.486909 -0.815937  0.743381\n",
       "2000-02-19 -0.288886  2.560776 -0.864528  0.727740"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_date.head(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在索引為時間型態的情況下，如果你想要把前 3 週的樣本取出，可以使用 `first` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-01-01</th>\n",
       "      <td>-0.143945</td>\n",
       "      <td>-0.020783</td>\n",
       "      <td>1.434651</td>\n",
       "      <td>1.044208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-08</th>\n",
       "      <td>-0.901508</td>\n",
       "      <td>-1.082932</td>\n",
       "      <td>-1.035743</td>\n",
       "      <td>0.334431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-01-15</th>\n",
       "      <td>-1.544868</td>\n",
       "      <td>1.183136</td>\n",
       "      <td>-0.913117</td>\n",
       "      <td>-1.199199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A         B         C         D\n",
       "2000-01-01 -0.143945 -0.020783  1.434651  1.044208\n",
       "2000-01-08 -0.901508 -1.082932 -1.035743  0.334431\n",
       "2000-01-15 -1.544868  1.183136 -0.913117 -1.199199"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_date.first('3W')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "十分方便的函式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基本數據處理與轉換"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在了解如何選取想要的數據以後，你可以透過這節的介紹來熟悉 pandas 裡一些常見的數據處理方式。這章節也是我認為使用 pandas 處理數據時最令人愉快的部分之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 對某一軸套用相同運算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你時常會需要對 DataFrame 裡頭的每一個欄位（縱軸）或是每一列（橫軸）做相同的運算。\n",
    "\n",
    "比方說你想將鐵達尼號資料集內的 `Survived` 數值欄位轉換成人類容易理解的字串："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>存活</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>死亡</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>倖存</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>倖存</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>倖存</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>死亡</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>死亡</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  存活  Survived  Pclass     Sex\n",
       "0            1  死亡         0       3    male\n",
       "1            2  倖存         1       1  female\n",
       "2            3  倖存         1       3  female\n",
       "3            4  倖存         1       1  female\n",
       "4            5  死亡         0       3    male\n",
       "5            6  死亡         0       3    male"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 重新讀取鐵達尼號數據\n",
    "df_titanic = pd.read_csv('http://bit.ly/kaggletrain')\n",
    "df_titanic = df_titanic.drop(\"Name\", axis=1)\n",
    "\n",
    "# 複製一份副本 DataFrame\n",
    "df = df_titanic.copy()\n",
    "columns = df.columns.tolist()[:4]\n",
    "\n",
    "# 好戲登場\n",
    "new_col = '存活'\n",
    "columns.insert(1, new_col)  # 調整欄位順序用\n",
    "df[new_col] = df.Survived.apply(lambda x: '倖存' if x else '死亡')\n",
    "df.loc[:5, columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過 `apply` 函式，我們把一個匿名函式 `lambda` 套用到整個 `df.Survived` Series 之上，並以此建立一個新的 `存活` 欄位。\n",
    "\n",
    "值得一提的是當你需要追加新的欄位但又不想影響到原始 DataFrame 時，可以使用 `copy` 函式複製一份副本另行操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 對每一個樣本做自定義運算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上小節我們用 `apply` 函式對 DataFrame 裡頭的**某個 Series **做運算並生成新欄位：\n",
    "\n",
    "```python\n",
    "df[new_col] = df.Survived.apply(...\n",
    "```\n",
    "\n",
    "不過你時常會想要把樣本（row）裡頭的**多個**欄位一次取出做運算並產生一個新的值。這時你可以自定義一個 Python function 並將 `apply` 函式套用到整個 DataFrame 之上："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3    male  22.0      1      0   \n",
       "1            2         1       1  female  38.0      1      0   \n",
       "2            3         1       3  female  26.0      0      0   \n",
       "3            4         1       1  female  35.0      1      0   \n",
       "4            5         0       3    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>描述</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>一名 22.0 歲的男性</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>一名 38.0 歲的女性</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>一名 26.0 歲的女性</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>一名 35.0 歲的女性</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>一名 35.0 歲的男性</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Sex   Age  SibSp  Parch            Ticket     Fare Cabin Embarked  \\\n",
       "0    male  22.0      1      0         A/5 21171   7.2500   NaN        S   \n",
       "1  female  38.0      1      0          PC 17599  71.2833   C85        C   \n",
       "2  female  26.0      0      0  STON/O2. 3101282   7.9250   NaN        S   \n",
       "3  female  35.0      1      0            113803  53.1000  C123        S   \n",
       "4    male  35.0      0      0            373450   8.0500   NaN        S   \n",
       "\n",
       "             描述  \n",
       "0  一名 22.0 歲的男性  \n",
       "1  一名 38.0 歲的女性  \n",
       "2  一名 26.0 歲的女性  \n",
       "3  一名 35.0 歲的女性  \n",
       "4  一名 35.0 歲的男性  "
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "display(df.head())\n",
    "\n",
    "# apply custom function 可以說是 pandas 裡最重要的技巧之一\n",
    "d = {'male': '男性', 'female': '女性'} \n",
    "def generate_desc(row):\n",
    "    return f\"一名 {row['Age']} 歲的{d[row['Sex']]}\"\n",
    "\n",
    "df['描述'] = df.apply(generate_desc, axis=1)\n",
    "df.loc[:4, 'Sex':]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此例中 `apply` 函式將 `generate_desc` 函式**個別**套用到 DataFrame 裡頭的每一個樣本（row），結合 `Sex` 及 `Age` 兩欄位資訊，生成新的 `描述`。\n",
    "\n",
    "當然，將 `axis` 設置為 `0` 則可以對每一個欄位分別套用自定義的 Python function。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將連續數值轉換成分類數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時你會想把一個連續數值（numerical）的欄位分成多個 groups 以方便對每個 groups 做統計。這時候你可以使用 `pd.cut` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>年齡區間</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>631</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 10</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27042</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>A23</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>852</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 10</td>\n",
       "      <td>74.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>347060</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>113509</td>\n",
       "      <td>61.9792</td>\n",
       "      <td>B30</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>97</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 9</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17754</td>\n",
       "      <td>34.6542</td>\n",
       "      <td>A5</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>494</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 9</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17609</td>\n",
       "      <td>49.5042</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>族群 9</td>\n",
       "      <td>70.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370369</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass   Sex   年齡區間   Age  SibSp  Parch    Ticket  \\\n",
       "0          631         1       1  male  族群 10  80.0      0      0     27042   \n",
       "1          852         0       3  male  族群 10  74.0      0      0    347060   \n",
       "2           55         0       1  male   族群 9  65.0      0      1    113509   \n",
       "3           97         0       1  male   族群 9  71.0      0      0  PC 17754   \n",
       "4          494         0       1  male   族群 9  71.0      0      0  PC 17609   \n",
       "5          117         0       3  male   族群 9  70.5      0      0    370369   \n",
       "\n",
       "      Fare Cabin Embarked  \n",
       "0  30.0000   A23        S  \n",
       "1   7.7750   NaN        S  \n",
       "2  61.9792   B30        C  \n",
       "3  34.6542    A5        C  \n",
       "4  49.5042   NaN        C  \n",
       "5   7.7500   NaN        Q  "
      ]
     },
     "execution_count": 290,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "\n",
    "# 為了方便比較新舊欄位\n",
    "columns = df.columns.tolist()\n",
    "new_col = '年齡區間'\n",
    "columns.insert(4, new_col)\n",
    "\n",
    "# 將 numerical 轉換成 categorical 欄位\n",
    "labels = [f'族群 {i}' for i in range(1, 11)]\n",
    "df[new_col] = pd.cut(x=df.Age, \n",
    "                     bins=10, \n",
    "                     labels=labels)\n",
    "\n",
    "# 可以排序切割後的 categorical 欄位\n",
    "(df.sort_values(new_col, ascending=False)\n",
    "   .reset_index()\n",
    "   .loc[:5, columns]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上所示，使用 `pd.cut` 函式建立出來的每個分類 `族群 X` 有大小之分，因此你可以輕易地使用 `sort_values` 函式排序樣本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CategoricalDtype(categories=['族群 1', '族群 2', '族群 3', '族群 4', '族群 5', '族群 6', '族群 7',\n",
       "                  '族群 8', '族群 9', '族群 10'],\n",
       "                 ordered=True)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[new_col].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將 DataFrame 隨機切成兩個子集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有時你會想將手上的 DataFrame 隨機切成兩個獨立的子集。選取其中一個子集來訓練機器學習模型是一個常見的情境。\n",
    "\n",
    "要做到這件事情有很多種方法，你可以使用 [scikit-learn 的 `train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) 或是 numpy 的 `np.random.randn`，但假如你想要純 pandas 解法，可以使用 `sample` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>295</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349233</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>200</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>248747</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>865</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>233866</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625</th>\n",
       "      <td>626</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>61.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36963</td>\n",
       "      <td>32.3208</td>\n",
       "      <td>D50</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>638</th>\n",
       "      <td>639</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>41.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>3101295</td>\n",
       "      <td>39.6875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch   Ticket  \\\n",
       "294          295         0       3    male  24.0      0      0   349233   \n",
       "199          200         0       2  female  24.0      0      0   248747   \n",
       "864          865         0       2    male  24.0      0      0   233866   \n",
       "625          626         0       1    male  61.0      0      0    36963   \n",
       "638          639         0       3  female  41.0      0      5  3101295   \n",
       "\n",
       "        Fare Cabin Embarked  \n",
       "294   7.8958   NaN        S  \n",
       "199  13.0000   NaN        S  \n",
       "864  13.0000   NaN        S  \n",
       "625  32.3208   D50        S  \n",
       "638  39.6875   NaN        S  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>PP 9549</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>G6</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>113783</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>C103</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5. 2151</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>350406</td>\n",
       "      <td>7.8542</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>382652</td>\n",
       "      <td>29.1250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch     Ticket  \\\n",
       "10           11         1       3  female   4.0      1      1    PP 9549   \n",
       "11           12         1       1  female  58.0      0      0     113783   \n",
       "12           13         0       3    male  20.0      0      0  A/5. 2151   \n",
       "14           15         0       3  female  14.0      0      0     350406   \n",
       "16           17         0       3    male   2.0      4      1     382652   \n",
       "\n",
       "       Fare Cabin Embarked  \n",
       "10  16.7000    G6        S  \n",
       "11  26.5500  C103        S  \n",
       "12   8.0500   NaN        S  \n",
       "14   7.8542   NaN        S  \n",
       "16  29.1250   NaN        Q  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "各 DataFrame 大小： 891 713 178\n"
     ]
    }
   ],
   "source": [
    "df_train = df_titanic.sample(frac=0.8, random_state=5566)\n",
    "df_test = df_titanic.drop(df_train.index)\n",
    "\n",
    "# 顯示結果，無特殊操作\n",
    "display(df_train.head())\n",
    "display(df_test.head())\n",
    "print('各 DataFrame 大小：', \n",
    "      len(df_titanic), len(df_train), len(df_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個解法的前提是原來的 DataFrame `df_titanic` 裡頭的索引是獨一無二的。另外記得設定 `random_state` 以方便別人重現你的結果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用 SQL 的方式合併兩個 DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候你會想要將兩個 DataFrames 依照某個共通的欄位（鍵值）合併成單一 DataFrame 以整合資訊。\n",
    "\n",
    "比方說給定以下兩個 DataFrames："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>密蘇里州</td>\n",
       "      <td>堪薩斯城</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>亞利桑那州</td>\n",
       "      <td>鳳凰城</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>肯塔基州</td>\n",
       "      <td>路易維爾</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>紐約州</td>\n",
       "      <td>紐約市</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   state  city\n",
       "0   密蘇里州  堪薩斯城\n",
       "1  亞利桑那州   鳳凰城\n",
       "2   肯塔基州  路易維爾\n",
       "3    紐約州   紐約市"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>population</th>\n",
       "      <th>feature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>路易維爾</td>\n",
       "      <td>741096</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>堪薩斯城</td>\n",
       "      <td>481420</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>鳳凰城</td>\n",
       "      <td>4039182</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   city  population feature\n",
       "0  路易維爾      741096       a\n",
       "1  堪薩斯城      481420       b\n",
       "2   鳳凰城     4039182       c"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_city = pd.DataFrame({\n",
    "    'state': ['密蘇里州', '亞利桑那州', '肯塔基州', '紐約州'],\n",
    "    'city': ['堪薩斯城', '鳳凰城', '路易維爾', '紐約市']\n",
    "})\n",
    "df_info = pd.DataFrame({\n",
    "    'city': ['路易維爾', '堪薩斯城', '鳳凰城'],\n",
    "    'population': [741096, 481420, 4039182],\n",
    "    'feature': list('abc')})\n",
    "\n",
    "display(df_city)\n",
    "display(df_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrame `df_city` 包含了幾個美國城市以及其對應的州名（state）；DataFrame `df_info` 則包含城市名稱以及一些數據。如果你想將這兩個 DataFrames 合併（merge），可以使用非常方便的 `merge` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>city</th>\n",
       "      <th>population</th>\n",
       "      <th>feature</th>\n",
       "      <th>_merge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>密蘇里州</td>\n",
       "      <td>堪薩斯城</td>\n",
       "      <td>481420.0</td>\n",
       "      <td>b</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>亞利桑那州</td>\n",
       "      <td>鳳凰城</td>\n",
       "      <td>4039182.0</td>\n",
       "      <td>c</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>肯塔基州</td>\n",
       "      <td>路易維爾</td>\n",
       "      <td>741096.0</td>\n",
       "      <td>a</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>紐約州</td>\n",
       "      <td>紐約市</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>left_only</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   state  city  population feature     _merge\n",
       "0   密蘇里州  堪薩斯城    481420.0       b       both\n",
       "1  亞利桑那州   鳳凰城   4039182.0       c       both\n",
       "2   肯塔基州  路易維爾    741096.0       a       both\n",
       "3    紐約州   紐約市         NaN     NaN  left_only"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(left=df_city,\n",
    "         right=df_info,\n",
    "         how=\"left\", # left outer join\n",
    "         on=\"city\", # 透過此欄位合併\n",
    "         indicator=True # 顯示結果中每一列的來源\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "沒錯，`merge` 函式運作方式就像 [SQL](https://leemeng.tw/why-you-need-to-learn-sql-as-a-data-scientist.html) 一樣，可以讓你透過更改 `how` 參數來做：\n",
    "- `left`：left outer join\n",
    "- `right`：right outer join\n",
    "- `outer`: full outer join\n",
    "- `inner`： inner join\n",
    "\n",
    "注意合併後的 DataFrame 的最後一列：因為是 left join，就算右側的 `df_info` 裡頭並沒有紐約市的資訊，我們也能把該城市保留在 `merge` 後的結果。你還可以透過 `indicator=True` 的方式讓 pandas 幫我們新增一個 `_merge` 欄位，輕鬆了解紐約市只存在左側的 `df_city` 裡。\n",
    "\n",
    "`merge` 函式強大之處在於能跟 SQL 一樣為我們抽象化如何合併兩個 DataFrames 的運算。如果你想要了解 SQL 跟 Python 本質上的差異，可以參考[為何資料科學家需要學習 SQL](https://leemeng.tw/why-you-need-to-learn-sql-as-a-data-scientist.html) 一文。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 存取並操作每一個樣本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們前面看過，雖然一般可以直接使用 [`apply` 函式](#對某一軸套用相同運算)來對每個樣本作運算，有時候你就是會想用 for 迴圈的方式把每個樣本取出處理。\n",
    "\n",
    "這種時候你可以用 `itertuples` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "堪薩斯城是密蘇里州裡頭的一個城市\n",
      "鳳凰城是亞利桑那州裡頭的一個城市\n",
      "路易維爾是肯塔基州裡頭的一個城市\n",
      "紐約市是紐約州裡頭的一個城市\n"
     ]
    }
   ],
   "source": [
    "for row in df_city.itertuples(name='City'):\n",
    "    print(f'{row.city}是{row.state}裡頭的一個城市')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "顧名思義，`itertuples` 函式回傳的是 Python [namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple)，也是一個你應該已經很熟悉的資料型態："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "City = namedtuple('City', ['Index', 'state', 'city'])\n",
    "c = City(3, '紐約州', '紐約市')\n",
    "c == row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 簡單匯總 & 分析數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在對數據做些基本處理以後，你會想要從手上的 DataFrame 匯總或整理出一些有用的統計數據。本節介紹一些常用的數據匯總技巧。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 取出某欄位 top k 的值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這你在[選取某欄位為 top-k 值的樣本](#選取某欄位為-top-k-值的樣本)小節應該就看過了。\n",
    "\n",
    "但因為這個使用情境實在太常出現，讓我們再次嘗試將鐵達尼號數據集裡頭 `Ticket` 欄位最常出現的值取出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3    male  22.0      1      0   \n",
       "1            2         1       1  female  38.0      1      0   \n",
       "2            3         1       3  female  26.0      0      0   \n",
       "3            4         1       1  female  35.0      1      0   \n",
       "4            5         0       3    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Ticket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>347082</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1601</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3101295</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CA 2144</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      index  Ticket\n",
       "0    347082       7\n",
       "1      1601       7\n",
       "2  CA. 2343       7\n",
       "3   3101295       6\n",
       "4   CA 2144       6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "display(df.head())\n",
    "display(df.Ticket.value_counts().head(5).reset_index())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`value_counts` 函式預設就會把欄位裡頭的值依照出現頻率**由高到低**排序，因此搭配 `head` 函式就可以把最常出現的 top k 值選出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一行描述數值欄位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "當你想要快速了解 DataFrame 裡所有數值欄位的統計數據（最小值、最大值、平均和中位數等）時可以使用 `describe` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>32.204208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>49.693429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.910400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   891.000000  891.000000  891.000000  714.000000  891.000000   \n",
       "mean    446.000000    0.383838    2.308642   29.699118    0.523008   \n",
       "std     257.353842    0.486592    0.836071   14.526497    1.102743   \n",
       "min       1.000000    0.000000    1.000000    0.420000    0.000000   \n",
       "25%     223.500000    0.000000    2.000000   20.125000    0.000000   \n",
       "50%     446.000000    0.000000    3.000000   28.000000    0.000000   \n",
       "75%     668.500000    1.000000    3.000000   38.000000    1.000000   \n",
       "max     891.000000    1.000000    3.000000   80.000000    8.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  891.000000  891.000000  \n",
       "mean     0.381594   32.204208  \n",
       "std      0.806057   49.693429  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    7.910400  \n",
       "50%      0.000000   14.454200  \n",
       "75%      0.000000   31.000000  \n",
       "max      6.000000  512.329200  "
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以用[取得想要關注的數據](#取得想要關注的數據_1)一節的技巧來選取自己關心的統計數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>29.699118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>14.526497</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Survived    Pclass        Age\n",
       "mean  0.383838  2.308642  29.699118\n",
       "std   0.486592  0.836071  14.526497"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().loc[['mean', 'std'], 'Survived':'Age']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 找出欄位裡所有出現過的值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "針對特定欄位使用 `unique` 函式即可："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['male', 'female'], dtype=object)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Sex.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分組匯總結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多時候你會想要把 DataFrame 裡頭的樣本依照某些特性分門別類，並依此匯總各組（group）的統計數據。這種時候你可以用 `groupby` 函式。\n",
    "\n",
    "讓我們再次拿出鐵達尼號數據集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3    male  22.0      1      0   \n",
       "1            2         1       1  female  38.0      1      0   \n",
       "2            3         1       3  female  26.0      0      0   \n",
       "3            4         1       1  female  35.0      1      0   \n",
       "4            5         0       3    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以將所有乘客（列）依照它們的 `Pclass` 欄位值分組，並計算每組裡頭乘客們的平均年齡："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pclass\n",
       "1    38.233441\n",
       "2    29.877630\n",
       "3    25.140620\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"Pclass\").Age.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以搭配剛剛看過的 `describe` 函式來匯總各組的統計數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>314.0</td>\n",
       "      <td>0.742038</td>\n",
       "      <td>0.438211</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>577.0</td>\n",
       "      <td>0.188908</td>\n",
       "      <td>0.391775</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        count      mean       std  min  25%  50%  75%  max\n",
       "Sex                                                       \n",
       "female  314.0  0.742038  0.438211  0.0  0.0  1.0  1.0  1.0\n",
       "male    577.0  0.188908  0.391775  0.0  0.0  0.0  0.0  1.0"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"Sex\").Survived.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以依照多個欄位分組，並利用 `size` 函式迅速地取得各組包含的樣本數："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pclass</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>94</td>\n",
       "      <td>76</td>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>122</td>\n",
       "      <td>108</td>\n",
       "      <td>347</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pclass    1    2    3\n",
       "Sex                  \n",
       "female   94   76  144\n",
       "male    122  108  347"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"Sex\", 'Pclass']).size().unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以用 `agg` 函式（**agg**regate，匯總）搭配 `groupby` 函式來將每一組樣本依照多種方式匯總："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th>Pclass</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">female</th>\n",
       "      <th>1</th>\n",
       "      <td>2.00</td>\n",
       "      <td>63.0</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.00</td>\n",
       "      <td>57.0</td>\n",
       "      <td>74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.75</td>\n",
       "      <td>63.0</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">male</th>\n",
       "      <th>1</th>\n",
       "      <td>0.92</td>\n",
       "      <td>80.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.67</td>\n",
       "      <td>70.0</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.42</td>\n",
       "      <td>74.0</td>\n",
       "      <td>253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                min   max  count\n",
       "Sex    Pclass                   \n",
       "female 1       2.00  63.0     85\n",
       "       2       2.00  57.0     74\n",
       "       3       0.75  63.0    102\n",
       "male   1       0.92  80.0    101\n",
       "       2       0.67  70.0     99\n",
       "       3       0.42  74.0    253"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"Sex\", 'Pclass']).Age.agg(['min', 'max', 'count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過 `unstack` 函式能讓你產生跟 `pivot_table` 函式相同的結果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">min</th>\n",
       "      <th colspan=\"3\" halign=\"left\">max</th>\n",
       "      <th colspan=\"3\" halign=\"left\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>2.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>63.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>85</td>\n",
       "      <td>74</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.92</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.42</td>\n",
       "      <td>80.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>101</td>\n",
       "      <td>99</td>\n",
       "      <td>253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         min               max             count         \n",
       "Pclass     1     2     3     1     2     3     1   2    3\n",
       "Sex                                                      \n",
       "female  2.00  2.00  0.75  63.0  57.0  63.0    85  74  102\n",
       "male    0.92  0.67  0.42  80.0  70.0  74.0   101  99  253"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"Sex\", 'Pclass']).Age.agg(['min', 'max', 'count']).unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "當然，你也可以直接使用 `pivot_table` 函式來匯總各組數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">min</th>\n",
       "      <th colspan=\"3\" halign=\"left\">max</th>\n",
       "      <th colspan=\"3\" halign=\"left\">count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>2.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>63.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>85</td>\n",
       "      <td>74</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.92</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.42</td>\n",
       "      <td>80.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>101</td>\n",
       "      <td>99</td>\n",
       "      <td>253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         min               max             count         \n",
       "Pclass     1     2     3     1     2     3     1   2    3\n",
       "Sex                                                      \n",
       "female  2.00  2.00  0.75  63.0  57.0  63.0    85  74  102\n",
       "male    0.92  0.67  0.42  80.0  70.0  74.0   101  99  253"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot_table(index='Sex', \n",
    "               columns='Pclass', \n",
    "               values='Age', \n",
    "               aggfunc=['min', 'max', 'count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "依照背景不同，每個人會有偏好的 pandas 使用方式。選擇對你來說最直覺又好記的方式吧！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 結合原始數據與匯總結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不管是上節的 `groupby` 搭配 `agg` 還是 `pivot_table`，匯總結果都會以另外一個全新的 DataFrame 表示。\n",
    "\n",
    "有時候你會想直接把各組匯總的結果放到原本的 DataFrame 裡頭，方便比較原始樣本與匯總結果的差異。這時你可以使用 `transform` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col1 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col8 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col9 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col1 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col8 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col9 {\n",
       "            background-color:  rgb(153, 255, 51);\n",
       "        }</style><table id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >Sex</th>        <th class=\"col_heading level0 col1\" >Age</th>        <th class=\"col_heading level0 col2\" >SibSp</th>        <th class=\"col_heading level0 col3\" >Parch</th>        <th class=\"col_heading level0 col4\" >Ticket</th>        <th class=\"col_heading level0 col5\" >Fare</th>        <th class=\"col_heading level0 col6\" >Cabin</th>        <th class=\"col_heading level0 col7\" >Embarked</th>        <th class=\"col_heading level0 col8\" >Avg_age</th>        <th class=\"col_heading level0 col9\" >Above_avg_age</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col0\" class=\"data row0 col0\" >male</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col1\" class=\"data row0 col1\" >22</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col4\" class=\"data row0 col4\" >A/5 21171</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col5\" class=\"data row0 col5\" >7.25</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col6\" class=\"data row0 col6\" >nan</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col7\" class=\"data row0 col7\" >S</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col8\" class=\"data row0 col8\" >30.7266</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row0_col9\" class=\"data row0 col9\" >no</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col0\" class=\"data row1 col0\" >female</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col1\" class=\"data row1 col1\" >38</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col2\" class=\"data row1 col2\" >1</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col4\" class=\"data row1 col4\" >PC 17599</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col5\" class=\"data row1 col5\" >71.2833</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col6\" class=\"data row1 col6\" >C85</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col7\" class=\"data row1 col7\" >C</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col8\" class=\"data row1 col8\" >27.9157</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row1_col9\" class=\"data row1 col9\" >yes</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col0\" class=\"data row2 col0\" >female</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col1\" class=\"data row2 col1\" >26</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col3\" class=\"data row2 col3\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col4\" class=\"data row2 col4\" >STON/O2. 3101282</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col5\" class=\"data row2 col5\" >7.925</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col6\" class=\"data row2 col6\" >nan</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col7\" class=\"data row2 col7\" >S</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col8\" class=\"data row2 col8\" >27.9157</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row2_col9\" class=\"data row2 col9\" >no</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col0\" class=\"data row3 col0\" >female</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col1\" class=\"data row3 col1\" >35</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col2\" class=\"data row3 col2\" >1</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col3\" class=\"data row3 col3\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col4\" class=\"data row3 col4\" >113803</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col5\" class=\"data row3 col5\" >53.1</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col6\" class=\"data row3 col6\" >C123</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col7\" class=\"data row3 col7\" >S</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col8\" class=\"data row3 col8\" >27.9157</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row3_col9\" class=\"data row3 col9\" >yes</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col0\" class=\"data row4 col0\" >male</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col1\" class=\"data row4 col1\" >35</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col2\" class=\"data row4 col2\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col3\" class=\"data row4 col3\" >0</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col4\" class=\"data row4 col4\" >373450</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col5\" class=\"data row4 col5\" >8.05</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col6\" class=\"data row4 col6\" >nan</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col7\" class=\"data row4 col7\" >S</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col8\" class=\"data row4 col8\" >30.7266</td>\n",
       "                        <td id=\"T_51aa6300_b014_11e9_8b6c_8c8590794fe2row4_col9\" class=\"data row4 col9\" >yes</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1a2e879cc0>"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "df['Avg_age'] = df.groupby(\"Sex\").Age.transform(\"mean\")\n",
    "df['Above_avg_age'] = df.apply(lambda x: 'yes' if x.Age > x.Avg_age else 'no', \n",
    "                               axis=1)\n",
    "# styling\n",
    "(df.loc[:4, 'Sex':]\n",
    " .style\n",
    " .highlight_max(subset=['Avg_age'])\n",
    " .applymap(lambda x: 'background-color: rgb(153, 255, 51)', \n",
    "           subset=pd.IndexSlice[[0, 4], ['Age', 'Above_avg_age']])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此例將所有乘客依照性別 `Sex` 分組之後，計算各組的平均年齡 `Age`，並利用 `transform` 函式將各組結果插入對應的乘客（列）裡頭。你會發現兩名男乘客跟平均男性壽命 `Avg_age` 欄位相比正好一上一下，這差異則反映到 `Above_avg_age` 欄位裡頭。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 對時間數據做匯總"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "給定一個跟時間相關的 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000-03-31</th>\n",
       "      <td>-20.465499</td>\n",
       "      <td>-5.402825</td>\n",
       "      <td>4.790400</td>\n",
       "      <td>-6.949846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-06-30</th>\n",
       "      <td>-3.573217</td>\n",
       "      <td>2.091491</td>\n",
       "      <td>12.725446</td>\n",
       "      <td>-10.215842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-09-30</th>\n",
       "      <td>-3.377268</td>\n",
       "      <td>3.600146</td>\n",
       "      <td>13.607341</td>\n",
       "      <td>6.309234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000-12-31</th>\n",
       "      <td>-11.181567</td>\n",
       "      <td>-16.471324</td>\n",
       "      <td>8.659892</td>\n",
       "      <td>5.619725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001-03-31</th>\n",
       "      <td>7.148111</td>\n",
       "      <td>8.887097</td>\n",
       "      <td>-0.117279</td>\n",
       "      <td>-1.628799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001-06-30</th>\n",
       "      <td>9.765045</td>\n",
       "      <td>-1.160198</td>\n",
       "      <td>4.898406</td>\n",
       "      <td>12.054339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001-09-30</th>\n",
       "      <td>-8.067456</td>\n",
       "      <td>-9.746304</td>\n",
       "      <td>-12.500050</td>\n",
       "      <td>-8.970445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001-12-31</th>\n",
       "      <td>-3.300273</td>\n",
       "      <td>-11.148755</td>\n",
       "      <td>-13.194654</td>\n",
       "      <td>12.852938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-03-31</th>\n",
       "      <td>-6.421518</td>\n",
       "      <td>12.420474</td>\n",
       "      <td>-3.546766</td>\n",
       "      <td>20.694348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002-06-30</th>\n",
       "      <td>11.967359</td>\n",
       "      <td>9.522958</td>\n",
       "      <td>1.557142</td>\n",
       "      <td>-2.778456</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    A          B          C          D\n",
       "2000-03-31 -20.465499  -5.402825   4.790400  -6.949846\n",
       "2000-06-30  -3.573217   2.091491  12.725446 -10.215842\n",
       "2000-09-30  -3.377268   3.600146  13.607341   6.309234\n",
       "2000-12-31 -11.181567 -16.471324   8.659892   5.619725\n",
       "2001-03-31   7.148111   8.887097  -0.117279  -1.628799\n",
       "2001-06-30   9.765045  -1.160198   4.898406  12.054339\n",
       "2001-09-30  -8.067456  -9.746304 -12.500050  -8.970445\n",
       "2001-12-31  -3.300273 -11.148755 -13.194654  12.852938\n",
       "2002-03-31  -6.421518  12.420474  -3.546766  20.694348\n",
       "2002-06-30  11.967359   9.522958   1.557142  -2.778456"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_date = pd.util.testing.makeTimeDataFrame(freq='Q').head(10) * 10\n",
    "df_date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以利用 `resample` 函式來依照不同時間粒度匯總這個時間 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>A</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2000-12-31</td>\n",
       "      <td>-3.377268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2001-12-31</td>\n",
       "      <td>9.765045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2002-12-31</td>\n",
       "      <td>11.967359</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        year          A\n",
       "0 2000-12-31  -3.377268\n",
       "1 2001-12-31   9.765045\n",
       "2 2002-12-31  11.967359"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df_date.resample('Y').A.max()\n",
    " .reset_index()\n",
    " .rename({'index': 'year'}, axis=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此例中將不同年份（**Y**ear）的樣本分組，並從每一組的欄位 `A` 中選出最大值。你可以查看[官方 resampling 說明文件](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#resampling)來了解還有什麼時間粒度可以選擇（分鐘、小時、月份等等）。\n",
    "\n",
    "另外小細節是你可以利用 [numpy 的 broadcasting 運算](https://numpy.org/devdocs/user/theory.broadcasting.html)輕鬆地將 DataFrame 裡的所有數值做操作（初始 `df_date` 時的用到的 `* 10`）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 簡易繪圖並修改預設樣式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 Python 世界裡有很多數據視覺化工具供你選擇，比方說經典的 [Matplotlib](https://matplotlib.org/) 以及在[淺談神經機器翻譯一文](https://leemeng.tw/neural-machine-translation-with-transformer-and-tensorflow2.html)中被我拿來視覺化矩陣運算的 [Seaborn](https://seaborn.pydata.org/)。你也可以使用前面看過的 [Chartify](https://leemeng.tw/chartify-a-simple-yet-powerful-python-data-visualization-tool-which-boost-your-productivity-as-a-data-scientist.html) 搭配 pandas 畫出美麗圖表。\n",
    "\n",
    "但有時，你只需要 pandas 內建的 `plot` 函式就能輕鬆地將一個 DataFrame 轉成統計圖：\n",
    "\n",
    "```python\n",
    "# 用 `plot` 一行畫圖。前兩行只是改變預設樣式\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use(\"ggplot\")\n",
    "df.groupby(\"Pclass\").Survived.count().plot(\"barh\");\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/pandas_plot.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們都是視覺動物，pandas 的 `plot` 函式讓你在進行[探索型數據分析（**E**xploratory **D**ata **A**nalysis, EDA）](https://en.wikipedia.org/wiki/Exploratory_data_analysis)、試著快速了解手上數據集時十分方便。我們在[淺談資料視覺化以及 ggplot2 實踐](https://leemeng.tw/data-visualization-from-matplotlib-to-ggplot2.html)一文也已經探討過類似概念：\n",
    "\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- ggplot2/data-vis-purpose.png\n",
    "- 在做 EDA 時只需要包含最重要且你自己能夠解讀的視覺變數，重點是效率\n",
    "- https://leemeng.tw/data-visualization-from-matplotlib-to-ggplot2.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/>\n",
    "\n",
    "另外 pandas 底層預設使用 Matplotlib 繪圖，而用過 Matplotlib 的人都知道其初始的繪圖樣式實在不太討喜。你可以透過 `plt.style.available` 查看所有可供使用的繪圖樣式（style），並將喜歡的樣式透過 `plt.style.use()` 套用到所有 DataFrames 的 `plot` 函式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['seaborn-dark',\n",
       " 'seaborn-darkgrid',\n",
       " 'seaborn-ticks',\n",
       " 'fivethirtyeight',\n",
       " 'seaborn-whitegrid',\n",
       " 'classic',\n",
       " '_classic_test',\n",
       " 'fast',\n",
       " 'seaborn-talk',\n",
       " 'seaborn-dark-palette',\n",
       " 'seaborn-bright',\n",
       " 'seaborn-pastel',\n",
       " 'grayscale',\n",
       " 'seaborn-notebook',\n",
       " 'ggplot',\n",
       " 'seaborn-colorblind',\n",
       " 'seaborn-muted',\n",
       " 'seaborn',\n",
       " 'Solarize_Light2',\n",
       " 'seaborn-paper',\n",
       " 'bmh',\n",
       " 'tableau-colorblind10',\n",
       " 'seaborn-white',\n",
       " 'dark_background',\n",
       " 'seaborn-poster',\n",
       " 'seaborn-deep']"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plt.style.available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 與 pandas 相得益彰的實用工具"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前面幾個章節介紹了不少 pandas 的使用技巧與操作概念，這節則介紹一些我認為十分適合跟 pandas 一起搭配使用的數據工具 / 函式庫。在說明每個工具的功能時，我都會使用你已經十分熟悉的鐵達尼號數據集作為範例 DataFrame："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch  \\\n",
       "0            1         0       3    male  22.0      1      0   \n",
       "1            2         1       1  female  38.0      1      0   \n",
       "2            3         1       3  female  26.0      0      0   \n",
       "3            4         1       1  female  35.0      1      0   \n",
       "4            5         0       3    male  35.0      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  \n",
       "0         A/5 21171   7.2500   NaN        S  \n",
       "1          PC 17599  71.2833   C85        C  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3            113803  53.1000  C123        S  \n",
       "4            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_titanic.copy()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [tqdm](https://github.com/tqdm/tqdm)：了解你的數據處理進度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[tqdm](https://github.com/tqdm/tqdm) 是一個十分強大的 Python 進度條工具，且有[整合 pandas](https://github.com/tqdm/tqdm#pandas-integration)。此工具可以幫助我們了解 DataFrame `apply` 函式的進度。回想一下我們在[對某一軸套用相同運算](#對某一軸套用相同運算)一節做的一個簡單 `apply` 運算："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>存活</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>死亡</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>倖存</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>倖存</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>倖存</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>死亡</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>死亡</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass     Sex   Age  SibSp  Parch            Ticket     Fare  \\\n",
       "0         0       3    male  22.0      1      0         A/5 21171   7.2500   \n",
       "1         1       1  female  38.0      1      0          PC 17599  71.2833   \n",
       "2         1       3  female  26.0      0      0  STON/O2. 3101282   7.9250   \n",
       "3         1       1  female  35.0      1      0            113803  53.1000   \n",
       "4         0       3    male  35.0      0      0            373450   8.0500   \n",
       "5         0       3    male   NaN      0      0            330877   8.4583   \n",
       "\n",
       "  Cabin Embarked  存活  \n",
       "0   NaN        S  死亡  \n",
       "1   C85        C  倖存  \n",
       "2   NaN        S  倖存  \n",
       "3  C123        S  倖存  \n",
       "4   NaN        S  死亡  \n",
       "5   NaN        Q  死亡  "
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['存活'] = df.Survived.apply(lambda x: '倖存' if x else '死亡')\n",
    "df.loc[:5, 'Survived':'存活']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這個不到 1,000 筆的 DataFrame 做這樣的簡單運算不用一秒鐘，但實務上你可能常常需要對幾十萬、幾百萬筆數據分別做複雜的運算，這時候了解執行進度就是一件非常重要的事情。首先透過 [Anaconda](https://www.anaconda.com/) 安裝 tqdm：\n",
    "\n",
    "```bash\n",
    "conda install -c conda-forge tqdm \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你是在 Juypter 筆記本裡執行 pandas，可以用以下的方式將 tqdm 進度條整合到 `apply` 函式：\n",
    "\n",
    "```python\n",
    "from tqdm import tqdm_notebook\n",
    "tqdm_notebook().pandas()\n",
    "clear_output()\n",
    "\n",
    "# 只需將 `apply` 替換成 `progress_apply`\n",
    "df['存活'] = df.Survived.progress_apply(lambda x: '倖存' if x else '死亡')\n",
    "df.loc[:5, 'Survived':'存活']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "from tqdm import tqdm_notebook\n",
    "tqdm_notebook().pandas()\n",
    "df['存活'] = df.Survived.progress_apply(lambda x: '倖存' if x else '死亡')\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/tqdm_progress_apply.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "透過使用 `progress_apply` 函式，我們可以得到跟使用 `apply` 函式一樣的結果，附贈進度條。相信我，在你 `apply` 的函式很複雜且樣本數很大時，你會很感謝有進度條的存在。你可以查看 [tqdm repo](https://github.com/tqdm/tqdm) 了解更多使用案例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/tqdm-jupyter-1.gif"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [swifter](https://github.com/jmcarpenter2/swifter)：加速你的數據處理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[swifter](https://github.com/jmcarpenter2/swifter) 函式庫能以最有效率的方式執行 `apply` 函式。同樣可以使用 Anaconda 安裝：\n",
    "\n",
    "```bash\n",
    "conda install -c conda-forge swifter\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接著讓我建立一個有 100 萬筆樣本的 DataFrame ，測試 swifter 與原版 `apply` 函式的效能差異："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "240 ms ± 5.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "5 ms ± 565 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "import swifter\n",
    "df = pd.DataFrame(pd.np.random.rand(1000000, 1), columns=['x'])\n",
    "\n",
    "%timeit -n 10 df['x2'] = df['x'].apply(lambda x: x**2)\n",
    "%timeit -n 10 df['x2'] = df['x'].swifter.apply(lambda x: x**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這個簡單的平方運算中，swifter 版的 `apply` 函式在我的 mac 上的效率是原始 `apply` 函式的 48 倍。而要使用 swifter 你也只需要加上 `swifter` 關鍵字即可，何樂而不為？\n",
    "\n",
    "喔別擔心，剛剛的 tqdm 進度條仍然會繼續顯示：\n",
    "\n",
    "```python\n",
    "df = pd.DataFrame(pd.np.random.rand(100000, 1), columns=['x'])\n",
    "df['x2'] = df['x'].swifter.apply(lambda x: x**2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/swifter_with_tqdm.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "魚與熊掌兩者皆得。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [qgrid](https://github.com/quantopian/qgrid)：即時排序、篩選及編輯你的 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[qgrid](https://github.com/quantopian/qgrid) 是一個能讓你在 Jupyter 筆記本裡使用簡單 UI 瀏覽並操作 DataFrame 的 widget。一樣透過 Anaconda 安裝：\n",
    "\n",
    "```bash\n",
    "conda install -c tim_shawver qgrid\n",
    "jupyter nbextension enable --py --sys-prefix widgetsnbextension\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "實際的使用方式也非常簡單：\n",
    "\n",
    "```python\n",
    "import qgrid\n",
    "qgrid.set_grid_option('maxVisibleRows', 7)\n",
    "q = qgrid.show_grid(df_titanic)\n",
    "q\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/qgrid_demo.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以簡單瀏覽 DataFrame 內容，也能依照欄位值篩選及排序樣本，甚至編輯裡頭的值。\n",
    "\n",
    "如果你想要取得當前狀態的 DataFrame，也能使用 qgrid 裡頭的 `get_changed_df` 函式：\n",
    "\n",
    "```python\n",
    "# 取得更改後的 DataFrame\n",
    "df_changed = q.get_changed_df()\n",
    "\n",
    "# styling\n",
    "(df_changed.iloc[:8, :]\n",
    " .style\n",
    " .applymap(lambda x: 'color:red',\n",
    "           subset=pd.IndexSlice[829, 'SibSp'])\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- dark\n",
    "- pandas/qgrid_get_changed_df.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意我剛剛在 qgrid 裡頭將 `0` 改成 `2`，而這個變動被反映到新的 DataFrame `df_changed` 裡頭。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [pandas-profiling](https://github.com/pandas-profiling/pandas-profiling)：你的一鍵 EDA 神器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "廢話不多說，先附上安裝指令：\n",
    "\n",
    "```bash\n",
    "conda install -c conda-forge pandas-profiling\n",
    "```\n",
    "\n",
    "當碰到新的資料集時，你常會需要做 EDA 來探索並了解手上的數據。常見的基礎步驟有：\n",
    "- 隨機抽樣 DataFrame 並用肉眼觀察樣本\n",
    "- 了解樣本數有多少、各類型的變數有多少\n",
    "- 了解 DataFrame 裡有多少空值\n",
    "- 看數值欄位的分佈，欄位之間的相關係數\n",
    "- 看分類型欄位的有多少不同的值，分佈為何\n",
    "\n",
    "等等。這些在你熟悉本文的 pandas 技巧以後都應該跟吃飯喝水一樣，但很多時候就是例行公事。\n",
    "\n",
    "這時你可以使用 [pandas-profiling](https://github.com/pandas-profiling/pandas-profiling) 讓它自動產生簡單報表，總結 DataFrame 裡的數據性質：\n",
    "\n",
    "```python\n",
    "import pandas_profiling\n",
    "df = df_titanic.copy()\n",
    "\n",
    "# 一行報表：將想觀察的 DataFrame 丟進去就完工了\n",
    "pandas_profiling.ProfileReport(df)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!mp4\n",
    "- dark\n",
    "- images/pandas/pandas_profiling.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了自動生成的數據分析報表，你不再需要自己寫繁瑣的程式碼就能「健檢」手上的數據集。這讓你能對數據更快地展開 domain-specific 的研究與分析，大大提升效率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 結語：實際運用所學"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "呼！本文的 pandas 旅程到此告一段落啦！\n",
    "\n",
    "我想在其他地方你應該是找不到跟本文一樣囉哩八唆的 pandas 教學文章了。本文雖長，但涵蓋的都是我認為十分實用的 pandas 使用手法，希望你有從中學到些東西，並開始自己的數據處理與分析之旅。如果你之後想要複習，可以點開頁面左側的章節傳送門，測試自己還記得多少技巧。\n",
    "\n",
    "接下來最重要的是培養你自己的 pandas **肌肉記憶**：重複應用你在本文學到的東西，分析自己感興趣的任何數據並內化這些知識。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!image\n",
    "- pandas/danial-ricaros-FCHlYvR5gJI-unsplash.jpg\n",
    "- Now it's your show time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas 是一個非常強大的資料處理工具，本文雖長，仍無法涵蓋所有東西。不過你現在已經學會作為一個資料科學家所需要的所有基礎 pandas 手法。除了多加練習找手感以外，我能給你的最後一個建議就是在 Jupyter 筆記本裡頭多加利用 `Shift` + `tab` 查看 pandas 函式的 documentation，很多時候你會有新的發現。\n",
    "\n",
    "如果你有任何其他 pandas 技巧，也請不吝留言與我分享！我懂的技巧不多，而現在輪到你教我了：）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ignore\n",
    "- todo:\n",
    "\n",
    "    - 結語\n",
    "        - 建議：多看 func 的 documentation，會有意想不到的收穫\n",
    "        馬上將這張的東西應用到你目前的數據及。\n",
    "            \n",
    "- output\n",
    "    - windows friendly output: `to_csv(encoding=\"cp932\")`\n",
    "- powerful tools\n",
    "    - Facets\n",
    "        - https://www.evernote.com/l/AETaGMqtguRAKbEF9z4hWtF9HazEVkWr70c\n",
    "    - cufflinks and plotly\n",
    "        - https://www.kdnuggets.com/2019/07/10-simple-hacks-speed-data-analysis-python.html\n",
    "\n",
    "- post-release\n",
    "    - 修改 chartify 連結，連回 pandsa 文章\n",
    "    - 檢查 pandas 文內 chartify 以及 ggplot 的連結對不對\n",
    "\n",
    "- 回訊息 maya"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}