talk_2017_08_RMACC_GotPandas/nb/2_importing_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** NAVIGATION **\n",
    "\n",
    "**Got Pandas? _Practical Data Wrangling with Pandas_**\n",
    "\n",
    "* [Introduction](./0_introduction.ipynb)\n",
    "1. [Data Structures](./1_data_structures.ipynb)\n",
    "2. **Importing Data**\n",
    "3. [Manipulating DataFrames](./3_dataframe_operations.ipynb)\n",
    "4. [Wrap Up](4_wrapping_up.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev1 toc-item\"><a href=\"#Importing-Data-in-Pandas\" data-toc-modified-id=\"Importing-Data-in-Pandas-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Importing Data in Pandas</a></div><div class=\"lev2 toc-item\"><a href=\"#Importing-Pandas\" data-toc-modified-id=\"Importing-Pandas-11\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Importing Pandas</a></div><div class=\"lev2 toc-item\"><a href=\"#Loading-CSV-and-Excel\" data-toc-modified-id=\"Loading-CSV-and-Excel-12\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Loading CSV and Excel</a></div><div class=\"lev3 toc-item\"><a href=\"#CSV\" data-toc-modified-id=\"CSV-121\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>CSV</a></div><div class=\"lev3 toc-item\"><a href=\"#Accessing-column-data-by-label\" data-toc-modified-id=\"Accessing-column-data-by-label-122\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>Accessing column data by label</a></div><div class=\"lev2 toc-item\"><a href=\"#Excel\" data-toc-modified-id=\"Excel-13\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Excel</a></div><div class=\"lev2 toc-item\"><a href=\"#JSON\" data-toc-modified-id=\"JSON-14\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>JSON</a></div><div class=\"lev2 toc-item\"><a href=\"#SQL\" data-toc-modified-id=\"SQL-15\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>SQL</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Data in Pandas\n",
    "\n",
    "Pandas supports a number of data formats  out of the box including:\n",
    "\n",
    "* CSV, Excel\n",
    "* JSON\n",
    "* HDF5\n",
    "* SQL databases\n",
    "* and others\n",
    "\n",
    "The major benefit for using Pandas to load these data is that it provides a simple, consistent mechanism for each of them and loads them directly into the Pandas DataFrame in a single operation, reducing the need to go elsewhere to perform the same operations with more code or overhead.\n",
    "\n",
    "Pandas I/O supports loading these data formats directly from local storage or using a URL containing such data.  The convenience being that the resource string used can be either a  local/network file string or a URL.\n",
    "\n",
    "**NOTEBOOK OBJECTIVES**\n",
    "\n",
    "In this notebook we'll:\n",
    "\n",
    "* load a local and remote csv file, \n",
    "* load Excel datafile,\n",
    "* load JSON data,\n",
    "* load data via SQL queries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Pandas\n",
    "\n",
    "You will most often load the Pandas library with the following line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading CSV and Excel\n",
    "\n",
    "### CSV\n",
    "\n",
    "CSV files are still a staple in data file formats.  They're portable, flexible, flat, usually easy to parse and ubiquitous.  We will begin by showing how to use Pandas to load CSV directly into a DataFrame.\n",
    "\n",
    "**DATA SOURCE**\n",
    "\n",
    "US Baseball Statistics Archive by Sean Lahman (CCBY-SA 3.0):\n",
    "\n",
    "* [http://seanlahman.com/baseball-archive/statistics/](http://seanlahman.com/baseball-archive/statistics/)\n",
    "* [https://github.com/chadwickbureau/baseballdatabank](https://github.com/chadwickbureau/baseballdatabank)\n",
    "\n",
    "We have put the dataset for [batting data](./datasets/Batting.csv) into our local `datasets` folder.\n",
    "\n",
    "Loading this into a Pandas DataFrame will require us to use the [`read_csv`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html#pandas.read_csv) function, which will attempt to load the CSV data directly into the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"./datasets/Batting.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we inspect this DataFrame, will get exactly what we expect -- each line corresponding to the row in file.  __NOTE__: where there are missing values, Pandas will automatically fill the data with `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerID</th>\n",
       "      <th>yearID</th>\n",
       "      <th>stint</th>\n",
       "      <th>teamID</th>\n",
       "      <th>lgID</th>\n",
       "      <th>G</th>\n",
       "      <th>AB</th>\n",
       "      <th>R</th>\n",
       "      <th>H</th>\n",
       "      <th>2B</th>\n",
       "      <th>...</th>\n",
       "      <th>RBI</th>\n",
       "      <th>SB</th>\n",
       "      <th>CS</th>\n",
       "      <th>BB</th>\n",
       "      <th>SO</th>\n",
       "      <th>IBB</th>\n",
       "      <th>HBP</th>\n",
       "      <th>SH</th>\n",
       "      <th>SF</th>\n",
       "      <th>GIDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>abercda01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>addybo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>118</td>\n",
       "      <td>30</td>\n",
       "      <td>32</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>13.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>allisar01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>137</td>\n",
       "      <td>28</td>\n",
       "      <td>40</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>allisdo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>133</td>\n",
       "      <td>28</td>\n",
       "      <td>44</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ansonca01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>120</td>\n",
       "      <td>29</td>\n",
       "      <td>39</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>armstbo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>49</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>barkeal01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>barnero01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31</td>\n",
       "      <td>157</td>\n",
       "      <td>66</td>\n",
       "      <td>63</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>13</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>barrebi01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>barrofr01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18</td>\n",
       "      <td>86</td>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>bassjo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>89</td>\n",
       "      <td>18</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>battijo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>bealsto01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>36</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>beaveed01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>bechtge01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>PH1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>94</td>\n",
       "      <td>24</td>\n",
       "      <td>33</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>bellast01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>128</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>berkena01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>PH1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>berryto01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>PH1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>berthha01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>73</td>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>biermch01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>birdge01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>106</td>\n",
       "      <td>19</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>birdsda01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>152</td>\n",
       "      <td>51</td>\n",
       "      <td>46</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>brainas01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30</td>\n",
       "      <td>134</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>brannmi01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CH1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>burrohe01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>careyto01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>87</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>carleji01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>127</td>\n",
       "      <td>31</td>\n",
       "      <td>32</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>conefr01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>77</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>connone01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>33</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>cravebi01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>118</td>\n",
       "      <td>26</td>\n",
       "      <td>38</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102786</th>\n",
       "      <td>wittgni01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>MIA</td>\n",
       "      <td>NL</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102787</th>\n",
       "      <td>wolteto01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>COL</td>\n",
       "      <td>NL</td>\n",
       "      <td>71</td>\n",
       "      <td>205</td>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>21</td>\n",
       "      <td>53.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102788</th>\n",
       "      <td>wongko01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>SLN</td>\n",
       "      <td>NL</td>\n",
       "      <td>121</td>\n",
       "      <td>313</td>\n",
       "      <td>39</td>\n",
       "      <td>75</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102789</th>\n",
       "      <td>woodal02</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>LAN</td>\n",
       "      <td>NL</td>\n",
       "      <td>15</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102790</th>\n",
       "      <td>woodbl01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CIN</td>\n",
       "      <td>NL</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102791</th>\n",
       "      <td>woodtr01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CHN</td>\n",
       "      <td>NL</td>\n",
       "      <td>81</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102792</th>\n",
       "      <td>worleva01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>BAL</td>\n",
       "      <td>AL</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102793</th>\n",
       "      <td>worthda01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>HOU</td>\n",
       "      <td>AL</td>\n",
       "      <td>16</td>\n",
       "      <td>39</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102794</th>\n",
       "      <td>wrighda03</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>NYN</td>\n",
       "      <td>NL</td>\n",
       "      <td>37</td>\n",
       "      <td>137</td>\n",
       "      <td>18</td>\n",
       "      <td>31</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>26</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102795</th>\n",
       "      <td>wrighda04</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CIN</td>\n",
       "      <td>NL</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102796</th>\n",
       "      <td>wrighda04</td>\n",
       "      <td>2016</td>\n",
       "      <td>2</td>\n",
       "      <td>LAA</td>\n",
       "      <td>AL</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102797</th>\n",
       "      <td>wrighmi01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>BAL</td>\n",
       "      <td>AL</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102798</th>\n",
       "      <td>wrighst01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>BOS</td>\n",
       "      <td>AL</td>\n",
       "      <td>25</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102799</th>\n",
       "      <td>yateski01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>NYA</td>\n",
       "      <td>AL</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102800</th>\n",
       "      <td>yelicch01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>MIA</td>\n",
       "      <td>NL</td>\n",
       "      <td>155</td>\n",
       "      <td>578</td>\n",
       "      <td>78</td>\n",
       "      <td>172</td>\n",
       "      <td>38</td>\n",
       "      <td>...</td>\n",
       "      <td>98.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>72</td>\n",
       "      <td>138.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102801</th>\n",
       "      <td>ynoaga01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>NYN</td>\n",
       "      <td>NL</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102802</th>\n",
       "      <td>ynoami01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CHA</td>\n",
       "      <td>AL</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102803</th>\n",
       "      <td>ynoara01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>COL</td>\n",
       "      <td>NL</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102804</th>\n",
       "      <td>youngch03</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>KCA</td>\n",
       "      <td>AL</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102805</th>\n",
       "      <td>youngch04</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>BOS</td>\n",
       "      <td>AL</td>\n",
       "      <td>76</td>\n",
       "      <td>203</td>\n",
       "      <td>29</td>\n",
       "      <td>56</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>21</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102806</th>\n",
       "      <td>younger03</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>NYA</td>\n",
       "      <td>AL</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102807</th>\n",
       "      <td>youngma03</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>ATL</td>\n",
       "      <td>NL</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102808</th>\n",
       "      <td>zastrro01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CHN</td>\n",
       "      <td>NL</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102809</th>\n",
       "      <td>zieglbr01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>ARI</td>\n",
       "      <td>NL</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102810</th>\n",
       "      <td>zieglbr01</td>\n",
       "      <td>2016</td>\n",
       "      <td>2</td>\n",
       "      <td>BOS</td>\n",
       "      <td>AL</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102811</th>\n",
       "      <td>zimmejo02</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>DET</td>\n",
       "      <td>AL</td>\n",
       "      <td>19</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102812</th>\n",
       "      <td>zimmery01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>115</td>\n",
       "      <td>427</td>\n",
       "      <td>60</td>\n",
       "      <td>93</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>46.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>29</td>\n",
       "      <td>104.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102813</th>\n",
       "      <td>zobribe01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>CHN</td>\n",
       "      <td>NL</td>\n",
       "      <td>147</td>\n",
       "      <td>523</td>\n",
       "      <td>94</td>\n",
       "      <td>142</td>\n",
       "      <td>31</td>\n",
       "      <td>...</td>\n",
       "      <td>76.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>96</td>\n",
       "      <td>82.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102814</th>\n",
       "      <td>zuninmi01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>SEA</td>\n",
       "      <td>AL</td>\n",
       "      <td>55</td>\n",
       "      <td>164</td>\n",
       "      <td>16</td>\n",
       "      <td>34</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102815</th>\n",
       "      <td>zychto01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>SEA</td>\n",
       "      <td>AL</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>102816 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         playerID  yearID  stint teamID lgID    G   AB   R    H  2B  ...   \\\n",
       "0       abercda01    1871      1    TRO  NaN    1    4   0    0   0  ...    \n",
       "1        addybo01    1871      1    RC1  NaN   25  118  30   32   6  ...    \n",
       "2       allisar01    1871      1    CL1  NaN   29  137  28   40   4  ...    \n",
       "3       allisdo01    1871      1    WS3  NaN   27  133  28   44  10  ...    \n",
       "4       ansonca01    1871      1    RC1  NaN   25  120  29   39  11  ...    \n",
       "5       armstbo01    1871      1    FW1  NaN   12   49   9   11   2  ...    \n",
       "6       barkeal01    1871      1    RC1  NaN    1    4   0    1   0  ...    \n",
       "7       barnero01    1871      1    BS1  NaN   31  157  66   63  10  ...    \n",
       "8       barrebi01    1871      1    FW1  NaN    1    5   1    1   1  ...    \n",
       "9       barrofr01    1871      1    BS1  NaN   18   86  13   13   2  ...    \n",
       "10       bassjo01    1871      1    CL1  NaN   22   89  18   27   1  ...    \n",
       "11      battijo01    1871      1    CL1  NaN    1    3   0    0   0  ...    \n",
       "12      bealsto01    1871      1    WS3  NaN   10   36   6    7   0  ...    \n",
       "13      beaveed01    1871      1    TRO  NaN    3   15   7    6   0  ...    \n",
       "14      bechtge01    1871      1    PH1  NaN   20   94  24   33   9  ...    \n",
       "15      bellast01    1871      1    TRO  NaN   29  128  26   32   3  ...    \n",
       "16      berkena01    1871      1    PH1  NaN    1    4   0    0   0  ...    \n",
       "17      berryto01    1871      1    PH1  NaN    1    4   0    1   0  ...    \n",
       "18      berthha01    1871      1    WS3  NaN   17   73  17   17   1  ...    \n",
       "19      biermch01    1871      1    FW1  NaN    1    2   0    0   0  ...    \n",
       "20       birdge01    1871      1    RC1  NaN   25  106  19   28   2  ...    \n",
       "21      birdsda01    1871      1    BS1  NaN   29  152  51   46   3  ...    \n",
       "22      brainas01    1871      1    WS3  NaN   30  134  24   30   4  ...    \n",
       "23      brannmi01    1871      1    CH1  NaN    3   14   2    1   0  ...    \n",
       "24      burrohe01    1871      1    WS3  NaN   12   63  11   15   2  ...    \n",
       "25      careyto01    1871      1    FW1  NaN   19   87  16   20   2  ...    \n",
       "26      carleji01    1871      1    CL1  NaN   29  127  31   32   8  ...    \n",
       "27       conefr01    1871      1    BS1  NaN   19   77  17   20   3  ...    \n",
       "28      connone01    1871      1    TRO  NaN    7   33   6    7   0  ...    \n",
       "29      cravebi01    1871      1    TRO  NaN   27  118  26   38   8  ...    \n",
       "...           ...     ...    ...    ...  ...  ...  ...  ..  ...  ..  ...    \n",
       "102786  wittgni01    2016      1    MIA   NL   48    0   0    0   0  ...    \n",
       "102787  wolteto01    2016      1    COL   NL   71  205  27   53  15  ...    \n",
       "102788   wongko01    2016      1    SLN   NL  121  313  39   75   7  ...    \n",
       "102789   woodal02    2016      1    LAN   NL   15   16   2    4   0  ...    \n",
       "102790   woodbl01    2016      1    CIN   NL   70    2   0    0   0  ...    \n",
       "102791   woodtr01    2016      1    CHN   NL   81   11   0    2   0  ...    \n",
       "102792  worleva01    2016      1    BAL   AL   35    0   0    0   0  ...    \n",
       "102793  worthda01    2016      1    HOU   AL   16   39   4    7   2  ...    \n",
       "102794  wrighda03    2016      1    NYN   NL   37  137  18   31   8  ...    \n",
       "102795  wrighda04    2016      1    CIN   NL    4    5   0    0   0  ...    \n",
       "102796  wrighda04    2016      2    LAA   AL    5    0   0    0   0  ...    \n",
       "102797  wrighmi01    2016      1    BAL   AL   18    0   0    0   0  ...    \n",
       "102798  wrighst01    2016      1    BOS   AL   25    4   0    0   0  ...    \n",
       "102799  yateski01    2016      1    NYA   AL   41    0   0    0   0  ...    \n",
       "102800  yelicch01    2016      1    MIA   NL  155  578  78  172  38  ...    \n",
       "102801   ynoaga01    2016      1    NYN   NL   10    3   0    0   0  ...    \n",
       "102802   ynoami01    2016      1    CHA   AL   23    0   0    0   0  ...    \n",
       "102803   ynoara01    2016      1    COL   NL    3    5   0    0   0  ...    \n",
       "102804  youngch03    2016      1    KCA   AL   34    1   0    0   0  ...    \n",
       "102805  youngch04    2016      1    BOS   AL   76  203  29   56  18  ...    \n",
       "102806  younger03    2016      1    NYA   AL    6    1   2    0   0  ...    \n",
       "102807  youngma03    2016      1    ATL   NL    8    0   0    0   0  ...    \n",
       "102808  zastrro01    2016      1    CHN   NL    8    3   0    0   0  ...    \n",
       "102809  zieglbr01    2016      1    ARI   NL   36    0   0    0   0  ...    \n",
       "102810  zieglbr01    2016      2    BOS   AL   33    0   0    0   0  ...    \n",
       "102811  zimmejo02    2016      1    DET   AL   19    4   0    1   0  ...    \n",
       "102812  zimmery01    2016      1    WAS   NL  115  427  60   93  18  ...    \n",
       "102813  zobribe01    2016      1    CHN   NL  147  523  94  142  31  ...    \n",
       "102814  zuninmi01    2016      1    SEA   AL   55  164  16   34   7  ...    \n",
       "102815   zychto01    2016      1    SEA   AL   12    0   0    0   0  ...    \n",
       "\n",
       "         RBI    SB   CS  BB     SO  IBB  HBP   SH   SF  GIDP  \n",
       "0        0.0   0.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "1       13.0   8.0  1.0   4    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "2       19.0   3.0  1.0   2    5.0  NaN  NaN  NaN  NaN   NaN  \n",
       "3       27.0   1.0  1.0   0    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "4       16.0   6.0  2.0   2    1.0  NaN  NaN  NaN  NaN   NaN  \n",
       "5        5.0   0.0  1.0   0    1.0  NaN  NaN  NaN  NaN   NaN  \n",
       "6        2.0   0.0  0.0   1    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "7       34.0  11.0  6.0  13    1.0  NaN  NaN  NaN  NaN   NaN  \n",
       "8        1.0   0.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "9       11.0   1.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "10      18.0   0.0  1.0   3    4.0  NaN  NaN  NaN  NaN   NaN  \n",
       "11       0.0   0.0  0.0   1    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "12       1.0   2.0  0.0   2    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "13       5.0   2.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "14      21.0   4.0  0.0   2    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "15      23.0   4.0  4.0   9    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "16       0.0   0.0  0.0   0    3.0  NaN  NaN  NaN  NaN   NaN  \n",
       "17       0.0   0.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "18       8.0   3.0  1.0   4    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "19       0.0   0.0  0.0   1    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "20      13.0   1.0  0.0   3    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "21      24.0   6.0  0.0   4    4.0  NaN  NaN  NaN  NaN   NaN  \n",
       "22      21.0   4.0  0.0   7    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "23       0.0   0.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "24      14.0   0.0  0.0   1    1.0  NaN  NaN  NaN  NaN   NaN  \n",
       "25      10.0   5.0  0.0   2    1.0  NaN  NaN  NaN  NaN   NaN  \n",
       "26      18.0   2.0  1.0   8    3.0  NaN  NaN  NaN  NaN   NaN  \n",
       "27      16.0  12.0  1.0   8    2.0  NaN  NaN  NaN  NaN   NaN  \n",
       "28       2.0   0.0  0.0   0    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "29      26.0   6.0  3.0   3    0.0  NaN  NaN  NaN  NaN   NaN  \n",
       "...      ...   ...  ...  ..    ...  ...  ...  ...  ...   ...  \n",
       "102786   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102787  30.0   4.0  1.0  21   53.0  2.0  0.0  4.0  0.0   1.0  \n",
       "102788  23.0   7.0  0.0  34   52.0  2.0  9.0  0.0  5.0   3.0  \n",
       "102789   2.0   0.0  0.0   1    9.0  0.0  0.0  2.0  0.0   0.0  \n",
       "102790   0.0   0.0  0.0   0    2.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102791   1.0   0.0  0.0   1    5.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102792   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102793   1.0   0.0  0.0   1    6.0  0.0  0.0  0.0  0.0   1.0  \n",
       "102794  14.0   3.0  2.0  26   55.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102795   0.0   0.0  0.0   0    2.0  0.0  0.0  1.0  0.0   0.0  \n",
       "102796   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102797   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102798   0.0   0.0  0.0   0    3.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102799   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102800  98.0   9.0  4.0  72  138.0  4.0  4.0  0.0  5.0  20.0  \n",
       "102801   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102802   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102803   0.0   0.0  0.0   0    2.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102804   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102805  24.0   4.0  2.0  21   50.0  0.0  3.0  0.0  0.0   4.0  \n",
       "102806   0.0   1.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102807   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102808   0.0   0.0  0.0   0    2.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102809   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102810   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "102811   0.0   0.0  0.0   0    2.0  0.0  0.0  1.0  0.0   0.0  \n",
       "102812  46.0   4.0  1.0  29  104.0  1.0  5.0  0.0  6.0  12.0  \n",
       "102813  76.0   6.0  4.0  96   82.0  6.0  4.0  4.0  4.0  17.0  \n",
       "102814  31.0   0.0  0.0  21   65.0  0.0  6.0  0.0  1.0   0.0  \n",
       "102815   0.0   0.0  0.0   0    0.0  0.0  0.0  0.0  0.0   0.0  \n",
       "\n",
       "[102816 rows x 22 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will soon learn that Pandas, supports some typical \"Pythonic\" use cases for accesing data.  The first we will encounter is with `len()`. We can get the size of this dataset (in rows) with the standard Python `len()` function, which will return exactly what we expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "102816"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every DataFrame will have a `columns` attribute, which contains the _column index_ for our dataset.  Thus, getting the length of that attribute returns, again, what we expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['playerID', 'yearID', 'stint', 'teamID', 'lgID', 'G', 'AB', 'R', 'H',\n",
       "       '2B', '3B', 'HR', 'RBI', 'SB', 'CS', 'BB', 'SO', 'IBB', 'HBP', 'SH',\n",
       "       'SF', 'GIDP'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want both column and row counts [`DataFrame.shape`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shape.html#pandas.DataFrame.shape) will return the tuple to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102816, 22)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which returns what we expect (yet again).\n",
    "\n",
    "Much like Python slicing of lists, if we want to the first _n_ rows of data, we can use the shorthand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerID</th>\n",
       "      <th>yearID</th>\n",
       "      <th>stint</th>\n",
       "      <th>teamID</th>\n",
       "      <th>lgID</th>\n",
       "      <th>G</th>\n",
       "      <th>AB</th>\n",
       "      <th>R</th>\n",
       "      <th>H</th>\n",
       "      <th>2B</th>\n",
       "      <th>...</th>\n",
       "      <th>RBI</th>\n",
       "      <th>SB</th>\n",
       "      <th>CS</th>\n",
       "      <th>BB</th>\n",
       "      <th>SO</th>\n",
       "      <th>IBB</th>\n",
       "      <th>HBP</th>\n",
       "      <th>SH</th>\n",
       "      <th>SF</th>\n",
       "      <th>GIDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>abercda01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>addybo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>118</td>\n",
       "      <td>30</td>\n",
       "      <td>32</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>13.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>allisar01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>137</td>\n",
       "      <td>28</td>\n",
       "      <td>40</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>allisdo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>WS3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>133</td>\n",
       "      <td>28</td>\n",
       "      <td>44</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ansonca01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>120</td>\n",
       "      <td>29</td>\n",
       "      <td>39</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>armstbo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>49</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>barkeal01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>barnero01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31</td>\n",
       "      <td>157</td>\n",
       "      <td>66</td>\n",
       "      <td>63</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>13</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>barrebi01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>barrofr01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>BS1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18</td>\n",
       "      <td>86</td>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    playerID  yearID  stint teamID lgID   G   AB   R   H  2B  ...    RBI  \\\n",
       "0  abercda01    1871      1    TRO  NaN   1    4   0   0   0  ...    0.0   \n",
       "1   addybo01    1871      1    RC1  NaN  25  118  30  32   6  ...   13.0   \n",
       "2  allisar01    1871      1    CL1  NaN  29  137  28  40   4  ...   19.0   \n",
       "3  allisdo01    1871      1    WS3  NaN  27  133  28  44  10  ...   27.0   \n",
       "4  ansonca01    1871      1    RC1  NaN  25  120  29  39  11  ...   16.0   \n",
       "5  armstbo01    1871      1    FW1  NaN  12   49   9  11   2  ...    5.0   \n",
       "6  barkeal01    1871      1    RC1  NaN   1    4   0   1   0  ...    2.0   \n",
       "7  barnero01    1871      1    BS1  NaN  31  157  66  63  10  ...   34.0   \n",
       "8  barrebi01    1871      1    FW1  NaN   1    5   1   1   1  ...    1.0   \n",
       "9  barrofr01    1871      1    BS1  NaN  18   86  13  13   2  ...   11.0   \n",
       "\n",
       "     SB   CS  BB   SO  IBB  HBP  SH  SF  GIDP  \n",
       "0   0.0  0.0   0  0.0  NaN  NaN NaN NaN   NaN  \n",
       "1   8.0  1.0   4  0.0  NaN  NaN NaN NaN   NaN  \n",
       "2   3.0  1.0   2  5.0  NaN  NaN NaN NaN   NaN  \n",
       "3   1.0  1.0   0  2.0  NaN  NaN NaN NaN   NaN  \n",
       "4   6.0  2.0   2  1.0  NaN  NaN NaN NaN   NaN  \n",
       "5   0.0  1.0   0  1.0  NaN  NaN NaN NaN   NaN  \n",
       "6   0.0  0.0   1  0.0  NaN  NaN NaN NaN   NaN  \n",
       "7  11.0  6.0  13  1.0  NaN  NaN NaN NaN   NaN  \n",
       "8   0.0  0.0   0  0.0  NaN  NaN NaN NaN   NaN  \n",
       "9   1.0  0.0   0  0.0  NaN  NaN NaN NaN   NaN  \n",
       "\n",
       "[10 rows x 22 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or just like slicing a list, we can do more complex slicing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerID</th>\n",
       "      <th>yearID</th>\n",
       "      <th>stint</th>\n",
       "      <th>teamID</th>\n",
       "      <th>lgID</th>\n",
       "      <th>G</th>\n",
       "      <th>AB</th>\n",
       "      <th>R</th>\n",
       "      <th>H</th>\n",
       "      <th>2B</th>\n",
       "      <th>...</th>\n",
       "      <th>RBI</th>\n",
       "      <th>SB</th>\n",
       "      <th>CS</th>\n",
       "      <th>BB</th>\n",
       "      <th>SO</th>\n",
       "      <th>IBB</th>\n",
       "      <th>HBP</th>\n",
       "      <th>SH</th>\n",
       "      <th>SF</th>\n",
       "      <th>GIDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>abercda01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>armstbo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>49</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>bassjo01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>89</td>\n",
       "      <td>18</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>bellast01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>128</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>birdge01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25</td>\n",
       "      <td>106</td>\n",
       "      <td>19</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>careyto01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>FW1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>87</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>cuthbne01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>PH1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28</td>\n",
       "      <td>150</td>\n",
       "      <td>47</td>\n",
       "      <td>37</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>ewellge01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>CL1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>flowedi01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>TRO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21</td>\n",
       "      <td>105</td>\n",
       "      <td>39</td>\n",
       "      <td>33</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>fulmech01</td>\n",
       "      <td>1871</td>\n",
       "      <td>1</td>\n",
       "      <td>RC1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16</td>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     playerID  yearID  stint teamID lgID   G   AB   R   H  2B  ...    RBI  \\\n",
       "0   abercda01    1871      1    TRO  NaN   1    4   0   0   0  ...    0.0   \n",
       "5   armstbo01    1871      1    FW1  NaN  12   49   9  11   2  ...    5.0   \n",
       "10   bassjo01    1871      1    CL1  NaN  22   89  18  27   1  ...   18.0   \n",
       "15  bellast01    1871      1    TRO  NaN  29  128  26  32   3  ...   23.0   \n",
       "20   birdge01    1871      1    RC1  NaN  25  106  19  28   2  ...   13.0   \n",
       "25  careyto01    1871      1    FW1  NaN  19   87  16  20   2  ...   10.0   \n",
       "30  cuthbne01    1871      1    PH1  NaN  28  150  47  37   7  ...   30.0   \n",
       "35  ewellge01    1871      1    CL1  NaN   1    3   0   0   0  ...    0.0   \n",
       "40  flowedi01    1871      1    TRO  NaN  21  105  39  33   5  ...   18.0   \n",
       "45  fulmech01    1871      1    RC1  NaN  16   63  11  17   1  ...    3.0   \n",
       "\n",
       "      SB   CS  BB   SO  IBB  HBP  SH  SF  GIDP  \n",
       "0    0.0  0.0   0  0.0  NaN  NaN NaN NaN   NaN  \n",
       "5    0.0  1.0   0  1.0  NaN  NaN NaN NaN   NaN  \n",
       "10   0.0  1.0   3  4.0  NaN  NaN NaN NaN   NaN  \n",
       "15   4.0  4.0   9  2.0  NaN  NaN NaN NaN   NaN  \n",
       "20   1.0  0.0   3  2.0  NaN  NaN NaN NaN   NaN  \n",
       "25   5.0  0.0   2  1.0  NaN  NaN NaN NaN   NaN  \n",
       "30  16.0  2.0  10  2.0  NaN  NaN NaN NaN   NaN  \n",
       "35   0.0  0.0   0  0.0  NaN  NaN NaN NaN   NaN  \n",
       "40   8.0  2.0   4  0.0  NaN  NaN NaN NaN   NaN  \n",
       "45   0.0  0.0   5  1.0  NaN  NaN NaN NaN   NaN  \n",
       "\n",
       "[10 rows x 22 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:50:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing column data by label\n",
    "\n",
    "One of the nice things about Pandas is that we can reference the columns of data by their names (or labels).  For example, we have a `yearID` label, `teamID` label, `G` label for game counts, and so on.  For our dataset to learn what the labels are in detail see the documentation for the provided links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1871\n",
       "1    1871\n",
       "2    1871\n",
       "3    1871\n",
       "4    1871\n",
       "5    1871\n",
       "6    1871\n",
       "7    1871\n",
       "8    1871\n",
       "9    1871\n",
       "Name: yearID, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.yearID[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "102806      6\n",
       "102807      8\n",
       "102808      8\n",
       "102809     36\n",
       "102810     33\n",
       "102811     19\n",
       "102812    115\n",
       "102813    147\n",
       "102814     55\n",
       "102815     12\n",
       "Name: G, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.G[-10:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we want all the player data for the [Washington Nationals](https://www.mlb.com/nationals) from 2015 and 2016 where a player played in 100 or more games:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerID</th>\n",
       "      <th>yearID</th>\n",
       "      <th>stint</th>\n",
       "      <th>teamID</th>\n",
       "      <th>lgID</th>\n",
       "      <th>G</th>\n",
       "      <th>AB</th>\n",
       "      <th>R</th>\n",
       "      <th>H</th>\n",
       "      <th>2B</th>\n",
       "      <th>...</th>\n",
       "      <th>RBI</th>\n",
       "      <th>SB</th>\n",
       "      <th>CS</th>\n",
       "      <th>BB</th>\n",
       "      <th>SO</th>\n",
       "      <th>IBB</th>\n",
       "      <th>HBP</th>\n",
       "      <th>SH</th>\n",
       "      <th>SF</th>\n",
       "      <th>GIDP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100193</th>\n",
       "      <td>desmoia01</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>156</td>\n",
       "      <td>583</td>\n",
       "      <td>69</td>\n",
       "      <td>136</td>\n",
       "      <td>27</td>\n",
       "      <td>...</td>\n",
       "      <td>62.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>45</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100250</th>\n",
       "      <td>escobyu01</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>139</td>\n",
       "      <td>535</td>\n",
       "      <td>75</td>\n",
       "      <td>168</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>56.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>45</td>\n",
       "      <td>70.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100251</th>\n",
       "      <td>espinda01</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>118</td>\n",
       "      <td>367</td>\n",
       "      <td>59</td>\n",
       "      <td>88</td>\n",
       "      <td>21</td>\n",
       "      <td>...</td>\n",
       "      <td>37.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>33</td>\n",
       "      <td>106.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100422</th>\n",
       "      <td>harpebr03</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>153</td>\n",
       "      <td>521</td>\n",
       "      <td>118</td>\n",
       "      <td>172</td>\n",
       "      <td>38</td>\n",
       "      <td>...</td>\n",
       "      <td>99.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>124</td>\n",
       "      <td>131.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100950</th>\n",
       "      <td>ramoswi01</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>128</td>\n",
       "      <td>475</td>\n",
       "      <td>41</td>\n",
       "      <td>109</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>68.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21</td>\n",
       "      <td>101.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100993</th>\n",
       "      <td>robincl01</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>126</td>\n",
       "      <td>309</td>\n",
       "      <td>44</td>\n",
       "      <td>84</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>37</td>\n",
       "      <td>52.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101176</th>\n",
       "      <td>taylomi02</td>\n",
       "      <td>2015</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>138</td>\n",
       "      <td>472</td>\n",
       "      <td>49</td>\n",
       "      <td>108</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>63.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>35</td>\n",
       "      <td>158.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101725</th>\n",
       "      <td>espinda01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>157</td>\n",
       "      <td>516</td>\n",
       "      <td>66</td>\n",
       "      <td>108</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>72.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>54</td>\n",
       "      <td>174.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101895</th>\n",
       "      <td>harpebr03</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>147</td>\n",
       "      <td>506</td>\n",
       "      <td>84</td>\n",
       "      <td>123</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>86.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>108</td>\n",
       "      <td>117.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102245</th>\n",
       "      <td>murphda08</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>142</td>\n",
       "      <td>531</td>\n",
       "      <td>88</td>\n",
       "      <td>184</td>\n",
       "      <td>47</td>\n",
       "      <td>...</td>\n",
       "      <td>104.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>35</td>\n",
       "      <td>57.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102429</th>\n",
       "      <td>ramoswi01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>131</td>\n",
       "      <td>482</td>\n",
       "      <td>58</td>\n",
       "      <td>148</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>35</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102449</th>\n",
       "      <td>rendoan01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>156</td>\n",
       "      <td>567</td>\n",
       "      <td>91</td>\n",
       "      <td>153</td>\n",
       "      <td>38</td>\n",
       "      <td>...</td>\n",
       "      <td>85.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>65</td>\n",
       "      <td>117.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102451</th>\n",
       "      <td>reverbe01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>103</td>\n",
       "      <td>350</td>\n",
       "      <td>44</td>\n",
       "      <td>76</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>18</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102472</th>\n",
       "      <td>robincl01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>104</td>\n",
       "      <td>196</td>\n",
       "      <td>16</td>\n",
       "      <td>46</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102763</th>\n",
       "      <td>werthja01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>143</td>\n",
       "      <td>525</td>\n",
       "      <td>84</td>\n",
       "      <td>128</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>69.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>71</td>\n",
       "      <td>139.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102812</th>\n",
       "      <td>zimmery01</td>\n",
       "      <td>2016</td>\n",
       "      <td>1</td>\n",
       "      <td>WAS</td>\n",
       "      <td>NL</td>\n",
       "      <td>115</td>\n",
       "      <td>427</td>\n",
       "      <td>60</td>\n",
       "      <td>93</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>46.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>29</td>\n",
       "      <td>104.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         playerID  yearID  stint teamID lgID    G   AB    R    H  2B  ...   \\\n",
       "100193  desmoia01    2015      1    WAS   NL  156  583   69  136  27  ...    \n",
       "100250  escobyu01    2015      1    WAS   NL  139  535   75  168  25  ...    \n",
       "100251  espinda01    2015      1    WAS   NL  118  367   59   88  21  ...    \n",
       "100422  harpebr03    2015      1    WAS   NL  153  521  118  172  38  ...    \n",
       "100950  ramoswi01    2015      1    WAS   NL  128  475   41  109  16  ...    \n",
       "100993  robincl01    2015      1    WAS   NL  126  309   44   84  15  ...    \n",
       "101176  taylomi02    2015      1    WAS   NL  138  472   49  108  15  ...    \n",
       "101725  espinda01    2016      1    WAS   NL  157  516   66  108  15  ...    \n",
       "101895  harpebr03    2016      1    WAS   NL  147  506   84  123  24  ...    \n",
       "102245  murphda08    2016      1    WAS   NL  142  531   88  184  47  ...    \n",
       "102429  ramoswi01    2016      1    WAS   NL  131  482   58  148  25  ...    \n",
       "102449  rendoan01    2016      1    WAS   NL  156  567   91  153  38  ...    \n",
       "102451  reverbe01    2016      1    WAS   NL  103  350   44   76   9  ...    \n",
       "102472  robincl01    2016      1    WAS   NL  104  196   16   46   4  ...    \n",
       "102763  werthja01    2016      1    WAS   NL  143  525   84  128  28  ...    \n",
       "102812  zimmery01    2016      1    WAS   NL  115  427   60   93  18  ...    \n",
       "\n",
       "          RBI    SB    CS   BB     SO   IBB   HBP   SH    SF  GIDP  \n",
       "100193   62.0  13.0   5.0   45  187.0   0.0   3.0  6.0   4.0   9.0  \n",
       "100250   56.0   2.0   2.0   45   70.0   0.0   8.0  1.0   2.0  24.0  \n",
       "100251   37.0   5.0   2.0   33  106.0   5.0   6.0  3.0   3.0   6.0  \n",
       "100422   99.0   6.0   4.0  124  131.0  15.0   5.0  0.0   4.0  15.0  \n",
       "100950   68.0   0.0   0.0   21  101.0   2.0   0.0  0.0   8.0  16.0  \n",
       "100993   34.0   0.0   0.0   37   52.0   4.0   5.0  0.0   1.0   6.0  \n",
       "101176   63.0  16.0   3.0   35  158.0   9.0   1.0  1.0   2.0   5.0  \n",
       "101725   72.0   9.0   2.0   54  174.0  12.0  20.0  7.0   4.0   4.0  \n",
       "101895   86.0  21.0  10.0  108  117.0  20.0   3.0  0.0  10.0  11.0  \n",
       "102245  104.0   5.0   3.0   35   57.0  10.0   8.0  0.0   8.0   4.0  \n",
       "102429   80.0   0.0   0.0   35   79.0   2.0   2.0  0.0   4.0  17.0  \n",
       "102449   85.0  12.0   6.0   65  117.0   2.0   7.0  0.0   8.0   5.0  \n",
       "102451   24.0  14.0   5.0   18   34.0   0.0   3.0  2.0   2.0  12.0  \n",
       "102472   26.0   0.0   0.0   20   38.0   0.0   2.0  1.0   5.0   4.0  \n",
       "102763   69.0   5.0   1.0   71  139.0   0.0   4.0  0.0   6.0  17.0  \n",
       "102812   46.0   4.0   1.0   29  104.0   1.0   5.0  0.0   6.0  12.0  \n",
       "\n",
       "[16 rows x 22 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_was = df[(df.yearID > 2014) & (df.teamID=='WAS') & (df.G > 99)]\n",
    "df_was"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll put all these things in motion later, but for now put a thumbnail on this for future reference. __NOTE__: we'll need to access the dataset that crosswalks the `PlayerID` with the actual player name and vitals, but we'll leave that as an exercise for the interested (hint: take a look [in this dataset](https://github.com/chadwickbureau/baseballdatabank/blob/master/core/People.csv))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excel\n",
    "\n",
    "Loading Excel data is nearly as easy as CSV data.  This time we'll use a different data source and show how to access it in a slightly different manner.  Instead of the _local_ file source, we will use a _remote URL_ for the resource. This will show us exactly how easy it is to seamlessly interchange various data resources. \n",
    "\n",
    "**DATA SOURCES**\n",
    "\n",
    "* [US Bureau of Transportation Statistics | Airline Employment Data](https://www.bts.gov/newsroom/may-2017-passenger-airline-employment-data) which includes data for year-over-year percentage change in employment for workers in the passenger airline industry\n",
    "\n",
    "To read data from the data set we will access it by URL and use the [`pandas.read_excel()` method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html#pandas.read_excel) note we're using the `sheetname=None` parameter to read each sheet to be assigned its own key in a dictionary for easy lookup by sheet name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xl = pd.read_excel(\n",
    "    \"https://www.bts.gov/sites/bts.dot.gov/files/docs/newsroom/206581/airline-employment-press-tables-web.xlsx\",\n",
    "    sheetname=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice now, if we want to access the _sheet_ called `Table1` we can easily do this in a Pythonic way much like any other dictionary.  The result is the DataFrame representation of that  _sheet_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Table 1: Yearly Change in Scheduled Passenger Airline Full-time Equivalent Employees* by Airline Group</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "      <th>Unnamed: 4</th>\n",
       "      <th>Unnamed: 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Most recent 13 months - percent change from sa...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Network Airlines</td>\n",
       "      <td>Low-Cost Airlines</td>\n",
       "      <td>Regional Airlines</td>\n",
       "      <td>Other Airlines</td>\n",
       "      <td>All Passenger Airlines **</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>May 2015 - May 2016</td>\n",
       "      <td>2.3</td>\n",
       "      <td>10.7</td>\n",
       "      <td>0.2</td>\n",
       "      <td>9.3</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jun 2015 - Jun 2016</td>\n",
       "      <td>2.3</td>\n",
       "      <td>11</td>\n",
       "      <td>0.9</td>\n",
       "      <td>10.6</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Jul 2015 - Jul 2016</td>\n",
       "      <td>2.4</td>\n",
       "      <td>11.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>11.2</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aug 2015 - Aug 2016</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>3.3</td>\n",
       "      <td>11.9</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Sep 2015 - Sep 2016</td>\n",
       "      <td>2.6</td>\n",
       "      <td>10.6</td>\n",
       "      <td>2.9</td>\n",
       "      <td>13</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Oct 2015 - Oct 2016</td>\n",
       "      <td>2.7</td>\n",
       "      <td>10.3</td>\n",
       "      <td>0.3</td>\n",
       "      <td>12.7</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Nov 2015 - Nov 2016</td>\n",
       "      <td>2.3</td>\n",
       "      <td>9.8</td>\n",
       "      <td>0.2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Dec 2015 - Dec 2016</td>\n",
       "      <td>2.4</td>\n",
       "      <td>9.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Jan 2016 - Jan 2017</td>\n",
       "      <td>2.3</td>\n",
       "      <td>9.7</td>\n",
       "      <td>1.9</td>\n",
       "      <td>12.7</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Feb 2016 - Feb 2017</td>\n",
       "      <td>2.4</td>\n",
       "      <td>9.4</td>\n",
       "      <td>2.4</td>\n",
       "      <td>11.8</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Mar 2016 - Mar 2017</td>\n",
       "      <td>2.7</td>\n",
       "      <td>9.1</td>\n",
       "      <td>2</td>\n",
       "      <td>11.7</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Apr 2016 - Apr 2017</td>\n",
       "      <td>2.6</td>\n",
       "      <td>8.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>10.7</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>May 2016 - May 2017</td>\n",
       "      <td>2.4</td>\n",
       "      <td>8.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>4.2</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Source: Bureau of Transportation Statistics</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>* Full-time Equivalent Employee (FTE) calculat...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>** Includes network, low-cost, regional and ot...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Note: Percent changes based on numbers prior t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Note: See Table 2 for all passenger airlines, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Table 1: Yearly Change in Scheduled Passenger Airline Full-time Equivalent Employees* by Airline Group  \\\n",
       "0   Most recent 13 months - percent change from sa...                                                       \n",
       "1                                                 NaN                                                       \n",
       "2                                 May 2015 - May 2016                                                       \n",
       "3                                 Jun 2015 - Jun 2016                                                       \n",
       "4                                 Jul 2015 - Jul 2016                                                       \n",
       "5                                 Aug 2015 - Aug 2016                                                       \n",
       "6                                 Sep 2015 - Sep 2016                                                       \n",
       "7                                 Oct 2015 - Oct 2016                                                       \n",
       "8                                 Nov 2015 - Nov 2016                                                       \n",
       "9                                 Dec 2015 - Dec 2016                                                       \n",
       "10                                Jan 2016 - Jan 2017                                                       \n",
       "11                                Feb 2016 - Feb 2017                                                       \n",
       "12                                Mar 2016 - Mar 2017                                                       \n",
       "13                                Apr 2016 - Apr 2017                                                       \n",
       "14                                May 2016 - May 2017                                                       \n",
       "15        Source: Bureau of Transportation Statistics                                                       \n",
       "16  * Full-time Equivalent Employee (FTE) calculat...                                                       \n",
       "17  ** Includes network, low-cost, regional and ot...                                                       \n",
       "18  Note: Percent changes based on numbers prior t...                                                       \n",
       "19  Note: See Table 2 for all passenger airlines, ...                                                       \n",
       "\n",
       "          Unnamed: 1         Unnamed: 2         Unnamed: 3      Unnamed: 4  \\\n",
       "0                NaN                NaN                NaN             NaN   \n",
       "1   Network Airlines  Low-Cost Airlines  Regional Airlines  Other Airlines   \n",
       "2                2.3               10.7                0.2             9.3   \n",
       "3                2.3                 11                0.9            10.6   \n",
       "4                2.4               11.3                3.3            11.2   \n",
       "5                2.5                 11                3.3            11.9   \n",
       "6                2.6               10.6                2.9              13   \n",
       "7                2.7               10.3                0.3            12.7   \n",
       "8                2.3                9.8                0.2            13.5   \n",
       "9                2.4                9.5                0.2            13.7   \n",
       "10               2.3                9.7                1.9            12.7   \n",
       "11               2.4                9.4                2.4            11.8   \n",
       "12               2.7                9.1                  2            11.7   \n",
       "13               2.6                8.5                2.1            10.7   \n",
       "14               2.4                8.3                2.5             4.2   \n",
       "15               NaN                NaN                NaN             NaN   \n",
       "16               NaN                NaN                NaN             NaN   \n",
       "17               NaN                NaN                NaN             NaN   \n",
       "18               NaN                NaN                NaN             NaN   \n",
       "19               NaN                NaN                NaN             NaN   \n",
       "\n",
       "                   Unnamed: 5  \n",
       "0                         NaN  \n",
       "1   All Passenger Airlines **  \n",
       "2                         3.7  \n",
       "3                         3.9  \n",
       "4                         4.3  \n",
       "5                         4.3  \n",
       "6                         4.3  \n",
       "7                           4  \n",
       "8                         3.7  \n",
       "9                         3.7  \n",
       "10                        3.9  \n",
       "11                        3.9  \n",
       "12                          4  \n",
       "13                        3.9  \n",
       "14                        3.6  \n",
       "15                        NaN  \n",
       "16                        NaN  \n",
       "17                        NaN  \n",
       "18                        NaN  \n",
       "19                        NaN  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xl_tbl1 = xl['Table1']\n",
    "xl_tbl1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One problem we have here is that the data is not exactly as clean as we want it to be.  We'll spend more time talking about the `iloc`() method in the next section, but for now, let's get a flavor for how we might clean this up so it is more usable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Table 1: Yearly Change in Scheduled Passenger Airline Full-time Equivalent Employees* by Airline Group\n",
      "2                                 May 2015 - May 2016                                                    \n",
      "3                                 Jun 2015 - Jun 2016                                                    \n",
      "4                                 Jul 2015 - Jul 2016                                                    \n",
      "5                                 Aug 2015 - Aug 2016                                                    \n",
      "6                                 Sep 2015 - Sep 2016                                                    \n",
      "7                                 Oct 2015 - Oct 2016                                                    \n",
      "8                                 Nov 2015 - Nov 2016                                                    \n",
      "9                                 Dec 2015 - Dec 2016                                                    \n",
      "10                                Jan 2016 - Jan 2017                                                    \n",
      "11                                Feb 2016 - Feb 2017                                                    \n",
      "12                                Mar 2016 - Mar 2017                                                    \n",
      "13                                Apr 2016 - Apr 2017                                                    \n",
      "14                                May 2016 - May 2017                                                    \n",
      "Unnamed: 1             Network Airlines\n",
      "Unnamed: 2            Low-Cost Airlines\n",
      "Unnamed: 3            Regional Airlines\n",
      "Unnamed: 4               Other Airlines\n",
      "Unnamed: 5    All Passenger Airlines **\n",
      "Name: 1, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# lets select the (row) index \n",
    "idx = xl_tbl1.iloc[2:15, 0:1]\n",
    "\n",
    "# lets select the (col) index\n",
    "col = xl_tbl1.iloc[1,1:]\n",
    "\n",
    "print(idx)\n",
    "print(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['May 2015 - May 2016', 'Jun 2015 - Jun 2016', 'Jul 2015 - Jul 2016',\n",
       "       'Aug 2015 - Aug 2016', 'Sep 2015 - Sep 2016', 'Oct 2015 - Oct 2016',\n",
       "       'Nov 2015 - Nov 2016', 'Dec 2015 - Dec 2016', 'Jan 2016 - Jan 2017',\n",
       "       'Feb 2016 - Feb 2017', 'Mar 2016 - Mar 2017', 'Apr 2016 - Apr 2017',\n",
       "       'May 2016 - May 2017'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we'll create the index object\n",
    "idxs = pd.Index([v[0] for v in idx.values])\n",
    "idxs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Network Airlines',\n",
       " 'Low-Cost Airlines',\n",
       " 'Regional Airlines',\n",
       " 'Other Airlines',\n",
       " 'All Passenger Airlines **']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set the columns\n",
    "cols = [v for v in col.values]\n",
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.3, 10.7, 0.2, 9.3, 3.7],\n",
       "       [2.3, 11, 0.9, 10.6, 3.9],\n",
       "       [2.4, 11.3, 3.3, 11.2, 4.3],\n",
       "       [2.5, 11, 3.3, 11.9, 4.3],\n",
       "       [2.6, 10.6, 2.9, 13, 4.3],\n",
       "       [2.7, 10.3, 0.3, 12.7, 4],\n",
       "       [2.3, 9.8, 0.2, 13.5, 3.7],\n",
       "       [2.4, 9.5, 0.2, 13.7, 3.7],\n",
       "       [2.3, 9.7, 1.9, 12.7, 3.9],\n",
       "       [2.4, 9.4, 2.4, 11.8, 3.9],\n",
       "       [2.7, 9.1, 2, 11.7, 4],\n",
       "       [2.6, 8.5, 2.1, 10.7, 3.9],\n",
       "       [2.4, 8.3, 2.5, 4.2, 3.6]], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now for the data\n",
    "data = xl_tbl1.iloc[2:15,1:].values\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Network Airlines</th>\n",
       "      <th>Low-Cost Airlines</th>\n",
       "      <th>Regional Airlines</th>\n",
       "      <th>Other Airlines</th>\n",
       "      <th>All Passenger Airlines **</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>May 2015 - May 2016</th>\n",
       "      <td>2.3</td>\n",
       "      <td>10.7</td>\n",
       "      <td>0.2</td>\n",
       "      <td>9.3</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jun 2015 - Jun 2016</th>\n",
       "      <td>2.3</td>\n",
       "      <td>11</td>\n",
       "      <td>0.9</td>\n",
       "      <td>10.6</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jul 2015 - Jul 2016</th>\n",
       "      <td>2.4</td>\n",
       "      <td>11.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>11.2</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aug 2015 - Aug 2016</th>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>3.3</td>\n",
       "      <td>11.9</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sep 2015 - Sep 2016</th>\n",
       "      <td>2.6</td>\n",
       "      <td>10.6</td>\n",
       "      <td>2.9</td>\n",
       "      <td>13</td>\n",
       "      <td>4.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Oct 2015 - Oct 2016</th>\n",
       "      <td>2.7</td>\n",
       "      <td>10.3</td>\n",
       "      <td>0.3</td>\n",
       "      <td>12.7</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nov 2015 - Nov 2016</th>\n",
       "      <td>2.3</td>\n",
       "      <td>9.8</td>\n",
       "      <td>0.2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dec 2015 - Dec 2016</th>\n",
       "      <td>2.4</td>\n",
       "      <td>9.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jan 2016 - Jan 2017</th>\n",
       "      <td>2.3</td>\n",
       "      <td>9.7</td>\n",
       "      <td>1.9</td>\n",
       "      <td>12.7</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb 2016 - Feb 2017</th>\n",
       "      <td>2.4</td>\n",
       "      <td>9.4</td>\n",
       "      <td>2.4</td>\n",
       "      <td>11.8</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar 2016 - Mar 2017</th>\n",
       "      <td>2.7</td>\n",
       "      <td>9.1</td>\n",
       "      <td>2</td>\n",
       "      <td>11.7</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr 2016 - Apr 2017</th>\n",
       "      <td>2.6</td>\n",
       "      <td>8.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>10.7</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>May 2016 - May 2017</th>\n",
       "      <td>2.4</td>\n",
       "      <td>8.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>4.2</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Network Airlines Low-Cost Airlines Regional Airlines  \\\n",
       "May 2015 - May 2016              2.3              10.7               0.2   \n",
       "Jun 2015 - Jun 2016              2.3                11               0.9   \n",
       "Jul 2015 - Jul 2016              2.4              11.3               3.3   \n",
       "Aug 2015 - Aug 2016              2.5                11               3.3   \n",
       "Sep 2015 - Sep 2016              2.6              10.6               2.9   \n",
       "Oct 2015 - Oct 2016              2.7              10.3               0.3   \n",
       "Nov 2015 - Nov 2016              2.3               9.8               0.2   \n",
       "Dec 2015 - Dec 2016              2.4               9.5               0.2   \n",
       "Jan 2016 - Jan 2017              2.3               9.7               1.9   \n",
       "Feb 2016 - Feb 2017              2.4               9.4               2.4   \n",
       "Mar 2016 - Mar 2017              2.7               9.1                 2   \n",
       "Apr 2016 - Apr 2017              2.6               8.5               2.1   \n",
       "May 2016 - May 2017              2.4               8.3               2.5   \n",
       "\n",
       "                    Other Airlines All Passenger Airlines **  \n",
       "May 2015 - May 2016            9.3                       3.7  \n",
       "Jun 2015 - Jun 2016           10.6                       3.9  \n",
       "Jul 2015 - Jul 2016           11.2                       4.3  \n",
       "Aug 2015 - Aug 2016           11.9                       4.3  \n",
       "Sep 2015 - Sep 2016             13                       4.3  \n",
       "Oct 2015 - Oct 2016           12.7                         4  \n",
       "Nov 2015 - Nov 2016           13.5                       3.7  \n",
       "Dec 2015 - Dec 2016           13.7                       3.7  \n",
       "Jan 2016 - Jan 2017           12.7                       3.9  \n",
       "Feb 2016 - Feb 2017           11.8                       3.9  \n",
       "Mar 2016 - Mar 2017           11.7                         4  \n",
       "Apr 2016 - Apr 2017           10.7                       3.9  \n",
       "May 2016 - May 2017            4.2                       3.6  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# putting it all together ...\n",
    "df_tbl1 = pd.DataFrame(data=xl_tbl1.iloc[2:15,1:].values,\n",
    "                       columns=[v for v in col.values], \n",
    "                       index=pd.Index([v[0] for v in idx.values]))\n",
    "df_tbl1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "May 2015 - May 2016    2.3\n",
       "Jun 2015 - Jun 2016    2.3\n",
       "Jul 2015 - Jul 2016    2.4\n",
       "Aug 2015 - Aug 2016    2.5\n",
       "Sep 2015 - Sep 2016    2.6\n",
       "Oct 2015 - Oct 2016    2.7\n",
       "Nov 2015 - Nov 2016    2.3\n",
       "Dec 2015 - Dec 2016    2.4\n",
       "Jan 2016 - Jan 2017    2.3\n",
       "Feb 2016 - Feb 2017    2.4\n",
       "Mar 2016 - Mar 2017    2.7\n",
       "Apr 2016 - Apr 2017    2.6\n",
       "May 2016 - May 2017    2.4\n",
       "Name: Network Airlines, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tbl1['Network Airlines']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON\n",
    "\n",
    "JSON has become a standard format format for many web data sources.  It is succinct, readable and very portable -- there are libraries in nearly every modern language that can parse JSON, Python being no exception.  We'll load a remote JSON data source to demonstrate remote access as well as the capabilities of using Pandas to load such a source.\n",
    "\n",
    "**JSON DATA SOURCE**\n",
    "\n",
    "* [Quotes for developers](https://github.com/fortrabbit/quotes) by _fortrabbit_\n",
    "\n",
    "If we haven't noticed the pattern yet, loading JSON data will come as no surprise via the [`pandas.read_json()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html#pandas.read_json).\n",
    "\n",
    "With JSON data you may get the best results with relatively _flat_ JSON objects.  If you need to obtain different results (or you're getting results that are not as expected), you might instead into the `orient` parameter to get different resulting DataFrames.  We'll load the data as-is and reshape our DataFrame for some extra practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Martin Golding</td>\n",
       "      <td>Always code as if the guy who ends up maintain...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>All computers wait at the same speed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>A misplaced decimal point will always end up w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>A good programmer looks both ways before cross...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>A computer program does what you tell it to do...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>\"Intel Inside\" is a Government Warning require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Arthur Godfrey</td>\n",
       "      <td>Common sense gets a lot of credit that belongs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Chuck Norris doesn’t go hunting. Chuck Norris ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Chuck Norris counted to infinity... twice.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>C is quirky, flawed, and an enormous success.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Beta is Latin for still doesn’t work.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>ASCII stupid question, get a stupid ANSI!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Artificial Intelligence usually beats natural ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Ted Nelson</td>\n",
       "      <td>Any fool can use a computer. Many do.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Hey! It compiles! Ship it!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Martin Luther King Junior</td>\n",
       "      <td>Hate cannot drive out hate; only love can do t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Guns don’t kill people. Chuck Norris kills peo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>God is real, unless declared integer.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>John Johnson</td>\n",
       "      <td>First, solve the problem. Then, write the code.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Oscar Wilde</td>\n",
       "      <td>Experience is the name everyone gives to their...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Miguel de Icaza</td>\n",
       "      <td>Every piece of software written today is likel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Computers make very fast, very accurate mistakes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Computers do not solve problems, they execute ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>I have NOT lost my mind—I have it backed up on...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>If brute force doesn’t solve your problems, th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>It works on my machine.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Java is, in many ways, C++??.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Keyboard not found...Press any key to continue.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Life would be so much easier if we only had th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Mac users swear by their Mac, PC users swear a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>Paul Graham</td>\n",
       "      <td>OO programming offers a sustainable way to wri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>Nikita Popov</td>\n",
       "      <td>Ruby is rubbish! PHP is phpantastic!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>Douglas Adams</td>\n",
       "      <td>So long and thanks for all the fish!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>Cicero</td>\n",
       "      <td>If I had more time, I would have written a sho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>Jeff Atwood</td>\n",
       "      <td>The best reaction to \"this is confusing, where...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>Jeff Atwood</td>\n",
       "      <td>The older I get, the more I believe that the o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>Douglas Crockford</td>\n",
       "      <td>\"That hardly ever happens\" is another way of s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>Anna Debenham</td>\n",
       "      <td>Hello, PHP, my old friend.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>Melvin Conway</td>\n",
       "      <td>Organizations which design systems are constra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>Melvin Conway</td>\n",
       "      <td>In design, complexity is toxic.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>Jeffrey Zeldman</td>\n",
       "      <td>Good is the enemy of great, but great is the e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>Rick Lemons</td>\n",
       "      <td>Don't make the user provide information that t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Donald E. Knuth</td>\n",
       "      <td>You're bound to be unhappy if you optimize eve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>Anna Nachesa</td>\n",
       "      <td>If the programmers like each other, they play ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>Edsger W. Dijkstra</td>\n",
       "      <td>Simplicity is prerequisite for reliability.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>Jordi Boggiano</td>\n",
       "      <td>Focus on WHY instead of WHAT in your code will...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>Andrei Herasimchuk</td>\n",
       "      <td>The best engineers I know are artists at heart...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>Barry Boehm</td>\n",
       "      <td>Poor management can increase software costs mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>Daniel Bryant</td>\n",
       "      <td>If you can't deploy your services independentl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>Daniel Bryant</td>\n",
       "      <td>If you can't deploy your services independentl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>Jeff Atwood</td>\n",
       "      <td>No one hates software more than software devel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>Robert C. Martin</td>\n",
       "      <td>The proper use of comments is to compensate fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>Cory House</td>\n",
       "      <td>Code is like humor. When you have to explain i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>Steve Maguire</td>\n",
       "      <td>Fix the cause, not the symptom.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>David Heinemeier Hansson</td>\n",
       "      <td>Programmers are constantly making things more ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>Linus Torvalds</td>\n",
       "      <td>People will realize that software is not a pro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>Ron Fein</td>\n",
       "      <td>Design is choosing how you will fail.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>Steve Jobs</td>\n",
       "      <td>Focus is saying no to 1000 good ideas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>Ron Jeffries</td>\n",
       "      <td>Code never lies, comments sometimes do.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>Unknown</td>\n",
       "      <td>Be careful with each other, so you can be dang...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>189 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        author  \\\n",
       "0               Martin Golding   \n",
       "1                      Unknown   \n",
       "2                      Unknown   \n",
       "3                      Unknown   \n",
       "4                      Unknown   \n",
       "5                      Unknown   \n",
       "6               Arthur Godfrey   \n",
       "7                      Unknown   \n",
       "8                      Unknown   \n",
       "9                      Unknown   \n",
       "10                     Unknown   \n",
       "11                     Unknown   \n",
       "12                     Unknown   \n",
       "13                  Ted Nelson   \n",
       "14                     Unknown   \n",
       "15   Martin Luther King Junior   \n",
       "16                     Unknown   \n",
       "17                     Unknown   \n",
       "18                John Johnson   \n",
       "19                 Oscar Wilde   \n",
       "20             Miguel de Icaza   \n",
       "21                     Unknown   \n",
       "22                     Unknown   \n",
       "23                     Unknown   \n",
       "24                     Unknown   \n",
       "25                     Unknown   \n",
       "26                     Unknown   \n",
       "27                     Unknown   \n",
       "28                     Unknown   \n",
       "29                     Unknown   \n",
       "..                         ...   \n",
       "159                Paul Graham   \n",
       "160               Nikita Popov   \n",
       "161              Douglas Adams   \n",
       "162                     Cicero   \n",
       "163                Jeff Atwood   \n",
       "164                Jeff Atwood   \n",
       "165          Douglas Crockford   \n",
       "166              Anna Debenham   \n",
       "167              Melvin Conway   \n",
       "168              Melvin Conway   \n",
       "169            Jeffrey Zeldman   \n",
       "170                Rick Lemons   \n",
       "171            Donald E. Knuth   \n",
       "172               Anna Nachesa   \n",
       "173         Edsger W. Dijkstra   \n",
       "174             Jordi Boggiano   \n",
       "175         Andrei Herasimchuk   \n",
       "176                Barry Boehm   \n",
       "177              Daniel Bryant   \n",
       "178              Daniel Bryant   \n",
       "179                Jeff Atwood   \n",
       "180           Robert C. Martin   \n",
       "181                 Cory House   \n",
       "182              Steve Maguire   \n",
       "183   David Heinemeier Hansson   \n",
       "184             Linus Torvalds   \n",
       "185                   Ron Fein   \n",
       "186                 Steve Jobs   \n",
       "187               Ron Jeffries   \n",
       "188                    Unknown   \n",
       "\n",
       "                                                  text  \n",
       "0    Always code as if the guy who ends up maintain...  \n",
       "1                All computers wait at the same speed.  \n",
       "2    A misplaced decimal point will always end up w...  \n",
       "3    A good programmer looks both ways before cross...  \n",
       "4    A computer program does what you tell it to do...  \n",
       "5    \"Intel Inside\" is a Government Warning require...  \n",
       "6    Common sense gets a lot of credit that belongs...  \n",
       "7    Chuck Norris doesn’t go hunting. Chuck Norris ...  \n",
       "8           Chuck Norris counted to infinity... twice.  \n",
       "9        C is quirky, flawed, and an enormous success.  \n",
       "10               Beta is Latin for still doesn’t work.  \n",
       "11           ASCII stupid question, get a stupid ANSI!  \n",
       "12   Artificial Intelligence usually beats natural ...  \n",
       "13               Any fool can use a computer. Many do.  \n",
       "14                          Hey! It compiles! Ship it!  \n",
       "15   Hate cannot drive out hate; only love can do t...  \n",
       "16   Guns don’t kill people. Chuck Norris kills peo...  \n",
       "17               God is real, unless declared integer.  \n",
       "18     First, solve the problem. Then, write the code.  \n",
       "19   Experience is the name everyone gives to their...  \n",
       "20   Every piece of software written today is likel...  \n",
       "21   Computers make very fast, very accurate mistakes.  \n",
       "22   Computers do not solve problems, they execute ...  \n",
       "23   I have NOT lost my mind—I have it backed up on...  \n",
       "24   If brute force doesn’t solve your problems, th...  \n",
       "25                             It works on my machine.  \n",
       "26                       Java is, in many ways, C++??.  \n",
       "27     Keyboard not found...Press any key to continue.  \n",
       "28   Life would be so much easier if we only had th...  \n",
       "29   Mac users swear by their Mac, PC users swear a...  \n",
       "..                                                 ...  \n",
       "159  OO programming offers a sustainable way to wri...  \n",
       "160               Ruby is rubbish! PHP is phpantastic!  \n",
       "161               So long and thanks for all the fish!  \n",
       "162  If I had more time, I would have written a sho...  \n",
       "163  The best reaction to \"this is confusing, where...  \n",
       "164  The older I get, the more I believe that the o...  \n",
       "165  \"That hardly ever happens\" is another way of s...  \n",
       "166                         Hello, PHP, my old friend.  \n",
       "167  Organizations which design systems are constra...  \n",
       "168                    In design, complexity is toxic.  \n",
       "169  Good is the enemy of great, but great is the e...  \n",
       "170  Don't make the user provide information that t...  \n",
       "171  You're bound to be unhappy if you optimize eve...  \n",
       "172  If the programmers like each other, they play ...  \n",
       "173        Simplicity is prerequisite for reliability.  \n",
       "174  Focus on WHY instead of WHAT in your code will...  \n",
       "175  The best engineers I know are artists at heart...  \n",
       "176  Poor management can increase software costs mo...  \n",
       "177  If you can't deploy your services independentl...  \n",
       "178  If you can't deploy your services independentl...  \n",
       "179  No one hates software more than software devel...  \n",
       "180  The proper use of comments is to compensate fo...  \n",
       "181  Code is like humor. When you have to explain i...  \n",
       "182                    Fix the cause, not the symptom.  \n",
       "183  Programmers are constantly making things more ...  \n",
       "184  People will realize that software is not a pro...  \n",
       "185              Design is choosing how you will fail.  \n",
       "186             Focus is saying no to 1000 good ideas.  \n",
       "187            Code never lies, comments sometimes do.  \n",
       "188  Be careful with each other, so you can be dang...  \n",
       "\n",
       "[189 rows x 2 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_json(\n",
    "    \"https://raw.githubusercontent.com/fortrabbit/quotes/master/quotes.json\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though not a best practice, say we wanted to set the author as the index and the  quote of the text the value.  In this dataset, we're going to have repeated index values, and it might make sense if we wanted to access the data this way, but be _very careful doing this in practice_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Martin Golding</th>\n",
       "      <td>Always code as if the guy who ends up maintain...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>All computers wait at the same speed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>A misplaced decimal point will always end up w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>A good programmer looks both ways before cross...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>A computer program does what you tell it to do...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>\"Intel Inside\" is a Government Warning require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arthur Godfrey</th>\n",
       "      <td>Common sense gets a lot of credit that belongs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Chuck Norris doesn’t go hunting. Chuck Norris ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Chuck Norris counted to infinity... twice.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>C is quirky, flawed, and an enormous success.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Beta is Latin for still doesn’t work.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>ASCII stupid question, get a stupid ANSI!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Artificial Intelligence usually beats natural ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ted Nelson</th>\n",
       "      <td>Any fool can use a computer. Many do.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Hey! It compiles! Ship it!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Martin Luther King Junior</th>\n",
       "      <td>Hate cannot drive out hate; only love can do t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Guns don’t kill people. Chuck Norris kills peo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>God is real, unless declared integer.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>John Johnson</th>\n",
       "      <td>First, solve the problem. Then, write the code.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Oscar Wilde</th>\n",
       "      <td>Experience is the name everyone gives to their...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Miguel de Icaza</th>\n",
       "      <td>Every piece of software written today is likel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Computers make very fast, very accurate mistakes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Computers do not solve problems, they execute ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>I have NOT lost my mind—I have it backed up on...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>If brute force doesn’t solve your problems, th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>It works on my machine.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Java is, in many ways, C++??.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Keyboard not found...Press any key to continue.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Life would be so much easier if we only had th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Mac users swear by their Mac, PC users swear a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Paul Graham</th>\n",
       "      <td>OO programming offers a sustainable way to wri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nikita Popov</th>\n",
       "      <td>Ruby is rubbish! PHP is phpantastic!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Douglas Adams</th>\n",
       "      <td>So long and thanks for all the fish!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cicero</th>\n",
       "      <td>If I had more time, I would have written a sho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jeff Atwood</th>\n",
       "      <td>The best reaction to \"this is confusing, where...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jeff Atwood</th>\n",
       "      <td>The older I get, the more I believe that the o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Douglas Crockford</th>\n",
       "      <td>\"That hardly ever happens\" is another way of s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Anna Debenham</th>\n",
       "      <td>Hello, PHP, my old friend.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Melvin Conway</th>\n",
       "      <td>Organizations which design systems are constra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Melvin Conway</th>\n",
       "      <td>In design, complexity is toxic.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jeffrey Zeldman</th>\n",
       "      <td>Good is the enemy of great, but great is the e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rick Lemons</th>\n",
       "      <td>Don't make the user provide information that t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Donald E. Knuth</th>\n",
       "      <td>You're bound to be unhappy if you optimize eve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Anna Nachesa</th>\n",
       "      <td>If the programmers like each other, they play ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Edsger W. Dijkstra</th>\n",
       "      <td>Simplicity is prerequisite for reliability.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jordi Boggiano</th>\n",
       "      <td>Focus on WHY instead of WHAT in your code will...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Andrei Herasimchuk</th>\n",
       "      <td>The best engineers I know are artists at heart...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Barry Boehm</th>\n",
       "      <td>Poor management can increase software costs mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Daniel Bryant</th>\n",
       "      <td>If you can't deploy your services independentl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Daniel Bryant</th>\n",
       "      <td>If you can't deploy your services independentl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jeff Atwood</th>\n",
       "      <td>No one hates software more than software devel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Robert C. Martin</th>\n",
       "      <td>The proper use of comments is to compensate fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cory House</th>\n",
       "      <td>Code is like humor. When you have to explain i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steve Maguire</th>\n",
       "      <td>Fix the cause, not the symptom.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>David Heinemeier Hansson</th>\n",
       "      <td>Programmers are constantly making things more ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Linus Torvalds</th>\n",
       "      <td>People will realize that software is not a pro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ron Fein</th>\n",
       "      <td>Design is choosing how you will fail.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steve Jobs</th>\n",
       "      <td>Focus is saying no to 1000 good ideas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ron Jeffries</th>\n",
       "      <td>Code never lies, comments sometimes do.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Be careful with each other, so you can be dang...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>189 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                        text\n",
       "author                                                                      \n",
       "Martin Golding             Always code as if the guy who ends up maintain...\n",
       "Unknown                                All computers wait at the same speed.\n",
       "Unknown                    A misplaced decimal point will always end up w...\n",
       "Unknown                    A good programmer looks both ways before cross...\n",
       "Unknown                    A computer program does what you tell it to do...\n",
       "Unknown                    \"Intel Inside\" is a Government Warning require...\n",
       "Arthur Godfrey             Common sense gets a lot of credit that belongs...\n",
       "Unknown                    Chuck Norris doesn’t go hunting. Chuck Norris ...\n",
       "Unknown                           Chuck Norris counted to infinity... twice.\n",
       "Unknown                        C is quirky, flawed, and an enormous success.\n",
       "Unknown                                Beta is Latin for still doesn’t work.\n",
       "Unknown                            ASCII stupid question, get a stupid ANSI!\n",
       "Unknown                    Artificial Intelligence usually beats natural ...\n",
       "Ted Nelson                             Any fool can use a computer. Many do.\n",
       "Unknown                                           Hey! It compiles! Ship it!\n",
       "Martin Luther King Junior  Hate cannot drive out hate; only love can do t...\n",
       "Unknown                    Guns don’t kill people. Chuck Norris kills peo...\n",
       "Unknown                                God is real, unless declared integer.\n",
       "John Johnson                 First, solve the problem. Then, write the code.\n",
       "Oscar Wilde                Experience is the name everyone gives to their...\n",
       "Miguel de Icaza            Every piece of software written today is likel...\n",
       "Unknown                    Computers make very fast, very accurate mistakes.\n",
       "Unknown                    Computers do not solve problems, they execute ...\n",
       "Unknown                    I have NOT lost my mind—I have it backed up on...\n",
       "Unknown                    If brute force doesn’t solve your problems, th...\n",
       "Unknown                                              It works on my machine.\n",
       "Unknown                                        Java is, in many ways, C++??.\n",
       "Unknown                      Keyboard not found...Press any key to continue.\n",
       "Unknown                    Life would be so much easier if we only had th...\n",
       "Unknown                    Mac users swear by their Mac, PC users swear a...\n",
       "...                                                                      ...\n",
       "Paul Graham                OO programming offers a sustainable way to wri...\n",
       "Nikita Popov                            Ruby is rubbish! PHP is phpantastic!\n",
       "Douglas Adams                           So long and thanks for all the fish!\n",
       "Cicero                     If I had more time, I would have written a sho...\n",
       "Jeff Atwood                The best reaction to \"this is confusing, where...\n",
       "Jeff Atwood                The older I get, the more I believe that the o...\n",
       "Douglas Crockford          \"That hardly ever happens\" is another way of s...\n",
       "Anna Debenham                                     Hello, PHP, my old friend.\n",
       "Melvin Conway              Organizations which design systems are constra...\n",
       "Melvin Conway                                In design, complexity is toxic.\n",
       "Jeffrey Zeldman            Good is the enemy of great, but great is the e...\n",
       "Rick Lemons                Don't make the user provide information that t...\n",
       "Donald E. Knuth            You're bound to be unhappy if you optimize eve...\n",
       "Anna Nachesa               If the programmers like each other, they play ...\n",
       "Edsger W. Dijkstra               Simplicity is prerequisite for reliability.\n",
       "Jordi Boggiano             Focus on WHY instead of WHAT in your code will...\n",
       "Andrei Herasimchuk         The best engineers I know are artists at heart...\n",
       "Barry Boehm                Poor management can increase software costs mo...\n",
       "Daniel Bryant              If you can't deploy your services independentl...\n",
       "Daniel Bryant              If you can't deploy your services independentl...\n",
       "Jeff Atwood                No one hates software more than software devel...\n",
       "Robert C. Martin           The proper use of comments is to compensate fo...\n",
       "Cory House                 Code is like humor. When you have to explain i...\n",
       "Steve Maguire                                Fix the cause, not the symptom.\n",
       "David Heinemeier Hansson   Programmers are constantly making things more ...\n",
       "Linus Torvalds             People will realize that software is not a pro...\n",
       "Ron Fein                               Design is choosing how you will fail.\n",
       "Steve Jobs                            Focus is saying no to 1000 good ideas.\n",
       "Ron Jeffries                         Code never lies, comments sometimes do.\n",
       "Unknown                    Be careful with each other, so you can be dang...\n",
       "\n",
       "[189 rows x 1 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df.set_index(df['author']).drop('author', axis=1)\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though we haven't talked about it, there is a very interesting and useful mechanism for filtering data using the [`apply()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.apply.html#pandas.Series.apply) method.  In this case, we're going to write a cute anonymous function that finds all the quotes by the author `Unknown` with `java` in the quote."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>author</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Unknown</th>\n",
       "      <td>Java is, in many ways, C++??.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  text\n",
       "author                                \n",
       "Unknown  Java is, in many ways, C++??."
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.loc[\"Unknown\"][df1.loc[\"Unknown\"][\"text\"]\n",
    "                   .apply(lambda v: \"jav\" in v.lower())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Loading SQL data into a DataFrame is also supported by Pandas. You might need to take a look at the [SQLAlchemy](http://www.sqlalchemy.org/) and the [documentation on creating database engines](http://docs.sqlalchemy.org/en/latest/core/engines.html), as this is the framework supported directly by Pandas.\n",
    "\n",
    "**SQL DATA SOURCE**\n",
    "\n",
    "* [Jeopardy! Data Analysis](https://github.com/cmohamma/jeopardy) - a sqlite database by _cmohamma_\n",
    "\n",
    "This file contains a number of tables that contain the Jeopardy! game clues, players, wins, categories, etc.  We will only use a fraction of the data to demonstrate the SQL capabilities.\n",
    "\n",
    "Our example will use a [SQLite database](https://sqlite.org/) so we can demonstrate the example in a standalone context.  We'll show reading a table in full using the [`read_sql_table()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html#pandas.read_sql_table) and then how to do ad hoc queries using [`read_sql_query()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html#pandas.read_sql_query)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine\n",
    "engine = create_engine('sqlite:///datasets/database.sqlite')\n",
    "\n",
    "with engine.connect() as conn, conn.begin():\n",
    "    data = pd.read_sql_table('final', conn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>game_id</th>\n",
       "      <th>clue_id</th>\n",
       "      <th>value</th>\n",
       "      <th>category</th>\n",
       "      <th>clue</th>\n",
       "      <th>strike1</th>\n",
       "      <th>strike2</th>\n",
       "      <th>strike3</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>280</td>\n",
       "      <td>16720</td>\n",
       "      <td>100</td>\n",
       "      <td>HIGH ROLLERS</td>\n",
       "      <td>After an 1891 roulette run, Charles Wells was ...</td>\n",
       "      <td>What is Atlantic City?</td>\n",
       "      <td>What is Las Vegas?</td>\n",
       "      <td>What is Monaco?</td>\n",
       "      <td>Monte Carlo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>429</td>\n",
       "      <td>25403</td>\n",
       "      <td>100</td>\n",
       "      <td>OH, CRAPS!</td>\n",
       "      <td>The combo that totals one shy of \"boxcars\"</td>\n",
       "      <td>What is 11?</td>\n",
       "      <td>What is 10?</td>\n",
       "      <td>What is 9?</td>\n",
       "      <td>5 &amp; 6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>866</td>\n",
       "      <td>51549</td>\n",
       "      <td>100</td>\n",
       "      <td>ROCK &amp; POP</td>\n",
       "      <td>It was the last decade in which Cher didn't ha...</td>\n",
       "      <td>What are the 1980s?</td>\n",
       "      <td>What are the 1970s?</td>\n",
       "      <td>What are the 1990s?</td>\n",
       "      <td>1950s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1018</td>\n",
       "      <td>60582</td>\n",
       "      <td>100</td>\n",
       "      <td>LET'S HAVE A BALL</td>\n",
       "      <td>Sink it &amp; you've scratched</td>\n",
       "      <td>Um...</td>\n",
       "      <td>What is the pinball?</td>\n",
       "      <td>What is the 8-ball?</td>\n",
       "      <td>the cue ball</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1069</td>\n",
       "      <td>63644</td>\n",
       "      <td>100</td>\n",
       "      <td>WHAT A YEAR!</td>\n",
       "      <td>Dewaele won the Tour de France, Coco Chanel wa...</td>\n",
       "      <td>What is 1933?</td>\n",
       "      <td>What is 1987?</td>\n",
       "      <td>What is 1927?</td>\n",
       "      <td>1929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1473</td>\n",
       "      <td>84364</td>\n",
       "      <td>100</td>\n",
       "      <td>EUROPEAN HISTORY</td>\n",
       "      <td>A former Socialist, he formed the anti-Communi...</td>\n",
       "      <td>Who was Lenin?</td>\n",
       "      <td>Who was Franco?</td>\n",
       "      <td>Who was Hitler?</td>\n",
       "      <td>Benito Mussolini</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1635</td>\n",
       "      <td>93864</td>\n",
       "      <td>100</td>\n",
       "      <td>CHRISTIANITY</td>\n",
       "      <td>According to tradition, Dismas &amp; Gestas were t...</td>\n",
       "      <td>Who are the thieves?</td>\n",
       "      <td>What is Cavalry?</td>\n",
       "      <td>What is Mt. Olive?</td>\n",
       "      <td>Calvary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4166</td>\n",
       "      <td>242419</td>\n",
       "      <td>100</td>\n",
       "      <td>NAME THE DECADE</td>\n",
       "      <td>Paul Revere &amp; William Dawes warn colonists tha...</td>\n",
       "      <td>What is the 16th century?</td>\n",
       "      <td>What is the 18th century?</td>\n",
       "      <td>What is the 18th century?</td>\n",
       "      <td>the 1770s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>112</td>\n",
       "      <td>6679</td>\n",
       "      <td>200</td>\n",
       "      <td>ODD ALPHABETS</td>\n",
       "      <td>In alphabet radio code, \"B\" is Bravo and \"F\" s...</td>\n",
       "      <td>What's the Flamingo?</td>\n",
       "      <td>What's a Fandango?</td>\n",
       "      <td>What's the Flamenco? - you have it written the...</td>\n",
       "      <td>Foxtrot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>354</td>\n",
       "      <td>20984</td>\n",
       "      <td>200</td>\n",
       "      <td>SPORTS</td>\n",
       "      <td>A filly becomes a mare at this age</td>\n",
       "      <td>What is 3?</td>\n",
       "      <td>What is 1?</td>\n",
       "      <td>What is 2?</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   game_id  clue_id  value           category  \\\n",
       "0      280    16720    100       HIGH ROLLERS   \n",
       "1      429    25403    100         OH, CRAPS!   \n",
       "2      866    51549    100         ROCK & POP   \n",
       "3     1018    60582    100  LET'S HAVE A BALL   \n",
       "4     1069    63644    100       WHAT A YEAR!   \n",
       "5     1473    84364    100   EUROPEAN HISTORY   \n",
       "6     1635    93864    100       CHRISTIANITY   \n",
       "7     4166   242419    100    NAME THE DECADE   \n",
       "8      112     6679    200      ODD ALPHABETS   \n",
       "9      354    20984    200             SPORTS   \n",
       "\n",
       "                                                clue  \\\n",
       "0  After an 1891 roulette run, Charles Wells was ...   \n",
       "1         The combo that totals one shy of \"boxcars\"   \n",
       "2  It was the last decade in which Cher didn't ha...   \n",
       "3                         Sink it & you've scratched   \n",
       "4  Dewaele won the Tour de France, Coco Chanel wa...   \n",
       "5  A former Socialist, he formed the anti-Communi...   \n",
       "6  According to tradition, Dismas & Gestas were t...   \n",
       "7  Paul Revere & William Dawes warn colonists tha...   \n",
       "8  In alphabet radio code, \"B\" is Bravo and \"F\" s...   \n",
       "9                 A filly becomes a mare at this age   \n",
       "\n",
       "                     strike1                    strike2  \\\n",
       "0     What is Atlantic City?         What is Las Vegas?   \n",
       "1                What is 11?                What is 10?   \n",
       "2        What are the 1980s?        What are the 1970s?   \n",
       "3                      Um...       What is the pinball?   \n",
       "4              What is 1933?              What is 1987?   \n",
       "5             Who was Lenin?            Who was Franco?   \n",
       "6       Who are the thieves?           What is Cavalry?   \n",
       "7  What is the 16th century?  What is the 18th century?   \n",
       "8       What's the Flamingo?         What's a Fandango?   \n",
       "9                 What is 3?                 What is 1?   \n",
       "\n",
       "                                             strike3            answer  \n",
       "0                                    What is Monaco?       Monte Carlo  \n",
       "1                                         What is 9?             5 & 6  \n",
       "2                                What are the 1990s?             1950s  \n",
       "3                                What is the 8-ball?      the cue ball  \n",
       "4                                      What is 1927?              1929  \n",
       "5                                    Who was Hitler?  Benito Mussolini  \n",
       "6                                 What is Mt. Olive?           Calvary  \n",
       "7                          What is the 18th century?         the 1770s  \n",
       "8  What's the Flamenco? - you have it written the...           Foxtrot  \n",
       "9                                         What is 2?                 4  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now say we want to find out the distribution of occupations of players over the years.  When we look into the `players` table, we can see we can create a query that allows for us to aggregate these occupations easily. \n",
    "\n",
    "Using [`read_sql_query()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html#pandas.read_sql_query) we can get the job done and dump this into a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "query = \"\"\"\n",
    "    SELECT occupation, count(occupation) as freq FROM players\n",
    "    WHERE occupation != ''\n",
    "    GROUP BY occupation \n",
    "    ORDER BY count(occupation) DESC \n",
    "    \"\"\"\n",
    "\n",
    "with engine.connect() as conn, conn.begin():\n",
    "    occupation_data = pd.read_sql_query(query, conn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>occupation</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>attorney</td>\n",
       "      <td>380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>senior</td>\n",
       "      <td>228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>graduate student</td>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>writer</td>\n",
       "      <td>176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>teacher</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>junior</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>law student</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>lawyer</td>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>homemaker</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>actor</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         occupation  freq\n",
       "0          attorney   380\n",
       "1            senior   228\n",
       "2  graduate student   212\n",
       "3            writer   176\n",
       "4           teacher   159\n",
       "5            junior   158\n",
       "6       law student   120\n",
       "7            lawyer   112\n",
       "8         homemaker   101\n",
       "9             actor    97"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "occupation_data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look closely, we can see that there are many occupations that are the same, but labeled differently.  For example, \"attorney\" and \"lawyer\", or the various kinds of \"teachers\".  Thus, if we just look at the frequency from above, we might be deceived in thinking that these frequencies are correct for the groupings that make sense at a slightly higher level of granularity than has been captured.\n",
    "\n",
    "So let's do some data munging with Pandas and see how we might group all the \"teachers\" together.\n",
    "\n",
    "To to this we'll need to do a few things:\n",
    "\n",
    "* find all occupations that have `\"teach\"` in them (or `\"teacher\"` if you'd like)\n",
    "* remove all of those from the data frame\n",
    "* add just the aggregate and apply the generic label \"teacher\"\n",
    "* as a bonus, we'll generate the percentages as an additional column\n",
    "\n",
    "Let's get going!\n",
    "\n",
    "We are going to make use of a nice convenience attribution `str` of the `Series` object.  It operates much like the `String` object in Python and has a [`contains()`](http://pandas.pydata.org/pandas-docs/stable/api.html#string-handling) method, which will allow us to determine if the substring we're looking for is contained as a substring in any of the values of the Series.  These methods are indeed very useful to have!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "freq_all_occupations = occupation_data.freq.sum()\n",
    "\n",
    "combined_teacher_freq = \\\n",
    "        occupation_data[\n",
    "            occupation_data['occupation']\n",
    "                .str.contains('teach')]\\\n",
    "        .sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "occupation    teacherhigh school teacherhigh school English ...\n",
       "freq                                                        830\n",
       "dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_teacher_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the occupation is the concatenation of all those teachers.  We want to change that to a single label `\"teacher\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "combined_teacher_freq['occupation'] = 'teacher'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "occupation    teacher\n",
       "freq              830\n",
       "dtype: object"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_teacher_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need only append the data to our original DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "occupation_data = \\\n",
    "    occupation_data[\n",
    "        ~occupation_data['occupation']\n",
    "            .str.contains('teach')] \\\n",
    "    .append(combined_teacher_freq, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>occupation</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4205</th>\n",
       "      <td>writer for an online magazine</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4206</th>\n",
       "      <td>writer's assistant</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4207</th>\n",
       "      <td>writer-producer</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4208</th>\n",
       "      <td>writing instructor</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4209</th>\n",
       "      <td>yoga instructor</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4210</th>\n",
       "      <td>yogurt franchise operator</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4211</th>\n",
       "      <td>youth ministry consultant</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4212</th>\n",
       "      <td>zoo docent</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4213</th>\n",
       "      <td>zoo educator</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4214</th>\n",
       "      <td>teacher</td>\n",
       "      <td>830</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         occupation  freq\n",
       "4205  writer for an online magazine     1\n",
       "4206             writer's assistant     1\n",
       "4207                writer-producer     1\n",
       "4208             writing instructor     1\n",
       "4209                yoga instructor     1\n",
       "4210      yogurt franchise operator     1\n",
       "4211      youth ministry consultant     1\n",
       "4212                     zoo docent     1\n",
       "4213                   zoo educator     1\n",
       "4214                        teacher   830"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "occupation_data[-10:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add the percentage column and call it `pct`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "occupation_data['pct'] = occupation_data['freq']/occupation_data.freq.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>occupation</th>\n",
       "      <th>freq</th>\n",
       "      <th>pct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4214</th>\n",
       "      <td>teacher</td>\n",
       "      <td>830</td>\n",
       "      <td>0.078905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>attorney</td>\n",
       "      <td>380</td>\n",
       "      <td>0.036125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>senior</td>\n",
       "      <td>228</td>\n",
       "      <td>0.021675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>graduate student</td>\n",
       "      <td>212</td>\n",
       "      <td>0.020154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>writer</td>\n",
       "      <td>176</td>\n",
       "      <td>0.016732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>junior</td>\n",
       "      <td>158</td>\n",
       "      <td>0.015020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>law student</td>\n",
       "      <td>120</td>\n",
       "      <td>0.011408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>lawyer</td>\n",
       "      <td>112</td>\n",
       "      <td>0.010647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>homemaker</td>\n",
       "      <td>101</td>\n",
       "      <td>0.009602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>actor</td>\n",
       "      <td>97</td>\n",
       "      <td>0.009221</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            occupation  freq       pct\n",
       "4214           teacher   830  0.078905\n",
       "0             attorney   380  0.036125\n",
       "1               senior   228  0.021675\n",
       "2     graduate student   212  0.020154\n",
       "3               writer   176  0.016732\n",
       "4               junior   158  0.015020\n",
       "5          law student   120  0.011408\n",
       "6               lawyer   112  0.010647\n",
       "7            homemaker   101  0.009602\n",
       "8                actor    97  0.009221"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "occupation_data.sort_values(by='pct', ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore how you might make a more complex filter by looking at [`apply`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html#pandas.DataFrame.apply), [`applymap`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.applymap.html#pandas.DataFrame.applymap) and [`aggregate`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.aggregate.html#pandas.DataFrame.aggregate). &Xi;"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "160px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": true,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}