talk_2017_08_RMACC_GotPandas/nb/4_wrapping_up.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** NAVIGATION **\n",
    "\n",
    "**Got Pandas? _Practical Data Wrangling with Pandas_**\n",
    "\n",
    "* [Introduction](./0_introduction.ipynb)\n",
    "1. [Data Structures](./1_data_structures.ipynb)\n",
    "2. [Importing Data](./2_importing_data.ipynb)\n",
    "3. [Manipulating DataFrames](./3_dataframe_operations.ipynb)\n",
    "4. **Wrap Up**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTEBOOK OBJECTIVES**\n",
    "\n",
    "In this notebook we'll:\n",
    "\n",
    "* explore exporting DataFrames,\n",
    "* explore basic visualization capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exporting Data\n",
    "\n",
    "We've see [that data importing](./1_importing_data.ipynb) is very easy to do and for a variety of formats.\n",
    "\n",
    "We're going to show exporting of CSV data to Excel, SQL and JSON.  There are a few other exporters that may be of interest to the reader:\n",
    "\n",
    "* [to_xarray()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_xarray.html#pandas.DataFrame.to_xarray): a method for converting to xarrays\n",
    "* [to_latex()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_latex.html#pandas.DataFrame.to_latex) : a convenience method for making pretty $\\LaTeX$ from data \n",
    "* [to_pickel()](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_pickle.html#pandas.DataFrame.to_pickle): a method for pickling (serializing) data to file\n",
    "\n",
    "In all our basic examples here, we will be using methods over DataFrames.\n",
    "\n",
    "We're going to go back to our baseball batting data file and learn how to convert this CSV file into something perhaps more interesting - in particular, Excel, SQL and JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "df = pd.read_csv(\"./datasets/Batting.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excel\n",
    "\n",
    "An [`ExcelWriter`]() object is required to perform the export to Microsoft Excel, but once it is created, writing to the file is a cinch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "writer = pd.ExcelWriter('export/batting.xlsx')\n",
    "df.to_excel(writer)\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SQL\n",
    "\n",
    "Dumping data to a database is nearly as easy as it was to read it.  We need to use the SQLAlchemy engines [as before](./1_importing_data.ipynb#SQL)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine\n",
    "engine = create_engine('sqlite:///export/demo.sqlite')\n",
    "\n",
    "with engine.connect() as conn, conn.begin():\n",
    "    try:\n",
    "        df.to_sql('batting', conn)\n",
    "    except ValueError: # table already exists\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON\n",
    "\n",
    "Exporting to JSON is also very straightforward, but because the more intricate structure that can be communicated in JSON, we have several options regarding how the data is organized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default [`to_json()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) structures the object in a way that _column_ labels are represented as the keys and the values for each column are represented as an object with the index as the key and the value for that (index, column) pair as the value.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "json.loads(df[:5].to_json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are encouraged to [read the documentation](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) further, but you can orient the output data along _records_ or _rows_ and obtain a slightly different output which may be more suitable for your JSON needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "json.loads(df[:5].to_json(orient='records'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic visualization\n",
    "\n",
    "By now you are probably very happy to have been acquainted with Pandas, but in this tutorial, we've just scratched the surface.\n",
    "\n",
    "If all the other data wrangling features of Pandas weren't enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Making basic scatter plots is very easy with the [`plot()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html#pandas.DataFrame.plot) method.  As a convenience we can also use the `plot.scatter()`, which is equivalent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "df[(df.teamID=='WAS') & (df.yearID==2016)][['G', 'AB']].plot.scatter(x='G', y='AB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making basic line plots is similarly easy with the  `plot.line()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df[(df.teamID=='WAS') & (df.yearID==2016)][['H', 'AB']].plot.line(x='H', y='AB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# plot the number of cummulative HR by team by year\n",
    "df_was_hr = df[(df.teamID=='WAS') & (df.yearID>2006)][['yearID', 'HR']].groupby('yearID').sum()\n",
    "df_was_hr.plot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "* Pandas plotting provides the basics ...\n",
    "* for more, you won't escape the grips of [matplotlib](http://matplotlib.org/)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additional resources\n",
    "\n",
    "As can be imagined, there are many great resources to use to learn more about Pandas:\n",
    "\n",
    "* the [pydata documentation](http://pandas.pydata.org) is complete, if not overwhelming for the beginner\n",
    "* [Pandas Cookbook](https://github.com/jvns/pandas-cookbook) on Github by Julia Evans (also on pydata.org)\n",
    "* [Data Wrangling with Pandas cheat sheet](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf) by pydata.org\n",
    "* [Pandas for Data Science cheat sheet](https://www.datacamp.com/community/blog/python-pandas-cheat-sheet) by DataCamp.com"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "123px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}