talk_2017_08_RMACC_GotPandas/nb/4_wrapping_up.ipynb

NAVIGATION

Got Pandas? Practical Data Wrangling with Pandas

• Introduction

1. Data Structures
2. Importing Data
3. Manipulating DataFrames
4. Wrap Up

---

NOTEBOOK OBJECTIVES

In this notebook we'll:

• explore exporting DataFrames,
• explore basic visualization capabilities.

Exporting Data¶

We've see that data importing is very easy to do and for a variety of formats.

We're going to show exporting of CSV data to Excel, SQL and JSON. There are a few other exporters that may be of interest to the reader:

• to_xarray(): a method for converting to xarrays
• to_latex() : a convenience method for making pretty $\LaTeX$ from data
• to_pickel(): a method for pickling (serializing) data to file

In all our basic examples here, we will be using methods over DataFrames.

We're going to go back to our baseball batting data file and learn how to convert this CSV file into something perhaps more interesting - in particular, Excel, SQL and JSON.

In [1]:

import pandas as pd 
df = pd.read_csv("./datasets/Batting.csv")

Excel¶

An ExcelWriter object is required to perform the export to Microsoft Excel, but once it is created, writing to the file is a cinch.

In [2]:

writer = pd.ExcelWriter('export/batting.xlsx')
df.to_excel(writer)
writer.save()

SQL¶

Dumping data to a database is nearly as easy as it was to read it. We need to use the SQLAlchemy engines as before.

In [ ]:

from sqlalchemy import create_engine
engine = create_engine('sqlite:///export/demo.sqlite')

with engine.connect() as conn, conn.begin():
    try:
        df.to_sql('batting', conn)
    except ValueError: # table already exists
        pass

JSON¶

Exporting to JSON is also very straightforward, but because the more intricate structure that can be communicated in JSON, we have several options regarding how the data is organized.

The default to_json() structures the object in a way that column labels are represented as the keys and the values for each column are represented as an object with the index as the key and the value for that (index, column) pair as the value.

In [ ]:

import json
json.loads(df[:5].to_json())

You are encouraged to read the documentation further, but you can orient the output data along records or rows and obtain a slightly different output which may be more suitable for your JSON needs.

In [ ]:

json.loads(df[:5].to_json(orient='records'))

Basic visualization¶

By now you are probably very happy to have been acquainted with Pandas, but in this tutorial, we've just scratched the surface.

If all the other data wrangling features of Pandas weren't enough.

In [ ]:

Making basic scatter plots is very easy with the [`plot()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html#pandas.DataFrame.plot) method.  As a convenience we can also use the `plot.scatter()`, which is equivalent.

In [ ]:

%matplotlib inline
df[(df.teamID=='WAS') & (df.yearID==2016)][['G', 'AB']].plot.scatter(x='G', y='AB')

Making basic line plots is similarly easy with the plot.line():

In [ ]:

df[(df.teamID=='WAS') & (df.yearID==2016)][['H', 'AB']].plot.line(x='H', y='AB')

In [ ]:

# plot the number of cummulative HR by team by year
df_was_hr = df[(df.teamID=='WAS') & (df.yearID>2006)][['yearID', 'HR']].groupby('yearID').sum()
df_was_hr.plot.bar()

• Pandas plotting provides the basics ...
• for more, you won't escape the grips of matplotlib!

Additional resources¶

As can be imagined, there are many great resources to use to learn more about Pandas:

• the pydata documentation is complete, if not overwhelming for the beginner
• Pandas Cookbook on Github by Julia Evans (also on pydata.org)
• Data Wrangling with Pandas cheat sheet by pydata.org
• Pandas for Data Science cheat sheet by DataCamp.com