talk_2017_08_RMACC_GotPandas/nb/1_data_structures.ipynb

NAVIGATION

Got Pandas? Practical Data Wrangling with Pandas

• Introduction

1. Data Structures
2. Importing Data
3. Manipulating DataFrames
4. Wrap Up

---

Table of Contents¶

1  Core Pandas Data Structures

1.1  Series

1.2  DataFrames

1.2.1  [] operator for basic slicing

1.2.2  iloc[]

1.2.3  More sophisticated slicing

1.2.4  loc()

---

NOTEBOOK OBJECTIVES

In this notebook we'll:

• explore the Series and DataFrame data structures,
• understand basic selection and slicing operation in each.

Core Pandas Data Structures¶

As we move through the content, we will be working with two of Pandas primary data structures. There are more, but we will only focus on these two:

• Series
• DataFrames

The structures we won't have time to explore are Panels, which should be explored when you're ready to do so.

There are some key ideas behind the data structures provided by Pandas:

• data may be heterogeneous
• when data is numeric, convenience functions exist to provide aggregate statistical operations (min(), max(), cumsum(), median(), mode(), etc.),
• data structures are decomposable and composable, that is making DataFrames from Series or Series from DataFrame is supported natively,
• data structures are translatable, that is you can create Numpy NDArray.

Series¶

The Pandas Series data structure is a one dimensional structure much like a vector, that has axis labels. Series objects can be initialized from an array-like data object, dictionary or scalar value.

In [1]:

import numpy as np
import pandas as pd

In [2]:

np_ints = np.random.randint(0,51,10)
pd.Series(np_ints)

Out[2]:

0    43
1     3
2    15
3    20
4     3
5    45
6    44
7    30
8    25
9    48
dtype: int32

Data in a series do not have to be numeric:

In [3]:

import string
random_letters = [''
                  .join([string.ascii_letters[c] for c in np.random.randint(0,51,10)]) 
                  for i in range(10)]
pd.Series(random_letters)

Out[3]:

0    WhTIhhttjy
1    ubtcrWeUrE
2    pPzpwOsKpm
3    pQLCcUiotK
4    AiOwCuildy
5    DCkniyiWqp
6    TOCDhTYkFw
7    ziJpNNTbRo
8    UjveUhQFFm
9    EDEqSQpCKV
dtype: object

We can specify an index if we'd like that will allow us to have meaningful labels to access the data in the Series:

In [4]:

index = ['city', 'state', 'zip', 'neigborhood', 'area_code']
data  = ['Denver', 'CO', '80023', 'Furhman', '303']
s = pd.Series(data, index)
s

Out[4]:

city            Denver
state               CO
zip              80023
neigborhood    Furhman
area_code          303
dtype: object

Now we can access the data by its index label ...

In [5]:

s['city'], s['state']

Out[5]:

('Denver', 'CO')

Accessing data in a series is much like accessing data in a Python list. The usual slicing operator is available to get at the data in the Series.

In [6]:

s[1]

Out[6]:

'CO'

In [7]:

s[0:3]

Out[7]:

city     Denver
state        CO
zip       80023
dtype: object

In [8]:

s[0:3:2]

Out[8]:

city    Denver
zip      80023
dtype: object

More sophisticated slicing by index using integers can be achieved with iloc. Here are some simple examples:

In [9]:

s.iloc[0:3]

Out[9]:

city     Denver
state        CO
zip       80023
dtype: object

In [10]:

s.iloc[0:3:2]

Out[10]:

city    Denver
zip      80023
dtype: object

We can pass a list of the indices we'd like just as easily ...

In [11]:

s.iloc[[1,2,4]]

Out[11]:

state           CO
zip          80023
area_code      303
dtype: object

To get at all the values of the Series as an NDArray, simply do

In [12]:

s.values

Out[12]:

array(['Denver', 'CO', '80023', 'Furhman', '303'], dtype=object)

which then allows us to convert to a list

In [13]:

s.values.tolist()

Out[13]:

['Denver', 'CO', '80023', 'Furhman', '303']

DataFrames¶

DataFrames are a natural extension to Series in that they are 2-dimensional, and similarly to matrices (from vectors). They have many of the same operations (extended to 2 dimensions), but have some additional properties and operations. We'll cover the basics here.

• a DataFrame has a row and column axis, which defaults to numeric values (0 and 1),
• a DataFrame axis can be multi-level, that is multi-level indices can be created for row, column or both,
• DataFrames can be converted to NDArray and thus be converted to lists and dictionaries,
• indexing is achieved either by integer value or index label (both where applicable),
• DataFrame values may be heterogeous.

Let's first begin by building a DataFrame from a 2D Numpy array ...

In [14]:

df = pd.DataFrame(np.random.randint(1,100,100).reshape(10,10))
df

Out[14]:

	0	1	2	3	4	5	6	7	8	9
0	58	6	91	11	22	29	25	36	55	87
1	83	30	8	55	43	62	82	74	12	5
2	83	95	28	33	95	28	7	32	6	2
3	17	25	82	3	65	39	73	63	6	49
4	13	63	18	86	29	35	97	24	71	50
5	33	39	84	85	15	42	68	45	26	69
6	55	27	17	44	78	19	38	63	31	60
7	41	19	48	36	92	35	41	97	98	10
8	59	65	67	58	36	84	8	45	16	76
9	56	36	98	63	73	54	36	61	56	73

[] operator for basic slicing¶

The slicing operator [] works on DataFrames over row slices. Getting at data in the DataFrame is otherwise done with the iloc[] and loc[] selectors.

Be mindful to use this operator sparingly it is not consistent with loc and iloc, and may create confusing code if mixed arbitrarily with those selectors. Let's see a few basic cases for [] ...

In [15]:

df[1:4] # selecting the rows index 1 to 4

Out[15]:

	0	1	2	3	4	5	6	7	8	9
1	83	30	8	55	43	62	82	74	12	5
2	83	95	28	33	95	28	7	32	6	2
3	17	25	82	3	65	39	73	63	6	49

In [16]:

df[0:4][1] # selecting rows index 0 to 4, column index 1

Out[16]:

0     6
1    30
2    95
3    25
Name: 1, dtype: int32

In [17]:

df[1] # selecting column 1, rows 0 .. n

Out[17]:

0     6
1    30
2    95
3    25
4    63
5    39
6    27
7    19
8    65
9    36
Name: 1, dtype: int32

In [1]:

df[1][4] # selecting the value at column index 1, row index 4

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-d68189ac4f7e> in <module>()
----> 1 df[1][4] # selecting the value at column index 1, row index 4

NameError: name 'df' is not defined

The [] operator is merely a convenience and does not provide the functionality of iloc() and loc() discussed below.

iloc[]¶

iloc is an integer-based selector. As such, you will need to know the integer values of the row or column indices as necessary. You may see some correspondence with the [] selector, but it provides much more.

In [19]:

df.iloc[1] # row index 1, returns the full ROW

Out[19]:

0    83
1    30
2     8
3    55
4    43
5    62
6    82
7    74
8    12
9     5
Name: 1, dtype: int32

In [20]:

df.iloc[1,2] # row index 1, column index 2

Out[20]:

8

In [21]:

df.iloc[1,:] # row index 1, as above

Out[21]:

0    83
1    30
2     8
3    55
4    43
5    62
6    82
7    74
8    12
9     5
Name: 1, dtype: int32

In [22]:

df.iloc[1,2:4] # row index 2, column index 2:3

Out[22]:

2     8
3    55
Name: 1, dtype: int32

In [23]:

df.iloc[1:2,:] # row index 1:2, column index 0:-1 same as above

Out[23]:

	0	1	2	3	4	5	6	7	8	9
1	83	30	8	55	43	62	82	74	12	5

More sophisticated slicing¶

More sophisticated slicing can be done over integer indices. For example, if we wanted specific rows and columns slicing, we can something like the following. Remember that the first argument to the selector is the row and the second, the column.

In [24]:

df.iloc[1:2,2:5] # row index 1:2, column index 2:5

Out[24]:

	2	3	4
1	8	55	43

In [25]:

df.iloc[[1,3,7], [2,5,6]] # row indices 1,3,7 column indices 2,5,6

Out[25]:

	2	5	6
1	8	62	82
3	82	39	73
7	48	35	41

loc()¶

loc() is a label-based selector, and provides a much richer experience in selecting data. It improves the overall readibility of analysis code and also allows for multi-indices to become more easily understood, as complex multi-indices that are numeric are often difficult to follow when complexity increases.

We will create a DataFrame with explicit index and column labels, filling in the values of our previous DataFrame df above.

In [26]:

df_si = pd.DataFrame(df.values, 
                     index=['r{}'.format(i) for i in range(0,10)],
                     columns=['c{}'.format(i) for i in range(0,10)])

In [27]:

df_si

Out[27]:

	c0	c1	c2	c3	c4	c5	c6	c7	c8	c9
r0	58	6	91	11	22	29	25	36	55	87
r1	83	30	8	55	43	62	82	74	12	5
r2	83	95	28	33	95	28	7	32	6	2
r3	17	25	82	3	65	39	73	63	6	49
r4	13	63	18	86	29	35	97	24	71	50
r5	33	39	84	85	15	42	68	45	26	69
r6	55	27	17	44	78	19	38	63	31	60
r7	41	19	48	36	92	35	41	97	98	10
r8	59	65	67	58	36	84	8	45	16	76
r9	56	36	98	63	73	54	36	61	56	73

As a convenience, the [] selector is capable of also dealing with labels, though we will not go any further than this basic example.

In [28]:

df_si['r3':'r6']['c4']

Out[28]:

r3    65
r4    29
r5    15
r6    78
Name: c4, dtype: int32

Selecting contiguous slices (indices are sorted), is very straightforward.

In [29]:

df_si.loc['r3':'r4',]

Out[29]:

	c0	c1	c2	c3	c4	c5	c6	c7	c8	c9
r3	17	25	82	3	65	39	73	63	6	49
r4	13	63	18	86	29	35	97	24	71	50

As expected we can slice the columns as well.

In [30]:

df_si.loc['r3':'r4', 'c2':'c3']

Out[30]:

	c2	c3
r3	82	3
r4	18	86

This wraps up our basic discussion of selection, we will return to this discussion in part 3, when we talk about more complex boolean slicing and multi-index slicing. Ξ