M07 - Series vs DataFrame; tidy data

CSCI 1109 — Practical Data Science

Frank Rudzicz

Where we are

  • We’ve finished the Orientation & Python foundations cluster (C01).
  • You’ve seen:
    • Basic Python (variables, types, control flow, functions).
    • Doctests and a first taste of debugging.
  • Next cluster (C02) is about pandas: the main tool we’ll use for tabular data.

Note

In most real data work, although fancy ML models may be your focus, to train those models correctly will require a fair amount of ‘data wrangling’ of the type we talk about here.

Outcomes

By the end of this module, you should be able to:

  • Explain the difference between a Series and a DataFrame.
  • Construct small Series/DataFrames from lists and dicts.
  • Use .head(), .shape, .columns, and .index to quickly inspect a DataFrame.
  • Explain what tidy data means:
    each variable a column, each observation a row, each unit a table.
  • Recognize when a table is not tidy, and describe how you’d fix it (even if you don’t write all the code yet).

Tables, in the wild

Real-world tables everywhere

Real projects you might work on:

  • A city analysing bike-share trips:
    • trip_id, start_time, end_time, start_station, end_station, user_type, price.
  • A clinic tracking patient vitals:
    • patient_id, date, systolic_bp, diastolic_bp, heart_rate.
  • An app tracking daily steps and sleep:
    • user_id, date, steps, sleep_hours, city.

All of these are tables: rows are records, columns are variables.

Tip

pandas gives us a language for working with tables in Python.

Pandas in one slide

🧠 Quick mental model

  • pandas.Series → 1D, labelled vector of values.
  • pandas.DataFrame → 2D, table made of aligned Series.
  • Every Series/DataFrame has:
    • an index (labels for rows),
    • optional names for columns and Series.
Code 
import pandas as pd

steps = pd.Series(
    [8123, 10450, 9800],
    index=["Mon", "Tue", "Wed"],
    name="steps"
)
steps
Mon     8123
Tue    10450
Wed     9800
Name: steps, dtype: int64

Note

For now: think “Series = column with labels”, “DataFrame = whole table”.

DataFrames as tables

Constructing a tiny DataFrame

Code 
import pandas as pd

data = {
    "user_id": [101, 102, 103],
    "city": ["Halifax", "Halifax", "Toronto"],
    "steps": [8123, 10450, 9800],
}

df = pd.DataFrame(data)
df
user_id city steps
0 101 Halifax 8123
1 102 Halifax 10450
2 103 Toronto 9800

We can quickly inspect:

Code 
df.shape      # (rows, columns)
df.columns    # Index of column names
df.index      # row labels (here: 0, 1, 2 by default)
df.head()     # first 5 rows
user_id city steps
0 101 Halifax 8123
1 102 Halifax 10450
2 103 Toronto 9800

Tip

When you get a new dataset, your first moves should feel like saying hello: look at .head(), .shape, .columns, and .dtypes (more on dtypes in M08).

Series vs DataFrame: a story

Same information, different shapes

Code 
steps = pd.Series(
    [8123, 10450, 9800],
    index=["Mon", "Tue", "Wed"],
    name="steps"
)

df_steps = steps.to_frame()  # turn Series into 1-column DataFrame
  • steps is a Series: just one column of data with an index.
  • df_steps is a DataFrame: still one column, but:
    • we can now add more columns (sleep, city, etc.).
    • many pandas functions expect a DataFrame.

✏️

When is it enough to use a Series, and when do I want a full DataFrame?

Tidy data: the core idea

Tidy vs messy

Tidy data (from 🔗 Hadley Wickham):
- Each variable → a column.
- Each observation → a row.
- Each type of observational unit → a table.

Example (daily app data):

Tidy:

user_id date city steps sleep_hours
101 2025-09-01 Halifax 8123 7.5
102 2025-09-01 Halifax 10450 6.8
101 2025-09-02 Halifax 9800 7.9

Messy example (wide):

user_id city steps_2025_09_01 steps_2025_09_02 sleep_1 sleep_2
101 Halifax 8123 9800 7.5 7.9
102 Halifax 10450 NaN 6.8 NaN

Warning

Both tables contain the same information, but tidy tables are much easier to: - filter, - group, - visualise, - and feed into models.

Case study: student stress & sleep

A realistic mini dataset

Imagine a campus study:

  • students log their sleep hours and stress level each day for 2 weeks.
  • we also know their program (CS, Biology, etc.) and year.

A tidy table might look like:

student_id date program year sleep_hours stress_level
201 2025-09-01 CS 1 6.0 7
201 2025-09-02 CS 1 7.5 5
305 2025-09-01 Biology 2 8.0 3

Questions we can answer easily from tidy data:

  • “On average, how many hours do first-year CS students sleep?”
  • “Is average stress higher on weekdays than weekends?”
  • “Do students with < 7 hours sleep report higher stress?”

In later modules, we’ll use pandas to actually run these summaries.

Wide ↔︎ tidy: steps example

Wide format in code

Code 
steps_wide = pd.DataFrame({
    "user_id": [101, 102, 103],
    "steps_mon": [8123, 10450, 9800],
    "steps_tue": [9800, 9000, 12050],
    "steps_wed": [7500, 10100, 11500],
})
steps_wide
user_id steps_mon steps_tue steps_wed
0 101 8123 9800 7500
1 102 10450 9000 10100
2 103 9800 12050 11500

This is nice for one person per row, but awkward if we want:

  • average steps per day-of-week,
  • a line plot of steps vs day.

Hinting at tidy

Code 
steps_tidy = pd.melt(
    steps_wide,
    id_vars="user_id",
    var_name="day",
    value_name="steps",
)
steps_tidy
user_id day steps
0 101 steps_mon 8123
1 102 steps_mon 10450
2 103 steps_mon 9800
3 101 steps_tue 9800
4 102 steps_tue 9000
5 103 steps_tue 12050
6 101 steps_wed 7500
7 102 steps_wed 10100
8 103 steps_wed 11500

Now we have:

  • one row per (user_id, day),
  • a single steps column.

We’ll get more systematic about reshaping in M10, but it’s useful to see the idea now.

Checking your understanding

Quick questions to self-check

  • In tidy data, what does a row represent?
  • If you see columns like temp_morning, temp_afternoon, temp_evening, what might a tidy version look like?
  • For our step-tracking app:
    • Would you store city as a column or split it into multiple tables?
    • Why?

Note

You don’t need to memorise every pandas method yet; just make sure the concept of tidy tables feels clear.

Looking ahead in C02 (pandas)

Where this is going

  • M08 — Loading real datasets (CSV/JSON); describe(), info(), data types.
  • M09 — Indexing, selection, boolean masks.
  • M10 — groupby, aggregations, windows, pivots and reshaping.
  • M11 — A bigger pandas case study (e.g., flights or movies).

Your job after this module:

  • Be comfortable with Series vs DataFrame.
  • Be able to say whether a table is tidy or messy, and why that matters.
  • Be a little curious about how to transform messy → tidy, even if some of the code is still new.