---
title: "Lists and Data Frames"
author: "IND218"
date: today
format:
html:
toc: true
toc-depth: 3
code-fold: false
code-tools: true
---
## Introduction to Complex Data Structures
While vectors are fundamental, real-world data analysis often requires more complex structures. In this section, we'll explore two essential data structures that build upon vectors:
- **Lists**: Flexible containers that can hold different data types
- **Data frames**: The backbone of data analysis, organizing data in rows and columns
## Lists: Flexible Data Containers
### What are Lists?
Lists are collections that can contain elements of **different types**. Unlike vectors, which must contain elements of the same type, lists can mix numbers, characters, logicals, and even other lists!
### Creating Lists
```{r}
#| label: list-creation
# Basic list with different types
my_list <- list(
numbers = c(1, 2, 3, 4),
text = "Hello, world!",
logical_value = TRUE,
single_number = 42
)
print(my_list)
# Lists can contain vectors of different lengths
mixed_list <- list(
short_vector = c(1, 2),
long_vector = c("a", "b", "c", "d", "e"),
single_value = 3.14
)
print(mixed_list)
```
### List Structure and Properties
```{r}
#| label: list-structure
# Examine list structure
str(my_list)
# Check if it's a list
is.list(my_list)
# Get list length (number of elements)
length(my_list)
# Get element names
names(my_list)
```
### Complex Lists
Lists can contain other lists and complex objects:
```{r}
#| label: complex-lists
# List containing other lists
nested_list <- list(
personal_info = list(
name = "Alice Johnson",
age = 28,
city = "Boston"
),
scores = c(85, 92, 78, 96),
metadata = list(
date_created = Sys.Date(),
version = 1.0
)
)
str(nested_list)
```
### Accessing List Elements
There are three main ways to access list elements:
#### Using `[[]]` (Double Brackets)
```{r}
#| label: list-double-brackets
# Extract single elements (returns the actual object)
my_list[[1]] # First element
my_list[["numbers"]] # Element named "numbers"
my_list$numbers # Same as above, using $ notation
# Check types
typeof(my_list[[1]]) # Returns "double" (the vector)
typeof(my_list[1]) # Returns "list" (a list with one element)
```
#### Using `[]` (Single Brackets)
```{r}
#| label: list-single-brackets
# Returns a list containing the selected elements
my_list[1] # Returns a list with the first element
my_list[c(1, 3)] # Returns a list with elements 1 and 3
my_list[c("numbers", "text")] # Returns a list with named elements
# The difference is important!
class(my_list[[1]]) # "numeric" - the actual vector
class(my_list[1]) # "list" - a list containing the vector
```
#### Using `$` (Dollar Sign)
```{r}
#| label: list-dollar-sign
# Access by name (most common for named lists)
my_list$numbers
my_list$text
my_list$logical_value
# This only works with named elements
# my_list$1 # This would cause an error
```
### Modifying Lists
```{r}
#| label: list-modification
# Create a list to modify
student_data <- list(
name = "Bob Smith",
grades = c(85, 90, 78)
)
# Add new elements
student_data$age <- 20
student_data$major <- "Statistics"
# Modify existing elements
student_data$grades <- c(student_data$grades, 92) # Add new grade
# Remove elements (set to NULL)
student_data$age <- NULL
print(student_data)
```
### List Functions
```{r}
#| label: list-functions
# Create sample list
data_list <- list(
group_a = c(23, 25, 27, 29),
group_b = c(18, 22, 24, 26, 28),
group_c = c(30, 32, 34)
)
# Apply function to each element
lapply(data_list, mean) # Returns a list
sapply(data_list, mean) # Returns a vector when possible
vapply(data_list, mean, numeric(1)) # Specify output type
# Get lengths of each element
lapply(data_list, length)
sapply(data_list, length)
# Combine lists
list1 <- list(a = 1, b = 2)
list2 <- list(c = 3, d = 4)
combined <- c(list1, list2)
print(combined)
```
## Data Frames: The Heart of Data Analysis
### What are Data Frames?
Data frames are the most important data structure for data analysis. Think of them as spreadsheets or database tables - they organize data in rows and columns where:
- Each **column** represents a variable (like age, income, name)
- Each **row** represents an observation (like a person, transaction, measurement)
- All columns have the same length
- Different columns can contain different data types
### Creating Data Frames
```{r}
#| label: dataframe-creation
# Basic data frame
students <- data.frame(
name = c("Alice", "Bob", "Charlie", "Diana"),
age = c(20, 22, 19, 21),
major = c("Psychology", "Math", "Biology", "English"),
gpa = c(3.8, 3.6, 3.9, 3.7),
stringsAsFactors = FALSE # Keep strings as characters
)
print(students)
# Check structure
str(students)
```
### Data Frame Properties
```{r}
#| label: dataframe-properties
# Dimensions
nrow(students) # Number of rows
ncol(students) # Number of columns
dim(students) # Both dimensions
# Column and row names
colnames(students)
rownames(students)
names(students) # Same as colnames for data frames
# Quick overview
head(students) # First 6 rows (default)
tail(students) # Last 6 rows
summary(students) # Summary statistics
```
### Accessing Data Frame Elements
#### Column Access
```{r}
#| label: dataframe-column-access
# Access columns (multiple ways)
students$name # Using $
students[["name"]] # Using [[]]
students["name"] # Returns data frame with one column
students[, "name"] # Using row, column notation
# Multiple columns
students[c("name", "gpa")]
students[, c("name", "gpa")]
# All columns
students[, ] # All rows and columns
```
#### Row Access
```{r}
#| label: dataframe-row-access
# Access rows
students[1, ] # First row, all columns
students[c(1, 3), ] # Rows 1 and 3, all columns
students[1:2, ] # Rows 1 through 2
# Specific cells
students[1, "name"] # Row 1, column "name"
students[1, 2] # Row 1, column 2
students[c(1, 3), c("name", "gpa")] # Multiple rows and columns
```
#### Conditional Subsetting
```{r}
#| label: dataframe-conditional
# Filter rows based on conditions
high_gpa <- students[students$gpa > 3.7, ]
print(high_gpa)
# Multiple conditions
young_high_achievers <- students[students$age < 21 & students$gpa > 3.8, ]
print(young_high_achievers)
# Using subset() function (alternative)
subset(students, gpa > 3.7)
subset(students, age < 21 & gpa > 3.8)
# Filter and select specific columns
subset(students, gpa > 3.7, select = c("name", "gpa"))
```
### Modifying Data Frames
#### Adding Columns
```{r}
#| label: dataframe-add-columns
# Add new columns
students$year <- c("Sophomore", "Senior", "Freshman", "Junior")
students$credits <- c(45, 120, 15, 90)
# Calculate new columns from existing ones
students$gpa_category <- ifelse(students$gpa >= 3.8, "High",
ifelse(students$gpa >= 3.5, "Medium", "Low"))
print(students)
```
#### Adding Rows
```{r}
#| label: dataframe-add-rows
# Create new student data
new_student <- data.frame(
name = "Eve",
age = 20,
major = "Computer Science",
gpa = 3.95,
year = "Sophomore",
credits = 50,
gpa_category = "High",
stringsAsFactors = FALSE
)
# Add to existing data frame
students_expanded <- rbind(students, new_student)
print(students_expanded)
```
#### Modifying Values
```{r}
#| label: dataframe-modify-values
# Modify specific values
students$age[1] <- 21 # Change Alice's age
# Modify based on conditions
students$gpa[students$name == "Bob"] <- 3.65 # Update Bob's GPA
# Modify multiple values
students$credits <- students$credits + 3 # Add 3 credits to everyone
print(students)
```
### Data Frame Operations
#### Sorting
```{r}
#| label: dataframe-sorting
# Sort by GPA (ascending)
students_by_gpa <- students[order(students$gpa), ]
print(students_by_gpa)
# Sort by GPA (descending)
students_by_gpa_desc <- students[order(-students$gpa), ]
print(students_by_gpa_desc)
# Sort by multiple columns
students_sorted <- students[order(students$major, -students$gpa), ]
print(students_sorted)
```
#### Aggregation
```{r}
#| label: dataframe-aggregation
# Basic statistics
mean(students$gpa)
max(students$age)
min(students$credits)
# Statistics by group
aggregate(gpa ~ major, data = students, FUN = mean)
aggregate(age ~ gpa_category, data = students, FUN = mean)
# Count by category
table(students$major)
table(students$gpa_category)
```
## Working with Real Data
### Example 1: Sales Analysis
```{r}
#| label: sales-analysis
# Create sales data
sales_data <- data.frame(
date = as.Date(c("2024-01-01", "2024-01-02", "2024-01-03", "2024-01-04", "2024-01-05")),
product = c("Widget A", "Widget B", "Widget A", "Widget C", "Widget B"),
quantity = c(10, 15, 8, 12, 20),
price = c(25.99, 35.50, 25.99, 45.00, 35.50),
sales_rep = c("Alice", "Bob", "Alice", "Charlie", "Bob")
)
# Calculate revenue
sales_data$revenue <- sales_data$quantity * sales_data$price
# Add day of week
sales_data$day_of_week <- weekdays(sales_data$date)
print(sales_data)
# Analysis
total_revenue <- sum(sales_data$revenue)
best_day <- sales_data[which.max(sales_data$revenue), ]
revenue_by_rep <- aggregate(revenue ~ sales_rep, data = sales_data, FUN = sum)
cat("Total revenue: $", round(total_revenue, 2), "\n")
print("Best performing day:")
print(best_day)
print("Revenue by sales rep:")
print(revenue_by_rep)
```
### Example 2: Student Performance Analysis
```{r}
#| label: student-performance
# Create comprehensive student data
student_performance <- data.frame(
student_id = 1:20,
name = paste("Student", LETTERS[1:20]),
math_score = round(rnorm(20, mean = 85, sd = 10)),
science_score = round(rnorm(20, mean = 82, sd = 12)),
english_score = round(rnorm(20, mean = 88, sd = 8)),
attendance_rate = round(runif(20, min = 0.7, max = 1.0), 2),
study_hours = round(runif(20, min = 5, max = 25))
)
# Calculate derived variables
student_performance$average_score <- rowMeans(student_performance[, c("math_score", "science_score", "english_score")])
student_performance$grade <- cut(student_performance$average_score,
breaks = c(0, 60, 70, 80, 90, 100),
labels = c("F", "D", "C", "B", "A"))
# Identify high performers
high_performers <- subset(student_performance,
average_score >= 90 & attendance_rate >= 0.9)
# Performance by study hours
cor(student_performance$study_hours, student_performance$average_score)
print("High performers:")
print(high_performers[, c("name", "average_score", "attendance_rate", "study_hours")])
print("Grade distribution:")
print(table(student_performance$grade))
```
## Combining Lists and Data Frames
Sometimes you need to store data frames within lists or create complex nested structures:
```{r}
#| label: combined-structures
# List containing multiple data frames
analysis_results <- list(
raw_data = student_performance,
summary_stats = data.frame(
metric = c("Mean Score", "Median Score", "Std Dev", "Max Score"),
value = c(mean(student_performance$average_score),
median(student_performance$average_score),
sd(student_performance$average_score),
max(student_performance$average_score))
),
high_performers = high_performers,
metadata = list(
analysis_date = Sys.Date(),
total_students = nrow(student_performance),
subjects_analyzed = c("math", "science", "english")
)
)
# Access different components
str(analysis_results, max.level = 2)
print(analysis_results$summary_stats)
print(analysis_results$metadata$total_students)
```
## Best Practices and Common Pitfalls
### 1. Data Frame vs List Choice
```{r}
#| label: structure-choice
# Use data frames when:
# - All columns have the same length
# - You're working with rectangular data
# - You need to analyze relationships between variables
# Use lists when:
# - Elements have different lengths
# - You need to store heterogeneous objects
# - You're collecting results from multiple analyses
```
### 2. Factor Handling
```{r}
#| label: factor-handling
# Be careful with factors in older R versions
# Always use stringsAsFactors = FALSE when creating data frames
# Or convert factors to characters when needed
students_safe <- data.frame(
name = c("Alice", "Bob", "Charlie"),
major = c("Math", "Science", "Art"),
stringsAsFactors = FALSE
)
# Check if columns are factors
sapply(students_safe, class)
```
### 3. Consistent Data Types
```{r}
#| label: consistent-types
# Ensure consistent data types within columns
mixed_ages <- c(20, "twenty-one", 22) # This becomes all character!
print(mixed_ages)
# Better approach
clean_ages <- c(20, 21, 22)
special_case <- "unknown" # Handle special cases separately
```
## Exercises
### Exercise 1: List Practice
1. Create a list containing:
- A vector of your favorite colors
- Your age
- A logical value indicating if you like pizza
- A nested list with your address information
2. Practice accessing elements using different methods (`$`, `[[]]`, `[]`)
### Exercise 2: Data Frame Creation
Create a data frame with information about 10 movies including:
- Title, year, genre, rating (1-10), duration (minutes)
- Add calculated columns for decade and rating category
- Find movies from a specific decade with high ratings
### Exercise 3: Real-world Analysis
Given this employee data structure:
```{r}
#| eval: false
employees <- data.frame(
name = c("Alice", "Bob", "Charlie", "Diana", "Eve"),
department = c("Sales", "IT", "Sales", "HR", "IT"),
salary = c(50000, 75000, 55000, 60000, 80000),
years_experience = c(3, 7, 4, 5, 9),
performance_rating = c(4.2, 4.8, 3.9, 4.5, 4.9)
)
```
1. Calculate average salary by department
2. Find the highest paid employee in each department
3. Identify employees with above-average performance ratings
4. Create a bonus column (10% of salary for ratings > 4.5)
## Summary
Lists and data frames are essential data structures in R:
### Lists
- **Flexible**: Can contain different data types and structures
- **Nested**: Can contain other lists for complex hierarchies
- **Access**: Use `$`, `[[]]` for elements, `[]` for sub-lists
- **Use case**: Storing analysis results, configuration data, complex objects
### Data Frames
- **Rectangular**: Rows and columns like a spreadsheet
- **Mixed types**: Different columns can have different types
- **Consistent length**: All columns must have the same number of rows
- **Use case**: Primary structure for data analysis
### Key Operations
- **Creation**: `list()` and `data.frame()`
- **Access**: Multiple indexing methods for flexibility
- **Modification**: Add, remove, or change elements and columns
- **Analysis**: Built-in functions for summaries and aggregation
These structures form the foundation for data manipulation and analysis. Next, we'll learn about control structures to add logic and iteration to our data processing workflows!