---
title: "Working with Strings using stringr"
author: "IND218"
date: today
format:
html:
toc: true
toc-depth: 3
code-fold: false
code-tools: true
---
## Introduction to String Manipulation
Text data is everywhere in real-world datasets - names, addresses, comments, categories, and more. The `stringr` package provides a consistent, intuitive set of functions for working with strings that integrates seamlessly with the tidyverse.
```{r}
#| label: setup
#| message: false
library(tidyverse)
library(stringr)
# All stringr functions start with str_ for easy discovery
cat("Key stringr functions we'll explore:\n")
cat("- str_length(): get string length\n")
cat("- str_sub(): extract substrings\n")
cat("- str_detect(): find patterns\n")
cat("- str_replace(): replace patterns\n")
cat("- str_split(): split strings\n")
cat("- str_trim(): remove whitespace\n")
cat("- str_to_*(): change case\n")
```
## Basic String Operations
### String Length and Subsetting
```{r}
#| label: basic-operations
# Sample text data
customer_names <- c("John Smith", "Mary Johnson", "Bob O'Connor", "李小明", "José García")
product_codes <- c("ABC-123", "XYZ-456", "DEF-789", "GHI-012")
# Get string length
str_length(customer_names)
str_length(product_codes)
# Extract substrings
str_sub(customer_names, 1, 4) # First 4 characters
str_sub(customer_names, -5, -1) # Last 5 characters
str_sub(product_codes, 1, 3) # Extract prefix
str_sub(product_codes, 5, 7) # Extract suffix
# Modify substrings
modified_codes <- product_codes
str_sub(modified_codes, 1, 3) <- "NEW" # Replace first 3 characters
print(modified_codes)
```
### Case Conversion
```{r}
#| label: case-conversion
messy_names <- c("JOHN SMITH", "mary johnson", "Bob O'connor", "MARÍA garcía")
# Case conversion functions
str_to_lower(messy_names) # All lowercase
str_to_upper(messy_names) # All uppercase
str_to_title(messy_names) # Title Case
str_to_sentence(messy_names) # Sentence case
# Locale-aware conversion (important for international names)
str_to_title(messy_names, locale = "en")
str_to_title("maría garcía", locale = "es")
```
### Whitespace and Padding
```{r}
#| label: whitespace-padding
messy_text <- c(" John Smith ", "\tMary Johnson\n", "Bob Wilson", " ")
# Remove whitespace
str_trim(messy_text) # Remove leading/trailing
str_trim(messy_text, side = "left") # Remove only leading
str_trim(messy_text, side = "right") # Remove only trailing
str_squish(messy_text) # Remove all extra whitespace
# Add padding
names <- c("John", "Mary", "Bob")
str_pad(names, width = 10, side = "left", pad = " ")
str_pad(names, width = 10, side = "both", pad = "-")
str_pad(names, width = 8, side = "right", pad = ".")
```
## Pattern Detection and Matching
### Basic Pattern Detection
```{r}
#| label: pattern-detection
email_list <- c("john@email.com", "mary.wilson@company.org", "invalid-email",
"bob@test.co.uk", "sarah@domain", "alice@example.com")
# Detect patterns
str_detect(email_list, "@") # Contains @
str_detect(email_list, "\\.com") # Ends with .com
str_detect(email_list, "^[a-z]+@") # Starts with lowercase letters before @
# Count matches
str_count(email_list, "\\.") # Count dots
str_count(email_list, "[aeiou]") # Count vowels
# Find pattern locations
str_locate(email_list, "@") # First @ position
str_locate_all(email_list, "[aeiou]") # All vowel positions
```
### Working with Regular Expressions
```{r}
#| label: regex-patterns
# Sample data for regex practice
phone_numbers <- c("123-456-7890", "(555) 123-4567", "555.123.4567",
"1234567890", "+1-555-123-4567", "invalid")
# Basic regex patterns
str_detect(phone_numbers, "\\d{3}-\\d{3}-\\d{4}") # xxx-xxx-xxxx format
str_detect(phone_numbers, "\\(\\d{3}\\)") # Area code in parentheses
str_detect(phone_numbers, "^\\+1") # Starts with +1
# Extract parts using regex groups
phone_pattern <- "(\\d{3})[.-](\\d{3})[.-](\\d{4})"
str_extract(phone_numbers, phone_pattern)
# More complex patterns
email_pattern <- "([a-zA-Z0-9._%-]+)@([a-zA-Z0-9.-]+\\.[a-zA-Z]{2,})"
sample_emails <- c("user@domain.com", "test.email@company.co.uk", "invalid@", "@invalid.com")
str_detect(sample_emails, email_pattern)
str_extract(sample_emails, email_pattern)
```
### Common Regex Patterns
```{r}
#| label: common-regex
# Useful regex patterns for data cleaning
sample_text <- c("Call 555-123-4567 today!", "Email: user@test.com",
"Price: $29.99", "Date: 2024-01-15", "ID: ABC123XYZ")
# Phone numbers
phone_regex <- "\\b\\d{3}[.-]?\\d{3}[.-]?\\d{4}\\b"
str_extract(sample_text, phone_regex)
# Email addresses
email_regex <- "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b"
str_extract(sample_text, email_regex)
# Currency amounts
currency_regex <- "\\$\\d+\\.\\d{2}"
str_extract(sample_text, currency_regex)
# Dates (YYYY-MM-DD format)
date_regex <- "\\b\\d{4}-\\d{2}-\\d{2}\\b"
str_extract(sample_text, date_regex)
# Alphanumeric IDs
id_regex <- "\\b[A-Z]{3}\\d{3}[A-Z]{3}\\b"
str_extract(sample_text, id_regex)
```
## String Replacement and Transformation
### Basic Replacement
```{r}
#| label: string-replacement
messy_addresses <- c("123 Main St.", "456 Oak Ave", "789 Pine Street", "101 Elm St")
# Simple replacement
str_replace(messy_addresses, "St\\.", "Street") # Replace first match
str_replace_all(messy_addresses, "\\.", "") # Remove all periods
str_replace_all(messy_addresses, "St$", "Street") # Replace St at end
# Multiple replacements
cleanup_patterns <- c("St\\." = "Street", "Ave" = "Avenue", "\\b(\\d+)\\b" = "Number \\1")
str_replace_all(messy_addresses, cleanup_patterns)
# Case-insensitive replacement
str_replace_all(messy_addresses, regex("ST", ignore_case = TRUE), "Street")
```
### Advanced Replacement with Functions
```{r}
#| label: advanced-replacement
# Sample product descriptions
products <- c("LAPTOP-15INCH-8GB", "PHONE-ANDROID-64GB", "TABLET-IPAD-128GB")
# Replace with custom function
# Note: the function receives all matches at once, so it must be vectorised
str_replace_all(products, "\\b(\\d+)GB\\b", function(x) {
gb <- as.numeric(str_extract(x, "\\d+"))
case_when(
gb >= 128 ~ "High Storage",
gb >= 64 ~ "Medium Storage",
.default = "Low Storage"
)
})
# Clean and format product names
clean_products <- products %>%
str_replace_all("-", " ") %>%
str_to_title() %>%
str_replace_all("(\\d+)(Gb|GB)", "\\1 GB")
print(clean_products)
```
## String Splitting and Joining
### Splitting Strings
```{r}
#| label: string-splitting
# Sample data with delimiters
csv_data <- c("John,25,Engineer", "Mary,30,Manager", "Bob,28,Analyst")
pipe_data <- c("Apple|Red|Sweet", "Banana|Yellow|Sweet", "Lemon|Yellow|Sour")
# Split by delimiter
str_split(csv_data, ",") # Returns list
str_split(csv_data, ",", simplify = TRUE) # Returns matrix
str_split(pipe_data, "\\|", n = 2) # Limit to 2 pieces
# Split into tibble (very useful!)
str_split(csv_data, ",", simplify = TRUE) %>%
as_tibble(.name_repair = "minimal") %>%
set_names(c("name", "age", "job"))
# More complex splitting
full_names <- c("John A. Smith", "Mary Elizabeth Johnson", "Bob Wilson")
str_split(full_names, "\\s+") # Split on any whitespace
str_split(full_names, "\\s+", n = 2) # Split into first and last parts
```
### Joining Strings
```{r}
#| label: string-joining
# Sample data to join
first_names <- c("John", "Mary", "Bob")
last_names <- c("Smith", "Johnson", "Wilson")
titles <- c("Dr.", "Ms.", "Mr.")
# Basic joining
str_c(first_names, last_names, sep = " ") # Simple concatenation
str_c(titles, first_names, last_names, sep = " ") # Multiple parts
# Join with collapse (combine all into one string)
str_c(first_names, collapse = ", ") # "John, Mary, Bob"
str_c(first_names, collapse = " and ") # "John and Mary and Bob"
# Conditional joining (skip missing values)
mixed_data <- c("John", NA, "Mary", "", "Bob")
str_c("Name: ", mixed_data) # Creates NA
str_c("Name: ", mixed_data, sep = "") # Still creates NA
# Better approach with coalesce
str_c("Name: ", coalesce(mixed_data, "Unknown"))
```
## Real-World String Cleaning Examples
### Example 1: Cleaning Customer Data
```{r}
#| label: customer-data-cleaning
# Messy customer data
raw_customers <- tibble(
id = 1:6,
name = c(" JOHN SMITH ", "mary-johnson", "Bob O'Connor",
"SARAH WILSON", "mike.davis", "ANNA-MARIA GARCIA"),
email = c("JOHN@EMAIL.COM", "mary@company.org", "bob@COMPANY.com",
"sarah.wilson@test.CO.UK", "mike.davis@DOMAIN.com", "anna@example.COM"),
phone = c("(555) 123-4567", "555.123.4567", "5551234567",
"555-123-4567", "(555)123-4567", "555 123 4567"),
address = c("123 main st.", "456 OAK AVE", "789 pine street apt 2",
"101 ELM ST UNIT B", "202 maple ave.", "303 BIRCH ST")
)
print("Raw customer data:")
print(raw_customers)
# Clean the data step by step
cleaned_customers <- raw_customers %>%
mutate(
# Clean names
name_clean = name %>%
str_trim() %>% # Remove leading/trailing spaces
str_squish() %>% # Remove extra internal spaces
str_replace_all("[.-]", " ") %>% # Replace hyphens and dots with spaces
str_to_title(), # Convert to title case
# Clean emails
email_clean = email %>%
str_to_lower() %>% # Convert to lowercase
str_trim(), # Remove any spaces
# Clean phone numbers
phone_clean = phone %>%
str_replace_all("[^0-9]", "") %>% # Remove all non-digits
str_replace("(\\d{3})(\\d{3})(\\d{4})", "(\\1) \\2-\\3"), # Format as (xxx) xxx-xxxx
# Clean addresses
address_clean = address %>%
str_to_title() %>% # Convert to title case
str_replace_all("\\bSt\\b", "Street") %>% # Expand abbreviations
str_replace_all("\\bAve\\b", "Avenue") %>%
str_replace_all("\\bApt\\b", "Apartment") %>%
str_squish() # Clean up spaces
) %>%
select(id, name_clean, email_clean, phone_clean, address_clean)
print("Cleaned customer data:")
print(cleaned_customers)
```
### Example 2: Parsing Product Information
```{r}
#| label: product-parsing
# Product codes with embedded information
product_data <- tibble(
sku = c("LAPTOP-DELL-15INCH-8GB-256SSD-WIN11",
"PHONE-APPLE-IPHONE14-128GB-BLUE",
"TABLET-SAMSUNG-10INCH-64GB-ANDROID",
"LAPTOP-HP-14INCH-16GB-512SSD-WIN11",
"PHONE-SAMSUNG-GALAXY-256GB-BLACK"),
price = c("$1299.99", "$999.00", "$449.99", "$1599.99", "$799.99")
)
print("Raw product data:")
print(product_data)
# Parse information from SKU codes
parsed_products <- product_data %>%
mutate(
# Extract basic product type
product_type = str_extract(sku, "^[A-Z]+"),
# Extract brand
brand = str_extract(sku, "(?<=-)([A-Z]+)(?=-)"),
# Extract storage information
storage_info = str_extract(sku, "\\d+GB|\\d+SSD"),
storage_gb = as.numeric(str_extract(storage_info, "\\d+")),
storage_type = case_when(
str_detect(storage_info, "SSD") ~ "SSD",
str_detect(storage_info, "GB") ~ "Flash",
TRUE ~ "Unknown"
),
# Extract screen size for relevant products
screen_size = str_extract(sku, "\\d+INCH"),
screen_inches = as.numeric(str_extract(screen_size, "\\d+")),
# Clean price
price_numeric = as.numeric(str_replace_all(price, "[$,]", "")),
# Create clean product name
product_name = str_replace_all(sku, "-", " ") %>%
str_to_title() %>%
str_replace_all("\\b(\\d+)Gb\\b", "\\1 GB") %>%
str_replace_all("\\b(\\d+)Ssd\\b", "\\1 SSD") %>%
str_replace_all("\\b(\\d+)Inch\\b", "\\1 Inch")
) %>%
select(sku, product_name, product_type, brand, storage_gb, storage_type,
screen_inches, price_numeric)
print("Parsed product data:")
print(parsed_products)
# Summary analysis
cat("Product analysis:\n")
parsed_products %>%
group_by(product_type, brand) %>%
summarise(
count = n(),
avg_price = round(mean(price_numeric), 2),
avg_storage = round(mean(storage_gb, na.rm = TRUE), 0),
.groups = "drop"
) %>%
print()
```
### Example 3: Text Data Mining
```{r}
#| label: text-mining
# Customer feedback comments
feedback_data <- tibble(
customer_id = 1:8,
comment = c(
"Great product! Very satisfied with the quality and fast shipping.",
"Poor quality, arrived damaged. Customer service was unhelpful.",
"Good value for money. Would recommend to others.",
"Excellent! Fast delivery and great customer support.",
"Average product. Nothing special but does the job.",
"Terrible experience. Product broke after 2 days.",
"Love it! Best purchase I've made this year.",
"Okay product but overpriced. Customer service was good though."
),
rating = c(5, 1, 4, 5, 3, 1, 5, 3)
)
print("Customer feedback:")
print(feedback_data)
# Analyze sentiment and extract insights
feedback_analysis <- feedback_data %>%
mutate(
# Convert to lowercase for analysis
comment_clean = str_to_lower(comment),
# Extract sentiment indicators
positive_words = str_count(comment_clean, "\\b(great|excellent|good|love|best|satisfied|recommend)\\b"),
negative_words = str_count(comment_clean, "\\b(poor|terrible|bad|awful|hate|worst|disappointed|unhelpful)\\b"),
# Check for specific topics
mentions_quality = str_detect(comment_clean, "\\bquality\\b"),
mentions_shipping = str_detect(comment_clean, "\\b(shipping|delivery)\\b"),
mentions_service = str_detect(comment_clean, "\\b(service|support)\\b"),
mentions_price = str_detect(comment_clean, "\\b(price|value|money|expensive|cheap|overpriced)\\b"),
# Calculate sentiment score
sentiment_score = positive_words - negative_words,
# Classify sentiment
sentiment = case_when(
sentiment_score > 0 ~ "Positive",
sentiment_score < 0 ~ "Negative",
TRUE ~ "Neutral"
),
# Extract key phrases (simple approach)
key_phrases = str_extract_all(comment_clean, "\\b(very|extremely|really)\\s+\\w+"),
# Word count
word_count = str_count(comment_clean, "\\S+")
)
print("Feedback analysis:")
feedback_analysis %>%
select(customer_id, rating, sentiment, sentiment_score, mentions_quality,
mentions_shipping, mentions_service, mentions_price, word_count) %>%
print()
# Summary insights
cat("\nSentiment Analysis Summary:\n")
feedback_analysis %>%
group_by(sentiment) %>%
summarise(
count = n(),
avg_rating = round(mean(rating), 1),
avg_word_count = round(mean(word_count), 1)
) %>%
print()
cat("\nTopic Mentions:\n")
feedback_analysis %>%
summarise(
quality_mentions = sum(mentions_quality),
shipping_mentions = sum(mentions_shipping),
service_mentions = sum(mentions_service),
price_mentions = sum(mentions_price)
) %>%
print()
```
## Advanced String Techniques
### Working with Encoding and Special Characters
```{r}
#| label: encoding-special-chars
# Text with special characters and different encodings
international_text <- c("Café", "naïve", "résumé", "piñata", "Zürich", "北京", "Москва")
# String length (note: may differ from character count for some encodings)
str_length(international_text)
nchar(international_text)
# Detect and handle different character types
str_detect(international_text, "[^\\x00-\\x7F]") # Non-ASCII characters
str_detect(international_text, "\\p{L}") # Unicode letters
str_detect(international_text, "\\p{Han}") # Chinese characters
# Normalize text (remove accents)
# Note: This is a simplified example; real normalization needs specialized packages
simplified_text <- international_text %>%
str_replace_all("[àáâãäå]", "a") %>%
str_replace_all("[èéêë]", "e") %>%
str_replace_all("[ìíîï]", "i") %>%
str_replace_all("[òóôõö]", "o") %>%
str_replace_all("[ùúûü]", "u") %>%
str_replace_all("[ñ]", "n")
print(data.frame(original = international_text, simplified = simplified_text))
```
### String Validation
```{r}
#| label: string-validation
# Create validation functions using stringr
validate_email <- function(email) {
pattern <- "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
str_detect(email, pattern)
}
validate_phone <- function(phone) {
# Remove all non-digits and check if 10 digits remain
digits_only <- str_replace_all(phone, "[^0-9]", "")
str_length(digits_only) == 10
}
validate_postal_code <- function(postal_code, country = "US") {
if (country == "US") {
# US ZIP code: 12345 or 12345-6789
str_detect(postal_code, "^\\d{5}(-\\d{4})?$")
} else if (country == "CA") {
# Canadian postal code: A1A 1A1
str_detect(postal_code, "^[A-Z]\\d[A-Z]\\s?\\d[A-Z]\\d$")
} else if (country == "UK") {
# UK postal code (simplified): AA1 1AA or A1 1AA
str_detect(postal_code, "^[A-Z]{1,2}\\d{1,2}\\s?\\d[A-Z]{2}$")
}
}
# Test validation functions
test_emails <- c("user@domain.com", "invalid.email", "test@company.co.uk", "bad@")
test_phones <- c("(555) 123-4567", "555.123.4567", "12345", "555-123-4567")
test_postal <- c("12345", "12345-6789", "ABC", "90210-1234")
validation_results <- tibble(
email = test_emails,
email_valid = validate_email(test_emails),
phone = test_phones,
phone_valid = validate_phone(test_phones),
postal = test_postal,
postal_valid = validate_postal_code(test_postal, "US")
)
print(validation_results)
```
## Best Practices and Common Pitfalls
### Performance Tips
```{r}
#| label: performance-tips
# For large datasets, vectorized operations are much faster
large_text <- rep(c("Hello World", "Data Science", "R Programming"), 1000)
# Good: Use vectorized stringr functions
system.time({
result1 <- str_to_upper(large_text)
})
# Avoid: Loops for simple operations
system.time({
result2 <- character(length(large_text))
for (i in seq_along(large_text)) {
result2[i] <- toupper(large_text[i])
}
})
# Pre-compile regex patterns for repeated use
email_pattern <- regex("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")
test_emails <- rep(c("user@domain.com", "invalid"), 500)
system.time({
result3 <- str_detect(test_emails, email_pattern)
})
```
### Common Mistakes and Solutions
```{r}
#| label: common-mistakes
# Mistake 1: Forgetting to escape special regex characters
test_text <- c("price: $29.99", "cost: $15.50", "value: 10.00")
# Wrong: . matches any character in regex
str_detect(test_text, "$29.99") # This won't work as expected
# Right: Escape special characters
str_detect(test_text, "\\$29\\.99")
# Mistake 2: Not handling missing values
messy_data <- c("John Smith", NA, "Mary Johnson", "")
# This will create NA for missing values
str_to_upper(messy_data)
# Better: Handle missing values explicitly
coalesce(str_to_upper(messy_data), "UNKNOWN")
# Mistake 3: Assuming consistent formats
mixed_dates <- c("2024-01-15", "01/15/2024", "Jan 15, 2024")
# Wrong: Assuming one format
str_extract(mixed_dates, "\\d{4}-\\d{2}-\\d{2}")
# Better: Handle multiple formats
case_when(
str_detect(mixed_dates, "\\d{4}-\\d{2}-\\d{2}") ~ "ISO format",
str_detect(mixed_dates, "\\d{2}/\\d{2}/\\d{4}") ~ "US format",
str_detect(mixed_dates, "[A-Za-z]+ \\d{1,2}, \\d{4}") ~ "Written format",
TRUE ~ "Unknown format"
)
```
## Exercises
### Exercise 1: Email Validation and Cleaning
Given a list of messy email addresses, write code to:
1. Clean and standardize the format
2. Identify valid vs invalid emails
3. Extract domain names and categorize them
### Exercise 2: Product Name Standardization
You have product names in various formats. Create functions to:
1. Extract product categories, brands, and specifications
2. Standardize naming conventions
3. Identify products with missing information
### Exercise 3: Text Analysis
Analyze customer reviews to:
1. Calculate sentiment scores based on positive/negative words
2. Extract key features mentioned (price, quality, service, etc.)
3. Identify the most common complaint topics
### Exercise 4: Data Validation Pipeline
Create a comprehensive data validation system that:
1. Validates phone numbers, emails, and postal codes
2. Standardizes name formats
3. Flags potentially problematic entries for manual review
## Summary
The `stringr` package is essential for working with text data in R:
### Key Functions:
- **Basic operations**: `str_length()`, `str_sub()`, `str_trim()`, `str_pad()`
- **Case conversion**: `str_to_lower()`, `str_to_upper()`, `str_to_title()`
- **Pattern matching**: `str_detect()`, `str_count()`, `str_locate()`
- **Replacement**: `str_replace()`, `str_replace_all()`
- **Splitting/joining**: `str_split()`, `str_c()`
### Best Practices:
- **Use vectorized operations** for performance
- **Escape special regex characters** properly
- **Handle missing values** explicitly
- **Test regex patterns** thoroughly
- **Consider internationalization** for global data
### Regular Expressions:
- Learn common patterns for emails, phones, dates
- Use `regex()` function for case-insensitive matching
- Practice with simple patterns before complex ones
- Test patterns with edge cases
String manipulation is a crucial skill for data cleaning and preparation. With `stringr`, you can handle even the messiest text data efficiently and elegantly!
Next: **[Managing Factors with forcats](factors-forcats.qmd)**