Text Mining Basics

Token Cleanup

Text to Words

A text-mining workflow often starts by normalizing punctuation and case, then splitting text into word tokens.

Program

Play the script to change the minimum token length and see which words remain.

token_cleanup.R 
text <- "Data science uses data, code, and questions"
words <- strsplit(tolower(gsub("[^A-Za-z ]", "", text)), "\\s+")[[1]]
min_chars <- 
kept <- words[nchar(words) >= min_chars]
label <- paste(kept, collapse = ",")
cat(label, "\n", sep = "")

text <- "Data science uses data, code, and questions"
words <- strsplit(tolower(gsub("[^A-Za-z ]", "", text)), "\\s+")[[1]]
min_chars <- 
kept <- words[nchar(words) >= min_chars]
label <- paste(kept, collapse = ",")
cat(label, "\n", sep = "")

text <- "Data science uses data, code, and questions"
words <- strsplit(tolower(gsub("[^A-Za-z ]", "", text)), "\\s+")[[1]]
min_chars <- 
kept <- words[nchar(words) >= min_chars]
label <- paste(kept, collapse = ",")
cat(label, "\n", sep = "")
gsub `gsub("[^A-Za-z ]", "", text)` removes punctuation for this small example.
tolower `tolower` normalizes words before counting or filtering.
nchar `nchar(words) >= min_chars` filters tokens by length.