Text Mining Basics

Stopword Filter

Keep Content Words

Stopword filtering removes common words so the remaining tokens carry more topic signal.

Program

Play the script to choose a stopword set and compare the remaining content tokens.

stopword_filter.R 
tokens <- c("data", "is", "useful", "when", "data", "answers", "questions")
stop_choice <- 
stop_sets <- list(c("is", "when"), c("is", "when", "data"), c("is", "when", "questions"))
stopwords <- stop_sets[[stop_choice]]
content <- tokens[!(tokens %in% stopwords)]
summary <- paste(content, collapse = ",")
cat(summary, "\n", sep = "")

tokens <- c("data", "is", "useful", "when", "data", "answers", "questions")
stop_choice <- 
stop_sets <- list(c("is", "when"), c("is", "when", "data"), c("is", "when", "questions"))
stopwords <- stop_sets[[stop_choice]]
content <- tokens[!(tokens %in% stopwords)]
summary <- paste(content, collapse = ",")
cat(summary, "\n", sep = "")

tokens <- c("data", "is", "useful", "when", "data", "answers", "questions")
stop_choice <- 
stop_sets <- list(c("is", "when"), c("is", "when", "data"), c("is", "when", "questions"))
stopwords <- stop_sets[[stop_choice]]
content <- tokens[!(tokens %in% stopwords)]
summary <- paste(content, collapse = ",")
cat(summary, "\n", sep = "")
stopwords Stopwords are tokens the analysis chooses to ignore.
list `stop_sets[[stop_choice]]` picks one candidate vector from a list.
%in% `tokens %in% stopwords` marks which tokens should be removed.