The goal of {excluder} is to facilitate checking for, marking, and excluding rows of data frames for common exclusion criteria. This package applies to data collected from Qualtrics surveys, and default column names come from importing data with the {qualtRics} package.
This may be most useful for Mechanical Turk data to screen for duplicate entries from the same location/IP address or entries from locations outside of the United States. But it can be used more generally to exclude based on response durations, preview status, progress, or screen resolution.
More details are available on the package website and the getting started vignette.
Installation
You can install the stable released version of excluder from CRAN with:
install.packages("excluder")You can install developmental versions from GitHub with:
# install.packages("remotes")
remotes::install_github("ropensci/excluder")Verbs
This package provides three primary verbs:
-
markfunctions add a new column to the original data frame that labels the rows meeting the exclusion criteria. This is useful to label the potential exclusions for future processing without changing the original data frame. -
checkfunctions search for the exclusion criteria and output a message with the number of rows meeting the criteria and a data frame of the rows meeting the criteria. This is useful for viewing the potential exclusions. -
excludefunctions remove rows meeting the exclusion criteria. This is safest to do after checking the rows to ensure the exclusions are correct.
Exclusion types
This package provides seven types of exclusions based on Qualtrics metadata. If you have ideas for other metadata exclusions, please submit them as issues. Note, the intent of this package is not to develop functions for excluding rows based on survey-specific data but on general, frequently used metadata.
-
duplicatesworks with rows that have duplicate IP addresses and/or locations (latitude/longitude). -
durationworks with rows whose survey completion time is too short and/or too long. -
ipworks with rows whose IP addresses are not found in the specified country (note: this exclusion type requires an internet connection to download the country’s IP ranges). -
locationworks with rows whose latitude and longitude are not found in the United States. -
previewworks with rows that are survey previews. -
progressworks with rows in which the survey was not complete. -
resolutionworks with rows whose screen resolution is not acceptable.
Usage
The verbs and exclusion types combine with _ to create the functions, such as check_duplicates(), exclude_ip(), and mark_duration(). Multiple functions can be linked together using the {magrittr} pipe %>%. For datasets downloaded directly from Qualtrics, use remove_label_rows() to remove the first two rows of labels and convert date and numeric columns in the metadata, and use deidentify() to remove standard Qualtrics columns with identifiable information (e.g., IP addresses, geolocation).
Marking
The mark_*() functions output the original data set with a new column specifying rows that meet the exclusion criteria. These can be piped together with %>% for multiple exclusion types.
library(excluder)
# Mark preview and short duration rows
df <- qualtrics_text %>%
mark_preview() %>%
mark_duration(min_duration = 200)
#> ℹ 2 rows were collected as previews. It is highly recommended to exclude these rows before further processing.
#> ℹ 23 out of 100 rows took less time than 200.
tibble::glimpse(df)
#> Rows: 100
#> Columns: 18
#> $ StartDate <dttm> 2020-12-11 12:06:52, 2020-12-11 12:06:43, 202…
#> $ EndDate <dttm> 2020-12-11 12:10:30, 2020-12-11 12:11:27, 202…
#> $ Status <chr> "Survey Preview", "Survey Preview", "IP Addres…
#> $ IPAddress <chr> NA, NA, "73.23.43.0", "16.140.105.0", "107.57.…
#> $ Progress <dbl> 100, 100, 100, 100, 100, 100, 100, 100, 100, 1…
#> $ `Duration (in seconds)` <dbl> 465, 545, 651, 409, 140, 213, 177, 662, 296, 2…
#> $ Finished <lgl> TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE…
#> $ RecordedDate <dttm> 2020-12-11 12:10:30, 2020-12-11 12:11:27, 202…
#> $ ResponseId <chr> "R_xLWiuPaNuURSFXY", "R_Q5lqYw6emJQZx2o", "R_f…
#> $ LocationLatitude <dbl> 29.73694, 39.74107, 34.03852, 44.96581, 27.980…
#> $ LocationLongitude <dbl> -94.97599, -121.82490, -118.25739, -93.07187, …
#> $ UserLanguage <chr> "EN", "EN", "EN", "EN", "EN", "EN", "EN", "EN"…
#> $ Browser <chr> "Chrome", "Chrome", "Chrome", "Chrome", "Chrom…
#> $ Version <chr> "88.0.4324.41", "88.0.4324.50", "87.0.4280.88"…
#> $ `Operating System` <chr> "Windows NT 10.0", "Macintosh", "Windows NT 10…
#> $ Resolution <chr> "1366x768", "1680x1050", "1366x768", "1536x864…
#> $ exclusion_preview <chr> "preview", "preview", "", "", "", "", "", "", …
#> $ exclusion_duration <chr> "", "", "", "", "duration_quick", "", "duratio…Use the unite_exclusions() function to unite all of the marked columns into a single column.
# Collapse labels for preview and short duration rows
df <- qualtrics_text %>%
mark_preview() %>%
mark_duration(min_duration = 200) %>%
unite_exclusions()
#> ℹ 2 rows were collected as previews. It is highly recommended to exclude these rows before further processing.
#> ℹ 23 out of 100 rows took less time than 200.
tibble::glimpse(df)
#> Rows: 100
#> Columns: 17
#> $ StartDate <dttm> 2020-12-11 12:06:52, 2020-12-11 12:06:43, 202…
#> $ EndDate <dttm> 2020-12-11 12:10:30, 2020-12-11 12:11:27, 202…
#> $ Status <chr> "Survey Preview", "Survey Preview", "IP Addres…
#> $ IPAddress <chr> NA, NA, "73.23.43.0", "16.140.105.0", "107.57.…
#> $ Progress <dbl> 100, 100, 100, 100, 100, 100, 100, 100, 100, 1…
#> $ `Duration (in seconds)` <dbl> 465, 545, 651, 409, 140, 213, 177, 662, 296, 2…
#> $ Finished <lgl> TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE…
#> $ RecordedDate <dttm> 2020-12-11 12:10:30, 2020-12-11 12:11:27, 202…
#> $ ResponseId <chr> "R_xLWiuPaNuURSFXY", "R_Q5lqYw6emJQZx2o", "R_f…
#> $ LocationLatitude <dbl> 29.73694, 39.74107, 34.03852, 44.96581, 27.980…
#> $ LocationLongitude <dbl> -94.97599, -121.82490, -118.25739, -93.07187, …
#> $ UserLanguage <chr> "EN", "EN", "EN", "EN", "EN", "EN", "EN", "EN"…
#> $ Browser <chr> "Chrome", "Chrome", "Chrome", "Chrome", "Chrom…
#> $ Version <chr> "88.0.4324.41", "88.0.4324.50", "87.0.4280.88"…
#> $ `Operating System` <chr> "Windows NT 10.0", "Macintosh", "Windows NT 10…
#> $ Resolution <chr> "1366x768", "1680x1050", "1366x768", "1536x864…
#> $ exclusions <chr> "preview", "preview", "", "", "duration_quick"…Checking
The check_*() functions output messages about the number of rows that meet the exclusion criteria. Because checks return only the rows meeting the criteria, they should not be connected via pipes unless you want to subset the second check criterion within the rows that meet the first criterion. Thus, in general, check_*() functions should be used individually. If you want to view the potential exclusions for multiple criteria, use the mark_*() functions.
# Check for preview rows
qualtrics_text %>%
check_preview()
#> ℹ 2 rows were collected as previews. It is highly recommended to exclude these rows before further processing.
#> StartDate EndDate Status IPAddress Progress
#> 1 2020-12-11 12:06:52 2020-12-11 12:10:30 Survey Preview <NA> 100
#> 2 2020-12-11 12:06:43 2020-12-11 12:11:27 Survey Preview <NA> 100
#> Duration (in seconds) Finished RecordedDate ResponseId
#> 1 465 TRUE 2020-12-11 12:10:30 R_xLWiuPaNuURSFXY
#> 2 545 TRUE 2020-12-11 12:11:27 R_Q5lqYw6emJQZx2o
#> LocationLatitude LocationLongitude UserLanguage Browser Version
#> 1 29.73694 -94.97599 EN Chrome 88.0.4324.41
#> 2 39.74107 -121.82490 EN Chrome 88.0.4324.50
#> Operating System Resolution
#> 1 Windows NT 10.0 1366x768
#> 2 Macintosh 1680x1050Excluding
The exclude_*() functions remove the rows that meet exclusion criteria. These, too, can be piped together. Since the output of each function is a subset of the original data with the excluded rows removed, the order of the functions will influence the reported number of rows meeting the exclusion criteria.
# Exclude preview then incomplete progress rows
df <- qualtrics_text %>%
exclude_duration(min_duration = 100) %>%
exclude_progress()
#> ℹ 4 out of 100 rows of short and/or long duration were excluded, leaving 96 rows.
#> ℹ 4 out of 96 rows with incomplete progress were excluded, leaving 92 rows.
dim(df)
#> [1] 92 16
# Exclude incomplete progress then preview rows
df <- qualtrics_text %>%
exclude_progress() %>%
exclude_duration(min_duration = 100)
#> ℹ 6 out of 100 rows with incomplete progress were excluded, leaving 94 rows.
#> ℹ 2 out of 94 rows of short and/or long duration were excluded, leaving 92 rows.
dim(df)
#> [1] 92 16Though the order of functions should not influence the final data set, it may speed up processing large files by removing preview and incomplete progress data first and waiting to check IP addresses and locations after other exclusions have been performed.
# Exclude rows
df <- qualtrics_text %>%
exclude_preview() %>%
exclude_progress() %>%
exclude_duplicates() %>%
exclude_duration(min_duration = 100) %>%
exclude_resolution() %>%
exclude_ip() %>%
exclude_location()
#> ℹ 2 out of 100 preview rows were excluded, leaving 98 rows.
#> ℹ 6 out of 98 rows with incomplete progress were excluded, leaving 92 rows.
#> ℹ 9 out of 92 duplicate rows were excluded, leaving 83 rows.
#> ℹ 2 out of 83 rows of short and/or long duration were excluded, leaving 81 rows.
#> ℹ 3 out of 81 rows with unacceptable screen resolution were excluded, leaving 78 rows.
#> ℹ 2 out of 78 rows with IP addresses outside of US were excluded, leaving 76 rows.
#> ℹ 4 out of 76 rows outside of the US were excluded, leaving 72 rows.Citing this package
To cite excluder, use:
Stevens, J. R. (2021). excluder: An R package that checks for exclusion criteria in online data. Journal of Open Source Software, 6(67), 3893. https://doi.org/10.21105/joss.03893
Contributing to this package
Contributions to excluder are most welcome! Feel free to check out open issues for ideas. And pull requests are encouraged, but you may want to raise an issue or contact the maintainer first.
Please note that the excluder project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
Acknowledgments
I thank Francine Goh and Billy Lim for comments on an early version of the package, as well as the insightful feedback from rOpenSci editor Mauro Lepore and reviewers Joseph O’Brien and Julia Silge. This work was funded by US National Science Foundation grant NSF-1658837.