Review guide for analysis best practice, developed at rOpenSci unconf 2017
Related: checkers - a package to assess analysis
While every analysis is different, there are common elements which can strengthen validity, reproducibility, and reusability. These guidelines describe, prioritize and assist analysts in developing the strongest analyses and workflows possible while remaining flexible for a wide variety of applications and contexts.
Tiers are in descending order of importance: Focus on Tier 1 elements, then Tier 2, then Tier 3.
- Tier 1: Must Have - These elements are required for reliable and trustworthy analyses.
- Tier 2: Nice to Have - These elements are recommended for best practice and reproducibility and should be strongly considered.
- Tier 3: Recommended - These elements are ideal best practice.
Fully automatic: Included as elements of checkers.
Semi-automatic: Can be implemented as custom checks in checkers.
Human-powered: Cannot be automated. Analyst uses guidelines to make sure analysis and report fit best practice for specific context.
Clear research question(s) which can be answered by your available data.
This might include “What patterns do we see?” and other exploratory analyses as well as more formal hypotheses, but questions and analysis plans should be clearly defined.
Know your Data/Data Structure (T1, Human)
In order to conduct robust analyses and make reliable inferences, we have to understand our data.
- Is a data dictionary/codebook or other metadata available?
- Are column names reasonable - meaningful with no special characters?
- Is data tidy, with all information contained in rows/columns (not rownames or labels)?
- Have you visualized your data to better understand missingness and variable distributions?
dplyr and tidyr for tidying data; visdat for visualizing missingness
Quality Checks (T1, Human)
Check data for reasonable values.
Examples of unreasonable values:
- Pregnant males
- Lab measurements outside biological limits
- Variables which should be continuous containing character values (
Rawest data form (T1, Human)
To retain as much information and avoid as much duplication of effort and human error as possible, data should be loaded in the form closest to its original state as is reasonable.
- Downloading data from the web directly via API vs adding in an additional export step
- Comma separated files vs formatted spreadsheets
- Raw subject-level data rather than summary statistics
Check and control for updated source data (T2, Semi-Auto)
Do the column names and dimensions in your current data files match an expected set of names/dimensions? If rows or column are added or deleted, your results and the stability of your scripts and models might be affected.
dim() of all datasets
Ownership clearly identified (T3, Human)
At minimum, the source of the data should be noted, along with any necessary acknowledgements. If the data is publicly available and licensed, the license should be included.
Scripts and Code
Fail fast if it fails (code needs to break fast to help improve) (T2, Human)
Robust to changes (T2, Semi-Auto)
Don’t Repeat Yourself (T2, Semi-Auto)
Best Code Praxtices (T2, auto)
Platform agnostic (T2, auto)
Labelled chunks (T2, Auto)
Streamlining objects in memory (T3, Auto)
License and ownership (T3, Auto)
Packages / Organisation
Neglected Packages (T1, Auto)
Expressive Object names (T2, Human)
Filestructure (T2, Semi-Automatic)
Version control (T2, Auto)
README for complex data structures (T2, Auto)
Package versions (Know your options) (T2, Auto)
Readability (Specifically, naming conventions) (T2, Auto)
Readability (Style: Adequate whitespace / indentation) (T3, Auto)
Description of the root datasets (T1, human)
Description of all analyses (T1, Human)
Model Assumption Checks (T1, Semi-Auto)
Model Goodness of fit (T1, Semi-Auto)
Model Validation (T1, Semi-Auto)
Diagnostics (T1, Semi-Auto)
Plan to handle missingness (T1, Semi-Auto)
Model Results (T1, Semi-Auto)
Record of decisions (T2, Human)
Visualisation / Reporting
Informative Titles and Labels (T1, Human)
Final Product Geared for its intended audience (T1, Human)
Visualisation of conclusions (T1, Human)
Descriptive plots (T1, Semi-Auto)
Exploratory Data Analysis (T1, Semi-Auto)
Appropriately Conveyed Data Types (T1, Semi-Auto)
Clear Structure and Flow (T1, Automatic)
Reusable outputs (T2, Human)
Remove chartjunk (T3, Human)
Grammar and Spelling (T3, Auto)