Cleaning data using GBIF issues

rgbif now has the ability to clean data retrieved from GBIF based on GBIF issues. These issues are returned in data retrieved from GBIF, e.g., through the occ_search() function. Inspired by magrittr, we’ve setup a workflow for cleaning data based on using the operator %>%. You don’t have to use it, but as we show below, it can make the process quite easy.

Note that you can also query based on issues, e.g., occ_search(taxonKey=1, issue='DEPTH_UNLIKELY'). However, we imagine it’s more likely that you want to search for occurrences based on a taxonomic name, or geographic area, not based on issues, so it makes sense to pull data down, then clean as needed using the below workflow with occ_issues().

Note that occ_issues() only affects the data element in the gbif class that is returned from a call to occ_search(). Maybe in a future version we will remove the associated records from the hierarchy and media elements as they are remove from the data element.

occ_issues() also works with data from occ_download().

Get rgbif

Install from CRAN

install.packages("rgbif")

Or install the development version from GitHub

remotes::install_github("ropensci/rgbif")

Load rgbif

library('rgbif')

Get some data

Get taxon key for Helianthus annuus

(key <- name_suggest(q='Helianthus annuus', rank='species')$data$key[1])
#> [1] 9206251

Then pass to occ_search()

(res <- occ_search(taxonKey=key, limit=100))
#> Records found [87309] 
#> Records returned [100] 
#> No. unique hierarchies [1] 
#> No. media records [100] 
#> No. facets [0] 
#> Args [limit=100, offset=0, taxonKey=9206251, fields=all] 
#> # A tibble: 100 x 101
#>    key    scientificName  decimalLatitude decimalLongitude issues  datasetKey   
#>    <chr>  <chr>                     <dbl>            <dbl> <chr>   <chr>        
#>  1 30179… Helianthus ann…           29.4            -98.5  cdroun… 50c9509d-22c…
#>  2 30181… Helianthus ann…          -29.6             30.4  cdroun… 50c9509d-22c…
#>  3 30181… Helianthus ann…           50.8              4.10 cdroun… 50c9509d-22c…
#>  4 30316… Helianthus ann…           34.3           -118.   cdroun… 50c9509d-22c…
#>  5 30316… Helianthus ann…            3.04           102.   cdroun… 50c9509d-22c…
#>  6 30318… Helianthus ann…           35.1           -107.   cdroun… 50c9509d-22c…
#>  7 30319… Helianthus ann…           33.9           -117.   cdroun… 50c9509d-22c…
#>  8 30319… Helianthus ann…          -38.0            -59.2  cdroun… 50c9509d-22c…
#>  9 30320… Helianthus ann…            8.60           -83.4  cdroun… 50c9509d-22c…
#> 10 30320… Helianthus ann…           44.3            -78.4  cdroun… 50c9509d-22c…
#> # … with 90 more rows, and 95 more variables: publishingOrgKey <chr>,
#> #   installationKey <chr>, publishingCountry <chr>, protocol <chr>,
#> #   lastCrawled <chr>, lastParsed <chr>, crawlId <int>,
#> #   hostingOrganizationKey <chr>, basisOfRecord <chr>, occurrenceStatus <chr>,
#> #   taxonKey <int>, kingdomKey <int>, phylumKey <int>, classKey <int>,
#> #   orderKey <int>, familyKey <int>, genusKey <int>, speciesKey <int>,
#> #   acceptedTaxonKey <int>, acceptedScientificName <chr>, kingdom <chr>,
#> #   phylum <chr>, order <chr>, family <chr>, genus <chr>, species <chr>,
#> #   genericName <chr>, specificEpithet <chr>, taxonRank <chr>,
#> #   taxonomicStatus <chr>, iucnRedListCategory <chr>, dateIdentified <chr>,
#> #   coordinateUncertaintyInMeters <dbl>, stateProvince <chr>, year <int>,
#> #   month <int>, day <int>, eventDate <chr>, modified <chr>,
#> #   lastInterpreted <chr>, references <chr>, license <chr>, identifiers <chr>,
#> #   facts <chr>, relations <chr>, isInCluster <lgl>, geodeticDatum <chr>,
#> #   class <chr>, countryCode <chr>, recordedByIDs <chr>, identifiedByIDs <chr>,
#> #   country <chr>, rightsHolder <chr>, identifier <chr>,
#> #   http...unknown.org.nick <chr>, verbatimEventDate <chr>, datasetName <chr>,
#> #   verbatimLocality <chr>, gbifID <chr>, collectionCode <chr>,
#> #   occurrenceID <chr>, taxonID <chr>, catalogNumber <chr>, recordedBy <chr>,
#> #   http...unknown.org.occurrenceDetails <chr>, institutionCode <chr>,
#> #   rights <chr>, eventTime <chr>, reproductiveCondition <chr>,
#> #   identifiedBy <chr>, identificationID <chr>, name <chr>,
#> #   occurrenceRemarks <chr>, recordedByIDs.type <chr>,
#> #   recordedByIDs.value <chr>, identifiedByIDs.type <chr>,
#> #   identifiedByIDs.value <chr>, identificationRemarks <chr>, gadm <chr>,
#> #   informationWithheld <chr>, continent <chr>, municipality <chr>,
#> #   county <chr>, identificationVerificationStatus <chr>, language <chr>,
#> #   type <chr>, vernacularName <chr>, taxonConceptID <chr>, endDayOfYear <chr>,
#> #   locality <chr>, startDayOfYear <chr>, datasetID <chr>, accessRights <chr>,
#> #   bibliographicCitation <chr>, higherClassification <chr>

Examine issues

The dataset gbifissues can be retrieved using the function gbif_issues(). The dataset’s first column code is a code that is used by default in the results from occ_search(), while the second column issue is the full issue name given by GBIF. The third column is a full description of the issue.

head(gbif_issues())
#>    code                              issue
#> 1   bri            BASIS_OF_RECORD_INVALID
#> 2   ccm         CONTINENT_COUNTRY_MISMATCH
#> 3   cdc CONTINENT_DERIVED_FROM_COORDINATES
#> 4 conti                  CONTINENT_INVALID
#> 5  cdiv                 COORDINATE_INVALID
#> 6 cdout            COORDINATE_OUT_OF_RANGE
#>                                                                                                    description
#> 1 The given basis of record is impossible to interpret or seriously different from the recommended vocabulary.
#> 2                                                       The interpreted continent and country do not match up.
#> 3                  The interpreted continent is based on the coordinates, not the verbatim string information.
#> 4                                                                      Uninterpretable continent values found.
#> 5                                      Coordinate value given in some form but GBIF is unable to interpret it.
#> 6                                        Coordinate has invalid lat/lon values out of their decimal max range.
#>         type
#> 1 occurrence
#> 2 occurrence
#> 3 occurrence
#> 4 occurrence
#> 5 occurrence
#> 6 occurrence

You can query to get certain issues

gbif_issues()[ gbif_issues()$code %in% c('cdround','cudc','gass84','txmathi'), ]
#>       code                            issue
#> 10 cdround               COORDINATE_ROUNDED
#> 12    cudc COUNTRY_DERIVED_FROM_COORDINATES
#> 23  gass84     GEODETIC_DATUM_ASSUMED_WGS84
#> 39 txmathi           TAXON_MATCH_HIGHERRANK
#>                                                                                                                                 description
#> 10                                                                                  Original coordinate modified by rounding to 5 decimals.
#> 12                                                The interpreted country is based on the coordinates, not the verbatim string information.
#> 23 Indicating that the interpreted coordinates assume they are based on WGS84 datum as the datum was either not indicated or interpretable.
#> 39                                        Matching to the taxonomic backbone can only be done on a higher rank and not the scientific name.
#>          type
#> 10 occurrence
#> 12 occurrence
#> 23 occurrence
#> 39 occurrence

The code cdround represents the GBIF issue COORDINATE_ROUNDED, which means that

Original coordinate modified by rounding to 5 decimals.

The content for this information comes from https://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/OccurrenceIssue.html

Parse data based on issues

Now that we know a bit about GBIF issues, you can parse your data based on issues. Using the data generated above, and using the function %>% imported from magrittr, we can get only data with the issue gass84, or GEODETIC_DATUM_ASSUMED_WGS84 (Note how the records returned goes down to 98 instead of the initial 100).

res %>%
  occ_issues(gass84)
#> Records found [87309] 
#> Records returned [99] 
#> No. unique hierarchies [1] 
#> No. media records [100] 
#> No. facets [0] 
#> Args [limit=100, offset=0, taxonKey=9206251, fields=all] 
#> # A tibble: 99 x 101
#>    key    scientificName  decimalLatitude decimalLongitude issues  datasetKey   
#>    <chr>  <chr>                     <dbl>            <dbl> <chr>   <chr>        
#>  1 30179… Helianthus ann…           29.4            -98.5  cdroun… 50c9509d-22c…
#>  2 30181… Helianthus ann…          -29.6             30.4  cdroun… 50c9509d-22c…
#>  3 30181… Helianthus ann…           50.8              4.10 cdroun… 50c9509d-22c…
#>  4 30316… Helianthus ann…           34.3           -118.   cdroun… 50c9509d-22c…
#>  5 30316… Helianthus ann…            3.04           102.   cdroun… 50c9509d-22c…
#>  6 30318… Helianthus ann…           35.1           -107.   cdroun… 50c9509d-22c…
#>  7 30319… Helianthus ann…           33.9           -117.   cdroun… 50c9509d-22c…
#>  8 30319… Helianthus ann…          -38.0            -59.2  cdroun… 50c9509d-22c…
#>  9 30320… Helianthus ann…            8.60           -83.4  cdroun… 50c9509d-22c…
#> 10 30320… Helianthus ann…           44.3            -78.4  cdroun… 50c9509d-22c…
#> # … with 89 more rows, and 95 more variables: publishingOrgKey <chr>,
#> #   installationKey <chr>, publishingCountry <chr>, protocol <chr>,
#> #   lastCrawled <chr>, lastParsed <chr>, crawlId <int>,
#> #   hostingOrganizationKey <chr>, basisOfRecord <chr>, occurrenceStatus <chr>,
#> #   taxonKey <int>, kingdomKey <int>, phylumKey <int>, classKey <int>,
#> #   orderKey <int>, familyKey <int>, genusKey <int>, speciesKey <int>,
#> #   acceptedTaxonKey <int>, acceptedScientificName <chr>, kingdom <chr>,
#> #   phylum <chr>, order <chr>, family <chr>, genus <chr>, species <chr>,
#> #   genericName <chr>, specificEpithet <chr>, taxonRank <chr>,
#> #   taxonomicStatus <chr>, iucnRedListCategory <chr>, dateIdentified <chr>,
#> #   coordinateUncertaintyInMeters <dbl>, stateProvince <chr>, year <int>,
#> #   month <int>, day <int>, eventDate <chr>, modified <chr>,
#> #   lastInterpreted <chr>, references <chr>, license <chr>, identifiers <chr>,
#> #   facts <chr>, relations <chr>, isInCluster <lgl>, geodeticDatum <chr>,
#> #   class <chr>, countryCode <chr>, recordedByIDs <chr>, identifiedByIDs <chr>,
#> #   country <chr>, rightsHolder <chr>, identifier <chr>,
#> #   http...unknown.org.nick <chr>, verbatimEventDate <chr>, datasetName <chr>,
#> #   verbatimLocality <chr>, gbifID <chr>, collectionCode <chr>,
#> #   occurrenceID <chr>, taxonID <chr>, catalogNumber <chr>, recordedBy <chr>,
#> #   http...unknown.org.occurrenceDetails <chr>, institutionCode <chr>,
#> #   rights <chr>, eventTime <chr>, reproductiveCondition <chr>,
#> #   identifiedBy <chr>, identificationID <chr>, name <chr>,
#> #   occurrenceRemarks <chr>, recordedByIDs.type <chr>,
#> #   recordedByIDs.value <chr>, identifiedByIDs.type <chr>,
#> #   identifiedByIDs.value <chr>, identificationRemarks <chr>, gadm <chr>,
#> #   informationWithheld <chr>, continent <chr>, municipality <chr>,
#> #   county <chr>, identificationVerificationStatus <chr>, language <chr>,
#> #   type <chr>, vernacularName <chr>, taxonConceptID <chr>, endDayOfYear <chr>,
#> #   locality <chr>, startDayOfYear <chr>, datasetID <chr>, accessRights <chr>,
#> #   bibliographicCitation <chr>, higherClassification <chr>

Note also that we’ve set up occ_issues() so that you can pass in issue names without having to quote them, thereby speeding up data cleaning.

Next, we can remove data with certain issues just as easily by using a - sign in front of the variable, like this, removing data with issues depunl and mdatunl.

res %>%
  occ_issues(-depunl, -mdatunl)
#> Records found [87309] 
#> Records returned [100] 
#> No. unique hierarchies [1] 
#> No. media records [100] 
#> No. facets [0] 
#> Args [limit=100, offset=0, taxonKey=9206251, fields=all] 
#> # A tibble: 100 x 101
#>    key    scientificName  decimalLatitude decimalLongitude issues  datasetKey   
#>    <chr>  <chr>                     <dbl>            <dbl> <chr>   <chr>        
#>  1 30179… Helianthus ann…           29.4            -98.5  cdroun… 50c9509d-22c…
#>  2 30181… Helianthus ann…          -29.6             30.4  cdroun… 50c9509d-22c…
#>  3 30181… Helianthus ann…           50.8              4.10 cdroun… 50c9509d-22c…
#>  4 30316… Helianthus ann…           34.3           -118.   cdroun… 50c9509d-22c…
#>  5 30316… Helianthus ann…            3.04           102.   cdroun… 50c9509d-22c…
#>  6 30318… Helianthus ann…           35.1           -107.   cdroun… 50c9509d-22c…
#>  7 30319… Helianthus ann…           33.9           -117.   cdroun… 50c9509d-22c…
#>  8 30319… Helianthus ann…          -38.0            -59.2  cdroun… 50c9509d-22c…
#>  9 30320… Helianthus ann…            8.60           -83.4  cdroun… 50c9509d-22c…
#> 10 30320… Helianthus ann…           44.3            -78.4  cdroun… 50c9509d-22c…
#> # … with 90 more rows, and 95 more variables: publishingOrgKey <chr>,
#> #   installationKey <chr>, publishingCountry <chr>, protocol <chr>,
#> #   lastCrawled <chr>, lastParsed <chr>, crawlId <int>,
#> #   hostingOrganizationKey <chr>, basisOfRecord <chr>, occurrenceStatus <chr>,
#> #   taxonKey <int>, kingdomKey <int>, phylumKey <int>, classKey <int>,
#> #   orderKey <int>, familyKey <int>, genusKey <int>, speciesKey <int>,
#> #   acceptedTaxonKey <int>, acceptedScientificName <chr>, kingdom <chr>,
#> #   phylum <chr>, order <chr>, family <chr>, genus <chr>, species <chr>,
#> #   genericName <chr>, specificEpithet <chr>, taxonRank <chr>,
#> #   taxonomicStatus <chr>, iucnRedListCategory <chr>, dateIdentified <chr>,
#> #   coordinateUncertaintyInMeters <dbl>, stateProvince <chr>, year <int>,
#> #   month <int>, day <int>, eventDate <chr>, modified <chr>,
#> #   lastInterpreted <chr>, references <chr>, license <chr>, identifiers <chr>,
#> #   facts <chr>, relations <chr>, isInCluster <lgl>, geodeticDatum <chr>,
#> #   class <chr>, countryCode <chr>, recordedByIDs <chr>, identifiedByIDs <chr>,
#> #   country <chr>, rightsHolder <chr>, identifier <chr>,
#> #   http...unknown.org.nick <chr>, verbatimEventDate <chr>, datasetName <chr>,
#> #   verbatimLocality <chr>, gbifID <chr>, collectionCode <chr>,
#> #   occurrenceID <chr>, taxonID <chr>, catalogNumber <chr>, recordedBy <chr>,
#> #   http...unknown.org.occurrenceDetails <chr>, institutionCode <chr>,
#> #   rights <chr>, eventTime <chr>, reproductiveCondition <chr>,
#> #   identifiedBy <chr>, identificationID <chr>, name <chr>,
#> #   occurrenceRemarks <chr>, recordedByIDs.type <chr>,
#> #   recordedByIDs.value <chr>, identifiedByIDs.type <chr>,
#> #   identifiedByIDs.value <chr>, identificationRemarks <chr>, gadm <chr>,
#> #   informationWithheld <chr>, continent <chr>, municipality <chr>,
#> #   county <chr>, identificationVerificationStatus <chr>, language <chr>,
#> #   type <chr>, vernacularName <chr>, taxonConceptID <chr>, endDayOfYear <chr>,
#> #   locality <chr>, startDayOfYear <chr>, datasetID <chr>, accessRights <chr>,
#> #   bibliographicCitation <chr>, higherClassification <chr>

Expand issue codes to full names

Another thing we can do with occ_issues() is go from issue codes to full issue names in case you want those in your dataset (here, showing only a few columns to see the data better for this demo):

out <- res %>% occ_issues(mutate = "expand")
head(out$data[,c(1,5)])
#> # A tibble: 6 x 2
#>   key        issues                                         
#>   <chr>      <chr>                                          
#> 1 3017947125 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84
#> 2 3018105046 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84
#> 3 3018133231 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84
#> 4 3031652179 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84
#> 5 3031677301 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84
#> 6 3031889761 COORDINATE_ROUNDED,GEODETIC_DATUM_ASSUMED_WGS84

Add columns

Sometimes you may want to have each type of issue as a separate column.

Split out each issue type into a separate column, with number of columns equal to number of issue types

out <- res %>% occ_issues(mutate = "split")
head(out$data[,c(1,5:10)])
#> # A tibble: 6 x 7
#>   name      cdround gass84 scientificName   datasetKey      publishingOrgKey    
#>   <chr>     <chr>   <chr>  <chr>            <chr>           <chr>               
#> 1 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> 2 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> 3 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> 4 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> 5 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> 6 Helianth… y       y      Helianthus annu… 50c9509d-22c7-… 28eb1a3f-1c15-4a95-…
#> # … with 1 more variable: installationKey <chr>

Expand and add columns

Or you can expand each issue type into its full name, and split each issue into a separate column.

out <- res %>% occ_issues(mutate = "split_expand")
head(out$data[,c(1,5:10)])
#> # A tibble: 6 x 7
#>   name      COORDINATE_ROUND… GEODETIC_DATUM_AS… scientificName  datasetKey     
#>   <chr>     <chr>             <chr>              <chr>           <chr>          
#> 1 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> 2 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> 3 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> 4 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> 5 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> 6 Helianth… y                 y                  Helianthus ann… 50c9509d-22c7-…
#> # … with 2 more variables: publishingOrgKey <chr>, installationKey <chr>

Wrap up

We hope this helps users get just the data they want, and nothing more. Let us know if you have feedback on data cleaning functionality in rgbif at [email protected] or at https://github.com/ropensci/rgbif/issues.

Scott Chamberlain

2021-06-11