Skip to contents

rerddap is a general purpose R client for working with ERDDAP servers. ERDDAP is a server built on top of OPenDAP, which serves some NOAA data. You can get gridded data (griddap), which lets you query from gridded datasets, or table data (tabledap) which lets you query from tabular datasets. In terms of how we interface with them, there are similarties, but some differences too. We try to make a similar interface to both data types in rerddap.

NetCDF

rerddap supports NetCDF format, and is the default when using the griddap() function. NetCDF is a binary file format, and will have a much smaller footprint on your disk than csv. The binary file format means it’s harder to inspect, but the ncdf4 package makes it easy to pull data out and write data back into a NetCDF file. Note the the file extension for NetCDF files is .nc. Whether you choose NetCDF or csv for small files won’t make much of a difference, but will with large files.

Caching

Data files downloaded are cached in a single hidden directory ~/.rerddap on your machine. It’s hidden so that you don’t accidentally delete the data, but you can still easily delete the data if you like.

When you use griddap() or tabledap() functions, we construct a MD5 hash from the base URL, and any query parameters - this way each query is separately cached. Once we have the hash, we look in ~/.rerddap for a matching hash. If there’s a match we use that file on disk - if no match, we make a http request for the data to the ERDDAP server you specify.

ERDDAP servers

You can get a data.frame of ERDDAP servers using the function servers(). The list of ERDDAP servers is drawn from the Awesome ERDDAP site. If you know of more ERDDAP servers, follow the instructions on that page to add the server.

Install

Stable version from CRAN

install.packages("rerddap")

Or, the development version from GitHub

remotes::install_github("ropensci/rerddap")

First, you likely want to search for data, specify either griddadp or tabledap

ed_search(query = 'size', which = "table")
#> # A tibble: 9 × 2
#>   title                                                         dataset_id      
#>   <chr>                                                         <chr>           
#> 1 Channel Islands, Kelp Forest Monitoring, Size and Frequency,… erdCinpKfmSFNH  
#> 2 CalCOFI Larvae Sizes                                          erdCalCOFIlrvsiz
#> 3 CalCOFI Larvae Counts Positive Tows                           erdCalCOFIlrvcn…
#> 4 CalCOFI Tows                                                  erdCalCOFItows  
#> 5 GLOBEC NEP MOCNESS Plankton (MOC1) Data, 2000-2002            erdGlobecMoc1   
#> 6 GLOBEC NEP Vertical Plankton Tow (VPT) Data, 1997-2001        erdGlobecVpt    
#> 7 File Names from the AWS S3 noaa-goes16 Bucket                 awsS3NoaaGoes16 
#> 8 AN EXPERIMENTAL DATASET: Underway Sea Surface Temperature an… nodcPJJU        
#> 9 PacIOOS Beach Camera 001: Waikiki, Oahu, Hawaii               BEACHCAM-001
ed_search(query = 'size', which = "grid")
#> # A tibble: 5 × 2
#>   title                                                      dataset_id         
#>   <chr>                                                      <chr>              
#> 1 Extended AVHRR Polar Pathfinder Fundamental Climate Data … noaa_ngdc_da08_dcd…
#> 2 Extended AVHRR Polar Pathfinder Fundamental Climate Data … noaa_ngdc_0fe5_a4b…
#> 3 Extended AVHRR Polar Pathfinder Fundamental Climate Data … noaa_ngdc_5253_bf9…
#> 4 Extended AVHRR Polar Pathfinder Fundamental Climate Data … noaa_ngdc_0f24_2f8…
#> 5 SST and SST Anomaly, NOAA Global Coral Bleaching Monitori… NOAA_DHW_monthly

There is now a convenience function to search over a list of ERDDAP servers, designed to work with the function servers()

global_search(query, server_list, which_service)
#> Error in global_search(query, server_list, which_service): could not find function "global_search"

Information

Then you can get information on a single dataset

info('erdCalCOFIlrvsiz')
#> <ERDDAP info> erdCalCOFIlrvsiz 
#>  Base URL: https://upwell.pfeg.noaa.gov/erddap 
#>  Dataset Type: tabledap 
#>  Variables:  
#>      calcofi_species_code: 
#>          Range: 19, 946 
#>      common_name: 
#>      cruise: 
#>      itis_tsn: 
#>      larvae_10m2: 
...

griddap (gridded) data

First, get information on a dataset to see time range, lat/long range, and variables.

(out <- info('erdMBchla1day'))
#> <ERDDAP info> erdMBchla1day 
#>  Base URL: https://upwell.pfeg.noaa.gov/erddap 
#>  Dataset Type: griddap 
#>  Dimensions (range):  
#>      time: (2006-01-01T12:00:00Z, 2021-11-16T12:00:00Z) 
#>      altitude: (0.0, 0.0) 
#>      latitude: (-45.0, 65.0) 
#>      longitude: (120.0, 320.0) 
#>  Variables:  
#>      chlorophyll: 
#>          Units: mg m-3

Then query for gridded data using the griddap() function

(res <- griddap(out,
  time = c('2015-01-01','2015-01-03'),
  latitude = c(14, 15),
  longitude = c(125, 126)
))
#> <ERDDAP griddap> erdMBchla1day
#>    Path: [/Users/rmendels/Library/Caches/R/rerddap/4d844aa48552049c3717ac94ced5f9b8.nc]
#>    Last updated: [2021-11-18 15:58:06]
#>    File size:    [0.03 mb]
#>    Dimensions (dims/vars):   [4 X 1]
#>    Dim names: time, altitude, latitude, longitude
#>    Variable names: Chlorophyll Concentration in Sea Water
#>    data.frame (rows/columns):   [5043 X 5]
#> # A tibble: 5,043 × 5
#>    time                   lat   lon altitude chlorophyll
#>    <chr>                <dbl> <dbl>    <dbl>       <dbl>
#>  1 2015-01-01T12:00:00Z    14  125         0          NA
#>  2 2015-01-01T12:00:00Z    14  125.        0          NA
#>  3 2015-01-01T12:00:00Z    14  125.        0          NA
#>  4 2015-01-01T12:00:00Z    14  125.        0          NA
#>  5 2015-01-01T12:00:00Z    14  125.        0          NA
#>  6 2015-01-01T12:00:00Z    14  125.        0          NA
#>  7 2015-01-01T12:00:00Z    14  125.        0          NA
#>  8 2015-01-01T12:00:00Z    14  125.        0          NA
#>  9 2015-01-01T12:00:00Z    14  125.        0          NA
#> 10 2015-01-01T12:00:00Z    14  125.        0          NA
#> # … with 5,033 more rows

The output of griddap() is a list that you can explore further. Get the summary

res$summary
#> $filename
#> [1] "/Users/rmendels/Library/Caches/R/rerddap/4d844aa48552049c3717ac94ced5f9b8.nc"
#> 
#> $writable
#> [1] FALSE
#> 
#> $id
#> [1] 65536
#> 
#> $safemode
#> [1] FALSE
#> 
#> $format
#> [1] "NC_FORMAT_CLASSIC"
#> 
...

Get the dimension variables

names(res$summary$dim)
#> [1] "time"      "altitude"  "latitude"  "longitude"

Get the data.frame (beware: you may want to just look at the head of the data.frame if large)

head(res$data)
#>                   time lat     lon altitude chlorophyll
#> 1 2015-01-01T12:00:00Z  14 125.000        0          NA
#> 2 2015-01-01T12:00:00Z  14 125.025        0          NA
#> 3 2015-01-01T12:00:00Z  14 125.050        0          NA
#> 4 2015-01-01T12:00:00Z  14 125.075        0          NA
#> 5 2015-01-01T12:00:00Z  14 125.100        0          NA
#> 6 2015-01-01T12:00:00Z  14 125.125        0          NA

tabledap (tabular) data

(out <- info('erdCalCOFIlrvsiz'))
#> <ERDDAP info> erdCalCOFIlrvsiz 
#>  Base URL: https://upwell.pfeg.noaa.gov/erddap 
#>  Dataset Type: tabledap 
#>  Variables:  
#>      calcofi_species_code: 
#>          Range: 19, 946 
#>      common_name: 
#>      cruise: 
#>      itis_tsn: 
#>      larvae_10m2: 
...
(dat <- tabledap('erdCalCOFIlrvsiz', fields=c('latitude','longitude','larvae_size',
  'scientific_name'), 'time>=2011-01-01', 'time<=2011-12-31'))
#> <ERDDAP tabledap> erdCalCOFIlrvsiz
#>    Path: [/Users/rmendels/Library/Caches/R/rerddap/db7389db5b5b0ed9c426d5c13bc43d18.csv]
#>    Last updated: [2021-11-18 15:58:09]
#>    File size:    [0.05 mb]
#> # A tibble: 1,304 × 4
#>    latitude  longitude  larvae_size scientific_name     
#>    <chr>     <chr>      <chr>       <chr>               
#>  1 32.956665 -117.305   4.5         Engraulis mordax    
#>  2 32.91     -117.4     5.0         Merluccius productus
#>  3 32.511665 -118.21167 2.0         Merluccius productus
#>  4 32.511665 -118.21167 3.0         Merluccius productus
#>  5 32.511665 -118.21167 5.5         Merluccius productus
#>  6 32.511665 -118.21167 6.0         Merluccius productus
#>  7 32.511665 -118.21167 2.8         Merluccius productus
#>  8 32.511665 -118.21167 3.0         Sardinops sagax     
#>  9 32.511665 -118.21167 5.0         Sardinops sagax     
#> 10 32.511665 -118.21167 2.5         Engraulis mordax    
#> # … with 1,294 more rows

Since both griddap() and tabledap() give back data.frame’s, it’s easy to do downstream manipulation. For example, we can use dplyr to filter, summarize, group, and sort:

library("dplyr")
dat$larvae_size <- as.numeric(dat$larvae_size)
dat %>%
  group_by(scientific_name) %>%
  summarise(mean_size = mean(larvae_size)) %>%
  arrange(desc(mean_size))
#> # A tibble: 7 × 2
#>   scientific_name       mean_size
#>   <chr>                     <dbl>
#> 1 Anoplopoma fimbria        23.3 
#> 2 Engraulis mordax           9.26
#> 3 Sardinops sagax            7.28
#> 4 Merluccius productus       5.48
#> 5 Tactostoma macropus        5   
#> 6 Scomber japonicus          3.4 
#> 7 Trachurus symmetricus      3.29