Context

The Mangal project

The Mangal project aims at archiving published ecological networks and at easing their retrieval. To do so, Mangal:

  1. uses a data specification for ecological networks (described in Poisot et al. 2016);

  2. archives ecological networks in a PostgreSQL database;

  3. provides:

Currently, 172 datasets are including in the database representing over 1300 ecological networks. In 2016, the first paper describing the project was published and introduced the first release of rmangal (Poisot et al. 2016). Since then, the structure of the database has been improved (new fields have been added), several ecological networks added and the API entirely rewritten. Consequently, the first release of the rmangal is obsolete (and archived) and we introduce rmangal v2.0 in this vignette.

Data structure


Mangal data structure

The diagram on the left side represents the structure of the Mangal database. All references included in Mangal correspond to a specific publication that includes one or several dataset(s). This dataset is basically a collection of ecological networks whose nodes and interactions (edges) are stored in separate tables. Below, we briefly describe the content of each table.

References – Information pertaining to a reference (scientific article, book, online website, etc.) characterizing an original collection of ecological networks. URLs of data and publication sources are included as well as persistent identifiers (when available) such as digital object identifiers (DOIs). This allows the user to retrieve more details about original publications using appropriate R packages such as crossref.

Datasets – Metadata of the datasets attached to a reference. It includes a general description of the networks.

Networks – Metadata of the networks attached to a dataset. It provides the sampling location, date and specific description of the network.

Nodes – Information on the population, taxa or individu in the network. Each node has the original taxon name documented and taxonomic backbone provided by all services embedded in taxize (Chamberlain et al. 2019).

Interactions – Information on the interaction type (e.g. mutualism, predation, etc.), the strength, and the direction of the interaction between two nodes.

Authentication

So far, the rmangal package provides methods to get access to the data store. Data requests (performed via httr::GET()) do not require any authentication.

A bearer authentication strategy using ORCID credentials (as a third-party services) has been implemented on all POST, DELETE, PUT API operations to allow the user to add and delete new ecological to the data base. These features are not currently included in the rmangal package, but are under consideration for future major releases.

How to use rmangal

Overall approach

In order to efficiently retrieve networks from the database, rmangal includes 7 search functions querying the 5 tables described above as well as a table dedicated to the taxonomy backbone.

  1. search_references(): search in the reference table, for instance the user can look for a specific doi;
  2. search_datasets(): search among datasets using a keyword;
  3. search_networks() and search_networks_sf(): search networks based on a keyword or a geographical area;
  4. search_interactions(): list all networks containing a specific interaction type;
  5. search_nodes(): identify nodes based on nodes information;
  6. search_taxonomy(): identify nodes based on taxonomic names and unique identifiers.

All of these functions return specific class objects with the information needed to retrieve the corresponding set of ecological networks with get_collection(). Hence, the user can easily retrieve data in two steps:

Note that if there is only one network to be retrieved, get_collection() returns a mgNetwork object, otherwise it returns an object of class mgNetworksCollection which is a collection (a list) of mgNetwork objects. Below, we exemplify how to use the search functions, how to get a collection of networks and how to use other packages to carry out specific analyses.

Search functions

In rmangal, every functions queries a specific table and allow only one query at a time (see section Batch analysis to learn how to perform more than one query). All the functions offer two ways to query the corresponding table:

  1. a keyword: in this case, the entries returned are the partial or full keyword match of any strings contained in the table;
  2. a custom query: in this case, entries returned are exact matches.

Let’s load rmangal as well as two helper packages:

library(rmangal)
library(magrittr) # for the pipe %>%
library(tibble) # to use tibble (enhanced data frames)

Search and list available datasets

Let’s assume we are looking for ecological networks including species living in lagoons. If we have no idea about any existing data set, the best starting point is then to query the dataset table with lagoon as a keyword:

If the Mangal reference id containing the lagoon networks was known, we could build a custom query as follow:

Note that if an empty character is passed, i.e. "", all entries are returned. We can use this behavior to list all datasets available:

As shown in the diagram above, a dataset comes from a specific reference and search_references() queries the reference table directly. A handy argument of this function is doi as it allows to pass a Digital Object Identifier and so to retrieve all datasets attached to a specific publication.

zetina_2003 <- search_references(doi = "10.1016/s0272-7714(02)00410-9")

Finding a specific network

We can also search by keyword across all networks.

It is also possible to retrieve all networks based on interaction types involved:

search_networks_sf() handles spatial queries: argument query_sf takes a sf object as input and returns all networks included in the spatial extent of this object. For instance, one can retrieve all Californian networks included in Mangal like so:

library(sf)
library(mapview)
library(USAboundaries)
area <- us_states(state = "california")
in_CA <- search_networks_sf(area)

mapView(in_CA, legend = FALSE) + mapview(sf::st_cast(area,"MULTILINESTRING"), color = "red", legend = FALSE)

Search for a specific taxon

The user can easily identify networks including a specific taxonomic entity with search_taxonomy():

This function allows to search for a specific taxonomic entity using it’s validated name or unique identifiers, i.e. EOL, TSN, GBIF, COL, BOLD and NCBI IDs. Taxon names of the taxonomy table were validated with TNRS (see http://tnrs.iplantcollaborative.org/ and/or GNR (see https://resolver.globalnames.org/). The taxon names in this table might not be the taxon name documented in the original publication. In order to identify relevant networks with the original name, use [search_nodes()].

The validation of taxon names was performed by an automated procedure using taxize (Chamberlain et al. 2019) and if there is any doubt, the original names recorded by authors should be regarded as the most reliable information. Please report any issue related to taxonomy at https://github.com/mangal-wg/mangal-datasets/issues.

Note that in some case, one may need to find a dataset based on the original name included in the publication, in such case, search_nodes() must be used:

Get networks associated with a search_* object

Once the search performed, ecological networks are accessible from the object returned with get_collection():

Note that mgNetworksCollection objects are lists of mgNetwork object which are a list of five datasets reflecting the 5 tables presented in the diagram in the first section:

names(nets_lagoons[[1]])
#> [1] "network"      "nodes"        "interactions" "dataset"      "reference"
glimpse(nets_lagoons[[1]]$network)
#> Observations: 1
#> Variables: 13
#> $ network_id       <int> 86
#> $ name             <chr> "zetina_2003_20030101_86"
#> $ date             <chr> "2003-01-01T00:00:00.000Z"
#> $ description      <chr> "Dietary matrix of the Huizache–Caimanero lagoon"
#> $ public           <lgl> TRUE
#> $ all_interactions <lgl> FALSE
#> $ created_at       <chr> "2019-02-23T17:04:34.046Z"
#> $ updated_at       <chr> "2019-02-23T17:04:34.046Z"
#> $ dataset_id       <int> 22
#> $ user_id          <int> 3
#> $ geom_type        <chr> "Point"
#> $ geom_lon         <list> [-106.1099]
#> $ geom_lat         <list> [22.98531]
glimpse(nets_lagoons[[1]]$nodes)
#> Observations: 26
#> Variables: 19
#> $ node_id             <int> 4904, 4905, 4906, 4907, 4908, 4909, 4910, 4911, 4…
#> $ original_name       <chr> "Scianids", "Elopids", "Lutjanids", "Carangids", …
#> $ node_level          <chr> "taxon", "taxon", "taxon", "taxon", "taxon", "tax…
#> $ network_id          <int> 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 8…
#> $ taxonomy_id         <int> 4363, 4364, 4365, 4366, 4367, 4368, 4369, 4355, 3…
#> $ created_at          <chr> "2019-02-23T17:04:42.505Z", "2019-02-23T17:04:42.…
#> $ updated_at          <chr> "2019-02-23T17:04:42.505Z", "2019-02-23T17:04:42.…
#> $ taxonomy.id         <int> 4363, 4364, 4365, 4366, 4367, 4368, 4369, 4355, 3…
#> $ taxonomy.name       <chr> "Sciaenidae", "Elops", "Lutjanidae", "Carangidae"…
#> $ taxonomy.ncbi       <int> 30870, 7927, 30850, 8157, 8184, 31017, 30840, 825…
#> $ taxonomy.tsn        <int> 169237, 28630, 168845, 168584, 167642, 43998, 169…
#> $ taxonomy.eol        <int> 5211, 46561210, 5294, 5361, 5355, 5115, 5317, 517…
#> $ taxonomy.bold       <int> 1856, 4061, 1858, 1851, 586, 1313, 1855, 1126, 49…
#> $ taxonomy.gbif       <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
#> $ taxonomy.col        <chr> "81a86c329909d507edb5c296906ef3f4", "94532a14786a…
#> $ taxonomy.rank       <chr> "family", "genus", "family", "family", "family", …
#> $ taxonomy.created_at <chr> "2019-02-23T17:04:35.620Z", "2019-02-23T17:04:35.…
#> $ taxonomy.updated_at <chr> "2019-06-14T15:25:46.438Z", "2019-06-14T15:25:46.…
#> $ taxonomy            <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
glimpse(nets_lagoons[[1]]$edges)
#>  NULL
glimpse(nets_lagoons[[1]]$dataset)
#> Observations: 1
#> Variables: 8
#> $ dataset_id  <int> 22
#> $ name        <chr> "zetina_2003"
#> $ description <chr> "Dietary matrix of the Huizache–Caimanero lagoon"
#> $ public      <lgl> TRUE
#> $ created_at  <chr> "2019-02-23T17:04:32.017Z"
#> $ updated_at  <chr> "2019-02-23T17:04:32.017Z"
#> $ ref_id      <int> 22
#> $ user_id     <int> 3
glimpse(nets_lagoons[[1]]$reference)
#> Observations: 1
#> Variables: 11
#> $ ref_id       <int> 22
#> $ doi          <chr> "10.1016/s0272-7714(02)00410-9"
#> $ first_author <chr> "manuel j. zetina-rejon"
#> $ year         <chr> "2003"
#> $ jstor        <lgl> NA
#> $ pmid         <lgl> NA
#> $ bibtex       <chr> "@article{Zetina_Rej_n_2003, doi = {10.1016/s0272-7714(0…
#> $ paper_url    <chr> "https://doi.org/10.1016%2Fs0272-7714%2802%2900410-9"
#> $ data_url     <chr> "https://globalwebdb.com/"
#> $ created_at   <chr> "2019-02-23T17:04:28.307Z"
#> $ updated_at   <chr> "2019-02-23T17:04:28.307Z"

Integrated workflow with rmangal

Batch analysis

So far, the search functions of rmangal allow the user to perform only a single search at a time. The simplest way to do more than one search is to loop over a vector or a list of queries. Below we exemplify how to do so using lapply():

Geolocalize Mangal networks with sf

The function get_collection() has an argument as_sf than converts network metadata of mgNetwork objects to sf objects, which requires sf to be installed. This allows the user to easily geolocalize the networks retrieved from Mangal.

Taxonomic analysis with taxize

As Mangal includes taxonomic identifiers, rmangal can readily be combined with taxize (see taxize for more details about this package):

Network manipulation and visualization with tidygraph and ggraph

The package tidygraph treats networks as two tidy tables (one for the edges and one for the nodes) that can be modified using the grammar of data manipulation developed in the tidyverse. Moreover, tidygraph wraps over most of the igraph functions so that the user can call a vast variety of algorithms to properly analysis networks. Fortunately, objects of class mgNetwork can readily be converted into tbl_graph objects which allows the user to benefit from all the tools included in tidygraph:

library(tidygraph)
# NB the line below would not work with a mgNetworksCollection (use lapply)
tg_lagoons <-  as_tbl_graph(mg_lagoons[[1]]) %>%
  mutate(centrality_dg = centrality_degree(mode = 'in'))
tg_lagoons %E>% as_tibble
#> # A tibble: 189 x 19
#>     from    to interaction_id date  direction type  method attr_id value public
#>    <int> <int>          <int> <chr> <chr>     <chr> <lgl>    <int> <dbl> <lgl> 
#>  1     9     9          48376 2003… directed  pred… NA          12 0.026 TRUE  
#>  2     9    11          48377 2003… directed  pred… NA          12 0.025 TRUE  
#>  3     9    12          48378 2003… directed  pred… NA          12 0.003 TRUE  
#>  4     9    15          48379 2003… directed  pred… NA          12 0.009 TRUE  
#>  5     9    16          48380 2003… directed  pred… NA          12 0.009 TRUE  
#>  6     9    17          48381 2003… directed  pred… NA          12 0.016 TRUE  
#>  7     9    18          48382 2003… directed  pred… NA          12 0.284 TRUE  
#>  8     9    19          48383 2003… directed  pred… NA          12 0.231 TRUE  
#>  9     9    21          48384 2003… directed  pred… NA          12 0.079 TRUE  
#> 10     9    22          48385 2003… directed  pred… NA          12 0.09  TRUE  
#> # … with 179 more rows, and 9 more variables: network_id <int>,
#> #   created_at <chr>, updated_at <chr>, attribute.id <int>,
#> #   attribute.name <chr>, attribute.description <chr>, attribute.unit <lgl>,
#> #   attribute.created_at <chr>, attribute.updated_at <chr>
tg_lagoons %N>% as_tibble %>%
  select(original_name, taxonomy.tsn, centrality_dg)
#> # A tibble: 26 x 3
#>    original_name  taxonomy.tsn centrality_dg
#>    <chr>                 <int>         <dbl>
#>  1 Scianids             169237             1
#>  2 Elopids               28630             0
#>  3 Lutjanids            168845             1
#>  4 Carangids            168584             2
#>  5 Centropomids         167642             2
#>  6 Ariids                43998             1
#>  7 Haemulids            169055             4
#>  8 Pleuronectoids       172859             3
#>  9 Callinectes           13951             6
#> 10 Belonoids            165546             4
#> # … with 16 more rows

Another strong advantage of tbl_graph objects is that there are the objects used by the package ggraph that that offers various functions (theme, geoms, etc.) to efficiently visualize networks:

library(ggraph)
ggraph(tg_lagoons, layout = "stress") +
  geom_edge_parallel(end_cap = circle(.5), start_cap = circle(.5),
        arrow = arrow(length = unit(1, 'mm'), type = 'closed')) +
  geom_node_point(aes(colour = taxonomy.rank), size = 8) +
  theme_graph(background = "grey40", foreground = NA, text_colour = 'white')

References

Chamberlain, Scott, Eduard Szoecs, Zachary Foster, Zebulun Arendsee, Carl Boettiger, Karthik Ram, Ignasi Bartomeus, et al. 2019. Taxize: Taxonomic Information from Around the Web. https://github.com/ropensci/taxize.

Poisot, Timothée, Benjamin Baiser, Jennifer A. Dunne, Sonia Kéfi, François Massol, Nicolas Mouquet, Tamara N. Romanuk, Daniel B. Stouffer, Spencer A. Wood, and Dominique Gravel. 2016. “Mangal - Making Ecological Network Analysis Simple.” Ecography 39 (4): 384–90. https://doi.org/10.1111/ecog.00976.