ROpenSci submission status DOI Project Status: Active – The project has reached a stable, usable state and is being actively developed. Last-changedate GitHub issues CI - GitHub Actions CI - TravisCI CI - Appveyor Text coverage Hosted RStudio with ruODK

Especially in these trying times, it is important to ask “r u ODK?”.

ruODK is an R client to access and parse data from ODK Central.

OpenDataKit (ODK) is free-and open-source software that helps millions of people collect data quickly, accurately, offline, and at scale. The software is in active use in every country in the world and is supported by a large and helpful community.

ruODK is a community contribution to the ODK ecosystem, but not directly affiliated with ODK.

ruODK assumes some familiarity of its users with the ODK ecosystem and workflows. For a detailed overview, read the extensive ODK documentation and visit the friendly ODK forum.

ODK Central is a cloud-based data clearinghouse for digitally captured data, replacing the older software ODK Aggregate. ODK Central manages user accounts and permissions, stores form definitions, and allows data collection clients like ODK Collect to connect to it for form download and submission upload.

An ODK setup with ODK Build, Central, Collect, and ruODK

A typical ODK workflow: An XForm is designed e.g. in ODK Build, published to ODK Central, and downloaded onto an Android device running ODK Collect. After data have been captured digitally using ODK Collect, the data are uploaded and stored in ODK Central. The next step from there is to extract the data, optionally upload it into another data warehouse, and then to analyse and generate insight from it.

While data can be retrieved in bulk through the GUI, ODK Central’s API provides access to its data and functionality through both an OData and a RESTful API with a comprehensive and interactive documentation.

ruODK is aimed at the technically minded researcher who wishes to access and process data from ODK Central using the programming language

R.

Benefits of using the R ecosystem in combination with ODK:

  • Scalability: Both R and ODK are free and open source software. Scaling to many users does not incur license fees.
  • Ubiquity: R is known to many scientists and is widely taught at universities.
  • Automation: The entire data access and analysis workflow can be automated through R scripts.
  • Reproducible reporting (e.g.  Sweave, RMarkdown), interactive web apps (Shiny), workflow scaling (drake).
  • Rstudio-as-a-Service (RaaS) at Binder

ruODK’s scope:

  • To wrap all ODK Central API endpoints with a focus on data access.
  • To provide working examples of interacting with the ODK Central API.
  • To provide convenience helpers for the day to day tasks when working with ODK Central data in R: data munging the ODK Central API output into tidy R formats.

ruODK’s use cases:

  • Smaller projects: Example rOzCBI
    1. Data collection: ODK Collect
    2. Data clearinghouse: ODK Central
    3. Data analysis and reporting: Rmd (ruODK)
    4. Publishing and dissemination: ckanr, CKAN
  • Larger projects:
    1. Data collection: ODK Collect
    2. Data clearinghouse: ODK Central
    3. ETL pipeline into data warehouses: Rmd (ruODK)
    4. QA: in data warehouse
    5. Reporting: Rmd
    6. Publishing and dissemination: ckanr, CKAN

Out of scope:

  • To wrap “management” API endpoints. ODK Central is a VueJS/NodeJS application which provides a comprehensive graphical user interface for the management of users, roles, permissions, projects, and forms.
  • To provide extensive data visualisation. We show only minimal examples of data visualisation and presentation, mainly to illustrate the example data. Once the data is in your hands as tidy tibbles… urODK!

A quick preview

ruODK screencast

Install

You can install the development version (main branch) of ruODK with:

if (!requireNamespace("remotes")) install.packages("remotes")
remotes::install_github("ropensci/[email protected]", dependencies = TRUE)

ODK Central

Access to an ODK Central instance

First, we need an ODK Central instance and some data to play with!

Either request a free trial or follow the setup instructions to build and deploy your very own ODK Central instance.

ODK Central setup

The ODK Central user manual provides up-to-date descriptions of the steps below.

  • Create a web user account on an ODK Central instance. Your username will be an email address.
  • Create a project and give the web user at least read permissions.
  • Create an XForm, e.g. using ODK Build, or use the example forms provided by ruODK. The .odkbuild versions can be loaded into ODK Build, while the .xml versions can be directly imported into ODK Central.
  • Publish the form to ODK Central.
  • Collect some data for this form on ODK Collect and let ODK Collect submit the finalised forms to ODK Central.

Configure ruODK

Set up ruODK with an OData Service URL and credentials of a read-permitted ODK Central web user. Adjust verbosity to your liking.

ruODK::ru_setup(
  svc = "https://sandbox.central.getodk.org/v1/projects/14/forms/build_Flora-Quadrat-0-4_1564384341.svc",
  un = "[email protected]",
  pw = "...",
  tz = "Australia/Perth",
  verbose = TRUE # great for demo or debugging
)

For all available detailed options to configure ruODK, read vignette("setup", package = "ruODK").

Use ruODK

A quick example browsing projects, forms, submissions, and accessing the data:

library(ruODK)
# Part 1: Data discovery ------------------------------------------------------#
# List projects
proj <- ruODK::project_list()
proj %>% head() %>% knitr::kable(.)
id name forms app_users created_at updated_at last_submission archived
115 1Computer_Scince_UZ_Training 1 5 2020-09-18 15:59:11 NA NA FALSE
59 1Horizon test project 1 6 2020-01-13 20:45:32 NA 2020-09-14 07:08:29 FALSE
57 AB’s 1 1 2020-01-06 08:56:00 NA NA FALSE
102 AK_Ste-Cecile 1 2 2020-08-24 22:57:56 NA NA FALSE
8 AmenazasRD 1 2 2019-04-14 14:21:00 NA 2020-04-21 14:23:06 FALSE
88 Amit-wfp-test 3 1 2020-06-12 10:09:57 NA 2020-06-12 13:03:00 FALSE
# List forms of default project
frms <- ruODK::form_list()
frms %>% head() %>% knitr::kable(.)
name fid version state submissions created_at created_by_id created_by updated_at last_submission hash
Field data collection September 2 open 1 2020-10-16 10:13:40 57 2020-10-16 10:16:15 2020-10-16 10:22:39 889ef491773157acc87b1fb3b849f3e7
Flora Quadrat 0.1 build_Flora-Quadrat-0-1_1558330379 closing 1 2019-05-20 13:33:15 57 2020-10-16 10:16:15 2019-05-20 13:44:20 4f0036619468ef05b572631b04b94f06
Flora Quadrat 0.2 build_Flora-Quadrat-0-2_1558575936 open 2 2019-05-23 09:46:08 57 2020-10-16 10:16:15 2019-05-23 11:12:16 14e269a2374132392c275117efbe67b6
Flora Quadrat 0.3 build_Flora-Quadrat-0-3_1559119570 open 1 2019-05-29 16:48:15 57 2020-10-16 10:16:15 2019-05-29 16:55:59 d5a80cefb1895eefcd0cb86a12d8acb4
Flora Quadrat 0.4 build_Flora-Quadrat-0-4_1564384341 open 2 2019-08-19 15:58:28 57 2020-10-16 10:16:15 2019-09-18 16:51:07 1bb959d541ac6990e3f74893e38c855b
Locations build_Locations_no_submissions NA open 0 2020-08-16 15:00:55 57 2020-10-16 10:16:15 NA NA
# Form details of default form
frmd <- ruODK::form_detail()
frmd %>% knitr::kable(.)
name fid version state submissions created_at created_by_id created_by updated_at last_submission hash
Flora Quadrat 0.4 build_Flora-Quadrat-0-4_1564384341 open 2 2019-08-19T07:58:28.212Z 57 2020-10-16T02:16:15.738Z 2019-09-18T08:51:07.482Z 1bb959d541ac6990e3f74893e38c855b
# Form schema: defaults to version 0.8
meta <- ruODK::form_schema(odkc_version = get_test_odkc_version())
#> ℹ Form schema v0.8
meta %>% knitr::kable(.)
path name type binary ruodk_name
/meta meta structure NA meta
/meta/instanceID instanceID string NA meta_instance_id
/encounter_start_datetime encounter_start_datetime dateTime NA encounter_start_datetime
/reporter reporter string NA reporter
/device_id device_id string NA device_id
/location location structure NA location
/location/area_name area_name string NA location_area_name
/location/quadrat_photo quadrat_photo binary TRUE location_quadrat_photo
/location/corner1 corner1 geopoint NA location_corner1
/habitat habitat structure NA habitat
/habitat/morphological_type morphological_type select1 NA habitat_morphological_type
/habitat/morphological_type_photo morphological_type_photo binary TRUE habitat_morphological_type_photo
/vegetation_stratum vegetation_stratum repeat NA vegetation_stratum
/vegetation_stratum/nvis_level3_broad_floristic_group nvis_level3_broad_floristic_group select1 NA vegetation_stratum_nvis_level3_broad_floristic_group
/vegetation_stratum/max_height_m max_height_m decimal NA vegetation_stratum_max_height_m
/vegetation_stratum/foliage_cover foliage_cover select1 NA vegetation_stratum_foliage_cover
/vegetation_stratum/dominant_species_1 dominant_species_1 string NA vegetation_stratum_dominant_species_1
/vegetation_stratum/dominant_species_2 dominant_species_2 string NA vegetation_stratum_dominant_species_2
/vegetation_stratum/dominant_species_3 dominant_species_3 string NA vegetation_stratum_dominant_species_3
/vegetation_stratum/dominant_species_4 dominant_species_4 string NA vegetation_stratum_dominant_species_4
/perimeter perimeter structure NA perimeter
/perimeter/corner2 corner2 geopoint NA perimeter_corner2
/perimeter/corner3 corner3 geopoint NA perimeter_corner3
/perimeter/corner4 corner4 geopoint NA perimeter_corner4
/perimeter/mudmap_photo mudmap_photo binary TRUE perimeter_mudmap_photo
/taxon_encounter taxon_encounter repeat NA taxon_encounter
/taxon_encounter/field_name field_name string NA taxon_encounter_field_name
/taxon_encounter/photo_in_situ photo_in_situ binary TRUE taxon_encounter_photo_in_situ
/taxon_encounter/taxon_encounter_location taxon_encounter_location geopoint NA taxon_encounter_taxon_encounter_location
/taxon_encounter/life_form life_form select1 NA taxon_encounter_life_form
/taxon_encounter/voucher_specimen_barcode voucher_specimen_barcode barcode NA taxon_encounter_voucher_specimen_barcode
/taxon_encounter/voucher_specimen_label voucher_specimen_label string NA taxon_encounter_voucher_specimen_label
/encounter_end_datetime encounter_end_datetime dateTime NA encounter_end_datetime
# Part 2: Data access ---------------------------------------------------------#
# Form tables
srv <- ruODK::odata_service_get()
srv %>% knitr::kable(.)
name kind url
Submissions EntitySet Submissions
Submissions.vegetation_stratum EntitySet Submissions.vegetation_stratum
Submissions.taxon_encounter EntitySet Submissions.taxon_encounter
# Form submissions
data <- ruODK::odata_submission_get(local_dir = fs::path("vignettes/media"),
                                    odkc_version = get_test_odkc_version())
#> ℹ Downloading submissions...
#> ✔ Downloaded submissions.
#> ℹ Reading form schema...
#> ℹ Form schema v0.8
#> ℹ Parsing submissions...
#> ℹ Not unnesting geo fields: value_location_corner1, value_perimeter_corner2, value_perimeter_corner3, value_perimeter_corner4, value_taxon_encounter_taxon_encounter_location
#> ℹ Unnesting: value
#> ℹ Unnesting column "value"
#> ℹ Unnesting more list cols: value___system, value_meta, value_location, value_habitat, value_perimeter
#> ℹ Not unnesting geo fields: value_location_corner1, value_perimeter_corner2, value_perimeter_corner3, value_perimeter_corner4, value_taxon_encounter_taxon_encounter_location
#> ℹ Unnesting: value___system, value_meta, value_location, value_habitat, value_perimeter
#> ℹ Unnesting column "value___system"
#> ℹ Unnesting column "value_meta"
#> ℹ Unnesting column "value_location"
#> ℹ Unnesting column "value_habitat"
#> ℹ Unnesting column "value_perimeter"
#> ℹ Found date/times: encounter_start_datetime, encounter_end_datetime.
#> ℹ Found attachments: location_quadrat_photo, habitat_morphological_type_photo, perimeter_mudmap_photo.
#> ℹ Downloading attachments...
#> ℹ Using local directory "vignettes/media".
#> ◉ File already downloaded, keeping "vignettes/media/1568794395624.jpg".
#> ◉ File already downloaded, keeping "vignettes/media/1568786958640.jpg".
#> ℹ Using local directory "vignettes/media".
#> ◉ File already downloaded, keeping "vignettes/media/1568794560256.jpg".
#> ◉ File already downloaded, keeping "vignettes/media/1568787004467.jpg".
#> ℹ Using local directory "vignettes/media".
#> ◯ Filename is NA, skipping download.
#> ◉ File already downloaded, keeping "vignettes/media/1568787172983.jpg".
#> ℹ Found geopoints: location_corner1, perimeter_corner2, perimeter_corner3, perimeter_corner4.
#> ℹ Parsing location_corner1...
#> ℹ Parsing perimeter_corner2...
#> ℹ Parsing perimeter_corner3...
#> ℹ Parsing perimeter_corner4...
#> ℹ Found geotraces: .
#> ℹ Found geoshapes: .
#> ✔ Returning parsed submissions.
data %>% dplyr::select(-"odata_context") %>% knitr::kable(.)
id encounter_start_datetime reporter device_id encounter_end_datetime system_submission_date system_submitter_id system_submitter_name system_attachments_present system_attachments_expected meta_instance_id location_area_name location_quadrat_photo location_corner1_longitude location_corner1_latitude location_corner1_altitude location_corner1_accuracy location_corner1 habitat_morphological_type habitat_morphological_type_photo vegetation_stratum_odata_navigation_link perimeter_corner2_longitude perimeter_corner2_latitude perimeter_corner2_altitude perimeter_corner2_accuracy perimeter_corner2 perimeter_corner3_longitude perimeter_corner3_latitude perimeter_corner3_altitude perimeter_corner3_accuracy perimeter_corner3 perimeter_corner4_longitude perimeter_corner4_latitude perimeter_corner4_altitude perimeter_corner4_accuracy perimeter_corner4 perimeter_mudmap_photo taxon_encounter_odata_navigation_link
uuid:d5e78a78-34db-483d-978f-d9c9a3bc7b69 2019-09-18 16:12:21 Florian Mayer f73d2e1221ceaa06 2019-09-18 16:40:46 2019-09-18T08:51:07.482Z 241 flora 4 4 uuid:d5e78a78-34db-483d-978f-d9c9a3bc7b69 Kensington Carpark 01 vignettes/media/1568794395624.jpg 115.8846 -31.99606 6.40451 4.288 Point , 115.88456271 , -31.99605811 , 6.40451049804688, 4.288 mid-slope vignettes/media/1568794560256.jpg Submissions(‘uuid:d5e78a78-34db-483d-978f-d9c9a3bc7b69’)/vegetation_stratum 115.8844 -31.99623 -26.305695 4.288 Point , 115.88435548 , -31.99622531 , -26.3056945800781, 4.288 115.8844 -31.99615 -17.897552 4.288 Point , 115.88436077 , -31.99614759 , -17.8975524902344, 4.288 115.8843 -31.99610 -27.759338 4.288 Point , 115.88432984 , -31.99610179 , -27.7593383789062, 4.288 NA Submissions(‘uuid:d5e78a78-34db-483d-978f-d9c9a3bc7b69’)/taxon_encounter
uuid:529cb189-8bb2-4cf1-9041-dcde716efb4f 2019-09-18 14:08:43 Florian Mayer f73d2e1221ceaa06 2019-09-18 14:18:53 2019-09-18T06:20:25.781Z 241 flora 8 8 uuid:529cb189-8bb2-4cf1-9041-dcde716efb4f Kensington Planter Box 1 vignettes/media/1568786958640.jpg 115.8843 -31.99615 -17.37241 4.288 Point , 115.88433971 , -31.99614843 , -17.3724060058594, 4.288 flat vignettes/media/1568787004467.jpg Submissions(‘uuid:529cb189-8bb2-4cf1-9041-dcde716efb4f’)/vegetation_stratum 115.8844 -31.99620 -8.662476 4.288 Point , 115.88438584 , -31.99620073 , -8.6624755859375, 4.288 115.8844 -31.99622 -6.266144 4.288 Point , 115.88440369 , -31.99622144 , -6.26614379882812, 4.288 115.8844 -31.99621 -6.597748 4.288 Point , 115.88438399 , -31.99620546 , -6.59774780273438, 4.288 vignettes/media/1568787172983.jpg Submissions(‘uuid:529cb189-8bb2-4cf1-9041-dcde716efb4f’)/taxon_encounter

A more detailed walk-through with some data visualisation examples is available in the vignette("odata-api", package="ruODK").

See also vignette("restful-api", package="ruODK") for examples using the alternative RESTful API.

Try ruODK

Binder will launch a disposable, hosted RStudio instance with ruODK installed and the companion package urODK opened as starting point for a hands-on workshop or instant demo of ruODK usage.

Create a new RMarkdown workbook from ruODK template “ODK Central via OData” and follow the instructions within.

Contribute

Contributions through issues and PRs are welcome!

See the contributing guide on best practices and further readings for code contributions.

Attribution

ruODK was developed, and is maintained, by Florian Mayer for the Western Australian Department of Biodiversity, Conservation and Attractions (DBCA). The development was funded both by DBCA core funding and external funds from the North West Shelf Flatback Turtle Conservation Program.

To cite package ruODK in publications use:

citation("ruODK")
#> 
#> To cite ruODK in publications use:
#> 
#>   Mayer, Florian Wendelin. (2020, July 21).  ruODK: An R Client for the
#>   ODK Central API (Version 0.9.1).  Zenodo.
#>   https://doi.org/10.5281/zenodo.3953158
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Misc{,
#>     title = {ruODK: Client for the ODK Central API},
#>     author = {Florian W. Mayer},
#>     note = {R package version 0.9.1},
#>     year = {2020},
#>     url = {https://github.com/ropensci/ruODK},
#>   }

Acknowledgements

The Department of Biodiversity, Conservation and Attractions (DBCA) acknowledges the traditional owners of country throughout Western Australia and their continuing connection to the land, waters and community. We pay our respects to them, their culture and to their Elders past and present.

This software was created both as a contribution to the ODK ecosystem and for the conservation of the biodiversity of Western Australia, and in doing so, caring for country.

Package functionality

See vignette("comparison", package="ruODK") for a comprehensive comparison of ruODK to other software packages from both an ODK and an OData angle.