Parallelize spatial computation by hierarchy in input data

"Hierarchy" refers to a system, which divides the entire study region into multiple subregions. It is oftentimes reflected in an area code system (e.g., FIPS for US Census geographies and Nomenclature of Territorial Units for Statistics (NUTS), etc.). future::multisession, future::multicore, future::cluster, future.mirai::mirai_multisession in future::plan will parallelize the work by splitting lower level features into several higher level feature group. For details of the terminology in future package, please refer to future::plan documentation. Each thread will process the number of lower level features in each higher level feature. Please be advised that accessing the same file simultaneously with multiple processes may result in errors.

Usage

par_hierarchy(
  regions,
  regions_id = NULL,
  length_left = NULL,
  pad = 0,
  pad_y = FALSE,
  fun_dist,
  ...,
  .debug = FALSE
)

Arguments

regions

sf/SpatVector object. Computational regions. Only polygons are accepted.

regions_id

character(1). Name of unique ID field in regions. The regions will be split by the common level value.

length_left

integer(1). Length of the first characters of the regions_id values. Default is NULL, which will not manipulate the regions_id values. If the number of characters is not consistent (for example, numerics), the function will alert the user.

pad

numeric(1). Padding distance for each subregion defined by regions_id or trimmed regions_id values. in linear unit of coordinate system. Default is 0, which means each subregion is used as is. If the value is greater than 0, the subregion will be buffered by the value. The padding distance will be applied to x (pad_y = FALSE) or y (pad_y = TRUE) to filter the data.

pad_y

logical(1). Whether to filter y with the padded grid. Should be TRUE when x is where the values are calculated. Default is FALSE. In the reverse case, like terra::extent or exactextractr::exact_extract, the raster (x) should be scoped with the padded grid.

fun_dist

sf, terra, or chopin functions. This function should have x and y arguments.

...

Arguments passed to the argument fun_dist.

.debug

logical(1). Default is FALSE If a unit computation fails, the error message and the regions_id value where the error occurred will be included in the output.

Value

a data.frame object with computation results. For entries of the results, consult the function used in fun_dist argument.

Details

In dynamic dots (...), fun_dist arguments should include x and y where sf/terra class objects or file paths are accepted. Hierarchy is interpreted by the regions_id argument first. regions_id is assumed to be a field name in the x or y argument object. It is expected that regions represents the higher level boundaries and x or y in fun_dist is the lower level boundaries. However, if that is not the case, with trim argument, the function will generate the higher level codes from regions_id by extracting left-t Whether x or y is searched is determined by pad_y value. pad_y = TRUE will make the function attempt to find regions_id in x, whereas pad_y = FALSE will look for regions_id at y. If the regions_id doesn't exist in x or y, the function will utilize spatial relationship (intersects) to filter the data. Note that dispatching computation by subregions based on the spatial relationship may lead to a slight discrepancy in the result. For example, if the higher and lower level features are not perfectly aligned, there may be some features that are not included or duplicated in the subregions. The function will alert the user if spatial relation- ship is used to filter the data.

Note

Virtually any sf/terra functions that accept two arguments can be put in fun_dist, but please be advised that some spatial operations do not necessarily give the exact result from what would have been done with single thread. For example, distance calculated through this function may return the lower value than actual because the computational region was reduced. This would be the case especially where the target features are spatially sparsely distributed.

See also

future::multisession, future::multicore, future::cluster, future.mirai::mirai_multisession, future::plan, par_convert_f

Other Parallelization: par_cut_coords(), par_grid(), par_make_grid(), par_merge_grid(), par_multirasters(), par_pad_balanced(), par_pad_grid(), par_split_list()

Author

Insang Song geoissong@gmail.com

Examples

library(terra)
#> terra 1.7.78
library(sf)
library(future)
library(future.mirai)
options(sf_use_s2 = FALSE)
future::plan(future.mirai::mirai_multisession, workers = 2)

ncpath <- system.file("extdata/nc_hierarchy.gpkg", package = "chopin")
nccnty <- sf::st_read(ncpath, layer = "county")
#> Reading layer `county' from data source 
#>   `/usr/local/lib/R/site-library/chopin/extdata/nc_hierarchy.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 100 features and 1 field
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 1054155 ymin: 1341756 xmax: 1838923 ymax: 1690176
#> Projected CRS: NAD83 / Conus Albers
nctrct <- sf::st_read(ncpath, layer = "tracts")
#> Reading layer `tracts' from data source 
#>   `/usr/local/lib/R/site-library/chopin/extdata/nc_hierarchy.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 2672 features and 1 field
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 1054155 ymin: 1341756 xmax: 1838923 ymax: 1690176
#> Projected CRS: NAD83 / Conus Albers
ncelev <-
  system.file("extdata/nc_srtm15_otm.tif", package = "chopin")

ncsamp <-
  sf::st_sample(
    nccnty,
    size = 1e4L
  )
# sfc to sf
ncsamp <- sf::st_as_sf(ncsamp)
# assign ID
ncsamp$kid <- sprintf("K-%05d", seq_len(nrow(ncsamp)))
res <-
  par_hierarchy(
    regions = nccnty,
    regions_id = "GEOID",
    fun_dist = extract_at,
    y = nctrct,
    x = ncelev,
    id = "GEOID",
    func = "mean"
  )
#> ℹ Input is not a character.
#> ℹ GEOID is used to stratify the process.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37037 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37001 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37057 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37069 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37155 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37109 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37027 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37063 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37145 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37115 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37151 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37131 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37013 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37159 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37051 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37153 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37093 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37025 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37029 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37169 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37031 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37005 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37139 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37193 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37003 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37083 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37163 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37189 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37173 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37011 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37045 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37125 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37067 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37077 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37185 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37137 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37033 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37107 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37075 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37073 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37161 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37187 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37007 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37135 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37049 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37195 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37061 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37087 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37081 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37099 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37097 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37091 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37149 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37165 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37085 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37105 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37017 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37039 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37035 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37177 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37113 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37143 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37095 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37071 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37101 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37015 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37167 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37079 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37129 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37147 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37141 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37179 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37121 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37133 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37065 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37119 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37199 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37197 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37023 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37191 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37059 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37111 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37183 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37053 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37103 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37041 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37021 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37157 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37117 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37089 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37127 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37009 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37019 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37123 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37181 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37175 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37171 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37043 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37055 is dispatched.
#> Input is a character. Attempt to read it with terra::rast...
#> ℹ Your input function at 37047 is dispatched.