rfema: Getting Started
Dylan Turner
2023-11-12
Source:vignettes/getting_started.Rmd
getting_started.RmdIntroduction
This vignette provides a brief overview on using the
rfema package to obtain data from the Open FEMA API. The
rest of this vignette covers how to install the package, followed by
examples on using the package to obtain data for various objectives.
Installation
Right now, the best way to install and use the rfema
package is by installing directly from rOpenSci using
install.packages("rfema", repos = "https://ropensci.r-universe.dev").
The FEMA API does not require and API key, meaning no further setup
steps need be taken to start using the package
Available Datasets
For those unfamiliar with the data sets available through the FEMA
API, a good starting place is to visit the FEMA API
documentation page. However, if you are already familiar with the
data and want to quickly reference the data set names or another piece
of meta data, using the fema_data_sets() function to obtain
a tibble of available data sets along with associated meta data is a
convenient option.
# store the avaliable data sets as an object in your R environment that can be referenced later
data_sets <- fema_data_sets()
# view data
data_sets## # A tibble: 51 × 35
## identifier name title descr…¹ webSe…² dataD…³ keyword modif…⁴ publi…⁵ conta…⁶ mbox acces…⁷
## <chr> <chr> <chr> <chr> <chr> <chr> <list> <chr> <chr> <chr> <chr> <chr>
## 1 openfema-53 Registr… Regi… "This … https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 2 openfema-68 NfipCom… NFIP… "This … https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 3 openfema-14 Registr… Regi… "This … https:… https:… <list> 2023-1… Federa… OpenFE… open… public
## 4 openfema-45 HazardM… Haza… "The d… https:… https:… <list> 2023-1… Federa… OpenFE… open… public
## 5 openfema-54 Housing… Hous… "This … https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 6 openfema-55 Housing… Hous… "The d… https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 7 openfema-34 Individ… Indi… "This … https:… https:… <list> 2020-0… Federa… OpenFE… open… public
## 8 openfema-24 FemaWeb… FEMA… "This … https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 9 openfema-33 Mission… Miss… "1.1 W… https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## 10 openfema-25 FemaWeb… FEMA… "This … https:… https:… <list> 2023-0… Federa… OpenFE… open… public
## # … with 41 more rows, 23 more variables: landingPage <list>, temporal <list>, api <lgl>,
## # version <int>, bureauCode <chr>, programCode <chr>, license <list>, theme <list>,
## # dataQuality <chr>, accrualPeriodicity <list>, language <chr>, references <list>,
## # issued <list>, recordCount <list>, depDate <list>, depApiMessage <list>, depWebMessage <list>,
## # depNewURL <list>, hash <chr>, lastRefresh <chr>, id <chr>, lastDataSetRefresh <list>,
## # distribution <list>, and abbreviated variable names ¹description, ²webService,
## # ³dataDictionary, ⁴modified, ⁵publisher, ⁶contactPoint, ⁷accessLevel
# print out just the names of the avaliable data sets without all the other meta data
paste(data_sets$title, sep = ", ")## [1] "Registration Intake and Individuals Household Program (RI-IHP)"
## [2] "NFIP Community Assistance Visits"
## [3] "Registration Intake and Individuals Household Program (RI-IHP)"
## [4] "Hazard Mitigation Assistance Projects"
## [5] "Housing Assistance Program Data - Owners"
## [6] "Housing Assistance Program Data - Renters"
## [7] "Individuals and Households Program - Valid Registrations"
## [8] "FEMA Web Disaster Summaries"
## [9] "Mission Assignments"
## [10] "FEMA Web Disaster Declarations"
## [11] "Disaster Declarations Summaries"
## [12] "OpenFEMA Data Set Fields"
## [13] "Disaster Declarations Summaries"
## [14] "Hazard Mitigation Grant Program - Property Acquisitions"
## [15] "FEMA Web Declaration Areas"
## [16] "Housing Assistance Program Data - Renters"
## [17] "IPAWS Archived Alerts"
## [18] "Hazard Mitigation Assistance Projects"
## [19] "FIMA NFIP Redacted Claims"
## [20] "FIMA NFIP Redacted Claims"
## [21] "Emergency Management Performance Grants"
## [22] "FEMA Regions"
## [23] "Emergency Management Performance Grants"
## [24] "Non-Disaster and Assistance to Firefighter Grants"
## [25] "Hazard Mitigation Assistance Mitigated Properties"
## [26] "Hazard Mitigation Grant Program - Disaster Summaries"
## [27] "FEMA Regions"
## [28] "Hazard Mitigation Assistance Mitigated Properties"
## [29] "Individual Assistance Housing Registrants - Large Disasters"
## [30] "HMA Subapplications Project Site Inventories"
## [31] "HMA Subapplications"
## [32] "FIMA NFIP Redacted Policies"
## [33] "HMA Subapplications By NFIP CRS Communities"
## [34] "Hazard Mitigation Grants"
## [35] "Housing Assistance Program Data - Owners"
## [36] "NFIP Community Engagements"
## [37] "FIMA NFIP Redacted Policies"
## [38] "OpenFEMA Data Sets"
## [39] "Public Assistance Applicants"
## [40] "Hazard Mitigation Assistance Mitigated Properties"
## [41] "Declaration Denials"
## [42] "Hazard Mitigation Plan Statuses"
## [43] "NFIP Community Assistance Contacts"
## [44] "Public Assistance Funded Projects Details"
## [45] "Public Assistance Funded Project Summaries"
## [46] "Public Assistance Grant Award Activities"
## [47] "Hazard Mitigation Assistance Projects"
## [48] "Hazard Mitigation Grant Program - Disaster Summaries"
## [49] "Public Assistance Applicants Program Deliveries"
## [50] "Hazard Mitigation Assistance Projects by NFIP CRS Communities"
## [51] "NFIP Community Status Book"Example Workflow
Once we know what data set we want to access, or perhaps if we want
to know more about what data is available in a given data set, we can
use the fema_data_fields() function to get a look at the
available data fields in a given data set by setting the “data_set”
parameter to one of the “name” columns in the data frame returned by the
fema_data_sets() function.
# obtain all the data fields for the NFIP Policies data set
df <- fema_data_fields(data_set = "fimaNfipPolicies")##
Obtaining Data: 1 out of 2 iterations (50% complete)
Obtaining Data: 2 out of 2 iterations (100%
## complete)
# Note: the data set field is not case sensative, meaning you do not need to
# use camel case names despite that being the convention in the FEMA documentation.
df <- fema_data_fields(data_set = "fimanfippolicies")
# view the data fields
df## # A tibble: 127 × 15
## datase…¹ openF…² datas…³ name title descr…⁴ type sortO…⁵ isSea…⁶ isNes…⁷ isNul…⁸ prima…⁹ id
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 openfem… FimaNf… 2 poli… Poli… Date u… date 29 TRUE FALSE TRUE FALSE 3558…
## 2 openfem… FimaNf… 1 base… Base… Baseme… smal… 3 TRUE FALSE FALSE FALSE 689d…
## 3 openfem… FimaNf… 1 cond… Cond… This i… text 6 TRUE FALSE FALSE FALSE ca12…
## 4 openfem… FimaNf… 1 crsC… CRS … The Co… smal… 9 TRUE FALSE FALSE FALSE 1923…
## 5 openfem… FimaNf… 1 dedu… Dedu… The to… text 10 TRUE FALSE FALSE FALSE a3fb…
## 6 openfem… FimaNf… 1 dedu… Dedu… The to… text 11 TRUE FALSE FALSE FALSE 7c1b…
## 7 openfem… FimaNf… 1 elev… Elev… Yes (Y… bool… 12 TRUE FALSE FALSE FALSE e9b1…
## 8 openfem… FimaNf… 1 elev… Elev… Indica… text 13 TRUE FALSE FALSE FALSE b59e…
## 9 openfem… FimaNf… 1 floo… Floo… NFIP F… text 16 TRUE FALSE FALSE FALSE 42ac…
## 10 openfem… FimaNf… 1 loca… Loca… Code t… smal… 20 TRUE FALSE FALSE FALSE a72e…
## # … with 117 more rows, 2 more variables: lastRefresh <chr>, hash <chr>, and abbreviated variable
## # names ¹datasetId, ²openFemaDataSet, ³datasetVersion, ⁴description, ⁵sortOrder, ⁶isSearchable,
## # ⁷isNestedObject, ⁸isNullable, ⁹primaryKeyThe FEMA API limits the number of records that can be returned in a
single query to 1000, meaning if we want more observations than that, a
loop is necessary to iterate over multiple API calls. The
open_fema function handles this process automatically, but
by default will issue a warning letting you know how many records match
your criteria and how many API calls it will take to retrieve all those
records and ask you to confirm the request before it starts retrieving
data (this behavior can be turned off by setting the
ask_before_call argument to FALSE).
Additionally an estimated time will be issued to give you a sense of how
long it will take to complete the request. For example, requesting the
entire NFIP claims data set via
open_fema(data_set = "fimaNfipClaims") will yield the
following output in the R console.
Calculating estimated API call time...
2600579 matching records found. At 1000 records per call, it will take 2601 individual API calls to get all matching records. It's estimated that this will take approximately 2.12 hours. Continue?
1 - Yes, get that data!, 0 - No, let me rethink my API call: Note that the estimated time is based on network conditions at the
initial time the call is being made and may not be accurate for large
data requests that take long enough for network conditions to potential
change significantly during the request. As an aside, for large data
requests, like downloading the entire data set, it will usually be
faster to perform a bulk download using the bulk_dl
function.
Alternatively, we could specify the top_n argument to limit the
number of records returned. Specifying top_n greater than 1000 will
initiate the same message letting you know how many iterations it will
take to get your data. If top_n is less than 1000, the API
call will automatically be carried out. In the case below, we will
return the first 10 records from the NFIP Claims data.
df <- open_fema(data_set = "fimaNfipClaims", top_n = 10)
df## # A tibble: 10 × 73
## agricu…¹ asOfDate basem…² polic…³ crsCl…⁴ dateOfLoss eleva…⁵ eleva…⁶ eleva…⁷
## <chr> <dttm> <chr> <chr> <chr> <dttm> <chr> <chr> <chr>
## 1 FALSE 2020-01-22 00:00:00 NULL 1 8 1998-02-07 00:00:00 FALSE NULL NULL
## 2 FALSE 2020-01-22 00:00:00 NULL 1 8 2005-08-29 00:00:00 FALSE NULL NULL
## 3 FALSE 2020-01-22 00:00:00 NULL 1 9 1998-09-28 00:00:00 FALSE NULL NULL
## 4 FALSE 2019-09-19 00:00:00 1 1 9 1994-10-07 00:00:00 FALSE NULL NULL
## 5 FALSE 2019-09-19 00:00:00 NULL 1 8 1996-03-11 00:00:00 FALSE NULL NULL
## 6 FALSE 2020-01-22 00:00:00 NULL 1 NULL 1998-02-03 00:00:00 TRUE NULL NULL
## 7 FALSE 2020-01-22 00:00:00 NULL 1 5 2017-08-27 00:00:00 FALSE NULL NULL
## 8 FALSE 2019-10-19 00:00:00 NULL 1 NULL 1992-09-11 00:00:00 FALSE NULL NULL
## 9 FALSE 2019-10-19 00:00:00 NULL 1 8 1998-09-28 00:00:00 FALSE NULL NULL
## 10 FALSE 2019-09-19 00:00:00 NULL 1 8 1995-03-11 00:00:00 FALSE NULL NULL
## # … with 64 more variables: baseFloodElevation <chr>, ratedFloodZone <chr>, houseWorship <chr>,
## # locationOfContents <chr>, lowestAdjacentGrade <chr>, lowestFloorElevation <chr>,
## # numberOfFloorsInTheInsuredBuilding <chr>, nonProfitIndicator <chr>, obstructionType <chr>,
## # occupancyType <chr>, originalConstructionDate <dttm>, originalNBDate <dttm>,
## # amountPaidOnBuildingClaim <chr>, amountPaidOnContentsClaim <chr>,
## # amountPaidOnIncreasedCostOfComplianceClaim <chr>, postFIRMConstructionIndicator <chr>,
## # rateMethod <chr>, smallBusinessIndicatorBuilding <chr>, …If we wanted to limit the columns returned we could do so by passing
a character vector of data fields to be included in the returned data
frame. The data fields for a given data set can be retrieved using the
fema_data_fields() function.
data_fields <- fema_data_fields("fimanfipclaims")
data_fields## # A tibble: 113 × 15
## datase…¹ openF…² datas…³ name title descr…⁴ type sortO…⁵ isSea…⁶ isNes…⁷ isNul…⁸ prima…⁹ id
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 openfem… FimaNf… 2 wate… Wate… Depth … smal… 61 TRUE FALSE TRUE FALSE 7e46…
## 2 openfem… FimaNf… 1 base… Base… Baseme… smal… 4 TRUE FALSE FALSE FALSE be90…
## 3 openfem… FimaNf… 1 cond… Cond… This i… text 6 TRUE FALSE FALSE FALSE 8690…
## 4 openfem… FimaNf… 1 comm… Comm… The Co… smal… 9 TRUE FALSE FALSE FALSE 264c…
## 5 openfem… FimaNf… 1 elev… Elev… Indica… smal… 12 TRUE FALSE TRUE FALSE a63c…
## 6 openfem… FimaNf… 1 floo… Floo… Flood … text 15 TRUE FALSE TRUE FALSE f727…
## 7 openfem… FimaNf… 1 loca… Loca… Code t… smal… 18 TRUE FALSE TRUE FALSE 9720…
## 8 openfem… FimaNf… 1 numb… Numb… Code t… smal… 22 TRUE FALSE FALSE FALSE 56d6…
## 9 openfem… FimaNf… 1 obst… Obst… Code t… smal… 24 TRUE FALSE TRUE FALSE 85b2…
## 10 openfem… FimaNf… 1 occu… Occu… Code i… smal… 25 TRUE FALSE FALSE FALSE 7c3b…
## # … with 103 more rows, 2 more variables: lastRefresh <chr>, hash <chr>, and abbreviated variable
## # names ¹datasetId, ²openFemaDataSet, ³datasetVersion, ⁴description, ⁵sortOrder, ⁶isSearchable,
## # ⁷isNestedObject, ⁸isNullable, ⁹primaryKeyIn this case we will return only the policyCount and
countyCode columns.
## # A tibble: 10 × 2
## policyCount countyCode
## <chr> <chr>
## 1 1 06073
## 2 1 22071
## 3 1 12113
## 4 1 45013
## 5 1 12009
## 6 1 51810
## 7 1 48201
## 8 1 15003
## 9 1 12087
## 10 1 06053If we want to limit the rows returned rather than the columns, we can
also apply filters by specifying values of the columns to return. If we
want to quickly see the set of variables that can be used to filter API
queries with, we can use the valid_parameters() function to
return a tibble containing the variables that are “searchable” for a
particular data set.
params <- valid_parameters(data_set = "fimaNfipClaims")
params## # A tibble: 63 × 1
## params
## <chr>
## 1 waterDepth
## 2 basementEnclosureCrawlspaceType
## 3 crsClassificationCode
## 4 elevationCertificateIndicator
## 5 elevationDifference
## 6 ratedFloodZone
## 7 locationOfContents
## 8 numberOfFloorsInTheInsuredBuilding
## 9 obstructionType
## 10 buildingDeductibleCode
## # … with 53 more rowsWe can see from the above that both waterDepth and
ratedfloodZone are both searchable variables. Thus we can
specify a list that contains the values of each variable that we want
returned. Before doing that however, it can be useful to learn a bit
more about each parameter by using the parameter_values()
function.
# get more information onf the "ratedfloodZone" parameter from the NFIP Claims data set
parameter_values(data_set = "fimaNfipClaims",data_field = "ratedFloodZone")## Data Set: FimaNfipClaims
## Data Field: ratedFloodZone
## Data Field Description: Formerly called floodZone. NFIP Flood Zone derived from the Flood Insurance Rate Map (FIRM) used to rate the insured property.A - Special Flood with no Base Flood Elevation on FIRM; AE, A1-A30 - Special Flood with Base Flood Elevation on FIRM; A99 - Special Flood with Protection Zone; AH, AHB* - Special Flood with Shallow Ponding; AO, AOB* - Special Flood with Sheet Flow; X, B - Moderate Flood from primary water source. Pockets of areas subject to drainage problems; X, C - Minimal Flood from primary water source. Pockets of areas subject to drainage problems; D - Possible Flood; V - Velocity Flood with no Base Flood Elevation on FIRM; VE, V1-V30 - Velocity Flood with Base Flood Elevation on FIRM; AE, VE, X - New zone designations used on new maps starting January 1, 1986, in lieu of A1-A30, V1-V30, and B and C; AR - A Special Flood Hazard Area that results from the decertification of a previously accredited flood protection system that is determined to be in the process of being restored to provide base flood protection;AR Dual Zones - (AR/AE, AR/A1-A30, AR/AH, AR/AO, AR/A) Areas subject to flooding from failure of the flood protection system (Zone AR) which also overlap an existing Special Flood Hazard Area as a dual zone; *AHB, AOB, ARE, ARH, ARO, and ARA are not risk zones shown on a map, but are acceptable values for rating purposes
## Data Field Example Values: c("X", "AE", "VE", "NULL", "C", "V")
## More Information Available at: https://www.fema.gov/about/openfema/data-setsAs can be seen, parameter_values() returns the data set
name, the data field (i.e. the searchable parameter), a description of
the data field, and a vector of examples of the data field values which
can be useful for seeing how the values are formatted in the data.
We can see from the above that ratedFloodZone is a
character in the data and from the description we know that “AE” and “X”
are both valid values for the ratedFloodZone parameter. We
can thus define a filter to return only records from AE and X flood
zones.
# construct a filter that limits records to those in AE flood zones
my_filters <- list(ratedFloodZone = c("AE","X"))
# pass the filter to the open_fema function.
df <- open_fema(data_set = "fimaNfipclaims", top_n = 10,
select = c("policyCount","ratedFloodZone"),
filters = my_filters)
df## # A tibble: 10 × 2
## policyCount ratedFloodZone
## <chr> <chr>
## 1 1 X
## 2 1 X
## 3 1 X
## 4 1 X
## 5 1 X
## 6 1 AE
## 7 1 X
## 8 1 AE
## 9 1 AE
## 10 1 XMore Examples
Example: Return the first 100 NFIP claims for Florida that happened between 2010 and 2020.
df <- open_fema(data_set = "fimaNfipClaims",
top_n = 100,
filters = list(state = "FL",
yearOfLoss = ">= 2010",
yearOfLoss = "<= 2020"))
df## # A tibble: 100 × 73
## agricu…¹ asOfDate basem…² polic…³ crsCl…⁴ dateOfLoss eleva…⁵ eleva…⁶ eleva…⁷
## <chr> <dttm> <chr> <chr> <chr> <dttm> <chr> <chr> <chr>
## 1 FALSE 2020-01-22 00:00:00 NULL 1 7 2012-09-06 00:00:00 FALSE NULL 3
## 2 FALSE 2020-01-22 00:00:00 NULL 1 7 2017-10-07 00:00:00 FALSE NULL NULL
## 3 FALSE 2021-10-20 00:00:00 NULL 1 6 2014-04-30 00:00:00 TRUE NULL -3
## 4 FALSE 2020-01-22 00:00:00 NULL 1 5 2017-09-10 00:00:00 FALSE NULL NULL
## 5 FALSE 2022-06-17 00:00:00 NULL 1 7 2017-09-09 00:00:00 FALSE NULL NULL
## 6 FALSE 2020-01-22 00:00:00 NULL 1 6 2012-06-09 00:00:00 FALSE NULL NULL
## 7 FALSE 2020-01-22 00:00:00 NULL 1 8 2011-10-08 00:00:00 FALSE NULL NULL
## 8 FALSE 2020-01-22 00:00:00 NULL 1 6 2014-04-29 00:00:00 FALSE NULL NULL
## 9 FALSE 2020-01-22 00:00:00 NULL 1 5 2012-06-24 00:00:00 FALSE NULL NULL
## 10 FALSE 2023-07-31 00:00:00 NULL 1 7 2014-04-29 00:00:00 TRUE NULL 3
## # … with 90 more rows, 64 more variables: baseFloodElevation <chr>, ratedFloodZone <chr>,
## # houseWorship <chr>, locationOfContents <chr>, lowestAdjacentGrade <chr>,
## # lowestFloorElevation <chr>, numberOfFloorsInTheInsuredBuilding <chr>,
## # nonProfitIndicator <chr>, obstructionType <chr>, occupancyType <chr>,
## # originalConstructionDate <dttm>, originalNBDate <dttm>, amountPaidOnBuildingClaim <chr>,
## # amountPaidOnContentsClaim <chr>, amountPaidOnIncreasedCostOfComplianceClaim <chr>,
## # postFIRMConstructionIndicator <chr>, rateMethod <chr>, smallBusinessIndicatorBuilding <chr>, …Example: Get data on all Hazard Mitigation Assistance Projects associated with flood mitigation in Florida.
# see which parameter can be used for filtering the Hazard Mitigation Grants data set
valid_parameters("HazardMitigationAssistanceProjects") ## # A tibble: 29 × 1
## params
## <chr>
## 1 status
## 2 subrecipient
## 3 projectIdentifier
## 4 programArea
## 5 programFy
## 6 region
## 7 state
## 8 stateNumberCode
## 9 county
## 10 countyCode
## # … with 19 more rows
# see how values of "programArea" are formatted
params <- parameter_values(data_set = "HazardMitigationAssistanceProjects", data_field = "programArea", message = F)
params## # A tibble: 3 × 4
## `Data Set` `Data Field` `Data Field Description` Data F…¹
## <chr> <chr> <chr> <list>
## 1 HazardMitigationAssistanceProjects programArea Hazard Mitigation Assistance grant prog… <tibble>
## 2 HazardMitigationAssistanceProjects programArea Hazard Mitigation Assistance grant prog… <tibble>
## 3 HazardMitigationAssistanceProjects programArea Hazard Mitigation Assistance grant prog… <tibble>
## # … with abbreviated variable name ¹`Data Field Example Values`
# check to see how "state" is formatted
params <- parameter_values(data_set = "HazardMitigationAssistanceProjects", data_field = "state", message = F)
params## # A tibble: 3 × 4
## `Data Set` `Data Field` `Data Field Description` Data F…¹
## <chr> <chr> <chr> <list>
## 1 HazardMitigationAssistanceProjects state Full name of the State (e.g., Virginia)… <tibble>
## 2 HazardMitigationAssistanceProjects state Full name of the State (e.g., Virginia)… <tibble>
## 3 HazardMitigationAssistanceProjects state Full name of the State (e.g., Virginia)… <tibble>
## # … with abbreviated variable name ¹`Data Field Example Values`
# construct a list containing filters for Flood Mitigation Assistance projects in Florida
filter_list <- c(programArea = c("FMA"),
state = c("Florida"))
# pass filter_list to the open_fema function to retreieve data.
df <- open_fema(data_set = "HazardMitigationAssistanceProjects", filters = filter_list,
ask_before_call = FALSE)
df## # A tibble: 345 × 31
## projectIden…¹ progr…² progr…³ region state state…⁴ county count…⁵ disas…⁶ proje…⁷ proje…⁸ status
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 FMA-PJ-04-FL… FMA 2005 4 Flor… 12 Browa… 11 NULL BROWARD 405.1:… Closed
## 2 FMA-PJ-04-FL… FMA 2005 4 Flor… 12 Volus… 127 NULL VOLUSIA 200.1:… Closed
## 3 FMA-PJ-04-FL… FMA 2017 4 Flor… 12 Clay 19 NULL CLAY 200.1:… Closed
## 4 FMA-PJ-04-FL… FMA 2018 4 Flor… 12 Clay 19 NULL CLAY 202.1:… Oblig…
## 5 FMA-PJ-04-FL… FMA 2008 4 Flor… 12 Volus… 127 NULL VOLUSIA 202.2:… Closed
## 6 FMA-PJ-04-FL… FMA 2007 4 Flor… 12 Volus… 127 NULL VOLUSIA 202.1:… Closed
## 7 FMA-PJ-04-FL… FMA 2009 4 Flor… 12 Volus… 127 NULL VOLUSIA 202.1:… Closed
## 8 FMA-PJ-04-FL… FMA 2016 4 Flor… 12 Volus… 127 NULL VOLUSIA 200.1:… Closed
## 9 FMA-PJ-04-FL… FMA 2009 4 Flor… 12 Volus… 127 NULL VOLUSIA 200.2:… Closed
## 10 FMA-PJ-04-FL… FMA 2012 4 Flor… 12 Volus… 127 NULL VOLUSIA 202.1:… Closed
## # … with 335 more rows, 19 more variables: recipient <chr>, recipientTribalIndicator <chr>,
## # subrecipient <chr>, subrecipientTribalIndicator <chr>, dataSource <chr>, dateApproved <dttm>,
## # dateClosed <dttm>, dateInitiallyApproved <dttm>, projectAmount <chr>,
## # federalShareObligated <chr>, subrecipientAdminCostAmt <chr>, srmcObligatedAmt <chr>,
## # recipientAdminCostAmt <chr>, costSharePercentage <chr>, benefitCostRatio <chr>,
## # netValueBenefits <chr>, numberOfFinalProperties <chr>, numberOfProperties <chr>, id <chr>, and
## # abbreviated variable names ¹projectIdentifier, ²programArea, ³programFy, ⁴stateNumberCode, …Example: Determine how much money was awarded by FEMA for rental assistance following Hurricane Irma.
Get a dataset description for the
HousingAssistanceRenters data set to see if this is the
right data set for the question
# get meta data for the `HousingAssistanceRenters`
ds <- fema_data_sets()
ds <- ds[which(ds$name == "HousingAssistanceRenters"),]
# there are two entries corresponding to two versions of the data set,
# we want the most recent one
nrow(ds)## [1] 2
ds <- ds[which(ds$version == max(as.numeric(ds$version))),]
# now print out the data set description and make sure its the data set
# that applicable or our research question
print(ds$description)## [1] "The dataset was generated by FEMA’s Enterprise Coordination & Information Management (ECIM) Reporting team and is primarily composed of data from Housing Assistance Program reporting authority from FEMA registration renters and owners within the state, county, zip where the registration is valid. \n\nThis dataset contains aggregated, non-PII data on FEMA’s Housing Assistance Program within the state, county, zip where the registration is valid for the declarations, starting with disaster declarations number 4116. The data is divided into data for renters and data for property owners. Additional core data elements include number of applicants, county, zip code, severity of damage, owner or renter. Data is self-reported and as such is subject to human error. To learn more about disaster assistance please visit https://www.fema.gov/individual-disaster-assistance.\n\nThis is raw, unedited data from FEMA's National Emergency Management Information System (NEMIS) and as such is subject to a small percentage of human error. For example, when an applicant registers they enter their street and city address. The system runs a check and suggests a county. The applicant, if registering online can override that choice. If they are registering via the call center the Human Services Specialist (HSS) representatives are instructed to ask (not offer) what county they live in. So even though the system might suggest County A, an applicant has the right to choose County B. \n\nThe financial information is derived from NEMIS and not FEMA's official financial systems. Due to differences in reporting periods, status of obligations and how business rules are applied, this financial information may differ slightly from official publication on public websites such as usaspending.gov; this dataset is not intended to be used for any official federal financial reporting.\n\nCitation: The Agency’s preferred citation for datasets (API usage or file downloads) can be found on the OpenFEMA Terms and Conditions page, Citing Data section: https://www.fema.gov/about/openfema/terms-conditions.\n\nIf you have media inquiries about this dataset, please email the FEMA News Desk FEMA-News-Desk@fema.dhs.gov or call (202) 646-3272. For inquiries about FEMA's data and Open government program please contact the OpenFEMA team via email OpenFEMA@fema.dhs.gov."See which columns we can filter on to select just Hurricane Irma related grants
# see which parameter can be used for filtering the Housing Assistance for Renters
valid_parameters("HousingAssistanceRenters") ## # A tibble: 21 × 1
## params
## <chr>
## 1 disasterNumber
## 2 state
## 3 county
## 4 city
## 5 zipCode
## 6 validRegistrations
## 7 totalInspected
## 8 totalInspectedWithNoDamage
## 9 totalWithModerateDamage
## 10 totalWithMajorDamage
## # … with 11 more rowsAll we have in this data set is the disasterNumber.
Thus, to filter on a specific disaster we have to load the
FemaWebDisasterDeclarations data find the disaster number
associated with the event we are interested in.
# call the disaster declarations
dd <- rfema::open_fema(data_set = "FemaWebDisasterDeclarations", ask_before_call = F)##
Obtaining Data: 1 out of 5 iterations (20% complete)
Obtaining Data: 2 out of 5 iterations (40%
## complete)
Obtaining Data: 3 out of 5 iterations (60% complete)
Obtaining Data: 4 out of 5 iterations
## (80% complete)
Obtaining Data: 5 out of 5 iterations (100% complete)
# filter disaster declarations to those with "hurricane" in the name
hurricanes <- distinct(dd %>% filter(grepl("hurricane",tolower(disasterName))) %>% select(disasterName, disasterNumber))
hurricanes## # A tibble: 393 × 2
## disasterName disasterNumber
## <chr> <chr>
## 1 HURRICANE IDALIA 4734
## 2 HURRICANE IDALIA 4738
## 3 HURRICANE LANE 3399
## 4 HURRICANE IAN 4673
## 5 HURRICANE HENRI 3563
## 6 HURRICANE LEE 3599
## 7 HURRICANE LEE 3598
## 8 HURRICANE IDALIA 3597
## 9 HURRICANE NICOLE 4680
## 10 HURRICANE IAN 4677
## # … with 383 more rowsWe can see immediately that disaster numbers do not uniquely identify an event, since multiple disaster declarations may be declared for the same event, but in different locations. Thus to filter on a particular event, we need to collect all the disaster declaration numbers corresponding to that event (in this case Hurricane Irma).
# get all disaster declarations associated with hurricane irma.
# notice the use of grepl() which picked up a disaster declaration name
# that was different than all the others.
dd_irma <- hurricanes %>% filter(grepl("irma",tolower(disasterName)))
dd_irma## # A tibble: 13 × 2
## disasterName disasterNumber
## <chr> <chr>
## 1 HURRICANE IRMA 4346
## 2 HURRICANE IRMA - SEMINOLE TRIBE OF FLORIDA 4341
## 3 HURRICANE IRMA 4338
## 4 HURRICANE IRMA 3389
## 5 HURRICANE IRMA 4337
## 6 HURRICANE IRMA 4336
## 7 HURRICANE IRMA 3388
## 8 HURRICANE IRMA 3387
## 9 HURRICANE IRMA 3386
## 10 HURRICANE IRMA 4335
## 11 HURRICANE IRMA 3385
## 12 HURRICANE IRMA 3384
## 13 HURRICANE IRMA 3383
# get a vector of just the disaster declaration numbers
dd_nums_irma <- dd_irma$disasterNumberNow we are read to filter our API call for the
HousingAssistanceRenters data set.
# construct filter list
filter_list <- list(disasterNumber = dd_nums_irma)
# make the API call to get individual assistance grants awarded to renters for hurricane Irma damages.
assistance_irma <- open_fema(data_set = "HousingAssistanceRenters", filters = filter_list, ask_before_call = F)##
Obtaining Data: 1 out of 6 iterations (16.67% complete)
Obtaining Data: 2 out of 6 iterations
## (33.33% complete)
Obtaining Data: 3 out of 6 iterations (50% complete)
Obtaining Data: 4 out of 6
## iterations (66.67% complete)
Obtaining Data: 5 out of 6 iterations (83.33% complete)
Obtaining Data:
## 6 out of 6 iterations (100% complete)Check out the returned data
# check out the returned data
assistance_irma## # A tibble: 5,358 × 21
## disasterNum…¹ state county city zipCode valid…² total…³ total…⁴ total…⁵ total…⁶ total…⁷ appro…⁸
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 4335 VI St. J… ST J… 00083 1 0 1 0 0 0 1
## 2 4335 VI St. T… CHAL… 00801 1 0 1 0 0 0 0
## 3 4335 VI St. T… CHAR… 00801 1 1 1 0 0 0 0
## 4 4335 VI St. T… CHAR… 00801 10 8 6 3 0 0 5
## 5 4335 VI St. T… CHAR… 00801 1 1 1 0 0 0 1
## 6 4335 VI St. T… CHAR… 00801 1 1 1 0 0 0 1
## 7 4335 VI St. T… SAIN… 00801 7 6 7 0 0 0 1
## 8 4335 VI St. T… STT 00801 1 1 1 0 0 0 0
## 9 4335 VI St. T… STTH… 00801 6 4 4 2 0 0 2
## 10 4335 VI St. T… ST T… 00801 219 181 142 66 10 0 126
## # … with 5,348 more rows, 9 more variables: totalApprovedIhpAmount <chr>,
## # repairReplaceAmount <chr>, rentalAmount <chr>, otherNeedsAmount <chr>,
## # approvedBetween1And10000 <chr>, approvedBetween10001And25000 <chr>,
## # approvedBetween25001AndMax <chr>, totalMaxGrants <chr>, id <chr>, and abbreviated variable
## # names ¹disasterNumber, ²validRegistrations, ³totalInspected, ⁴totalInspectedWithNoDamage,
## # ⁵totalWithModerateDamage, ⁶totalWithMajorDamage, ⁷totalWithSubstantialDamage,
## # ⁸approvedForFemaAssistanceNow we can answer our original question: How much did FEMA awarded for rental assistance following Hurricane Irma?
# sum the rentalAmount Column
rent_assistance <- sum(as.numeric(assistance_irma$rentalAmount))
# scale to millions
rent_assistance <- rent_assistance/1000000
print(paste0("$",round(rent_assistance,2),
" million was awarded by FEMA for rental assistance following Hurricane Irma"))## [1] "$314.64 million was awarded by FEMA for rental assistance following Hurricane Irma"Clean one of the data sets with a nested structure
Some data sets that get returned from the FEMA API will be in a nested format. Data from the Integrated Public Alert & Warning System (IPAWS) is one such example of this. See for example the first column of the IPAWS data set, which is XML data returned as a character. Most of the useful information from this data set is in that first column, but isn’t in a form that will be useful for most R users.
# get the first ten entries from the IPAWS data set
ipaws <- rfema::open_fema("IpawsArchivedAlerts", top_n = 100)
ipaws## # A tibble: 100 × 18
## originalM…¹ ident…² sender sent status msgType source scope restr…³ addre…⁴ code note searc…⁵
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 "<?xml ver… NWS-14… w-nws… 2016… Actual Cancel NULL Publ… NULL NULL "lis… NULL "NULL"
## 2 "<?xml ver… NWS-14… w-nws… 2016… Actual Alert NULL Publ… NULL NULL "lis… NULL "list(…
## 3 "<?xml ver… NWS-14… w-nws… 2016… Actual Alert NULL Publ… NULL NULL "lis… NULL "list(…
## 4 "<?xml ver… NWS-14… w-nws… 2016… Actual Cancel NULL Publ… NULL NULL "lis… NULL "list(…
## 5 "<?xml ver… NWS-14… w-nws… 2016… Actual Update NULL Publ… NULL NULL "lis… NULL "list(…
## 6 "<?xml ver… NWS-14… w-nws… 2016… Actual Alert NULL Publ… NULL NULL "lis… NULL "NULL"
## 7 "<?xml ver… NWS-14… w-nws… 2016… Actual Alert NULL Publ… NULL NULL "lis… NULL "list(…
## 8 "<?xml ver… NWS-14… w-nws… 2016… Actual Alert NULL Publ… NULL NULL "lis… NULL "list(…
## 9 "<?xml ver… NWS-14… w-nws… 2016… Actual Update NULL Publ… NULL NULL "lis… NULL "NULL"
## 10 "<?xml ver… NWS-14… w-nws… 2016… Actual Update NULL Publ… NULL NULL "lis… NULL "NULL"
## # … with 90 more rows, 5 more variables: incidents <chr>, cogId <chr>, id <chr>, xmlns <chr>,
## # info <chr>, and abbreviated variable names ¹originalMessage, ²identifier, ³restriction,
## # ⁴addresses, ⁵searchGeometryThe following is one method for converting the xml data into tabular form.
library(dplyr)
library(XML)
# create function to unnest the ipaws entries
unnest_ipaws <- function(xml_entry){
# convert the raw xml data to a list
xml_data <- XML::xmlToList(xml_entry)
# get names of the list elements in xml data
names <- names(xml_data)
# get a summary of the data to id which elements are nested
data_sum <- summary(xml_data)
# put all the non nested elements into a data frame
df <- data.frame(xml_data[names[which(as.numeric(data_sum[,1]) == 1)]])
# get vector of elements that need to be unnested
needs_unnesting <- which(as.numeric(data_sum[,1]) > 1)
# loop over the elements identified above
for(k in needs_unnesting){
# unlist the nested data
unlisted_data <- t(unlist(xml_data[k], recursive = T, use.names = T))
# store the unlisted data as a data frame
temp_df <- data.frame(unlisted_data)
# add the unnested data frame to the existing "df" data frame
df <- cbind.data.frame(df,temp_df)
}
return(df)
}
# get the first 100 entries from the IPAWS alerts data set
ipaws <- rfema::open_fema("IpawsArchivedAlerts", top_n = 100)
# apply the `unnest_ipaws` function over all the XML entries in the returned `ipaws` object
ipaws_list <- sapply(ipaws$originalMessage, unnest_ipaws, simplify = T)
# convert the `ipaws_list` into a data frame
ipaws_df <- dplyr::bind_rows(ipaws_list)
# the number of columns can get unwieldy because of all the unique pieces of information
# in that "info" element that get tacked on
# dropping the geocoding columns could help simplify
ipaws_df <- ipaws_df %>% select(-contains("geocode"))
# dropping the "parameter value" columns would also help
# (depending on if those are needed or not)
ipaws_df <- ipaws_df %>% select(-contains("parameter.value"))
# view the final data
as_tibble(ipaws_df)## # A tibble: 100 × 37
## identi…¹ sender sent status msgType scope code refer…² info.…³ info.…⁴ info.…⁵ info.…⁶ info.…⁷
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 NWS-141… w-nws… 2016… Actual Cancel Publ… IPAW… w-nws.… Met Small … Avoid Expect… Minor
## 2 NWS-141… w-nws… 2016… Actual Alert Publ… IPAW… <NA> Met Severe… Shelter Immedi… Severe
## 3 NWS-141… w-nws… 2016… Actual Alert Publ… IPAW… <NA> Met Specia… Execute Expect… Modera…
## 4 NWS-141… w-nws… 2016… Actual Cancel Publ… IPAW… w-nws.… Met Severe… Shelter Immedi… Severe
## 5 NWS-141… w-nws… 2016… Actual Update Publ… IPAW… w-nws.… Met Severe… Shelter Immedi… Severe
## 6 NWS-141… w-nws… 2016… Actual Alert Publ… IPAW… <NA> Met Beach … Avoid Expect… Modera…
## 7 NWS-141… w-nws… 2016… Actual Alert Publ… IPAW… <NA> Met Severe… Shelter Immedi… Severe
## 8 NWS-141… w-nws… 2016… Actual Alert Publ… IPAW… <NA> Met Marine… Monitor Expect… Minor
## 9 NWS-141… w-nws… 2016… Actual Update Publ… IPAW… w-nws.… Met Small … Avoid Expect… Minor
## 10 NWS-141… w-nws… 2016… Actual Update Publ… IPAW… w-nws.… Met Hazard… Avoid Expect… Modera…
## # … with 90 more rows, 24 more variables: info.certainty <chr>, info.eventCode.valueName <chr>,
## # info.eventCode.value <chr>, info.eventCode.valueName.1 <chr>, info.eventCode.value.1 <chr>,
## # info.effective <chr>, info.onset <chr>, info.expires <chr>, info.senderName <chr>,
## # info.headline <chr>, info.description <chr>, info.instruction <chr>, info.web <chr>,
## # info.area.areaDesc <chr>, Signature.SignedInfo.CanonicalizationMethod.Algorithm <chr>,
## # Signature.SignedInfo.SignatureMethod.Algorithm <chr>,
## # Signature.SignedInfo.Reference.Transforms.Transform.Algorithm <chr>, …Bulk Downloads
In some cases bulk downloading a full data set file may be preferred. For particularly large data requests, its usually faster to bulk download the entire data set as a csv file and then load it into the R environment. In this case, users can use the bulk_dl() command to download a csv of the full data file and save it to a specified directory.
bulk_dl("femaRegions") # download a csv file containing all info on FEMA regions