Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/GlobalFishingWatch/gfwr
R package for accessing data from Global Fishing Watch APIs
https://github.com/GlobalFishingWatch/gfwr
api-wrapper globalfishingwatch mapping r
Last synced: about 2 months ago
JSON representation
R package for accessing data from Global Fishing Watch APIs
- Host: GitHub
- URL: https://github.com/GlobalFishingWatch/gfwr
- Owner: GlobalFishingWatch
- License: apache-2.0
- Created: 2022-01-21T20:46:18.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-11-08T17:35:16.000Z (about 2 months ago)
- Last Synced: 2024-11-13T02:35:52.582Z (about 2 months ago)
- Topics: api-wrapper, globalfishingwatch, mapping, r
- Language: HTML
- Homepage: https://globalfishingwatch.github.io/gfwr/
- Size: 7.39 MB
- Stars: 59
- Watchers: 19
- Forks: 7
- Open Issues: 11
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- Contributing: Contributing.md
- License: LICENSE.md
Awesome Lists containing this project
- open-sustainable-technology - gfwr - R package for accessing data from Global Fishing Watch APIs. (Biosphere / Marine Life and Fishery)
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
eval = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# `gfwr`: Access data from Global Fishing Watch APIs 
[](https://zenodo.org/badge/latestdoi/450635054)
[](https://www.repostatus.org/#active)
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/r-universe/globalfishingwatch/actions/workflows/sync.yml)
> **Important**
> This version of `gfwr` gives access to Global Fishing Watch API [version 3](https://globalfishingwatch.org/our-apis/documentation#version-3-api). Starting April 30th, 2024, this is the official API version. To install the previous version that communicated with API [version 2](https://globalfishingwatch.org/our-apis/documentation#version-2-api), please refer to branch `APIv2` in this repository.
> `remotes::install_github("GlobalFishingWatch/gfwr", ref = "APIv2")`
The `gfwr` R package is a simple wrapper for the Global Fishing Watch (GFW) [APIs](https://globalfishingwatch.org/our-apis/documentation#introduction). It provides convenient functions to freely pull GFW data directly into R in tidy formats.
The package currently works with the following APIs:
* [Vessels API](https://globalfishingwatch.org/our-apis/documentation#vessels-api): vessel search and identity based on AIS self reported data and public registry information
* [Events API](https://globalfishingwatch.org/our-apis/documentation#events-api): encounters, loitering, port visits, AIS-disabling events and fishing events based on AIS data
* [Gridded fishing effort (4Wings API)](https://globalfishingwatch.org/our-apis/documentation#map-visualization-4wings-api): apparent fishing effort based on AIS data
> **Note**:
> See the [Terms of Use](https://globalfishingwatch.org/our-apis/documentation#reference-data) page for GFW APIs for information on our API licenses and rate limits.
## Installation
You can install the most recent version of `gfwr` using:
```{r, eval = FALSE}
# Check/install remotes
if (!require("remotes"))
install.packages("remotes")
remotes::install_github("GlobalFishingWatch/gfwr")
```
`gfwr` is also in the rOpenSci [R-universe](https://globalfishingwatch.r-universe.dev/gfwr#),
and can be installed like this:
```{r eval = FALSE}
install.packages("gfwr",
repos = c("https://globalfishingwatch.r-universe.dev",
"https://cran.r-project.org"))
```
Once everything is installed, you can load and use `gfwr`
in your scripts with `library(gfwr)`
```{r, eval = FALSE}
library(gfwr)
```
```{r load_all, eval = TRUE, echo = FALSE, message = FALSE}
devtools::load_all()
```
## Authorization
The use of `gfwr` requires a GFW API token, which users can request from
the [GFW API Portal](https://globalfishingwatch.org/our-apis/tokens). Save
this token to your `.Renviron` file using `usethis::edit_r_environ()` and adding
a variable named `GFW_TOKEN` to the file (`GFW_TOKEN="PASTE_YOUR_TOKEN_HERE"`).
Save the `.Renviron` file and restart the R session to make the edit effective.
Then use the `gfw_auth()` helper function to inform the key on your function calls.
You can use `gfw_auth()` directly or save the information to an object
in your R workspace every time and pass it
to subsequent `gfwr` functions.
So you can do:
```{r auth, eval = TRUE}
key <- gfw_auth()
```
or this
```{r sys_getenv, eval = F}
key <- Sys.getenv("GFW_TOKEN")
```
> **Note**:
> `gfwr` functions are set to use `key = gfw_auth()` by default.
## Vessels API
The `get_vessel_info()` function allows you to get vessel identity details from
the [GFW Vessels API](https://globalfishingwatch.org/our-apis/documentation#introduction-vessels-api).
There are two search types: `search`, and `id`.
* `search` is performed by using parameters `query` for basic searches and
`where` for advanced searchers using SQL expressions
+ `query` takes a single identifier that can be the MMSI, IMO, callsign, or
shipname as input and identifies all vessels that match.
+ `where` search allows for the use of complex search with logical clauses
(AND, OR) and fuzzy matching with terms such as LIKE, using SQL syntax (see
examples in the function)
+ `includes` adds information from public registries. Options are "MATCH_CRITERIA",
"OWNERSHIP" and "AUTHORIZATIONS"
### Examples
To get information of a vessel using its MMSI, IMO number, callsign or name, the
search can be done directly using the number or the string. For example, to look
for a vessel with `MMSI = 224224000`:
```{r example_vessel_info_1, eval = TRUE}
get_vessel_info(query = 224224000,
search_type = "search",
key = key)
```
To do more specific searches (`imo = '8300949'`), combine different fields
(`imo = '8300949' AND ssvid = '214182732'`) and do fuzzy matching
(`"shipname LIKE '%GABU REEFE%' OR imo = '8300949'"`), use parameter `where`
instead of `query`:
```{r example_vessel_info_2, eval = TRUE}
get_vessel_info(where = "shipname LIKE '%GABU REEFE%' OR imo = '8300949'",
search_type = "search",
key = key)
```
* The `id` search allows the user to specify a vector of `vesselId`s
> **Note**:
> `vesselId` is an internal ID generated by GFW to connect data accross APIs
and involves a combination of vessel and tracking data information. It can be
retrieved using `get_vessel_info()` and fetching the vector of responses inside
`$selfReportedInfo$vesselId`. See the [identity vignette](https://globalfishingwatch.github.io/gfwr/articles/identity) for more information.
To search by `vesselId`, use parameter `ids` and specify `search_type = "id"`:
```{r example_vessel_info_3, eval = TRUE}
get_vessel_info(ids = "8c7304226-6c71-edbe-0b63-c246734b3c01",
search_type = "id",
key = key)
```
To specify more than one `vesselId`, you can submit a vector:
```{r example_vessel_info_4, eval = TRUE}
get_vessel_info(ids = c("8c7304226-6c71-edbe-0b63-c246734b3c01",
"6583c51e3-3626-5638-866a-f47c3bc7ef7c",
"71e7da672-2451-17da-b239-857831602eca"),
search_type = 'id',
key = key)
```
This is useful especially because a vessel can have different `vesselId`s in time.
Check the function documentation for examples with the other function arguments and
[our dedicated vignette](https://globalfishingwatch.github.io/gfwr/articles/identity)
for more information about vessel identity .
## Events API
The `get_event()` function allows you to get data on specific vessel activities
from the [GFW Events API](https://globalfishingwatch.org/our-apis/documentation#events-api).
Event types include apparent fishing events, potential transshipment events
(two-vessel encounters and loitering by refrigerated carrier vessels), port
visits, and AIS-disabling events ("gaps").
Find more information in our
[caveat documentation](https://globalfishingwatch.org/our-apis/documentation#data-caveat).
### Examples
The Events API uses `vesselId` as input, so you always need to use `get_vessel_info()`
first to extract `vesselId` from `$selfReportedInfo` in the response.
```{r example_id_event, eval = TRUE}
vessel_info <- get_vessel_info(query = 224224000, key = key)
id <- vessel_info$selfReportedInfo$vesselId[1]
```
To get a list of port visits for that vessel:
```{r example_event_type_1, eval = TRUE}
get_event(event_type = 'PORT_VISIT',
vessels = id,
confidences = 4,
key = key
)
```
> *Note*: Try narrowing your search using `start_date` and `end_date` if the
request is too large and returns a time out error (524)
We can also use more than one `vesselId`:
```{r example_event_type_2, , eval = TRUE}
get_event(event_type = 'PORT_VISIT',
vessels = c('8c7304226-6c71-edbe-0b63-c246734b3c01',
'6583c51e3-3626-5638-866a-f47c3bc7ef7c'),
confidences = 4,
start_date = "2020-01-01",
end_date = "2020-02-01",
key = key
)
```
Or get encounters for all vessels in a given date range:
```{r example_event_type_3, eval = TRUE}
get_event(event_type = 'ENCOUNTER',
start_date = "2020-01-01",
end_date = "2020-01-02",
key = key
)
```
When a date range is provided to `get_event()` using both `start_date` and
`end_date`, any event overlapping that range will be returned, including events
that start prior to `start_date` or end after `end_date`. If just `start_date`
or `end_date` are provided, results will include all events that end after
`start_date` or begin prior to `end_date`, respectively.
> **Note**:
> Because encounter events are events between two vessels, a single event will
be represented twice in the data, once for each vessel. To capture this
information and link the related data rows, the `id` field for encounter events
includes an additional suffix (1 or 2) separated by a period. The `vessel` field
will also contain different information specific to each vessel.
As another example, let's combine the Vessels and Events APIs to get fishing events for a list of 20 USA-flagged trawlers:
```{r example_event_type_4a}
# Download the list of USA trawlers
usa_trawlers <- get_vessel_info(
where = "flag='USA' AND geartypes='TRAWLERS'",
search_type = "search",
key = key
)
# Pass the vector of vessel ids to Events API
usa_trawler_ids <- usa_trawlers$selfReportedInfo$vesselId[1:20]
```
> *Note*: `get_event()` can receive up to 20 vessel ids at a time
Now get the list of fishing events for these trawlers in January, 2020:
```{r example_event_type_4b, eval=T}
get_event(event_type = 'FISHING',
vessels = usa_trawler_ids,
start_date = "2020-01-01",
end_date = "2020-02-01",
key = key
)
```
When no events are available, the `get_event()` function returns nothing.
```{r example_event_type_4c, eval=T}
get_event(event_type = 'FISHING',
vessels = usa_trawler_ids[2],
start_date = "2020-01-01",
end_date = "2020-01-01",
key = key
)
```
## Fishing effort API
The `get_raster()` function gets a raster from the [4Wings API](https://globalfishingwatch.org/our-apis/documentation#map-visualization-4wings-api)
and converts the response to a data frame. In order to use it, you should specify:
* The spatial resolution, which can be `LOW` (0.1 degree) or `HIGH` (0.01 degree)
* The temporal resolution, which can be `HOURLY`, `DAILY`, `MONTHLY`, `YEARLY` or `ENTIRE`.
* The variable to group by: `FLAG`, `GEARTYPE`, `FLAGANDGEARTYPE`, `MMSI` or `VESSEL_ID`
* The date range `note: this must be 366 days or less`
* The region polygon in `sf` format or the region code (such as an EEZ code) to filter the raster
* The source for the specified region. Currently, `EEZ`, `MPA`, `RFMO` or `USER_SHAPEFILE` (for `sf` shapefiles).
### Examples
You can load a sample shapefile inside `gfwr` to see how `'USER_SHAPEFILE'` works:
```{r example_map_1}
data("test_shape")
get_raster(
spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = '2021-01-01',
end_date = '2021-02-01',
region = test_shape,
region_source = 'USER_SHAPEFILE',
key = key
)
```
If you want raster data from a particular EEZ, you can use the `get_region_id()`
function to get the EEZ id, and enter that code in the `region` argument
of `get_raster()` instead of the region shapefile (ensuring you specify the `region_source`
as `'EEZ'`:
```{r example_map_2, eval= TRUE}
# use EEZ function to get EEZ code of Cote d'Ivoire
code_eez <- get_region_id(region_name = 'CIV', region_source = 'EEZ', key = key)
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-10-01",
region = code_eez$id,
region_source = 'EEZ',
key = key)
```
You could search for just one word in the name of the EEZ and then decide which
one you want:
```{r example_map_3, eval = TRUE}
(get_region_id(region_name = 'France', region_source = 'EEZ', key = key))
```
From the results above, let's say we're interested in the French Exclusive Economic Zone, `5677`
```{r fr_eez, eval = TRUE}
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-10-01",
region = 5677,
region_source = 'EEZ',
key = key)
```
A similar approach can be used to search for a specific Marine Protected Area,
in this case the Phoenix Island Protected Area (PIPA)
```{r example_map_4, eval= TRUE}
# use region id function to get MPA code of Phoenix Island Protected Area
code_mpa <- get_region_id(region_name = 'Phoenix', region_source = 'MPA', key = key)
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2015-01-01",
end_date = "2015-06-01",
region = code_mpa$id[1],
region_source = 'MPA',
key = key)
```
It is also possible to filter rasters to one of the five regional fisheries management
organizations (RFMO) that manage tuna and tuna-like species. These include `"ICCAT"`,
`"IATTC"`,`"IOTC"`, `"CCSBT"` and `"WCPFC"`.
```{r example_map_5, eval=T}
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'DAILY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-01-04",
region = 'ICCAT',
region_source = 'RFMO',
key = key)
```
The `get_region_id()` function also works in reverse. If a region id is passed as
a `numeric` to the function as the `region_name`, the corresponding region label
or iso3 code can be returned. This is especially useful when events are returned with
regions.
```{r example_region_id}
# using same example as above
get_event(event_type = 'FISHING',
vessels = usa_trawler_ids,
start_date = "2020-01-01",
end_date = "2020-02-01",
key = key
) %>%
# extract EEZ id code
dplyr::mutate(eez = as.character(purrr::map(purrr::map(regions, purrr::pluck, 'eez'),
paste0, collapse = ','))) %>%
dplyr::select(id, type, start, end, lat, lon, eez) %>%
dplyr::rowwise() %>%
dplyr::mutate(eez_name = get_region_id(region_name = as.numeric(eez),
region_source = 'EEZ',
key = key)$label) %>%
dplyr::select(-start, -end)
```
### When your API request times out
For API performance reasons, the `get_raster()` function restricts individual queries to a single year of data. However, even with this restriction, it is possible for API request to time out before it completes. When this occurs, the initial `get_raster()` call will return an HTTP 524 error, and subsequent API requests using any `gfwr` `get_` function will return an HTTP 429 error until the original request completes:
>
Error in `httr2::req_perform()`:
! HTTP 429 Too Many Requests.
• Your application token is not currently enabled to perform more than one concurrent report. If you need to generate more than one report concurrently, contact us at [email protected]
Although no data was received, the request is still being processed by the APIs and will become available when it completes. To account for this, `gfwr` includes the `get_last_report()` function, which lets users request the results of their last API request with `get_raster()`.
The `get_last_report()` function will tell you if the APIs are still processing your request and will download the results if the request has finished successfully. You will receive an error message if the request finished but resulted in an error or if it's been >30 minutes since the last report was generated using `get_raster()`. For more information, see the [Get last report generated endpoint](https://globalfishingwatch.org/our-apis/documentation#get-last-report-generated) documentation on the GFW API page.
## Contributing
We welcome all contributions to improve the package! Please read our
[Contribution Guide](https://github.com/GlobalFishingWatch/gfwr/blob/main/Contributing.md)
and reach out!