Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/globalfishingwatch/gfwr
R package for accessing data from Global Fishing Watch APIs
https://github.com/globalfishingwatch/gfwr
api-wrapper globalfishingwatch mapping r
Last synced: 5 days ago
JSON representation
R package for accessing data from Global Fishing Watch APIs
- Host: GitHub
- URL: https://github.com/globalfishingwatch/gfwr
- Owner: GlobalFishingWatch
- License: apache-2.0
- Created: 2022-01-21T20:46:18.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-12-03T22:12:22.000Z (19 days ago)
- Last Synced: 2024-12-12T18:40:20.441Z (10 days ago)
- Topics: api-wrapper, globalfishingwatch, mapping, r
- Language: HTML
- Homepage: https://globalfishingwatch.github.io/gfwr/
- Size: 9.13 MB
- Stars: 61
- Watchers: 18
- Forks: 8
- Open Issues: 10
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- Contributing: Contributing.md
- License: LICENSE.md
Awesome Lists containing this project
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
eval = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# `gfwr`: Access data from Global Fishing Watch APIs 
[](https://zenodo.org/badge/latestdoi/450635054)
[](https://www.repostatus.org/#active)
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/r-universe/globalfishingwatch/actions/workflows/sync.yml)
> **Important**
> This version of `gfwr` gives access to Global Fishing Watch API [version 3](https://globalfishingwatch.org/our-apis/documentation#version-3-api). Starting
April 30th, 2024, this is the official API version. For latest API releases,
please check our [API release notes](https://globalfishingwatch.org/our-apis/documentation#api-release-notes)
The `gfwr` R package is a simple wrapper for the Global Fishing Watch (GFW) [APIs](https://globalfishingwatch.org/our-apis/documentation#introduction). It
provides convenient functions to freely pull GFW data directly into R in tidy formats.
The package currently works with the following APIs:
* [Vessels API](https://globalfishingwatch.org/our-apis/documentation#vessels-api):
vessel search and identity based on AIS self reported data and public registry
information
* [Events API](https://globalfishingwatch.org/our-apis/documentation#events-api):
encounters, loitering, port visits, AIS-disabling events and fishing events
based on AIS data
* [Gridded fishing effort (4Wings API)](https://globalfishingwatch.org/our-apis/documentation#map-visualization-4wings-api):
apparent fishing effort based on AIS data
> **Note**:
> See the [Terms of Use](https://globalfishingwatch.org/our-apis/documentation#reference-data)
page for GFW APIs for information on our API licenses and rate limits.
## Installation
You can install the most recent version of `gfwr` using:
```{r, eval = FALSE}
# Check/install remotes
if (!require("remotes"))
install.packages("remotes")
remotes::install_github("GlobalFishingWatch/gfwr")
```
`gfwr` is also in the rOpenSci
[R-universe](https://globalfishingwatch.r-universe.dev/gfwr#), and can be
installed like this:
```{r eval = FALSE}
install.packages("gfwr",
repos = c("https://globalfishingwatch.r-universe.dev",
"https://cran.r-project.org"))
```
Once everything is installed, you can load and use `gfwr` in your scripts with
`library(gfwr)`
```{r, eval = FALSE}
library(gfwr)
```
```{r load_all, eval = TRUE, echo = FALSE, message = FALSE}
devtools::load_all()
```
## Authorization
The use of `gfwr` requires a GFW API token, which users can request from
the [GFW API Portal](https://globalfishingwatch.org/our-apis/tokens). Save
this token to your `.Renviron` file using `usethis::edit_r_environ()` and adding
a variable named `GFW_TOKEN` to the file (`GFW_TOKEN="PASTE_YOUR_TOKEN_HERE"`).
Save the `.Renviron` file and restart the R session to make the edit effective.
Then use the `gfw_auth()` helper function to inform the key on your function
calls. You can use `gfw_auth()` directly or save the information to an object in
your R workspace every time and pass it to subsequent `gfwr` functions.
So you can do:
```{r auth, eval = TRUE}
key <- gfw_auth()
```
or this
```{r sys_getenv, eval = F}
key <- Sys.getenv("GFW_TOKEN")
```
> **Note**:
> `gfwr` functions are set to use `key = gfw_auth()` by default.
## Vessels API
The `get_vessel_info()` function allows you to get vessel identity details from
the [GFW Vessels API](https://globalfishingwatch.org/our-apis/documentation#introduction-vessels-api).
There are two search types: `search`, and `id`.
* `search` is performed by using parameters `query` for basic searches and
`where` for advanced searchers using SQL expressions
+ `query` takes a single identifier that can be the MMSI, IMO, callsign, or
shipname as input and identifies all vessels that match.
+ `where` search allows for the use of complex search with logical clauses
(AND, OR) and fuzzy matching with terms such as LIKE, using SQL syntax (see
examples in the function)
+ `includes` adds information from public registries. Options are
"MATCH_CRITERIA", "OWNERSHIP" and "AUTHORIZATIONS"
### Examples
To get information of a vessel using its MMSI, IMO number, callsign or name, the
search can be done directly using the number or the string. For example, to look
for a vessel with `MMSI = 224224000`:
```{r example_vessel_info_1, eval = TRUE}
get_vessel_info(query = 224224000,
search_type = "search",
key = key)
```
To do more specific searches (`imo = '8300949'`), combine different fields
(`imo = '8300949' AND ssvid = '214182732'`) and do fuzzy matching
(`"shipname LIKE '%GABU REEFE%' OR imo = '8300949'"`), use parameter `where`
instead of `query`:
```{r example_vessel_info_2, eval = TRUE}
get_vessel_info(where = "shipname LIKE '%GABU REEFE%' OR imo = '8300949'",
search_type = "search",
key = key)
```
* The `id` search allows the user to specify a vector of `vesselId`s
> **Note**:
> `vesselId` is an internal ID generated by GFW to connect data accross APIs
and involves a combination of vessel and tracking data information. It can be
retrieved using `get_vessel_info()` and fetching the vector of responses inside
`$selfReportedInfo$vesselId`. See the
[identity vignette](https://globalfishingwatch.github.io/gfwr/articles/identity)
for more information.
To search by `vesselId`, use parameter `ids` and specify `search_type = "id"`:
```{r example_vessel_info_3, eval = TRUE}
get_vessel_info(ids = "8c7304226-6c71-edbe-0b63-c246734b3c01",
search_type = "id",
key = key)
```
To specify more than one `vesselId`, you can submit a vector:
```{r example_vessel_info_4, eval = TRUE}
get_vessel_info(ids = c("8c7304226-6c71-edbe-0b63-c246734b3c01",
"6583c51e3-3626-5638-866a-f47c3bc7ef7c",
"71e7da672-2451-17da-b239-857831602eca"),
search_type = 'id',
key = key)
```
This is useful especially because a vessel can have different `vesselId`s in time.
__Check the function documentation for examples with the other function arguments and
[our dedicated vignette](https://globalfishingwatch.github.io/gfwr/articles/identity)
for more information about vessel identity markers and the outputs retrieved.__
## Events API
The `get_event()` function allows you to get data on specific vessel activities
from the
[GFW Events API](https://globalfishingwatch.org/our-apis/documentation#events-api).
Event types include apparent fishing events, potential transshipment events
(two-vessel encounters and loitering by refrigerated carrier vessels), port
visits, and AIS-disabling events ("gaps").
Find more information about events in our
[caveat documentation](https://globalfishingwatch.org/our-apis/documentation#data-caveat).
### Events in a given time range
You can get events in a given date range. By not specifying `vessels`, the
response will return results for all vessels.
```{r example_event_type_3, eval = TRUE}
get_event(event_type = 'ENCOUNTER',
start_date = "2020-01-01",
end_date = "2020-01-02",
key = key
)
```
> *Note*: We do not recommend trying too large downloads, such as all
encounters for all vessels over a long period of time. This will possibly
return time out (524) errors. Our API team is working on a bulk download solution
for the future.
### Events in a specific area
You can provide a polygon in `sf` format or the region code (such as an EEZ
code) to filter the raster. Check the function documentation for more
information about parameters `region` and `region_source`
```{r events_shapefile}
# fishing events in user shapefile
test_polygon <- sf::st_bbox(c(xmin = -70, xmax = -40, ymin = -10, ymax = 5),
crs = 4326) |>
sf::st_as_sfc() |>
sf::st_as_sf()
get_event(event_type = 'FISHING',
start_date = "2020-10-01",
end_date = "2020-10-31",
region = test_polygon,
region_source = 'USER_SHAPEFILE',
key = gfw_auth())
```
### Events for specific vessels
To extract events for specific vessels, the Events API needs `vesselId` as
input, so you always need to use `get_vessel_info()` first to extract
`vesselId` from `$selfReportedInfo` in the response.
#### Single vessel events
```{r example_id_event, eval = TRUE}
vessel_info <- get_vessel_info(query = 224224000, key = key)
vessel_info$selfReportedInfo
```
The results show this vessel's story is grouped in two `vesselIds`.
To get a list of port visits for that vessel, you can use a single `vesselId`
of your interest:
```{r event_single_vesselid, eval = TRUE}
id <- vessel_info$selfReportedInfo$vesselId
id
get_event(event_type = 'PORT_VISIT',
vessels = id[1],
confidences = 4,
key = key
)
```
But to get the whole event history, you can also use the whole vector of
`vesselId` for that vessel:
```{r event_onevessel_allvesselids, eval = TRUE}
get_event(event_type = 'PORT_VISIT',
vessels = id, #using the whole vector of vesselIds
confidences = 4,
key = key
)
```
> *Note*: Try narrowing your search using `start_date` and `end_date` if the
request is too large and returns a time out error (524)
When a date range is provided to `get_event()` using both `start_date` and
`end_date`, any event overlapping that range will be returned, including events
that start prior to `start_date` or end after `end_date`. If just `start_date`
or `end_date` are provided, results will include all events that end after
`start_date` or begin prior to `end_date`, respectively.
> **Note**:
> Because encounter events are events between two vessels, a single event will
be represented twice in the data, once for each vessel. To capture this
information and link the related data rows, the `id` field for encounter events
includes an additional suffix (1 or 2) separated by a period. The `vessel` field
will also contain different information specific to each vessel.
#### Multiple vessel events
As another example, let's combine the Vessels and Events APIs to get fishing
events for a list of USA-flagged trawlers:
```{r example_event_type_4a}
# Download the list of USA trawlers
usa_trawlers <- get_vessel_info(
where = "flag='USA' AND geartypes='TRAWLERS'",
search_type = "search",
key = key,
quiet = TRUE # quiet = FALSE if you want an estimate progress of the download
)
```
This list returns `r nrow(usa_trawlers$selfReportedInfo)` `vesselIds` belonging
to `r nrow(usa_trawlers$dataset)` vessels.
```{r usa_trawlers_id}
usa_trawlers$selfReportedInfo
```
For clarity, we should try to send groups of `vesselIds` that belong to the same
vessels. For this, we should check the `index` column in the `$selfReportedInfo`
dataset.
> *Note*: `get_event()` can receive up to 20 vessel ids at a time
```{r usa_ten}
each_USA_trawler <- usa_trawlers$selfReportedInfo[, c("index", "vesselId")]
# how many vessels correspond to the first ten vessels.
each_USA_trawler %>% filter(index <= 10)
# It's exactly 20 in this case to we will request those.
ten_usa_trawlers <- each_USA_trawler %>% filter(index <= 10)
```
The first 20 `vesselIds` correspond to 10 vessels according to `index`.
Let's pass the vector of vessel ids to Events API. Now get the list of fishing
events for these trawlers in January, 2020:
```{r example_event_type_4b, eval=T}
events <- get_event(event_type = 'FISHING',
vessels = ten_usa_trawlers$vesselId,
start_date = "2020-01-01",
end_date = "2020-02-01",
key = key)
events
```
The columns starting by `vessel` have the vessel-related information for each
event: `vesselId`, `vessel_name`, `ssvid` (MMSI), `flag`, `vessel type` and
public authorizations.
```{r unnest_vessel}
events %>%
dplyr::select(starts_with("vessel"))
```
When no events are available, the `get_event()` function returns nothing.
```{r example_event_type_4c, eval=T}
get_event(event_type = 'FISHING',
vessels = ten_usa_trawlers$vesselId[2],
start_date = "2020-01-01",
end_date = "2020-01-01",
key = key
)
```
## Fishing effort API
The `get_raster()` function gets a raster from the [4Wings API](https://globalfishingwatch.org/our-apis/documentation#map-visualization-4wings-api)
and converts the response to a data frame. In order to use it, you should specify:
* The spatial resolution, which can be `LOW` (0.1 degree) or `HIGH` (0.01
degree)
* The temporal resolution, which can be `HOURLY`, `DAILY`, `MONTHLY`, `YEARLY`
or `ENTIRE`.
* The variable to group by: `FLAG`, `GEARTYPE`, `FLAGANDGEARTYPE`, `MMSI` or
`VESSEL_ID`
* The date range `note: this must be 366 days or less`
* The region polygon in `sf` format or the region code (such as an EEZ code) to
filter the raster
* The source for the specified region. Currently, `EEZ`, `MPA`, `RFMO` or
`USER_SHAPEFILE` (for `sf` shapefiles).
### Examples
We added a sample shapefile inside `gfwr` to show how `'USER_SHAPEFILE'` works:
```{r example_map_1}
data("test_shape")
get_raster(
spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = '2021-01-01',
end_date = '2021-02-01',
region = test_shape,
region_source = 'USER_SHAPEFILE',
key = key
)
```
If you want raster data from a particular EEZ, you can use the `get_region_id()`
function to get the EEZ id, and enter that code in the `region` argument
of `get_raster()` instead of the region shapefile (ensuring you specify the
`region_source` as `'EEZ'`:
```{r example_map_2, eval= TRUE}
# use EEZ function to get EEZ code of Cote d'Ivoire
code_eez <- get_region_id(region_name = 'CIV', region_source = 'EEZ', key = key)
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-10-01",
region = code_eez$id,
region_source = 'EEZ',
key = key)
```
You could search for just one word in the name of the EEZ and then decide which
one you want:
```{r example_map_3, eval = TRUE}
(get_region_id(region_name = 'France', region_source = 'EEZ', key = key))
```
From the results above, let's say we're interested in the French Exclusive
Economic Zone, `5677`
```{r fr_eez, eval = TRUE}
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-10-01",
region = 5677,
region_source = 'EEZ',
key = key)
```
A similar approach can be used to search for a specific Marine Protected Area,
in this case the Phoenix Island Protected Area (PIPA)
```{r example_map_4, eval= TRUE}
# use region id function to get MPA code of Phoenix Island Protected Area
code_mpa <- get_region_id(region_name = 'Phoenix',
region_source = 'MPA',
key = key)
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'YEARLY',
group_by = 'FLAG',
start_date = "2015-01-01",
end_date = "2015-06-01",
region = code_mpa$id[1],
region_source = 'MPA',
key = key)
```
It is also possible to filter rasters to one of the five regional fisheries
management organizations (RFMO) that manage tuna and tuna-like species. These
include `"ICCAT"`, `"IATTC"`,`"IOTC"`, `"CCSBT"` and `"WCPFC"`.
```{r example_map_5, eval=T}
get_raster(spatial_resolution = 'LOW',
temporal_resolution = 'DAILY',
group_by = 'FLAG',
start_date = "2021-01-01",
end_date = "2021-01-04",
region = 'ICCAT',
region_source = 'RFMO',
key = key)
```
The `get_region_id()` function also works in reverse. If a region id is passed as
a `numeric` to the function as the `region_name`, the corresponding region label
or iso3 code can be returned. This is especially useful when events are
returned with regions.
```{r example_region_id}
# using same example as above
get_event(event_type = 'FISHING',
vessels = ten_usa_trawlers$vesselId,
start_date = "2020-01-01",
end_date = "2020-02-01",
key = key
) %>%
# extract EEZ id code
dplyr::mutate(eez = as.character(
purrr::map(purrr::map(regions, purrr::pluck, 'eez'),
paste0, collapse = ','))) %>%
dplyr::select(eventId, eventType, start, end, lat, lon, eez) %>%
dplyr::rowwise() %>%
dplyr::mutate(eez_name = get_region_id(region_name = as.numeric(eez),
region_source = 'EEZ',
key = key)$label) %>%
dplyr::select(-start, -end)
```
### When your API request times out
For API performance reasons, the `get_raster()` function restricts individual
queries to a single year of data. However, even with this restriction, it is
possible for API request to time out before it completes. When this occurs, the
initial `get_raster()` call will return an HTTP 524 error, and subsequent API
requests using any `gfwr` `get_` function will return an HTTP 429 error until
the original request completes:
>
Error in `httr2::req_perform()`:
! HTTP 429 Too Many Requests.
• Your application token is not currently enabled to perform more than one
concurrent report. If you need to generate more than one report concurrently,
contact us at [email protected]
Although no data was received, the request is still being processed by the APIs
and will become available when it completes. To account for this, `gfwr`
includes the `get_last_report()` function, which lets users request the
results of their last API request with `get_raster()`.
The `get_last_report()` function will tell you if the APIs are still
processing your request and will download the results if the request has
finished successfully. You will receive an error message if the request
finished but resulted in an error or if it's been >30 minutes since the last
report was generated using `get_raster()`. For more information, see the
[Get last report generated endpoint](https://globalfishingwatch.org/our-apis/documentation#get-last-report-generated)
documentation on the GFW API page.
## Contributing
We welcome all contributions to improve the package! Please read our
[Contribution Guide](https://github.com/GlobalFishingWatch/gfwr/blob/main/Contributing.md)
and reach out!