{"id":19399928,"url":"https://github.com/worldbank/blackmarbler","last_synced_at":"2026-03-05T14:31:59.373Z","repository":{"id":205095302,"uuid":"710225299","full_name":"worldbank/blackmarbler","owner":"worldbank","description":"Georeferenced Rasters and Statistics of Nighttime Lights from NASA Black Marble","archived":false,"fork":false,"pushed_at":"2024-04-26T16:59:47.000Z","size":22041,"stargazers_count":14,"open_issues_count":5,"forks_count":3,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-05-04T09:29:21.264Z","etag":null,"topics":["nasa","nasa-data","nasa-earth-data","nighttime-lights","raster-data","viirs","worldbank","zonal-statistics"],"latest_commit_sha":null,"homepage":"https://worldbank.github.io/blackmarbler/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/worldbank.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-26T09:16:25.000Z","updated_at":"2024-05-09T22:23:43.243Z","dependencies_parsed_at":null,"dependency_job_id":"8ffc8569-df87-4ef1-813f-3c47930270dd","html_url":"https://github.com/worldbank/blackmarbler","commit_stats":{"total_commits":93,"total_committers":3,"mean_commits":31.0,"dds":"0.032258064516129004","last_synced_commit":"4fbb68ffdd6c709c46861a28cd1308e45354b91b"},"previous_names":["worldbank/blackmarbler"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2Fblackmarbler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2Fblackmarbler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2Fblackmarbler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worldbank%2Fblackmarbler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/worldbank","download_url":"https://codeload.github.com/worldbank/blackmarbler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240572740,"owners_count":19822714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nasa","nasa-data","nasa-earth-data","nighttime-lights","raster-data","viirs","worldbank","zonal-statistics"],"created_at":"2024-11-10T11:12:15.369Z","updated_at":"2026-03-05T14:31:59.315Z","avatar_url":"https://github.com/worldbank.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BlackMarbleR \u003cimg src=\"man/figures/logo.png\" align=\"right\" width=\"200\" /\u003e\n\n\u003c!-- badges: start --\u003e\n\n[![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/blackmarbler)](https://cran.r-project.org/package=blackmarbler)\n![downloads](https://cranlogs.r-pkg.org/badges/grand-total/blackmarbler)\n[![GitHub Repo stars](https://img.shields.io/github/stars/worldbank/blackmarbler)](https://github.com/worldbank/blackmarbler)\n[![activity](https://img.shields.io/github/commit-activity/m/worldbank/blackmarbler)](https://github.com/worldbank/blackmarbler/graphs/commit-activity)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/license/mit)\n[![R-CMD-check](https://github.com/worldbank/blackmarbler/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/worldbank/blackmarbler/actions/workflows/R-CMD-check.yaml)\n\u003c!-- badges: end --\u003e\n\n**BlackMarbleR** is a R package that provides a simple way to use nighttime lights data from NASA's Black Marble. [Black Marble](https://blackmarble.gsfc.nasa.gov) is a [NASA Earth Science Data Systems (ESDS)](https://www.earthdata.nasa.gov) project that provides a product suite of daily, monthly and yearly global [nighttime lights](https://www.earthdata.nasa.gov/topics/human-dimensions/nighttime-lights). This package automates the process of downloading all relevant tiles from the [NASA LAADS DAAC](https://ladsweb.modaps.eosdis.nasa.gov/) to cover a region of interest, converting and mosaicing the raw files (in HDF5 format) to georeferenced rasters.\n\n* [Installation](#installation)\n* [Bearer token](#token)\n* [Usage](#usage)\n  * [Setup](#setup)\n  * [Make raster](#raster)\n  * [Make raster across multiple time periods](#stack)\n  * [Make map](#map)\n  * [Make figure of trends in nighttime lights](#trends)\n  * [Workflow to update data](#update-data)\n* [Functions and arguments](#function-args)\n  * [Functions](#functions)\n  * [Required Arguments](#args-required)\n  * [Optional Arguments](#args-optional)\n  * [Argument only for `bm_extract`](#args-extract)\n* [Black Marble Resources](#resources)\n\n## Installation \u003ca name=\"installation\"\u003e\u003c/a\u003e\n\nThe package can be installed via CRAN.\n\n```r  \ninstall.packages(\"blackmarbler\")\n```\n\nTo install the development version from Github:\n\n```r\n# install.packages(\"devtools\")\ndevtools::install_github(\"worldbank/blackmarbler\")\n```\n\n## Bearer Token \u003ca name=\"token\"\u003e\n\nFollow the below steps to obtain a bearer token:\n\n1. Create a [NASA Earth Data account](https://ladsweb.modaps.eosdis.nasa.gov/) account. On the top right of the [webpage](https://ladsweb.modaps.eosdis.nasa.gov/), click \"Login\" then \"Earthdata Login\". Then click \"register\" (blue button).\n2. Enter the information in the registration page. You __must__ include the following information; this information is not required to create an account, but the bearer token will not work without this information:\n\n    - Study Area\n    - User Type\n    - Organization\n  \n3. Click \"Register for EarthData Login\" (green button at bottom). Check your email, and click the link in the email to activate the account.\n4. Go to the [Earth Data Login](https://urs.earthdata.nasa.gov/users) page and login.\n5. On the panel near the top, click \"EULAs\" then \"Accept New EULAs\". Accept:\n\n    - MERIS EULA\n    - Sentinel EULA\n  \n6. On the \"Profile Home\" page, you should see something like below. Information should be filled in for each category, and \"Agreed To Meris EULA\" and \"Agreed To Sentinel-3 EULA\" should be True.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"man/figures/nasa_profile_info.png\" alt=\"NASA Profile Home Information\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n7. Go to the [NASA LAADS Archive](https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/products/VNP46A4/) and login (login button on top right). You will see a page to authorize use of Sentinel3 and Meris. Click the green \"Authorize\" button.\n\n8. To obtain the bearer token, go to the [Earth Data Login](https://urs.earthdata.nasa.gov/users) page and login. On the top panel, click \"Generate token\". On this page, click \"Show Token\" to see the bearer token.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"man/figures/nasa_login_3.png\" alt=\"NASA LAADS Bearer Token\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n9. If the bearer token ever stops working, you make need to go to the \"Generate token\" page (see step 8), delete any existing tokens, and generate a new token.\n\n### Programmatically retrieve token \u003ca name=\"token-automatic\"\u003e\n\nAfter following the above steps, the bearer token can also be programmatically retrieved using the `get_nasa_token()` function and your usename and password.\n\n```r\nbearer \u003c- get_nasa_token(username = \"USERNAME-HERE\", \n                         password = \"PASSWORD-HERE\")\n```\n\n## Usage \u003ca name=\"usage\"\u003e\n\n### Setup \u003ca name=\"setup\"\u003e\n\nBefore downloading and extracting Black Marble data, we first load packages, define the NASA bearer token, and define a region of interest.\n\n```r\n#### Setup\n# Load packages\nlibrary(blackmarbler)\nlibrary(geodata)\nlibrary(sf)\nlibrary(terra)\nlibrary(ggplot2)\nlibrary(tidyterra)\nlibrary(lubridate)\n\n#### Define NASA bearer token\nbearer \u003c- \"BEARER-TOKEN-HERE\"\n\n### ROI\n# Define region of interest (roi). The roi must be (1) an sf polygon and (2)\n# in the WGS84 (epsg:4326) coordinate reference system. Here, we use the\n# getData function to load a polygon of Ghana\nroi_sf \u003c- gadm(country = \"GHA\", level=1, path = tempdir()) \n```\n\n### Make raster of nighttime lights \u003ca name=\"raster\"\u003e\n\nThe below example shows making daily, monthly, and annual rasters of nighttime\nlights for Ghana.\n\n```r\n### Daily data: raster for February 5, 2021\nr_20210205 \u003c- bm_raster(roi_sf = roi_sf,\n                        product_id = \"VNP46A2\",\n                        date = \"2021-02-05\",\n                        bearer = bearer)\n\n### Monthly data: raster for October 2021\nr_202110 \u003c- bm_raster(roi_sf = roi_sf,\n                      product_id = \"VNP46A3\",\n                      date = \"2021-10-01\", # The day is ignored\n                      bearer = bearer)\n\n### Annual data: raster for 2021\nr_2021 \u003c- bm_raster(roi_sf = roi_sf,\n                    product_id = \"VNP46A4\",\n                    date = 2021,\n                    bearer = bearer)\n```\n\n### Make raster of nighttime lights across multiple time periods \u003ca name=\"stack\"\u003e\n\nTo extract data for multiple time periods, add multiple time periods to `date`. The function will return a `SpatRaster` object with multiple bands, where each band corresponds to a different date. The below code provides examples getting data across multiple days, months, and years.\n\n```r\n#### Daily data in March 2021\nr_daily \u003c- bm_raster(roi_sf = roi_sf,\n                     product_id = \"VNP46A2\",\n                     date = seq.Date(from = ymd(\"2021-03-01\"), to = ymd(\"2021-03-31\"), by = \"day\"),\n                     bearer = bearer)\n\n#### Monthly aggregated data in 2021 and 2022\nr_monthly \u003c- bm_raster(roi_sf = roi_sf,\n                       product_id = \"VNP46A3\",\n                       date = seq.Date(from = ymd(\"2021-01-01\"), to = ymd(\"2022-12-01\"), by = \"month\"),\n                       bearer = bearer)\n\n#### Yearly aggregated data in 2012 and 2021\nr_annual \u003c- bm_raster(roi_sf = roi_sf,\n                      product_id = \"VNP46A4\",\n                      date = 2012:2021,\n                      bearer = bearer)\n```\n\n### Map of nighttime lights \u003ca name=\"map\"\u003e\n\nUsing one of the rasters, we can make a map of nighttime lights\n\n```r\n#### Make raster\nr \u003c- bm_raster(roi_sf = roi_sf,\n               product_id = \"VNP46A3\",\n               date = \"2021-10-01\",\n               bearer = bearer)\n\n#### Prep data\nr \u003c- r |\u003e terra::mask(roi_sf)\n\n## Distribution is skewed, so log\nr[] \u003c- log(r[] + 1)\n\n##### Map\nggplot() +\n  geom_spatraster(data = r) +\n  scale_fill_gradient2(low = \"black\",\n                       mid = \"yellow\",\n                       high = \"red\",\n                       midpoint = 4.5,\n                       na.value = \"transparent\") +\n  labs(title = \"Nighttime Lights: October 2021\") +\n  coord_sf() +\n  theme_void() +\n  theme(plot.title = element_text(face = \"bold\", hjust = 0.5),\n  legend.position = \"none\")\n```\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"man/figures/ntl_gha.png\" alt=\"Nighttime Lights Map\" width=\"800\"/\u003e\n\u003c/p\u003e\n\n### Trends over time \u003ca name=\"trends\"\u003e\n\nWe can use the `bm_extract` function to observe changes in nighttime lights over time. The `bm_extract` function leverages the [`exactextractr`](https://github.com/isciences/exactextractr) package to aggregate nighttime lights data to polygons. Below we show trends in annual nighttime lights data across Ghana's first administrative divisions.\n\n```r\n#### Extract annual data\nntl_df \u003c- bm_extract(roi_sf = roi_sf,\n                     product_id = \"VNP46A4\",\n                     date = 2012:2022,\n                     bearer = bearer)\n\n#### Trends over time\nntl_df |\u003e\n  ggplot() +\n  geom_col(aes(x = date,\n  y = ntl_mean),\n  fill = \"darkorange\") +\n  facet_wrap(~NAME_1) +\n  labs(x = NULL,\n       y = \"NTL Luminosity\",\n       title = \"Ghana Admin Level 1: Annual Average Nighttime Lights\") +\n  scale_x_continuous(labels = seq(2012, 2022, 4),\n                     breaks = seq(2012, 2022, 4)) +\n  theme_minimal() +\n  theme(strip.text = element_text(face = \"bold\"))\n```\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"man/figures/ntl_trends_gha.png\" alt=\"Nighttime Lights Trends\" width=\"800\"/\u003e\n\u003c/p\u003e\n\n### Workflow to update data \u003ca name=\"update-data\"\u003e\n\nSome users may want to monitor near-real-time changes in nighttime lights. For example, daily Black Marble nighttime lights data is updated regularly, where data is available roughly on a week delay; same use cases may require examining trends in daily nighttime lights data as new data becomes available. Below shows example code that could be regularly run to produce an updated daily dataset of nighttime lights.\n\nThe below code produces a dataframe of nighttime lights for each date, where average nighttime lights for Ghana's 1st administrative division is produced. The code will check whether data has already been downloaded/extracted for a specific date, and only download/extract new data.\n\n```r\n# Create directories to store data\ndir.create(file.path(getwd(), \"bm_files\"))\ndir.create(file.path(getwd(), \"bm_files\", \"daily\"))\n\n# Extract daily-level nighttime lights data for Ghana's first administrative divisions.\n# Save a separate dataset for each date in the `\"~/Desktop/bm_files/daily\"` directory.\n# The code extracts data from January 1, 2023 to today. Given that daily nighttime lights\n# data is produced on roughly a week delay, the function will only extract data that exists;\n# it will skip extracting data for dates where data has not yet been produced by NASA Black Marble.\nbm_extract(roi_sf = roi_sf,\n           product_id = \"VNP46A2\",\n           date = seq.Date(from = ymd(\"2023-01-01\"), to = Sys.Date(), by = 1),\n           bearer = bearer,\n           output_location_type = \"file\",\n           file_dir = file.path(getwd(), \"bm_files\", \"daily\"))\n\n# Append daily-level datasets into one file\nfile.path(getwd(), \"bm_files\", \"daily\") |\u003e\n  list.files(pattern = \"*.Rds\",\n  full.names = T) |\u003e\n  map_df(readRDS) |\u003e\n  saveRDS(file.path(getwd(), \"bm_files\", \"ntl_daily.Rds\"))\n```\n\n## Functions and arguments \u003ca name=\"function-args\"\u003e\n\n### Functions \u003ca name=\"functions\"\u003e\n\nThe package provides two functions.\n\n* `bm_raster` produces a raster of Black Marble nighttime lights.\n* `bm_extract` produces a dataframe of aggregated nighttime lights to a region of interest (e.g., average nighttime lights within US States).\n\nBoth functions take the following arguments:\n\n### Required arguments \u003ca name=\"args-required\"\u003e\n\n* **roi_sf:** Region of interest; sf polygon. Must be in the [WGS 84 (epsg:4326)](https://epsg.io/4326) coordinate reference system. For `bm_extract`, aggregates nighttime lights within each polygon of `roi_sf`.\n\n* **product_id:** One of the following:\n\n  * `\"VNP46A1\"`: Daily (raw)\n  * `\"VNP46A2\"`: Daily (corrected)\n  * `\"VNP46A3\"`: Monthly\n  * `\"VNP46A4\"`: Annual\n\n* **date:**  Date of raster data. Entering one date will produce a `SpatRaster` object. Entering multiple dates will produce a `SpatRaster` object with multiple bands; one band per date.\n\n  * For `product_id`s `\"VNP46A1\"` and `\"VNP46A2\"`, a date (eg, `\"2021-10-03\"`).\n  * For `product_id` `\"VNP46A3\"`, a date or year-month (e.g., `\"2021-10-01\"`, where the day will be ignored, or `\"2021-10\"`).\n  * For `product_id` `\"VNP46A4\"`, year or date  (e.g., `\"2021-10-01\"`, where the month and day will be ignored, or `2021`).\n\n* **bearer:** NASA bearer token. For instructions on how to create a token, see [here](https://github.com/worldbank/blackmarbler#bearer-token-).\n\n### Optional arguments \u003ca name=\"args-optional\"\u003e\n\n* **variable:** Variable to used to create raster (default: `NULL`). To see all variable choices, set `variable = \"\"` (this will create an error message that lists all valid variables). For additional information on all variable choices, see [here](https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/Document%20Archive/Science%20Data%20Product%20Documentation/VIIRS_Black_Marble_UG_v1.2_April_2021.pdf); for `VNP46A1`, see Table 3; for `VNP46A2` see Table 6; for `VNP46A3` and `VNP46A4`, see Table 9. If `NULL`, uses the following default variables:\n\n  * For `product_id` `\"VNP46A1\"`, uses `DNB_At_Sensor_Radiance_500m`.\n  * For `product_id` `\"VNP46A2\"`, uses `Gap_Filled_DNB_BRDF-Corrected_NTL`.\n  * For `product_id`s `\"VNP46A3\"` and `\"VNP46A4\"`, uses `NearNadir_Composite_Snow_Free`.\n\n* **quality_flag_rm:** Quality flag values to use to set values to `NA`. Each pixel has a quality flag value, where low quality values can be removed. Values are set to `NA` for each value in ther `quality_flag_rm` vector. (Default: `NULL`).\n\n  * For `VNP46A1` and `VNP46A2` (daily data):\n    * `0`: High-quality, Persistent nighttime lights\n    * `1`: High-quality, Ephemeral nighttime Lights\n    * `2`: Poor-quality, Outlier, potential cloud contamination, or other issues\n\n  * For `VNP46A3` and `VNP46A4` (monthly and annual data):\n    * `0`: Good-quality, The number of observations used for the composite is larger than 3\n    * `1`: Poor-quality, The number of observations used for the composite is less than or equal to 3\n    * `2`: Gap filled NTL based on historical data\n\n* **check_all_tiles_exist:** Check whether all Black Marble nighttime light tiles exist for the region of interest. Sometimes not all tiles are available, so the full region of interest may not be covered. If `TRUE`, skips cases where not all tiles are available. (Default: `TRUE`).\n* **interpol_na:** When data for more than one date is downloaded, whether to interpolate `NA` values in rasters using the [`terra::approximate`](https://www.rdocumentation.org/packages/raster/versions/3.6-26/topics/approxNA) function. Additional arguments for the [`terra::approximate`](https://www.rdocumentation.org/packages/raster/versions/3.6-26/topics/approxNA) function can also be passed into `bm_raster`/`bm_extract` (eg, `method`, `rule`, `f`, `ties`, `z`, `NA_rule`). (Default: `FALSE`).\n* **h5_dir:** Black Marble data are originally downloaded as `h5` files. If `h5_dir = NULL`, the function downloads to a temporary directory then deletes the directory. If `h5_dir` is set to a path, `h5` files are saved to that directory and not deleted. The function will then check if the needed `h5` file already exists in the directory; if it exists, the function will not re-download the `h5` file.\n* **download_method:** Method to download data (h5 files) from NASA LAADS Archive: \"`httr`\" or \"`wget`\". If `httr`, uses the `httr2` R package to download data. If `wget`, uses the `wget` command line tool. `httr` is fully integrated in R, while `wget` requires the `wget` system command. `wget` can be more efficient and can help avoid network issues. (Default: `\"httr\"`).\n\n* **output_location_type:** Where output should be stored (default: `r_memory`). Either:\n\n  * `r_memory` where the function will return an output in R\n  * `file` where the function will export the data as a file. For `bm_raster`, a `.tif` file will be saved; for `bm_extract`, a `.Rds` file will be saved. A file is saved for each date. Consequently, if `date = c(2018, 2019, 2020)`, three datasets will be saved: one for each year. Saving a dataset for each date can facilitate re-running the function later and only downloading data for dates where data have not been downloaded.\n\nIf `output_location_type = \"file\"`, the following arguments can be used:\n\n* **file_dir:** The directory where data should be exported (default: `NULL`, so the working directory will be used)\n* **file_prefix:** Prefix to add to the file to be saved. The file will be saved as the following: `[file_prefix][product_id]_t[date].[tif/Rds]`\n* **file_skip_if_exists:** Whether the function should first check wither the file already exists, and to skip downloading or extracting data if the data for that date if the file already exists (default: `TRUE`). If the function is first run with `date = c(2018, 2019, 2020)`, then is later run with `date = c(2018, 2019, 2020, 2021)`, the function will only download/extract data for 2021. Skipping existing files can facilitate re-running the function at a later date to download only more recent data.\n* **file_return_null:** Whether to return `NULL` instead of a output to R (`SpatRaster` or `dataframe`). When `output_location_type = 'file'`, the function will export data to the `file_dir` directory. When `file_return_null = FALSE`, the function will also return the queried data---so the data is available in R memory. Setting `file_return_null = TRUE`, data will be saved to `file_dir` but no data will be returned by the function to R memory (default: `FALSE`).\n\n  \n* **...:** Additional arguments for [`terra::approximate`](https://rspatial.github.io/terra/reference/approximate.html), if `interpol_na = TRUE`\n\n### Argument for `bm_extract` only \u003ca name=\"args-extract\"\u003e\n\n* **aggregation_fun:** A vector of functions to aggregate data (default: `\"mean\"`). The `exact_extract` function from the `exactextractr` package is used for aggregations; this parameter is passed to `fun` argument in `exactextractr::exact_extract`.\n* **add_n_pixels:** Whether to add a variable indicating the number of nighttime light pixels used to compute nighttime lights statistics (eg, number of pixels used to compute average of nighttime lights). When `TRUE`, it adds three values: `n_non_na_pixels` (the number of non-`NA` pixels used for computing nighttime light statistics); `n_pixels` (the total number of pixels); and `prop_non_na_pixels` the proportion of the two. (Default: `TRUE`).\n\n## Black Marble Resources \u003ca name=\"resources\"\u003e\n\nFor more information on NASA Black Marble, see:\n\n* [Academic paper](https://doi.org/10.1016/j.rse.2018.03.017)\n* [Substack Post](https://www.spatialedge.co/p/not-all-nightlight-datasets-are-the)\n* [Webinar](https://appliedsciences.nasa.gov/get-involved/training/english/arset-introduction-nasas-black-marble-night-lights-data)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbank%2Fblackmarbler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fworldbank%2Fblackmarbler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbank%2Fblackmarbler/lists"}