{"id":13724772,"url":"https://github.com/smnorris/bcfishobs","last_synced_at":"2025-04-14T00:55:05.142Z","repository":{"id":71794537,"uuid":"133072177","full_name":"smnorris/bcfishobs","owner":"smnorris","description":"Reference BC Known Fish Observations to the Freshwater Atlas stream network","archived":false,"fork":false,"pushed_at":"2024-10-18T22:47:28.000Z","size":335,"stargazers_count":6,"open_issues_count":5,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-29T20:31:57.737Z","etag":null,"topics":["british-columbia","fish-observations","fiss","fwa","stream-network"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smnorris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-05-11T18:00:01.000Z","updated_at":"2024-09-04T13:55:32.000Z","dependencies_parsed_at":"2023-12-31T01:42:12.912Z","dependency_job_id":"74f5aa64-31d9-490c-8fe0-36870d55b00a","html_url":"https://github.com/smnorris/bcfishobs","commit_stats":{"total_commits":150,"total_committers":1,"mean_commits":150.0,"dds":0.0,"last_synced_commit":"9da6e984c702042c4e6e0e1bf71144b60db72603"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smnorris%2Fbcfishobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smnorris%2Fbcfishobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smnorris%2Fbcfishobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smnorris%2Fbcfishobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smnorris","download_url":"https://codeload.github.com/smnorris/bcfishobs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222299797,"owners_count":16963090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["british-columbia","fish-observations","fiss","fwa","stream-network"],"created_at":"2024-08-03T01:02:03.207Z","updated_at":"2024-11-14T14:32:04.156Z","avatar_url":"https://github.com/smnorris.png","language":"Makefile","funding_links":[],"categories":["Biosphere"],"sub_categories":["Marine Life and Fishery"],"readme":"# bcfishobs\n\n[Known BC Fish Observations](https://catalogue.data.gov.bc.ca/dataset/known-bc-fish-observations-and-bc-fish-distributions) is documented as the *most current and comprehensive information source on fish presence for the province*. These scripts locate these observation points as linear referencing events on the most current and comprehensive stream network currently available for BC, the [Freshwater Atlas](https://www2.gov.bc.ca/gov/content/data/geographic-data-services/topographic-data/freshwater).\n\nThe scripts:\n\n- download `whse_fish.fiss_fish_obsrvtn_pnt_sp`, the latest observation data from DataBC\n- download a lookup table `whse_fish.wdic_waterbodies` used to match the 50k waterbody codes in the observations table to FWA waterbodies\n- download a lookup table `species_cd`, linking the fish species code found in the observation table to species name and scientific name\n- load above tables to a PostgreSQL database\n- discard any observations not coded as `point_type_code = 'Observation'` (`Summary` records are all duplicates of `Observation` records)\n- references the observation points to their position on the FWA stream network (as outlined below)\n- create two ouputs (see below for descriptions)\n\n### Matching logic, observations\n\n1. For observation points associated with a lake or wetland (according to `wbody_id`):\n\n    - match observations to the closest FWA stream in a waterbody that matches the observation's `wbody_id`, within 1500m\n    - if no FWA stream in a lake/wetland within 1500m matches the observation's `wbody_id`, match to the closest stream in any lake/wetland within 1500m\n\n2. For observation points associated with a stream:\n\n    - match to the closest FWA stream within 100m that has a matching watershed code (via `fwa_streams_20k_50k_xref`)\n    - for remaining unmatched records within 100m of an FWA stream, match to the closest stream regardless of a match via watershed code\n    - for remaining unmatched records between 100m to 500m of an FWA stream, match to the closest FWA stream that has a matching watershed code\n\nThis logic is based on the assumptions:\n\n- for observations noted as within a lake/wetland, we can use a relatively high distance threshold for matching to a stream because\n    -  an observation may be on a bank far from a waterbody flow line\n    -  as long as an observation is associated with the correct waterbody, it is not important to exactly locate it on the stream network within the waterbody\n- for observations on streams, the location of an observation should generally take priority over a match via the xref lookup because many points have been manually snapped to the 20k stream lines - the lookup is best used to prioritize instances of multiple matches within 100m and allow for confidence in making matches between 100 and 500m\n\n## General requirements\n\n- PostgreSQL/PostGIS \n- a FWA database created by [fwapg](https://github.com/smnorris/fwapg)\n- GDAL \u003e= 3.4\n- Python (\u003e=3.6)\n- [bcdata](https://github.com/smnorris/bcdata)\n\n\n## Run the scripts\n\nScripts presume that:\n\n- environment variable `DATABASE_URL` points to the appropriate db\n- FWA data are loaded to the db via `fwapg`\n\nTo set up the output tables and run the job:\n\n    $ git clone https://github.com/smnorris/bcfishobs.git\n    $ cd bcfishobs\n    $ make\n\n\nTo refresh the observation data from DataBC and re-run the analysis (without tearing down the output tables):\n\n    $ rm .make/fiss_fish_obsrvtn_pnt_sp\n    $ make\n\n\nTo tear down `bcfishobs` schema and re-run the analysis from scratch:\n\n    $ make clean\n    $ make\n\n\n## Outputs\n\nAll outputs are written to schema `bcfishobs`.\n\n#### `bcfishobs.fiss_fish_obsrvtn_events_vw`\n\nContains a record for each observation that is successfully matched to a stream \n(not just distinct locations), and commonly used columns.\nGeometries are located on the stream to which the observation is matched.\n\n```\n          Column           |          Type           |\n---------------------------+-------------------------+\n fish_observation_point_id | integer                 |\n fish_obsrvtn_event_id     | bigint                  |\n linear_feature_id         | bigint                  |\n wscode_ltree              | ltree                   |\n localcode_ltree           | ltree                   |\n blue_line_key             | integer                 |\n waterbody_key             | integer                 |\n downstream_route_measure  | double precision        |\n distance_to_stream        | double precision        |\n match_type                | text                    |\n watershed_group_code      | character varying(4)    |\n species_id                | integer                 |\n species_code              | character varying(6)    |\n agency_id                 | numeric                 |\n observation_date          | date                    |\n agency_name               | character varying(60)   |\n source                    | character varying(1000) |\n source_ref                | character varying(4000) |\n activity_code             | character varying(100)  |\n activity                  | character varying(300)  |\n life_stage_code           | character varying(100)  |\n life_stage                | character varying(300)  |\n acat_report_url           | character varying(254)  |\n geom                      | geometry(PointZM,3005)  |\n```\n\n#### `bcfishobs.fiss_fish_obsrvtn_events`\n\nDistinct locations of observations matched to streams.\nGeometries are located on the stream to which the observation is matched.\n\n```\n          Column          |          Type          |\n--------------------------+------------------------+\n fish_obsrvtn_event_id    | bigint                 |\n linear_feature_id        | integer                |\n wscode_ltree             | ltree                  |\n localcode_ltree          | ltree                  |\n blue_line_key            | integer                |\n watershed_group_code     | character varying(4)   |\n downstream_route_measure | double precision       |\n match_types              | text[]                 |\n obs_ids                  | integer[]              |\n species_codes            | text[]                 |\n species_ids              | integer[]              |\n maximal_species          | integer[]              |\n distances_to_stream      | double precision[]     |\n geom                     | geometry(PointZM,3005) |\nIndexes:\n    \"fiss_fish_obsrvtn_events_pkey\" PRIMARY KEY, btree (fish_obsrvtn_event_id)\n    \"fiss_fish_obsrvtn_events_blue_line_key_idx\" btree (blue_line_key)\n    \"fiss_fish_obsrvtn_events_linear_feature_id_idx\" btree (linear_feature_id)\n    \"fiss_fish_obsrvtn_events_localcode_ltree_idx\" btree (localcode_ltree)\n    \"fiss_fish_obsrvtn_events_obs_ids_idx\" gist (obs_ids gist__intbig_ops)\n    \"fiss_fish_obsrvtn_events_species_ids_idx\" gist (species_ids gist__intbig_ops)\n    \"fiss_fish_obsrvtn_events_wscode_ltree_idx\" btree (wscode_ltree)\n```\n\n#### `bcfishobs.fiss_fish_obsrvtn_unmatched`\n\nUnique observation locations that the scripts are unable to match to FWA streams.\n\n```\n            Column            |         Type         | Collation | Nullable | Default\n------------------------------+----------------------+-----------+----------+---------\n fish_obsrvtn_pnt_distinct_id | integer              |           | not null |\n obs_ids                      | integer[]            |           |          |\n species_ids                  | integer[]            |           |          |\n distance_to_stream           | double precision     |           |          |\n geom                         | geometry(Point,3005) |           |          |\nIndexes:\n    \"fiss_fish_obsrvtn_unmatched_pkey\" PRIMARY KEY, btree (fish_obsrvtn_pnt_distinct_id)\n    \"fiss_fish_obsrvtn_unmatched_geom_idx\" gist (geom)\n```\n\n#### `bcfishobs.summary`\n\nReport on total number of observations processed and matched to streams, and the type of match used.\n\n```\n      Column       |  Type   |\n-------------------+---------+\n match_type        | text    |\n n_distinct_events | integer |\n n_observations    | integer |\n```\n\n## Use the data\n\nWith the observations now linked to the Freswater Atlas, we can write queries to find fish observations relative to their location on the stream network.\n\n### Example 1\n\nList all species observed on the Cowichan River (`blue_line_key = 354155148`), downstream of Skutz Falls (`downstream_route_meaure = 34180`).\n\n```\nSELECT DISTINCT species_code\nFROM bcfishobs.fiss_fish_obsrvtn_events_vw\nWHERE blue_line_key = 354155148 AND\ndownstream_route_measure \u003c 34180\nORDER BY species_code;\n\n species_code\n--------------\n ACT\n AS\n BNH\n BT\n C\n CAL\n CAS\n CH\n CM\n CO\n CT\n DV\n EB\n GB\n KO\n L\n MARFAL\n RB\n SA\n SB\n ST\n TR\n TSB\n```\n\n### Example 2\n\nWhat is the slope (percent) of the stream at all distinct locations of Steelhead observations in `COWN` watershed group (on single line streams)?\n\n```\nSELECT \n  e.fish_obsrvtn_event_id,\n  s.gnis_name,\n  s.gradient\nFROM bcfishobs.fiss_fish_obsrvtn_events e\nINNER JOIN whse_basemapping.fwa_stream_networks_sp s\nON e.linear_feature_id = s.linear_feature_id\nWHERE e.species_codes \u0026\u0026 ARRAY['ST']\nAND e.watershed_group_code = 'COWN'\nAND s.edge_type = 1000\nORDER BY e.wscode_ltree, e.localcode_ltree, e.downstream_route_measure\n\n fish_obsrvtn_event_id |      gnis_name       | gradient \n-----------------------+----------------------+----------\n          675380033961 | Cowichan River       |   0.0071\n          675380034170 | Cowichan River       |   0.0614\n          641720026729 | Koksilah River       |   0.0058\n          641720036829 | Koksilah River       |   0.0369\n          641720037394 | Koksilah River       |   0.0761\n          581370001848 | Kelvin Creek         |   0.0061\n          581370006510 | Kelvin Creek         |   0.0167\n          564060000660 | Glenora Creek        |   0.0137\n          564060008058 | Glenora Creek        |   0.0843\n...\n```\n\n### Example 3\n\nWhat are the order, elevation and gradient of all Arctic Grayling observations in the Parsnip watershed group?\n\n```\nSELECT\n  fish_observation_point_id,\n  s.gradient,\n  s.stream_order,\n  round((ST_Z((ST_Dump(ST_LocateAlong(s.geom, e.downstream_route_measure))).geom))::numeric) as elevation\nFROM bcfishobs.fiss_fish_obsrvtn_events_vw e\nINNER JOIN whse_basemapping.fwa_stream_networks_sp s\nON e.linear_feature_id = s.linear_feature_id\nWHERE e.species_code = 'GR'\nAND e.watershed_group_code = 'PARS'\nORDER BY e.wscode_ltree, e.localcode_ltree, e.downstream_route_measure;\n\n fish_observation_point_id | gradient | stream_order | elevation \n---------------------------+----------+--------------+-----------\n                    233425 |        0 |            7 |       674\n                    233402 |   0.0007 |            7 |       675\n                    318578 |   0.0007 |            7 |       675\n                    233432 |   0.0004 |            7 |       685\n                     96418 |        0 |            6 |       694\n                    233458 |   0.0003 |            7 |       696\n...\n```\n\n## Warnings\n\n### `fish_observation_point_id`\n\nColumn `fish_observation_point_id` is not an immutable primary key in the source table.\n`fish_observation_point_id` is guaranteed to be unique for a given extract but values will change over time.\n- if current id values are required, run a fresh extract\n- if referring to a specific observation in communications, use some combination of `source`, `species_code`, `life_cycle_code`, coordinates, etc\n\n### Duplicates\n\nDuplicate rows (for all fields) exist in the source table and are replicated in the output view. \nUse observation counts with caution. \n\n## Scheduled job\n\nThe workflow is processed weekly and dumped to file.\nDownload the latest extract (as a geopackage) [here](https://bcfishpass.s3.us-west-2.amazonaws.com/bcfishobs.gpkg.zip).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmnorris%2Fbcfishobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmnorris%2Fbcfishobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmnorris%2Fbcfishobs/lists"}