https://github.com/bgpkit/bgpkit-broker-backend
BGPKIT Broker backend services and API
https://github.com/bgpkit/bgpkit-broker-backend
bgp rust
Last synced: about 1 year ago
JSON representation
BGPKIT Broker backend services and API
- Host: GitHub
- URL: https://github.com/bgpkit/bgpkit-broker-backend
- Owner: bgpkit
- License: mit
- Created: 2021-10-25T06:19:48.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-01-09T17:09:44.000Z (over 2 years ago)
- Last Synced: 2025-03-26T15:48:01.968Z (about 1 year ago)
- Topics: bgp, rust
- Language: Rust
- Size: 249 KB
- Stars: 8
- Watchers: 2
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.org
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
#+TITLE: BGPKIT Broker Backend
BGPKIT Broker Backend is the backend API and data collection implementation for
the [[https://bgpkit.com/broker][BGPKIT Broker project]].
Users can use this repository to deploy a BGPKIT Broker instance on their own
equipment, and it is free under MIT license! See more in [[#deployment][Deployment]] and [[#license][LICENSE]]
sections.
* Table of Content :TOC:
- [[#bgpkit-broker-api][BGPKIT Broker API]]
- [[#data-sources][Data Sources]]
- [[#deployment][Deployment]]
- [[#license][LICENSE]]
* BGPKIT Broker API
:PROPERTIES:
:ID: 279d0e5a-8de8-4906-9b45-71669a00469c
:END:
At a high level, the BGPKIT Broker is a REST API service that provides search
capability for BGP MRT archive files from various data collection projects. An
user can search for matching archive files with simple REST API calls.
For example, we want to learn about exactly what BGP updates were involved that
caused Facebook’s outage for multiple hours on Oct. 4th, 2021. We can use the
following REST call to the BGPKIT Broker API to gather the full list of data
files that may contain the relevant updates for the event:
#+begin_src restclient :exports both
GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1
#+end_src
#+RESULTS:
#+BEGIN_SRC js
{
"count": 5,
"page": 1,
"page_size": 5,
"error": null,
"data": [
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views2",
"data_type": "update",
"url": "http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 1153433,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views3",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views3/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 8388608,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views5",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views5/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 22528,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views.amsix",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views.amsix/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 7864320,
"exact_size": 0
},
{
"ts_start": "2021-10-04T15:30:00+00:00",
"ts_end": "2021-10-04T15:45:00+00:00",
"collector_id": "route-views.chicago",
"data_type": "update",
"url": "http://archive.routeviews.org/route-views.chicago/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2",
"rough_size": 1887436,
"exact_size": 0
}
]
}
// GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1
// HTTP/1.1 200 OK
// Content-Length: 1395
// Content-Type: application/json
// Date: Mon, 25 Apr 2022 21:54:29 GMT
// Server: Caddy
// Server: uvicorn
// Request duration: 1.434210s
#+END_SRC
Here is a break-down of the call parameters:
- ~ts_start=1633362000~: all BGP files after 15:40 UTC on that day
- ~ts_end=1633362022~: all BGP files before 16:00 UTC on that day
- ~data_type=update~: only return updates files
- ~page_size=5~: return 5 items per page
- ~page=1~: return the 1st page (pages starting from 1)
For more examples and API spec, please checkout our [[https://api.broker.bgpkit.com/v2/docs][documentation site]].
* Data Sources
:PROPERTIES:
:ID: 6c5be109-0165-4b04-9983-bbb33d5ab515
:END:
Broker backend currently support two data collection projects: [[http://archive.routeviews.org][RouteViews]] and
[[https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data][RIPE RIS]]. Specifically, we have designed tailored data collection procedures to
efficiently crawl the data publication sites from the two projects and collect
basic information of the published BGP MRT files.
The information we currently collect for each data file includes:
- data type: either BGP updates files, or BGP table dump files.
- timestamp: the time of the beginning of the data dump, usually aligns on the
full five-minute marks.
- url: the URL for the downloadable data file.
- collector ID: the ID of the data collector where that data comes from.
- rought and exact file sizes
Note that the data dump frequencies for RouteViews and RIPE RIS are different.
Here is a table summarizing their frequencies:
|------------+--------------------+----------------------|
| Project | Update frequency | Table dump frequency |
|------------+--------------------+----------------------|
| RouteViews | one per 15 minutes | one per 2 hours |
| RIPE RIS | one per 5 minutes | one per 8 hours |
|------------+--------------------+----------------------|
In essence, you would see more frequent BGP updates dumps from RIPE RIS and more
frequent full table dumps from RouteViews.
* Deployment
:PROPERTIES:
:ID: f400ff62-9b19-4416-ae3d-a358e71b937e
:END:
Here is a step-by-step guide for deploying BGPKIT Broker with Docker
1. Checkout the repository
2. run ~docker-compose -f deployment/docker-compose.yml -p bgpkit-broker up~
The initial database bootstrap phase would take about 30 minutes to one hour depending on
your deployment environment. It requires at least 30GB of free storage for the
container and build processes.
After the initial bootstrap phase is done, the API service should be up and
running, currently hosted at port ~18888~. You can modify the port in
~docker-compose.yml~ file. In the mean time, a cronjob service also started,
crawling collectors for recent data every 5 minutes. The frequency can be
configured in the ~update.cron~. It is not recommended to go more frequent than
one crawl per 5 minutes.
You can check out if the API is running by running:
#+begin_src bash
curl 0.0.0.0:18888/latest
curl "0.0.0.0:18888/search?ts_start=1643673600&ts_end=2022-02-02&collector_id=rrc00"
#+end_src
* LICENSE
:PROPERTIES:
:ID: d66943c0-30e1-40df-a02d-063806ca8d7d
:END:
BGPKIT Broker backend code is under [[file:LICENSE][MIT LICENSE]].
* Built with ❤️ by BGPKIT Team :noexport:
BGPKIT is a small-team focuses on building the best open-source tooling for BGP data processing and analysis in
Rust. Learn more about what we are offering at https://github.com/bgpkit.
[[https://spaces.bgpkit.org/assets/logos/wide-solid-200px.png]]