https://github.com/notnite/eightyeightthirtyone

Building a graph of the Internet, one button at a time
https://github.com/notnite/eightyeightthirtyone

Last synced: about 1 year ago
JSON representation

Building a graph of the Internet, one button at a time

• Host: GitHub
• URL: https://github.com/notnite/eightyeightthirtyone
• Owner: NotNite
• License: mit
• Created: 2023-12-23T18:04:16.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-09-09T16:42:10.000Z (almost 2 years ago)
• Last Synced: 2025-05-08T22:13:53.768Z (about 1 year ago)
• Language: Rust
• Homepage: https://eightyeightthirty.one/
• Size: 350 KB
• Stars: 138
• Watchers: 1
• Forks: 4
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# eightyeightthirtyone



  

  





Building a graph of the Internet, one button at a time. Website available [here](https://eightyeightthirty.one).

This project crawls the links between [88x31s](https://tekeye.uk/computer_history/powered-by) on the Internet, which are small badges on websites that link to other websites. It's split into three projects:

- A host server (Rust) that manages work between scraper nodes and talks to a Redis database using [axum](https://lib.rs/crates/axum) and [fred](https://lib.rs/crates/fred)

- A scraper (Rust) that talks to the server, fetches URLs, and returns information using [scraper](https://lib.rs/crates/scraper)

- A web app (TypeScript/React) to render the graph using [Cosmograph](https://cosmograph.app)

Websites are scanned for images, and images that match the 88x31 resolution and link to another site are logged. It respects robots.txt and is aware of redirects.

Scrapers can either run a WebDriver or just parse the HTML - note that the latter will break discovery for websites that use JavaScript to create the button elements (e.g. React apps).

## Opting out

The scrapers respect robots.txt, so block this user agent (or allow if you want to opt in):

```text

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 eightyeightthirtyone/1.0.0 (https://github.com/NotNite/eightyeightthirtyone)

```

Note that this only somewhat works:

- You will still appear on the graph if anyone else links to you.

- Previous entries in the database are not deleted - please email me your domain and affected URLs if known (see [Credits & contact](#credits--contact)).

## Credits & contact

This project wouldn't be possible without the following people:

- [![notnite](https://notnite.com/buttons/notnite.png)](https://notnite.com/)

  - for starting the project, hosting the website and server

- [![adryd](https://adryd.com/static/buttons/adryd.png)](https://adryd.com/)

  - for her help on the frontend, designing the project's 88x31, hosting a scraper node

- [![breq](https://breq.dev/badges/breq.png)](https://breq.dev/)

  - for her help designing the Redis database schema, contributions to the server

- [![Inferno Comms](https://notnite.com/buttons/infernocomms.png)](https://connormcf.com/)

  - for lending us a server to host a scraper node on

If you have any questions or concerns, you can send an email to me (NotNite), which is [available on my website](https://notnite.com) (not written here for spam concerns).

Thanks!