https://github.com/gr-menon/bigbanyantree

Gathering insights from Common Crawl using Apache Spark and LLMs.
https://github.com/gr-menon/bigbanyantree

apache-spark docker docker-compose llamafile

Last synced: 4 months ago
JSON representation

Gathering insights from Common Crawl using Apache Spark and LLMs.

• Host: GitHub
• URL: https://github.com/gr-menon/bigbanyantree
• Owner: GR-Menon
• Created: 2024-08-25T05:57:09.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-11-14T14:43:57.000Z (over 1 year ago)
• Last Synced: 2025-02-02T01:31:43.407Z (over 1 year ago)
• Topics: apache-spark, docker, docker-compose, llamafile
• Language: Jupyter Notebook
• Homepage:
• Size: 8.14 MB
• Stars: 3
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# BigBanyanTree



  

      




BigBanyanTree is an initiative to empower engineering colleges to set up their data engineering clusters and drive

interest in data processing and analysis using tools such as Apache Spark.


This project was made in collaboration with ![Suchit](https://www.linkedin.com/in/suchitg04/) under the guidance

of ![Mr. Harsh Singhal](https://www.linkedin.com/in/harshsinghal/).

The endeavour comprised of 3 main steps:

- Set up a ***dedicated Apache Spark cluster*** along with Jupyterlab interface to run Spark jobs.

- Parse a ***random 1% sample of the Common Crawl data dumps*** spanning the years 2018 to 2024, extracting various

  attributes.

- Perform **various analyses on the extracted datasets** and open-source our findings.

Check out the open-sourced HuggingFace datasets we created

at [huggingface.co/big-banyan-tree](https://huggingface.co/big-banyan-tree)

### Apache Cluster Setup

We first set up an Apache Spark cluster in standalone mode on a dedicated Hetzner server. The entire server setup was

made quite simple and straightforward by making use of `Docker` and `Docker Compose`.

To get a more in-depth understanding of our Apache Spark cluster setup, check out the following resources :

- [SparkBazaar](https://github.com/GR-Menon/Spark-Bazaar)

- [BigBanyanTree Setup Blog](https://datascience.fm/zero-to-spark-apache-spark-cluster-setup/)

- [LLM Service Setup](https://datascience.fm/llamafile-an-executable-llm/)

  


### CommonCrawl Data Processing

Common Crawl releases data dumps every few months, containing raw HTML source code of the literal Internet, and

open-sources this data using archival file storage formats such as **WARC** (Web Archive).

Under the BigBanyanTree project, we undertook two main data processing tasks:

- Extracting webpage JavaScript libraries from `src` tags within HTML `script` tags, among other fields.

- Enriching server IP & geolocation data using the MaxMind Database.

For a deep dive into both these topics, check out our blogs :

- [JSLib Extraction](https://datascience.fm/parsing-html-source-code-with-apache-spark-selectolax/)

- [Server MaxMind Data Enrichment](https://datascience.fm/bigbanyantree-enriching-warc-data-with-ip-information-from-maxmind/)

    - [Serializability in Spark](https://datascience.fm/serializability-in-spark-using-non-serializable-objects-in-spark-transformations/)

### Extracted Data Analysis

TODO