https://github.com/ctadeodev/spark-word-counter

A Dockerized PySpark application for counting word frequencies in an input PDF document
https://github.com/ctadeodev/spark-word-counter

docker pdf-document-processor pyspark python spark

Last synced: 3 months ago
JSON representation

A Dockerized PySpark application for counting word frequencies in an input PDF document

• Host: GitHub
• URL: https://github.com/ctadeodev/spark-word-counter
• Owner: ctadeodev
• License: mit
• Created: 2024-07-06T06:04:34.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-07-06T10:01:30.000Z (6 months ago)
• Last Synced: 2024-09-29T07:01:37.971Z (4 months ago)
• Topics: docker, pdf-document-processor, pyspark, python, spark
• Language: Python
• Homepage:
• Size: 31.3 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# PDF word counter with Spark

A Dockerized PySpark application for counting word frequencies in a PDF document

In this basic configuration, we will create the following:

1. A word_count.py script that list of words (top 10) that appear in a pdf document and the number of their occurence

2. A 3-node (1 master, 2 worker) spark cluster that runs in Docker

I ran these both on my local machine, but in theory, the driver and cluster can be on separate machines