{"id":26658895,"url":"https://github.com/denisecase/smart-sales-spark","last_synced_at":"2025-03-25T10:17:12.692Z","repository":{"id":265986052,"uuid":"896618328","full_name":"denisecase/smart-sales-spark","owner":"denisecase","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-01T23:09:37.000Z","size":1,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-01T23:31:48.379Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denisecase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-30T21:03:00.000Z","updated_at":"2024-12-01T23:09:40.000Z","dependencies_parsed_at":"2024-12-01T23:42:36.702Z","dependency_job_id":null,"html_url":"https://github.com/denisecase/smart-sales-spark","commit_stats":null,"previous_names":["denisecase/smart-sales-spark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fsmart-sales-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fsmart-sales-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fsmart-sales-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fsmart-sales-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denisecase","download_url":"https://codeload.github.com/denisecase/smart-sales-spark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245440537,"owners_count":20615633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-25T10:17:12.176Z","updated_at":"2025-03-25T10:17:12.676Z","avatar_url":"https://github.com/denisecase.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# smart-sales-spark\n\nThis project provides an introduction to using Apache Spark for data processing and analysis. \nIt includes hands-on examples for working with Spark DataFrames and instructions for setting up Spark on your local machine.\n\nApache Spark is a powerful distributed computing framework widely used for big data processing, machine learning, and real-time analytics. \n\nWe can use Spark with structured data using Spark DataFrames, query data with SQL, or process unstructured data using RDDs (Resilient Distributed Datasets).\n\n## Versions Matter!\n\nThis guide has been tested with:\n\n-Python 3.10.11 (newest is 3.12.4 - which does NOT work)\n-PySpark 3.5.3\n-Spark 3.5.3\n-JDK 17\n-Winutils for Spark 3\n\n## Apache Spark Homepage\n\nRead about Spark’s features and capabilities on the [Apache Spark Homepage](https://spark.apache.org/).\n\n## Apache Spark Examples\n\nVisit the [Apache Spark Examples Page](https://spark.apache.org/examples.html) to work through Spark’s official examples, including:\n- DataFrame Example -  demonstrates creating a Spark DataFrame and performing operations like filtering, aggregation, and adding columns.\n- SQL Example - illustrates how to query data using Spark SQL.\n- RDD Example - ntroduces RDDs for processing unstructured data.\n- Streaming Example - shows how to handle real-time data-in-motion with structured streaming.\n\n## Spark Pipeline Project Examples\n\nExplore how different Spark-based projects might be structured by checking the docs folder. \nEach project demonstrates the use of a scalable pipeline architecture tailored to a specific domain.\n\n- [Spark Sales Project](docs/spark-sales.md) - analyzes customer, product, and sales data to generate insights like sales trends and product performance.\n  \n- [Spark Social Media Project](docs/spark-social.md) - processes social media data to uncover user engagement patterns, hashtag trends, and sentiment analysis.\n- \n- [Spark Internet of Things (IOT) Project](docs/spark-iot.md) - aggregates and visualizes IoT sensor data to detect anomalies, monitor device usage, and analyze trends.\n\n## FIRST: Set Up Your Machine\n\nFollow the instructions to set up your system first:\n\n- [Setup for macOS/Linux](SETUP_SPARK_MAC_LINUX.md)\n- [Setup for Windows](SETUP_SPARK_WINDOWS.md)\n\n## SECOND: Activate your local Virtual Enviroment \u0026 Install Dependencies\n\nExamples of .gitignore and requirements.txt are provided (yours may vary).\n\nThe project assumes the smart sales respository is organized like the following (yours may vary - adjust your paths using pathlib to reflect your existing project layout). \n\n```\nproject/\n├── data/prepared\n│   ├── customers_data_prepared.csv\n│   ├── products_data_prepared.csv\n│   ├── sales_data_prepared.csv\n├── scripts/\n│   ├── step0-pipeline.py           # Orchestrate the pipeline\n│   ├── step1-extract.py            # Extract stage: Read data from sources\n│   ├── step2-transform.py          # Transform stage: Process data for insights\n│   ├── step3-load.py               # Load stage: Save results to storage\n│   └── step4-visualize.py          # Visualize results using seaborn or matplotlib\n├── notebooks/\n│   ├── insights.ipynb             # Notebook orchestrating extract-transform-load (ETL) + visualization\n├── .gitignore\n├── README.md\n└── requirements.txt\n```\n\nFollow the instructions to manage your local virtual environment:\n\n- [VIRTUAL_ENV](VIRTUAL_ENV.md)\n\n## THIRD: Run PySpark Basic Script\n\nWe keep our Python scripts in the scripts folder.\n\nCreate new files with these names in your scripts folder. \nPaste the contents from the file provided in this repo.\n\n### On Windows Machine\n\n1. In VS Code, open a PowerShell terminal in the root project folder. \n\n2. Activate your local project environment everytime you open a terminal to work on the project. \n\n```shell\n.\\.venv\\Scripts\\activate\n```\n\nProtip: After running the command once, you can usually get it back by typing just the initial dot and then hitting the right arrow key  - or use the up arrow to access prior commands. \n\n1. Execute the script.\n\n```shell\npy scripts\\step0_pipeline.py\n```\n\nProtip: After running the command once, you can usually get it back by typing just the initial py and then hitting the right arrow key - or use the up arrow to access prior commands. \n\nIf you get a Windows Firewall alert regarding the JDK, click Allow. \n\n\n### On Mac/LInux Machine\n\n1. In VS Code, open a terminal in the root project folder. \n\n2. Activate your local project environment everytime you open a terminal to work on the project. \n\n```zsh\nsource .venv/bin/activate\n \n```\n\n3. Execute the script\n\n```zsh\npython3 scripts/step0_pipeline.py\n```\n\n## FOURTH: ENHANCE FUNCTIONALITY\n\nAdd or update the files to make your own functionality. \n\nPaste the contents from the file provided in this repo.\n\nExecute your scripts - or experiment with a Jupyter notebook. \n\n\n## Troubleshooting\n\n```powershell\n$Env:HADOOP_HOME\n\nTest-Path \"$Env:HADOOP_HOME\\bin\\winutils.exe\"\n\n$env:Path = [System.Environment]::GetEnvironmentVariable(\"Path\", \"Machine\") + \";\" + [System.Environment]::GetEnvironmentVariable(\"Path\", \"User\")\n```\n\n\n# BYPASS LOCAL INSTALLATION - TRY IT ON THE WEB\n\n- \u003chttps://hub.ovh2.mybinder.org/user/apache-spark-rwmw01td/notebooks/python/docs/source/getting_started/quickstart_df.ipynb\u003e\n\n- \u003chttps://colab.research.google.com/drive/1fa2G3YuXx3Isqyby5kFETqmWotFwtqlH?usp=sharing\u003e\n\n- \u003chttps://github.com/apache/spark/tree/master/examples/src/main/python\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fsmart-sales-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenisecase%2Fsmart-sales-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fsmart-sales-spark/lists"}