{"id":22557491,"url":"https://github.com/vara-co/home_sales","last_synced_at":"2025-03-28T11:26:18.296Z","repository":{"id":241377377,"uuid":"806699397","full_name":"vara-co/Home_Sales","owner":"vara-co","description":"Module 22 challenge: Using Google Colab to work on Big Data queries with PySpark SQL, parquet, and cache partitions","archived":false,"fork":false,"pushed_at":"2024-06-01T15:20:05.000Z","size":48,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T11:45:44.922Z","etag":null,"topics":["big-data","big-data-analytics","cache","google-colab","google-colaboratory","parquet","pyspark","pyspark-sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vara-co.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-27T17:58:38.000Z","updated_at":"2024-06-02T04:41:57.000Z","dependencies_parsed_at":"2024-05-31T19:06:56.751Z","dependency_job_id":"6a81c738-64b4-494e-88cf-b9f09559fd45","html_url":"https://github.com/vara-co/Home_Sales","commit_stats":null,"previous_names":["vara-co/home_sales"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vara-co%2FHome_Sales","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vara-co%2FHome_Sales/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vara-co%2FHome_Sales/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vara-co%2FHome_Sales/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vara-co","download_url":"https://codeload.github.com/vara-co/Home_Sales/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246019672,"owners_count":20710609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","big-data-analytics","cache","google-colab","google-colaboratory","parquet","pyspark","pyspark-sql"],"created_at":"2024-12-07T20:07:24.767Z","updated_at":"2025-03-28T11:26:18.276Z","avatar_url":"https://github.com/vara-co.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eDU - University of Denver\u003cbr/\u003e\nData Analysis \u0026 Visualization Bootcamp\u003cbr/\u003e\u003c/h1\u003e\n\n--------------------------------\n\n\u003ch2 align=\"center\"\u003eBig Data and PySpark\u003cbr/\u003e\n(Using Google Colab)\nModule 22 Challenge\n\u003cbr/\u003e\nBy Laura Vara\u003c/h2\u003e\u003cbr/\u003e\n\n\nIn this challenge, you'll use your knowledge of SparkSQL to determine key metrics about home sales data. Then you'll use Spark to create temporary views, partition the data, cache and uncache a temporary table, and verify that the table has been uncached.\n\u003cbr/\u003e\n\u003cbr/\u003e\n**Note:** While there are two Jupyter Notebook formated files and both have the same code, the 'Home_Sales_colab.ipynb' was the were I ran the code via Google Colab. Therefore it's the only one of these two files that has the output to the queries.\n\n\n![GitReadMe_TxtImage](https://github.com/vara-co/Home_Sales/assets/152572519/037795f3-fc92-49a8-82e1-8b43b97dd047)\n\n---------------------------------\nINDEX\n---------------------------------\n1. Content of the repository\n2. Instructions for the Project\n3. References\n\n---------------------------------\nContent of the repository\n---------------------------------\n- Home_Sales.ipynb\n- Home_Sales_colab.ipynb \u003c-- This is the Jupyter Notebook that I actually worked on via Google Colab, thus it has the outputs to each cell of code.\n\n----------------------------------\nInstructions\n----------------------------------\n1. Rename the **Home_Sales_starter_code.ipynb** file as **Home_Sales.ipynb**\n2. Import the necessary PySpark SQL functions for this assignment.\n3. Read the **home_sales_revised.csv** data in the starter code into a Spark DataFrame.\n4. Create a temporary table called **home_sales**\n5. Answer the following questions using SparkSQL:\n    * What is the average price for a four-bedroom house sold for each year? Round off your answer to two decimal places\n    * What is the average price of a home for each year the home was built, that has three bedrooms and three bathrooms? Round off your answer to two decimal places.\n    * What is the average price of a home for each year the home was built, that has three betrooms, three bathrooms, two floors, and is greater than or equal to 2,000 square feet? Round off your answer to two decimal places.\n    * What is the average price of a home per \"view\" rating having an average home price greater than or equal to $350,000? Determine the run time for thi squery, and round off your naswer to two decimal places.\n6. Cache your temporary table **home_sales**\n7. Check if your temporary table is cached.\n8. Using the cached data, run the last query that calculates the average price of a home per \"view\" rating having an average home price greater than or equal to $350,000. Determine the runtime and compare it to uncached runtime.\n9. Partition by the \"date_built\" field on the formatted parquet home sales data.\n10. Create a temporary table for the parquet data.\n11. Run the last query that calculates the average price of a home per \"view\" rating having an average home price greater than or equal to $350,000. Determine the runtime and compare it to uncached runtime.\n12. Uncache the **home_sales** temporary table\n13. Verify that the **home_sales** temporary table is uncached using PySpark.\n14. Download your **Home_sales.ipynb** file and upload it to your \"Home_Sales\" GitHub repository.\n     * **(NOTE: I worked on the colab version due to the packages installed and the size of the data. Therefore this is the file with the outputs).**\n\n\n------------------------------------\nReferences\n------------------------------------\nEverything included in this project was covered in class. Regardless, these two resources were used to complete the challenge.\n\n- Number Padding in PySpark Dataframes: https://stackoverflow.com/questions/45400829/padding-in-a-pyspark-dataframe\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvara-co%2Fhome_sales","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvara-co%2Fhome_sales","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvara-co%2Fhome_sales/lists"}