{"id":17018336,"url":"https://github.com/devprojectekla/heliocity","last_synced_at":"2026-05-09T15:34:48.160Z","repository":{"id":238703166,"uuid":"797249762","full_name":"DevprojectEkla/HelioCity","owner":"DevprojectEkla","description":"HelioCity challenge. This project, developed as a challenge, focuses on importing data from .csv files into a PostgreSQL database and performing various data manipulation tasks.","archived":false,"fork":false,"pushed_at":"2024-06-20T16:25:18.000Z","size":66,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-22T16:11:45.454Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DevprojectEkla.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-07T13:32:35.000Z","updated_at":"2024-06-20T16:25:21.000Z","dependencies_parsed_at":"2024-06-21T09:19:31.210Z","dependency_job_id":"5c8a9702-f791-4858-b967-06f9fc3724c7","html_url":"https://github.com/DevprojectEkla/HelioCity","commit_stats":null,"previous_names":["devprojectekla/heliocity"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DevprojectEkla/HelioCity","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevprojectEkla%2FHelioCity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevprojectEkla%2FHelioCity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevprojectEkla%2FHelioCity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevprojectEkla%2FHelioCity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DevprojectEkla","download_url":"https://codeload.github.com/DevprojectEkla/HelioCity/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevprojectEkla%2FHelioCity/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32824402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T06:45:35.009Z","updated_at":"2026-05-09T15:34:48.141Z","avatar_url":"https://github.com/DevprojectEkla.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Heliocity: Backend Challenge\n\n## Description of Various Functionalities\n\n\u003e ### 0a. The `main.py` file provides an overview of a possible scenario (see section C below) combining all described functionalities.\n\n\u003e ### 0b. The `tests.py` file allows running tests for different functionalities.\n\n### 1. The `database_handler.py` file with the `DatabaseHandler` class and its `process_csv_file()` method\n\n- Imports `.csv` files into the database:\n\n    1. From the weather API\n    2. From the calculator\n\n\u003e ### Import Optimization:\n\u003e Uses various parallelism methods (`map_async`, `apply_async`, `map`) from Python's native `multiprocessing.Pool` class.\n\u003e    \n\u003e Strategies under development:\n\u003e - Splitting into smaller files\n\u003e - Implementation in a low-level language like Rust\n\n### 2. The `database_selector.py` file and its associated `DatabaseSelector` class with various methods\n\n- Adjusts weather data from a 5-minute to a 15-minute time step to ajust to the calculator's step.\n  \n\u003e Upcoming features:\n\u003e - Dynamic specification of initial and target time steps\n\u003e - Creates SQL sub-tables containing the selected data range (time range, temperature, etc.) generated from the original table.\n\n### 3. The `json_generator.py` file and its associated `JSONGenerator` class\n\n- Manipulates database data to generate a `.json` file for visualization.\n- Provides data preview with the option to filter out aberrant values.\n\n# Getting Started\n\n## Create a Virtual Environment\n\n### A. Prerequisites\n\n\u003e Guidelines for a Linux environment\n\n- Configured and running PostgreSQL server.\n- Creation and configuration of a new database.\n- Edit the `config.json` file with necessary parameters for connecting to the database.\n\n### B. Installation\n\n#### Clone files from the Git repo:\n\n```bash\ngit clone https://github.com/DevprojectEkla/HelioCity\ncd HelioCity\n```\n### Create a Virtual Environment:\n\n```bash\npython -m venv env\n```\n\n### Activate the Virtual Environment:\n\n```bash\nsource env/bin/activate  # On Linux\n```\n\n### Install Dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n### (Optional) Create a `data/` Folder for Your `.csv` Files:\n\n```bash\nmkdir data\n```\n\n## C. Usage Scenario Example Using Our Classes\n\nThe `main.py` file can be launched with arguments; otherwise, a series of prompts will ask for:\n\n- The table name (either an existing table name or the name for a new table to be created in the database from the imported file).\n- If applicable, the name of the `.csv` file to import into the database.\n- Optionally, use the `-f` flag to specify a simple import method; absence of the flag defaults to a parallelism-based import.\n\n```bash\npython main.py [table_name] [path_to_csv_file] [-f]\n```\n\n### Imagined Scenario Type:\n\n- Import a table from `./data/meteo_data.csv` in preprocessing or `./data/test_helio.csv` in post-processing.\n- Filter out aberrant data and specify a time interval.\n- Insert a new variable called `python_calc`* into a table for time-based representation.\n\u003e \u0026#42; In this scenario, it involves preprocessing wind chill temperature as a function of temperature, wind speed, and relative humidity. In post-processing, it's a test calculation (to be adjusted with a relevant formula).\n- Generate a `.json` file from this preview data for future use in another context.\n\n## D. Independent Usage of Different Scripts\n\n### Importing CSV Data into PostgreSQL\n\n#### Run `database_handler.py`:\n\n```bash\npython database_handler.py\n```\n\nYou will be prompted for:\n\n- The name of the new table to create (default: `meteo_data`).\n- The path to the `.csv` data file (default: `./data/meteo_data.csv`).\n- Specify data origin (weather or calculator); calculator column processing takes place in adjustable portions of the number of lines answered `'y'` if it's a large file. 'n' or '' in the case of a large file.\n\n\u003e Warning: Importing large `.csv` files from the calculator can take some time depending on the computer's memory capabilities. Adjust the value of the number of lines per portion to available memory.\n\n### Post and Pre-Processing Data Manipulations\n\n#### Using `DatabaseSelector` Class from `database_selector.py`:\n\nData manipulations can be performed using the `DatabaseSelector` class to create new tables in the database. It allows:\n\n- Creating sub-tables by interval of interest.\n- Aggregating weather data at the calculator's timestep.\n- Inserting calculated variables from existing table variables.\n\nFor a test, simply run the command:\n\n```bash\npython database_selector.py\n```\n\nFollow the instructions...\n\n#### Using `JSONGenerator` Class from `json_generator.py`:\n\nThis class only reads from the database and does not write to it. It facilitates easy manipulation of data in dataframes for visualization and is used to generate a `.json` format.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevprojectekla%2Fheliocity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevprojectekla%2Fheliocity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevprojectekla%2Fheliocity/lists"}