{"id":27890002,"url":"https://github.com/udaylab/aeros","last_synced_at":"2025-05-05T10:42:57.885Z","repository":{"id":258902455,"uuid":"875071101","full_name":"UdayLab/AEROS","owner":"UdayLab","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-21T02:48:10.000Z","size":25,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-10-21T05:59:46.870Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UdayLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-19T03:07:49.000Z","updated_at":"2024-10-21T02:48:13.000Z","dependencies_parsed_at":"2024-10-22T05:49:56.819Z","dependency_job_id":null,"html_url":"https://github.com/UdayLab/AEROS","commit_stats":null,"previous_names":["udaylab/aeros"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UdayLab%2FAEROS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UdayLab%2FAEROS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UdayLab%2FAEROS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UdayLab%2FAEROS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UdayLab","download_url":"https://codeload.github.com/UdayLab/AEROS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252484710,"owners_count":21755641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-05T10:42:57.206Z","updated_at":"2025-05-05T10:42:57.877Z","avatar_url":"https://github.com/UdayLab.png","language":"Python","readme":"# 1. Prerequisites\n\n1. Linux Operating System (E.g., Ubuntu)\n2. [PostGres and PostGIS](https://www.paulshapley.com/2022/12/install-postresql-14-and-postgis-3-on.html)\n3. Python 3.10 or above\n4. Psycopg2 and alive_progress Python Package\n    ```shell\n    pip install psycopg2-binary alive_progress\n    ```\n5. Google Maps API key\n6. [Data](https://1drv.ms/f/s!Ar09XhBKBP2MkvwTfy-AHGe3yEV1Ug?e=yPJW7g)\n\n# 2. Setting up the database\n\n1. Create a database in postgres to store the air pollution data.\n    ```sql\n    Create database soramame\n    ```\n        \n2. Connect to the database\n    ```sql\n    \\c soramame\n    ```\n3. Enable PostGIS extension to the database (admin privileges are needed)\n    ```sql\n    create extension postgis\n    ```\n4. Create a table to store the location information of the sensors. \n    ```sql\n    CREATE TABLE station_info(stationid varchar not null primary key, name varchar, address varchar, location geography(POINT,4326));\n    ``` \n5. Create a table to store the hourly observations of the sensors on a daily basis.\n    ```sql\n    create table hourly_observations(stationid varchar not null, obsDate timestamp, SO2 double precision, no double precision, no2 double precision, nox double precision, co double precision, ox double precision, nmhc double precision, ch4 double precision, thc double precision, spm double precision, pm25 double precision, sp double precision, wd varchar, ws double precision, temp double precision, hum double precision, constraint SOH unique (stationID,obsDate));\n    ```\n__Note:__ \n   - Do not establish primary key and foreign key relation for stationid attribute in station_info and hourly_observations tables\n   - Constraint SOH unique (stationID,obsDate) is used to prevent data repetition. A sensor cannot have multiple transactions with the same timestamp.\n\n6. __[Optional Steps:]__ We recommend it for the users who are not familar with the concepts of PostGres\n\n   - Create a user to work with this data\n        ```sql\n        CREATE USER aeros WITH PASSWORD 'aeros123';\n        ```\n   - Grant access for the new user to work on this data\n        ```sql\n        GRANT CONNECT ON DATABASE soramame TO aeros;\n        ```\n   - Grant permission to read and write data in the tables of the database\n        ```sql\n        GRANT SELECT, INSERT ON ALL TABLES IN SCHEMA public TO aeros;\n        ```\n9. Exit from the Postgres database.\n    ```console\n    control + D\n    ```\n# 3. Gathering the spatial location information of the stations\n\n1. Visit the sensors location page in [AEROS website](https://soramame.env.go.jp/station)\n2. Copy the first three columns (stationID, stationName, stationAddress) of all the stations and store them in an excel file.\n3. Carefully convert (or export) the Excel file into a csv file named 'rawStationsInfo.csv'.\n4. Execute the below Python program that reads the address of each sensors and derives the spatial location.\n```shell\npython3 address2LatLong.py rawStationsInfo.csv  finalStationsInfo.csv   \u003cgoogleMaps API key\u003e\n```\n       \n\n# 4. Storing the information of the stations in the database\n\n1. Open the Python file 'insertStationInfoInDatabase.py'\n    ```shell\n    vi insertStationInfoInDatabase.py\n    ```\n2. Go to the following line:\n    ```python \n    conn = psycopg2.connect(database=\"soramame\", user=\"\", password=\"\", host=\"\", port=5432)\n    ```\n3. Specify the appropriate database, user, password, host ipaddress of postgres, and port number.\n    ```python\n    conn = psycopg2.connect(database=\"soramame\", user=\"aeros\", password=\"aeros123\", host=\"163.143.165.136\", port=5432)\n    ```\n4. Save the file and exit.\n    ```console\n    :wq\n    ```\n5. Execute the following command to store the station information into the database.\n    ```shell\n    python3 insertStationInfoInDatabase.py finalStationsInfo.csv\n    ```\n# 5. Storing hourly observation data in the database\n\n## 5.1. Storing recent (or new) hourly data\n### 5.1.1. Download the hourly data\n1. Visit the download page of [AEROS website](https://soramame.env.go.jp/download)\n2. In the first dropdown menu choose any month other than current month. For example, if the current month is November, choose any previous months, such as August and September.\n3. In the second dropdown menu choose the default option, which is 'nationwide'.\n4. Click the download button.\n5. A zip file named 'data.zip' will be downloaded onto your local computer. \n6. Create a folder 'hourlyData'\n```shell\nmkdir hourlyData\n```\n7. Unzip the data.zip file by saving its contents in hourlyData folder.\n```shell\nunzip -d hourlyData ~/Downloads/data.zip\n```\n      \n8. __[Optional]__ If you have multiple zips representing various months, perform this two steps.\n   \n   - Enter into the hourlyData directory\n        ```shell\n        cd hourlyData\n        ```\n   - Create a shell script file using the following command\n     ```shell\n     vi uncompressNewZipFiles.sh\n     ```   \n   - Copy the below provided shell script code and paste it in the above file\n        ```shell\n        #add the below provided code\n        zipFiles=`ls *.zip` \n        for eachZipFile in $zipFiles\n        do\n           unzip $eachZipFile\n           rm $eachZipFile\n        done  \n        ```\n   - Execute the shell script\n     ```shell  \n     sh uncompressNewZipFiles.sh\n     ```\n   - remove the shell script file\n     ```shell\n     rm -rf uncompressNewZipFiles.sh\n     ```\n9. Delete or rename the zip file to yyyymm_00.zip for backup. \n    ```shell\n        #format: mv data.zip yyyymmdd_00.zip\n        mv data.zip 20240101_00.zip\n    ```\n10. Move back to the parent directory\n    ```shell\n    cd ..\n    ```\n### 5.1.2. Inserting the new hourly data into the database.\n1. Open the Python file 'insertNewHourlyObservationsData.py'\n    ```shell\n    vi insertNewHourlyObservationsData.py\n    ```\n2. Go to the following line:\n    ```python\n    conn = psycopg2.connect(database=\"\", user=\"\", password=\"\", host=\"\", port=5432)\n    ```\n3. Specify the appropriate database, user, password, host ipaddress of postgres, and port number.\n    ```python\n    conn = psycopg2.connect(database=\"soramame\", user=\"aeros\", password=\"aeros123\", host=\"163.143.165.136\", port=5432)\n    ```\n4. Save the file and exit.\n\n9. Run the Python program 'insertNewHourlyObservationsData.py' by specifying the folder.\n    ```shell\n    python3 insertNewHourlyObservationsData.py ./hourlyData\n    ```\n## 5.2. Storing old hourly data \nDuration of the data: 2018-01-01 to 2021-03-31\n\n### 5.2.1. Downloading the dld zip files [Truncated]\n1. Visit the download page of [AEROS website](https://soramame.env.go.jp/download)\n2. In the first dropdown menu choose any month other than current month. For example, if the current month is November, choose any previous months, such as August and September.\n3. In the second dropdown menu choose the default option, which is 'nationwide'.\n4. Click the download button.\n5. A zip file named 'data.zip' will be downloaded onto your local computer. \n\n### 5.2.2. Unzipping the downloaded zip files\n1. Create a temporary directory, say _temp_.\n    ```shell\n    mkdir temp\n    ```       \n2. Move or upload the zip files into the _temp_ directory.\n3. Enter into the temp directory\n    ```shell\n    cd temp\n    ```\n4. Create a shell script file to read every zip file and uncompress it.\n    ```shell\n    vi uncompressOldZipFiles.sh\n    ```\n5. Copy and paste the following shell script \n    ```shell       \n    #add the below provided code\n    zipFiles=`ls *.zip`\n    \n    for eachZipFile in $zipFiles\n    do\n      unzip $eachZipFile\n      rm $eachZipFile\n    done\n        \n    subZipFiles=`ls *.zip`\n    for eachZipfile in $subZipFiles\n    do\n          echo 'unzipping ' $eachZipfile\n          unzip $eachZipfile\n          rm -rf $eachZipfile\n    done\n    ```\n6. Execute the shell script.  \n    ```shell\n    sh uncompressOldZipFiles.sh\n    ```\n    The above program will create the folders '01' to '47'. Each folder represents a Prefecture in Japan.  \n\n7. Delete the shell script file\n    ```shell\n    rm -rf uncompressOldZipFiles.sh\n    ```\n8. Move back to the parent directory\n    ```shell\n    cd ..\n    ```\n### 5.2.3.  Inserting the old hourly data into the database.\n1. Open the Python file 'insertOldHourlyObservationsData.py'\n    ```shell\n    vi insertOldHourlyObservationsData.py\n    ```\n2. Go to the following line:\n    ```python\n    conn = psycopg2.connect(database=\"\", user=\"\", password=\"\", host=\"\", port=5432)\n    ```\n3. Specify the appropriate database, user, password, host ipaddress of postgres, and port number.\n    ```python\n    conn = psycopg2.connect(database=\"soramame\", user=\"aeros\", password=\"aeros123\", host=\"163.143.165.136\", port=5432)\n    ```\n4. Save the file and exit.\n\n9. Run the Python program 'insertNewHourlyObservationsData.py' by specifying the folder.\n    ```shell\n    python3 insertOldHourlyObservationsData.py ./temp\n    ```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fudaylab%2Faeros","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fudaylab%2Faeros","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fudaylab%2Faeros/lists"}