{"id":20065695,"url":"https://github.com/collaborative-ai/earthquake-forecasting","last_synced_at":"2025-03-02T10:43:57.461Z","repository":{"id":166195722,"uuid":"330204021","full_name":"Collaborative-AI/Earthquake-Forecasting","owner":"Collaborative-AI","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-03T14:54:11.000Z","size":305377,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-12T23:09:14.992Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Collaborative-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-16T16:20:39.000Z","updated_at":"2025-01-03T14:54:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"804c388e-3f88-499f-bb4d-3de6acd8f342","html_url":"https://github.com/Collaborative-AI/Earthquake-Forecasting","commit_stats":null,"previous_names":["collaborative-ai/earthquake-forecasting"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2FEarthquake-Forecasting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2FEarthquake-Forecasting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2FEarthquake-Forecasting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Collaborative-AI%2FEarthquake-Forecasting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Collaborative-AI","download_url":"https://codeload.github.com/Collaborative-AI/Earthquake-Forecasting/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241494142,"owners_count":19971870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T13:52:03.271Z","updated_at":"2025-03-02T10:43:57.443Z","avatar_url":"https://github.com/Collaborative-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SmartQuake\n\nSmartQuake is a research project aimed at predicting earthquakes on a global scale using the latest machine learning technologies. It compiles data from 14 global datasets, creating a robust dataset that can be leveraged for future earthquake prediction research.\n\n![Alt Text](smartquake_pipeline.png)\n**SmartQuake Data Pipeline**\n\n1. Visit the data_scraping/ folder\n2. Visit the data_processing/ folder\n3. Visit the data_merging/ folder\n\n---\n\n## Dataset Checkpoints\n\nAccess the dataset at various stages of acquisition:\n\n- [data_scraping](https://drive.google.com/drive/folders/1okZ_2QW58CqQwPA8JIDIwmaBbcOtLIMp?usp=sharing)\n- [data_processing](https://drive.google.com/drive/folders/1CnXaP9KgUxgQrrreYt3s1MSbCJKHmCQy?usp=sharing)\n- [data_merging](https://drive.google.com/drive/folders/1GUvjtBC2jBqHQGbAVy4rqSAd9fXe3sxT?usp=sharing)\n\n---\n\n## Table of Contents\n\n1. [Data Scraping](#data-scraping)\n2. [Data Processing](#data-processing)\n3. [Data Merging](#data-merging)\n4. [Data Source](#data-source)\n\n---\n\n# Data Scraping\n\n### Overview\n\nThis step involves scraping earthquake data from various sources, including text files, web pages, and PDFs. It utilizes **BeautifulSoup** for web scraping, **Pandas** for data manipulation, and **Tabula** for PDF data extraction.\n\n### Installation\n\n1. From Google Drive, download the raw datasets and place them under the `dataset/data_scraping/.../raw` folder.\n2. Install the required dependencies:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Run the scraping script:\n\n   ```bash\n   python dataset/main.py\n   ```\n\n   The scraped datasets will be saved under `dataset/data_scraping/.../clean`.\n\n### Usage\n\n1. **Initialization**: Create an instance of the `Scraper` class with parameters:\n   - `input_path`: Path to the input file (for text and PDF sources).\n   - `output_path`: Path to save the output CSV.\n   - `url`: Webpage URL to scrape (for web sources).\n   - `start_time` and `end_time`: Date range for filtering data.\n   - `header`: Column names for the output CSV.\n   - `separator`: Character for separating data in text files (default is space).\n\n2. **Scraping**:\n   - `find_quakes_txt(num_skips=0)`: For text files. `num_skips` skips initial lines.\n   - `find_quakes_web()`: For web pages. Scrapes data based on the body tag and predefined header.\n\n3. **Example**:\n\n   ```python\n   scraper = Scraper(input_path='input.txt', output_path='output.csv', url='http://example.com', header=['Date', 'Magnitude', 'Location'])\n   scraper.find_quakes_txt(num_skips=1)\n   scraper.find_quakes_web()\n   ```\n\n---\n\n# Data Processing\n\n### Overview\n\nData processing is the second step in the SmartQuake data pipeline. After scraping the datasets, they are compiled into a standardized format where all datasets share the same columns.\n\n### Data Standardization\n\nProcessed earthquake CSV files will contain the following columns:\n\n1. **Timestamp**: Stored as a `pd.Timestamp` string in UTC (format: YYYY-MM-DD HH:MM:SS.millisecond+00:00).\n2. **Magnitude**: Moment magnitude (Mw).\n3. **Latitude**: Range within [-90, 90].\n4. **Longitude**: Range within [-180, 180].\n5. **Depth**: Depth in kilometers (optional, may be `None` for older records).\n\nAll datasets are sorted chronologically and contain no duplicates.\n\n### File Organization\n\n- **data_processor.py**: Contains the `DataProcessor` class for standardized processing.\n- **run_processor.py**: Runs the `DataProcessor` on all scraped datasets.\n- **processed/**: Folder containing the processed output datasets (CSVs).\n\n### Running Data Processing\n\n1. Ensure that all `clean` datasets exist in the `dataset/data_scraping/.../clean` folder.\n2. Verify that the `processed/` folder exists in `data_processing/`.\n3. Run `run_processor.py`:\n\n   ```bash\n   python data_processing/run_processor.py\n   ```\n\n4. After completion, check for the processed CSVs in the `processed/` folder before proceeding to the merging step.\n\n---\n\n# Data Merging\n\nThe merging process combines all processed datasets into a single file for machine learning model input. This step preserves the same columns and ensures chronological order without duplicates.\n\n### File Organization\n\n- **helper.py**: Contains helper functions for merging.\n- **merge.py**: Merges non-USGS/SAGE datasets into `Various-Catalogs.csv`.\n- **usgs_pre_1950/**: Folder containing scripts for USGS data processing and merging.\n- **final/**: Folder containing `usgs_sage_various_merge.py`, which merges all datasets into `Completed-Merge.csv`.\n\n### Running Merge\n\n1. **Compile Processed Datasets**: Ensure all processed datasets are in `data_processing/processed/` (excluding USGS/SAGE datasets).\n2. **First Merge**: Run `merge.py` to create `Various-Catalogs.csv` and move it to the folder `data_merging/final`.\n3. **USGS Data Processing**:\n   - Visit the Google Drive and directly download [`USGS_SAGE_Merged.csv`](https://drive.google.com/file/d/1vZxxrXIYR7K7YWcuJUe4HGYJH8vDCTpX/view?usp=drive_link). Store the file in `data_merging/final/` for the next step.\n4. **Final Merge**: Run `usgs_sage_various_merge.py` to merge all datasets into `Completed-Merge.csv`.\n\n---\n\n# Data Source\n\n| Dataset       | Status      | Link                                                                         | Additional Comments                                |\n|---------------|-------------|------------------------------------------------------------------------------|---------------------------------------------------|\n| Argentina     | good        | [Link](https://doi.org/10.31905/YTIR1IED)                                    | Downloaded Manually |\n| Canada        | good        | [Link](https://earthquakescanada.nrcan.gc.ca/stndon/NEDB-BNDS/bulletin-en.php) | Downloaded Manually |\n| Japan         | good        | [Link](http://www-solid.eps.s.u-tokyo.ac.jp/~idehara/wtd0/Welcome.html)       | Downloaded Manually |\n| GHEA          | good        | [Link](http://evrrss.eri.u-tokyo.ac.jp/db/ghec/index.html)                   | Downloaded Manually |\n| NOAA          | good        | [Link](https://www.ngdc.noaa.gov/hazel/view/hazards/earthquake/search)       | Downloaded Manually |\n| SoCal         | good        | [Link](https://service.scedc.caltech.edu/ftp/catalogs/SCEC_DC/)              | Downloaded Manually |\n| Turkey        | good        | [Link](https://www.kaggle.com/datasets/atasaygin/turkey-earthquakes-19152021) | Downloaded Manually |\n| World Tremor  | good        | [Link](http://www-solid.eps.s.u-tokyo.ac.jp/~idehara/wtd0/Welcome.html)      | Downloaded Manually |\n| East Africa   | good        | [Link](https://www.isc.ac.uk/dataset_repository/view_submission.php?dsid=47) | Downloaded Manually |\n| Intensity     | good        | [Link](https://ngdc.noaa.gov/hazard/eq-intensity.shtml)                      | Downloaded Manually |\n| PNW Tremor    | good        | [Link](https://www.pnsn.org/tremor/)                                         | Downloaded Manually |\n| South Asia    | good        | [Link](https://link.springer.com/article/10.1007/s11069-016-2665-6#Sec11)    | Downloaded Manually |\n| Texas         | good        | [Link](https://catalog.texnet.beg.utexas.edu/)                               | Downloaded Manually |\n| USGS          | good        | [Link](https://earthquake.usgs.gov/fdsnws/event/1/)                          | Downloaded through python scraper, takes a lot of time to finish            |\n| SAGE          | deprecated  | [Link](http://service.iris.edu/fdsnws/event/docs/1/builder/)                              | Advised to use USGS according to the official webpage |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollaborative-ai%2Fearthquake-forecasting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollaborative-ai%2Fearthquake-forecasting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollaborative-ai%2Fearthquake-forecasting/lists"}