{"id":30861099,"url":"https://github.com/gperdrizet/ariel-data-challenge","last_synced_at":"2026-05-07T10:33:06.435Z","repository":{"id":312897391,"uuid":"1049204280","full_name":"gperdrizet/ariel-data-challenge","owner":"gperdrizet","description":"Kaggle competition: NeurIPS - Ariel Data Challenge 2025","archived":false,"fork":false,"pushed_at":"2025-10-25T22:55:29.000Z","size":65583,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-26T00:23:13.849Z","etag":null,"topics":["astronomy","data-science","exoplanets","kaggle","machine-learning","neurips-2025","python","spectroscopy"],"latest_commit_sha":null,"homepage":"https://gperdrizet.github.io/ariel-data-challenge/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gperdrizet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T16:27:14.000Z","updated_at":"2025-09-24T18:13:44.000Z","dependencies_parsed_at":"2025-09-02T18:25:41.517Z","dependency_job_id":"caebbcc6-1719-4d9f-807d-f295a5482851","html_url":"https://github.com/gperdrizet/ariel-data-challenge","commit_stats":null,"previous_names":["gperdrizet/ariel-data-challenge"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/gperdrizet/ariel-data-challenge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gperdrizet%2Fariel-data-challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gperdrizet%2Fariel-data-challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gperdrizet%2Fariel-data-challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gperdrizet%2Fariel-data-challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gperdrizet","download_url":"https://codeload.github.com/gperdrizet/ariel-data-challenge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gperdrizet%2Fariel-data-challenge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32733513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astronomy","data-science","exoplanets","kaggle","machine-learning","neurips-2025","python","spectroscopy"],"created_at":"2025-09-07T16:48:03.081Z","updated_at":"2026-05-07T10:33:06.430Z","avatar_url":"https://github.com/gperdrizet.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ariel Data Challenge\n\n[![Unittest](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/unittest.yml/badge.svg)](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/unittest.yml)\n[![PyPI release](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/pypi_release.yml/badge.svg)](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/pypi_release.yml)\n[![pages-build-deployment](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/gperdrizet/ariel-data-challenge/actions/workflows/pages/pages-build-deployment)\n\nThis repository contains my Ariel Data Challenge submission for NeurIPs 2025. The project has two main components. My Kaggle submission notebook, and an open source PyPI package for pre-processing the Ariel data. My project submission is mostly Jupyter notebooks and associated helper functions. See the project progress blog linked below and the `notebooks/` directory. The `ariel-data-preprocessing` package is my data pre-processing pipeline, refactored from notebooks and published to PyPI via GitHub workflows. You can find the (minimal) documentation on the PyPI project page linked below. It is pip installable and can be used independently of this repository.\n\n- [Project progress blog](https://gperdrizet.github.io/ariel-data-challenge/)\n- [Signal preprocessing package](https://pypi.org/project/ariel-data-preprocessing/)\n- [Kaggle competition page: Ariel Data Challenge 2025](https://www.kaggle.com/competitions/ariel-data-challenge-2025/overview)\n\n\n## 1. Setup\n\nAssumes the following base system configuration:\n\n- Python: 3.8.10\n- GPU: Tesla K80\n- Nvidia driver: 470.42.01\n- CUDA driver: 11.4\n- CUDA runtime: 11.4\n- CUDA compute capability: 3.7\n- cuDNN 8.1\n- GCC 9.4.0\n\nThe Python version is stuck at 3.8 in order to keep the old Tesla GPUs in my homelab data science/ML box running. If you have more modern hardware, feel free to update accordingly. If this is you, I'll assume you have a Nvidia driver, CUDA, cuDNN etc., already set up. In that case, just remove the version pins from all package installs and let pip do it's thing.\n\nAgain, if you are only here for the `ariel-data-preprocessing` package, you don't need to do any set-up just `pip install ariel-data-preprocessing`.\n\n\n### 1.1. Virtual environment\n\nCreate a Python 3.8 virtual environment:\n\n```bash\npython3.8 -m venv .venv\n```\n\n\n### 1.2. TensorFlow\n\nSet `LD_LIBRARY_PATH` from `.venv/bin/activate`:\n\n```bash\nexport LD_LIBRARY_PATH=/path/to/project/directory/.venv/lib/\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/project/directory/.venv/lib/python3.8/site-packages/tensorrt/\n```\n\nActivate the virtual environment, install TensorRT and TensorFlow:\n\n```bash\nsource .venv/bin/activate\npip install --upgrade pip\npip install nvidia-tensorrt==7.2.3.4\npip install tensorflow==2.11.0\n```\n\nTest TensorFlow with:\n\n```bash\npython -c \"import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))\"\n```\n\nYou should see something like:\n\n```bash\n[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]\n```\n\n\n### 1.3. LightGBM\n\nInstall LightGBM with CUDA support. For the `--config-settings` flag to work pip must be \u003e= 23.1.\n\n```bash\npip install lightgbm --no-binary lightgbm --config-settings=cmake.define.USE_CUDA=ON\n```\n\n\n### 1.4. Other requirements\n\n```bash\npip install -r requirements.txt\n```\n\n\n### 1.5. Optuna RDB storage\n\nOptuna needs an SQL database to store run information. Create a PostgreSQL database called `calories`:\n\n```bash\n$ sudo -u postgres_admin createdb \u003cPROJECT_NAME\u003e\n$ sudo -u postgres_admin psql \u003cPROJECT_NAME\u003e\n\npsql (17.2 (Ubuntu 17.2-1.pgdg20.04+1), server 16.6 (Ubuntu 16.6-1.pgdg20.04+1))\nType \"help\" for help.\n\ncalories=# ALTER USER postgres_user with encrypted password 'your_password';\nALTER ROLE\n\npostgres=# exit\n```\n\nModify `pg_hba.conf` to allow the machine running Optuna to access the database over the LAN and restart the database. Then, set environment variable for the following, to be read with `os.environ()` in `configuration.py`.\n\n1. `POSTGRES_USER`\n2. `POSTGRES_PASSWD`\n3. `POSTGRES_HOST`\n4. `POSTGRES_PORT`\n5. `STUDY_NAME`\n\nOnce run data is present, you can start the Optuna dashboard with:\n\n```bash\ngunicorn -b YOUR_LISTEN_IP --workers 2 functions.optuna_dashboard:application\n```\n\n\n## 2. Data acquisition\n\nNote: the Kaggle API cannot be used to download this dataset unless you have \u003e265 GB system memory. When calling `competition_download_files()` the python library appears to try and read the whole archive into memory before writing anything to disk. Unfortunately, I only have 128 GB system memory.\n\nGet the data the old fashioned way - manually download the archive by clicking 'Download all' link on the competition [data page](https://www.kaggle.com/competitions/ariel-data-challenge-2025/data). Then decompress with:\n\n```bash\nunzip ariel-data-challenge-2025.zip\n```\n\nBoth the zip archive and the extracted data are 247 GB on disk.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgperdrizet%2Fariel-data-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgperdrizet%2Fariel-data-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgperdrizet%2Fariel-data-challenge/lists"}