{"id":15904703,"url":"https://github.com/dmitryduev/tails","last_synced_at":"2025-09-11T22:31:23.904Z","repository":{"id":46588275,"uuid":"322133863","full_name":"dmitryduev/tails","owner":"dmitryduev","description":"Tails: chasing comets with ZTF and deep learning","archived":false,"fork":false,"pushed_at":"2022-09-11T21:12:41.000Z","size":61,"stargazers_count":6,"open_issues_count":7,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-25T01:02:38.550Z","etag":null,"topics":["astronomy","comets","deep-learning","solar-system","tails","ztf","ztf-ii"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmitryduev.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-17T00:12:44.000Z","updated_at":"2025-04-23T00:12:52.000Z","dependencies_parsed_at":"2023-01-18T04:16:08.141Z","dependency_job_id":null,"html_url":"https://github.com/dmitryduev/tails","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dmitryduev/tails","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitryduev%2Ftails","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitryduev%2Ftails/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitryduev%2Ftails/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitryduev%2Ftails/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmitryduev","download_url":"https://codeload.github.com/dmitryduev/tails/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitryduev%2Ftails/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274717358,"owners_count":25336875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astronomy","comets","deep-learning","solar-system","tails","ztf","ztf-ii"],"created_at":"2024-10-06T12:41:25.081Z","updated_at":"2025-09-11T22:31:23.526Z","avatar_url":"https://github.com/dmitryduev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tails: chasing comets with the Zwicky Transient Facility and Deep Learning\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4563226.svg)](https://doi.org/10.5281/zenodo.4563226)\n[![arXiv](https://img.shields.io/badge/arXiv-2102.13352-brightgreen)](https://arxiv.org/abs/2102.13352)\n\nTails is a deep-learning framework for the identification and localization of comets in the image data\nof the [Zwicky Transient Facility (ZTF)](https://ztf.caltech.edu), a robotic optical sky survey\nat the [Palomar Observatory](https://sites.astro.caltech.edu/palomar/about/) in California, USA.\n\nTails uses a custom EfficientDet-based architecture and is thus capable of finding comets in single images\nin near real time, rather than requiring multiple epochs as with traditional methods.\nIn production, we have observed 99% recall, \u003c0.01% false positive rate,\nand 1-2 pixel root mean square error in the predicted position.\n\nTails enabled the **first AI-assisted discovery of a comet** -\n[C/2020 T2](https://minorplanetcenter.net/mpec/K20/K20UH0.html).\n\n### Comet discoveries by Tails\n- [C/2020 T2](https://minorplanetcenter.net/mpec/K20/K20UH0.html)\n- [C/2022 P2](http://www.cbat.eps.harvard.edu/iau/cbet/005100/CBET005165.txt)\n- [C/2022 P3](http://www.cbat.eps.harvard.edu/iau/cbet/005100/CBET005169.txt)\n\n## Install and run Tails\n\nClone the repository:\n\n```bash\ngit clone https://github.com/dmitryduev/tails.git \u0026\u0026 cd tails\n```\n\nBefore proceeding, you may want to create/activate a virtual environment, for example:\n\n```bash\npython -m venv tails-env\nsource tails-env/bin/activate\n```\n\nInstall [`swarp`](https://www.astromatic.net/software/swarp). For example, with `conda`:\n\n```bash\nconda install -c conda-forge astromatic-swarp\n```\n\nInstall `Tails`:\n\n```bash\npython setup.py install\n```\n\nFetch pre-trained models:\n\n```bash\nmkdir models\ngsutil -m cp -r -n gs://tails-models/* models/\n```\n\nRun Tails on a (publicly accessible) ZTF observation of 2I/Borisov:\n\n```bash\ncd scripts\npython run_tails.py \\\n  --config=../config.defaults.yaml \\\n  --checkpoint=../models/tails-20210107/tails \\\n  --score_threshold=0.5 \\\n  --cleanup=none \\\n  --single_image=ztf_20191014495961_000570_zr_c05_o_q3\n```\n\nCheck out the `runs/20191014` directory for the output:\n- A csv file with the detection metadata\n- A 256x256 pix cutout image triplet (epochal, reference, and difference) containing the detection: an `.npy` file and a `.png` visualization:\n![ztf_20191014495961_000570_zr_c05_o_q3_0](https://user-images.githubusercontent.com/7557205/105624917-34802880-5dda-11eb-959e-8386142ac4a4.png)\n\n## ZTF sentinel service\n\nA containerized service that monitors\n[Kowalski](https://kowalski.caltech.edu)/[IRSA](https://irsa.ipac.caltech.edu/) for new ZTF data,\nexecutes Tails on them, and optionally posts the identified candidates to [Fritz](https://fritz.science).\n\nRequires Kowalski and IRSA accounts, see the [config file](config.defaults.yaml).\nAdditionally, a Fritz account is required if `sentinel.app.post_to_fritz` is set to `true`.\n\nFetch models from GCP:\n\n```bash\n./sentinel.py fetch-models\n```\n\nSpin up:\n\n```bash\n./sentinel.py up\n```\n\nShut down:\n\n```bash\n./sentinel.py down\n```\n\n---\n\nThe Sentinel:\n- Managed with the `sentinel.py` utility.\n- Implemented as a containerized service, orchestrated with `docker-compose`.\n- Monitors the `ZTF_ops` collection on `Kowalski` for new ZTF data (Twilight only by default).\n- Uses `dask.distributed` to process individual ZTF image frames (ccd-quads).\nEach worker is initialized with a `TailsWorker` instance that maintains a `Fritz` connection and preloads Tails.\nThe candidate comet detections, if any, are posted to `Fritz` together with auto-annotations\n(cross-matches from the MPC and SkyBot) and auxiliary data.\n  - By default, spins up a `dask.distributed.LocalCluster`, whose parameters are defined in `config.yaml`,\n  and a sentinel process, both managed with `supervisor` inside the `tails_sentinel_1` Docker container.\n  To use an external `dask.distributed` cluster, the user needs to simply provide its host and the scheduler port\n  and remove the `dask-cluster` `supervisor` program in the config.\n\n\nOn Fritz, the posted results look like the following:\n\n\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003eCandidates page\u003c/th\u003e\n\u003cth\u003eSource page\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003cimg width=\"1605\" alt=\"fritz-candidates\" src=\"https://user-images.githubusercontent.com/7557205/104243874-2283aa80-5416-11eb-98e1-8f5cc5224d9e.png\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg width=\"1606\" alt=\"fritz-source\" src=\"https://user-images.githubusercontent.com/7557205/104243884-29122200-5416-11eb-9c76-3f727f22683b.png\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitryduev%2Ftails","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmitryduev%2Ftails","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitryduev%2Ftails/lists"}