{"id":19746348,"url":"https://github.com/astrolabsoftware/fink-alert-schemas","last_synced_at":"2026-03-05T13:03:43.200Z","repository":{"id":92459550,"uuid":"569309304","full_name":"astrolabsoftware/fink-alert-schemas","owner":"astrolabsoftware","description":"Schemas for ztf/rubin alerts","archived":false,"fork":false,"pushed_at":"2024-09-17T08:24:26.000Z","size":238,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"4-add-support-for-kafka","last_synced_at":"2025-01-10T20:49:14.009Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astrolabsoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-22T14:37:57.000Z","updated_at":"2022-11-22T14:38:51.000Z","dependencies_parsed_at":"2023-06-02T12:45:39.181Z","dependency_job_id":null,"html_url":"https://github.com/astrolabsoftware/fink-alert-schemas","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-alert-schemas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-alert-schemas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-alert-schemas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-alert-schemas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astrolabsoftware","download_url":"https://codeload.github.com/astrolabsoftware/fink-alert-schemas/tar.gz/refs/heads/4-add-support-for-kafka","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241077233,"owners_count":19905726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T02:14:20.905Z","updated_at":"2026-03-05T13:03:43.150Z","avatar_url":"https://github.com/astrolabsoftware.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Avro Alert Schema\n\nApache Avro uses a schema to structure the data (define data types and protocols) that is being encoded (compact binary format). It has two different types of schema languages: one for human editing (Avro IDL) and another which is more machine-readable based on (JSON).\n\n\n## ZTF alert schema timeline\n\nThe ZTF alert schema evolves over time, to include the latest information. We provide a sample alert for each version (starting in 2019) in the `ztf` folder: `template_schema_ZTF_\u003cversion\u003e.avro`. One can find the latest description of fields on this [page](https://zwickytransientfacility.github.io/ztf-avro-alert/).\n\n| Period | Schema version | Change |\n|:--------|:-------|:-------|\n| Jul 2018 - Jan 2019 | version 3.1 | -- |\n| Jan 2019 - Jul 2019 | version 3.2 | [commit](https://github.com/ZwickyTransientFacility/ztf-avro-alert/commit/2b4af549fc99200e3117c24634a17b5ac04ed963) |\n| Jul 2019 - now | version 3.3 | [commit](https://github.com/ZwickyTransientFacility/ztf-avro-alert/commit/a4fa6a45621ccfc11e7a38f766a05c63681fd4e3#diff-c9550d5fad73447fc24ba47f95d1c6b7) |\n\n## Fink Distribution schema [ZTF]\n\nThe Fink distribution schema is the input ZTF alert schema plus additional fields describing the added values by Fink. This schema is mandatory to decode the alerts receive by the Fink client, and we release schema versions in the `ztf` folder: `distribution_schema_\u003cversion\u003e.avsc`. The fields are described on this [page](https://fink-broker.readthedocs.io/en/latest/science/added_values/).\n\n| Period | Schema version | Commit | Added values |\n|:--------|:-------|:-------|:-------|\n| Jul 2019 - Jan 2020 | version 0.1 | -- | `cdsxmatch` |\n| Jan 2020 - now | version 0.2 | [commit](https://github.com/astrolabsoftware/fink-broker/commit/bc5a03ae42513841c8c071a49f17bae1978e0e94) | `rfscore` |\n| Aug 2020 - now | version 0.3 | [commit](https://github.com/astrolabsoftware/fink-broker/commit/6385288158eeafe74a4a1d69b8ee857dc95f4c71) | `snn_snia_vs_nonia`, `snn_sn_vs_all`, `mulens.*`, `roid`, `nalerthist`, `fink_broker_version`, `fink_science_version` |\n\n## Simulator vs live streams\n\nWe run Fink in two modes: live (i.e. live data from ZTF), or simulation (replayed\nstreams using the Fink alert simulator). While we would expect the schema to be the same,\nthere are some small variations due to the way Spark handles input schema. The variations are as small as some missing comments in the schema - but nevertheless sufficient for the client to consider the data from live or simulation to need a different schema... Hence, for each schema version, we have two files:\n\n```\nschema_name_XpY.avsc --\u003e suitable for data from simulator\nschema_name_XpY-live.avsc --\u003e suitable for live data\n```\n\nWe intend to find a solution to merge the two in the future.\n\n## Troubleshooting\n\nUsing a wrong schema to decode alerts will lead to failures. Typically broker users will see this message:\n\n```bash\nCaused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.ArrayIndexOutOfBoundsException: -28\n\n\tat org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:424)\n\n\tat org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)\n\n\tat org.apache.avro.io.parsing.Parser.advance(Parser.java:88)\n\n\tat org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)\n\n\tat org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)\n\t...\n```\n\nand fink-client users could see:\n\n```bash\nIn [3]: topic, alert = consumer.poll(timeout=1)\n---------------------------------------------------------------------------\nUnicodeDecodeError                        Traceback (most recent call last)\n\u003cipython-input-4-d5d6e0affb36\u003e in \u003cmodule\u003e\n----\u003e 1 topic, alert = consumer.poll(timeout=ts)\n\n~/Documents/workspace/myrepos/fink-client/fink_client/consumer.py in poll(self, timeout)\n     92         topic = msg.topic()\n     93         avro_alert = io.BytesIO(msg.value())\n---\u003e 94         alert = _decode_avro_alert(avro_alert, self._parsed_schema)\n     95\n     96         return topic, alert\n\n~/Documents/workspace/myrepos/fink-client/fink_client/consumer.py in _decode_avro_alert(avro_alert, schema)\n    234     \"\"\"\n    235     avro_alert.seek(0)\n--\u003e 236     return fastavro.schemaless_reader(avro_alert, schema)\n\nfastavro/_read.pyx in fastavro._read.schemaless_reader()\n\nfastavro/_read.pyx in fastavro._read.schemaless_reader()\n\nfastavro/_read.pyx in fastavro._read._read_data()\n\nfastavro/_read.pyx in fastavro._read.read_record()\n\nfastavro/_read.pyx in fastavro._read._read_data()\n\nfastavro/_read.pyx in fastavro._read.read_union()\n\nfastavro/_read.pyx in fastavro._read._read_data()\n\nfastavro/_read.pyx in fastavro._read.read_utf8()\n\nfastavro/_six.pyx in fastavro._six.py3_btou()\n\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 13: invalid continuation byte\n```\n\nCheck you are using the relevant schema version according to the data you are processing. In fink-broker, you can always output the schema of an alert before it is sent by the broker using [get_kafka](https://github.com/astrolabsoftware/fink-broker/blob/master/fink_broker/distributionUtils.py#L29) and compare then to the schema you were using.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-alert-schemas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastrolabsoftware%2Ffink-alert-schemas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-alert-schemas/lists"}