{"id":22485405,"url":"https://github.com/astrolabsoftware/fink-object-api","last_synced_at":"2026-01-19T08:05:10.081Z","repository":{"id":264605363,"uuid":"892642866","full_name":"astrolabsoftware/fink-object-api","owner":"astrolabsoftware","description":"Fink REST API to access object data stored in HBase","archived":false,"fork":false,"pushed_at":"2026-01-16T13:20:39.000Z","size":71235,"stargazers_count":1,"open_issues_count":14,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-01-16T23:57:11.194Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astrolabsoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-22T13:56:09.000Z","updated_at":"2026-01-16T10:17:27.000Z","dependencies_parsed_at":"2024-11-25T10:36:12.815Z","dependency_job_id":"01de0f61-1a9e-4406-8e35-5ff0364a795e","html_url":"https://github.com/astrolabsoftware/fink-object-api","commit_stats":null,"previous_names":["astrolabsoftware/fink-object-api"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/astrolabsoftware/fink-object-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-object-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-object-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-object-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-object-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astrolabsoftware","download_url":"https://codeload.github.com/astrolabsoftware/fink-object-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-object-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28563298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T03:31:16.861Z","status":"ssl_error","status_checked_at":"2026-01-19T03:31:15.069Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T17:12:32.509Z","updated_at":"2026-01-19T08:05:10.065Z","avatar_url":"https://github.com/astrolabsoftware.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fink object API\n\n[![Sentinel](https://github.com/astrolabsoftware/fink-object-api/workflows/Sentinel/badge.svg)](https://github.com/astrolabsoftware/fink-object-api/actions?query=workflow%3ASentinel)\n\n![structure](.github/API_fink.png)\n\nThe object API is a Flask application used to access object data stored in tables in Apache HBase. The application relies internally on two components, the Java Gateway and the Fink cutout API. \n\nThe Java Gateway enables the Flask application to communicate with a JVM using [py4j](https://www.py4j.org/), where the Fink HBase client based on [Lomikel](https://github.com/hrivnac/Lomikel) is available. This client simplifies the interaction with HBase tables, where Fink aggregated alert data is stored.\n\nThe Fink cutout API is a Flask application to access cutouts from the Fink datalake. We only store cutout metadata in HBase, and this API retrieves the data from the raw parquet files stored on HDFS.\n\n_From 2019 to 2024, the development of this API was done in [fink-science-portal](https://github.com/astrolabsoftware/fink-science-portal). Check this repository for older issues and PR._\n\n## Documentation\n\nThere are several forms of documentation, depending on what you are looking for: \n\n- Tutorials/How-to guides: [Fink user manual](https://fink-broker.readthedocs.io/en/latest/services/search/getting_started/#quick-start-api)\n- API Reference guide: [https://api.fink-portal.org](https://api.fink-portal.org)\n- Notes for developpers and maintainers (auth required): [GitLab](https://gitlab.in2p3.fr/fink/rubin-performance-check/-/blob/main/portal/README.md?ref_type=heads)\n\n## Requirements and installation\n\nYou will need Python installed (\u003e=3.9) with requirements listed in [requirements.txt](requirements.txt). You will also need [fink-cutout-api](https://github.com/astrolabsoftware/fink-cutout-api) fully installed (which implies Hadoop installed on the machine, and Java 11/17). For the full installation and deployment, refer as to the [procedure](install/README.md).\n\n## Deployment\n\nThe input parameters can be found in [config.yml](config.yml). Make sure that the `SCHEMAVER` is the same you use for your tables in HBase.\n\n### Debug\n\nAfter starting the Fink Java Gateway and [fink-cutout-api](https://github.com/astrolabsoftware/fink-cutout-api) services, you can simply launch the API in debug mode using:\n\n```bash\npython app.py\n```\n\n### Production\n\nThe application is simply managed by `gunicorn` and `systemd` (see [install](install/README.md)), and you can manage it using:\n\n```bash\n# start the application\nsystemctl start fink_object_api\n\n# reload the application if code changed\nsystemctl restart fink_object_api\n\n# stop the application\nsystemctl stop fink_object_api\n```\n\n## Tests\n\nAll the routes are extensively tested. To trigger a test on a route, simply run:\n\n```bash\npython apps/routes/objects/test.py $HOST:$PORT\n```\n\nBy replacing `HOST` and `$PORT` with their values (could be the main API instance). If the program exits with no error or message, the test has been successful. Alternatively, you can launch all tests using:\n\n\n```bash\n./run_tests.sh --url $HOST:$PORT\n```\n\n## Profiling a route\n\nTo profile a route, simply use:\n\n```bash\nexport PYTHONPATH=$PYTHONPATH:$PWD\n./profile_route.sh --route apps/routes/v1/\u003croute\u003e\n```\n\nDepending on the route, you will see the details of the timings and a summary similar to:\n\n```python\nWrote profile results to profiling.py.lprof\nInspect results with:\npython -m line_profiler -rmt \"profiling.py.lprof\"\nTimer unit: 1e-06 s\n\nTotal time: 0.000241599 s\nFile: /home/peloton/codes/fink-object-api/apps/routes/v1/template/utils.py\nFunction: my_function at line 19\n\nLine #      Hits         Time  Per Hit   % Time  Line Contents\n==============================================================\n    19                                           @profile                                             \n    20                                           def my_function(payload):                            \n    21         1        241.6    241.6    100.0      return pd.DataFrame({payload[\"arg1\"]: [1, 2, 3]})\n\n\n  0.00 seconds - /home/peloton/codes/fink-object-api/apps/routes/v1/template/utils.py:19 - my_function\n```\n\n### Main route performance\n\nThe main route performance for a medium size object (14 alerts, about 130 columns):\n\n| request| time (second)|\n|--------|--------------|\n| Lightcurve data (3 cols) | 0.1 |\n| Lightcurve data (130 cols) | 0.3 |\n| Lightcurve \u0026 1 cutout data | 3.4 |\n| Lightcurve \u0026 3 cutout data | 5.4 |\n\nRequesting cutouts is costly! We have 14 alerts, which is about 0.25 second per cutout. Note that requesting 3 cutouts is faster then 3 times 1 cutout, as what drives the cost is to load the full block in HDFS in memory (see this [discussion](https://github.com/astrolabsoftware/fink-broker/issues/921) about the strategy behind). \n\nNote that for lightcurve data, the time is fortunately not linear with the number of alerts per object:\n\n| request| time (second)|\n|--------|--------------|\n| Lightcurve data (33 alerts, 130 cols) | 0.3 |\n| Lightcurve data (1575 alerts, 130 cols) | 1.8|\n\n\n### The power of the Gateway\n\nInitially, we loaded the client JAR using jpype at the application's start, sharing the client among all users. This approach caused several issues due to the client's lack of thread safety. To resolve this, we switched to an isolation mode, where a new client is created for each query instead of reusing a global client (see astrolabsoftware/fink-science-portal#516).\n\nWhile this strategy effectively prevents conflicts between sessions, it significantly slows down individual queries. For instance, when using the route `api/v1/objects`, the overall query time is primarily determined by the time taken to load the client.\n\nInstead of loading the client from scratch in the Python application for each query, we now spawn a JVM once (from outside the Python application), and access Java objects dynamically from the Python application using py4j. This has led to huge speed-up for most queries without the need for cutouts, e.g. for the `/api/v1/objects` route:\n\n| | time (second)|\n|--------|--------------|\n| Isolation mode | 3.4 |\n| Gateway | 0.3 | \n\n## Adding a new route\n\nYou find a [template](apps/routes/template) route to start a new route. Just copy this folder, and modify it with your new route. Alternatively, you can see how other routes are structured to get inspiration. Do not forget to add tests in the [test folder](tests/)!\n\n## Todo\n\n- [ ] configuration: Find a way to automatically sync schema with tables.\n- [ ] Make tests more verbose, even is successful.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-object-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastrolabsoftware%2Ffink-object-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-object-api/lists"}