{"id":50902415,"url":"https://github.com/notamitgamer/osma","last_synced_at":"2026-06-16T04:01:58.285Z","repository":{"id":354442547,"uuid":"1223664322","full_name":"notamitgamer/osma","owner":"notamitgamer","description":"Registry lookups, without the wait. OSMA is a lightning-fast, static search archive for querying open-source packages across NPM and PyPI, bypassing live registry bottlenecks.","archived":false,"fork":false,"pushed_at":"2026-06-07T11:44:28.000Z","size":12690,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-07T13:19:04.273Z","etag":null,"topics":["database","npm","pypi"],"latest_commit_sha":null,"homepage":"https://data.amit.is-a.dev","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/notamitgamer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["notamitgamer"]}},"created_at":"2026-04-28T14:35:38.000Z","updated_at":"2026-06-07T11:44:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/notamitgamer/osma","commit_stats":null,"previous_names":["notamitgamer/osma"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/notamitgamer/osma","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notamitgamer%2Fosma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notamitgamer%2Fosma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notamitgamer%2Fosma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notamitgamer%2Fosma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/notamitgamer","download_url":"https://codeload.github.com/notamitgamer/osma/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/notamitgamer%2Fosma/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34390052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","npm","pypi"],"created_at":"2026-06-16T04:01:53.639Z","updated_at":"2026-06-16T04:01:58.279Z","avatar_url":"https://github.com/notamitgamer.png","language":"Python","funding_links":["https://github.com/sponsors/notamitgamer"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\r\n  \u003cimg src=\"https://github.com/notamitgamer/osma/blob/main/docs/assets/osma.png\" alt=\"OSMA Icon\" width=\"120\" /\u003e\r\n\r\n  # OSMA (Open Source Module Archive)\r\n\r\n  \u003cp\u003e\r\n    \u003cimg src=\"https://img.shields.io/badge/NPM-3%2C881%2C465_Packages-cb3837?style=for-the-badge\u0026logo=npm\" alt=\"NPM Packages\" /\u003e\r\n    \u003cimg src=\"https://img.shields.io/badge/PyPI-793%2C648_Packages-3776ab?style=for-the-badge\u0026logo=python\u0026logoColor=white\" alt=\"PyPI Packages\" /\u003e\r\n    \u003cimg src=\"https://img.shields.io/badge/License-Apache_2.0-blue?style=for-the-badge\" alt=\"License\" /\u003e\r\n  \u003c/p\u003e\r\n\u003c/div\u003e\r\n\r\n**Registry lookups, without the wait.** OSMA is a lightning-fast, static search archive for querying open-source packages across NPM and PyPI, bypassing live registry bottlenecks.\r\n\r\n### Live Explorers \u0026 Docs\r\n* **Home:** [data.amit.is-a.dev](https://data.amit.is-a.dev)\r\n* **NPM Explorer:** [data.amit.is-a.dev/npm](https://data.amit.is-a.dev/npm)\r\n* **PyPI Explorer:** [data.amit.is-a.dev/pypi](https://data.amit.is-a.dev/pypi)\r\n* **API Documentation:** [data.amit.is-a.dev/docs](https://data.amit.is-a.dev/docs)\r\n* **GitHub Repository:** [notamitgamer/osma](https://github.com/notamitgamer/osma)\r\n\r\n---\r\n\r\n## Why I Built OSMA\r\n\r\nThis project was born out of pure frustration. I found myself constantly needing to look up package details, version numbers, and dependencies. But doing this on the official registries meant dealing with:\r\n1.  **Fastly Security Checks:** Being stalled by CAPTCHAs or security verification screens just to view a text page.\r\n2.  **The Multi-Click Tax:** Clicking through 2-3 different pages just to find the exact version tag or repository link I was looking for.\r\n\r\n**The Solution:** I took complete static snapshots of the registries **(2026-04-29 08:45 IST for PyPI and 2026-04-29 09:42 IST for NPM)**. By indexing these frozen lists into a custom backend, OSMA allows users to instantly search and verify packages without ever hitting a live registry rate-limit. *(Note: Clicking a package in the UI will still fetch its extended live details securely via a side-panel).*\r\n\r\n---\r\n\r\n## How the Backend Works\r\n\r\nThe backend is powered by **FastAPI**, serving static `.csv` snapshots securely downloaded from HuggingFace datasets. \r\n\r\n* **SQLite Indexing:** On the first boot, the server downloads the multi-gigabyte dataset and builds an optimized, indexed SQLite database (WAL mode). \r\n* **Speed:** Because the dataset is localized, lookups for searches (`/search`) and chunked pagination (`/browse`) respond in milliseconds.\r\n* **Rate Limiting:** To protect the free-tier infrastructure, strict rate limits (**10 requests/minute, 100 requests/hour**) are enforced via `slowapi`. Developers can bypass this by requesting an `X-Bypass-Token` via a GitHub issue.\r\n* **Ranking Logic:** Search results are strictly ranked:\r\n    1.  `Rank 0`: Exact match\r\n    2.  `Rank 1`: Starts with query\r\n    3.  `Rank 2`: Contains query\r\n* **Limitation:** Because this is an archive, packages published or deleted after April 2026 are not indexed. \r\n\r\n---\r\n\r\n## API Endpoints Documentation\r\n\r\nThe APIs are hosted publicly and support open CORS. No authentication is required for standard usage.\r\n\r\n### 1. Health \u0026 Stats\r\n* **`GET /health`**\r\n    * **Description:** Checks if the server is alive and returns the total indexed packages. Responds with `503` if the database is still initializing on a cold start.\r\n    * **Response (200):** `{\"status\": \"ok\", \"total_packages\": 3881465}`\r\n* **`GET /ping`** / **`GET /stats`**\r\n    * **Description:** Basic server alive/telemetry routes.\r\n\r\n### 2. Browse Packages\r\n* **`GET /browse`**\r\n    * **Description:** Returns packages sequentially, ordered by `package_no`. Used for pagination.\r\n    * **Parameters:** * `page` (int, default `1`): The page number (minimum 1).\r\n        * `limit` (int, default `200`): Items per page (maximum 500).\r\n    * **Response (200):** List of package objects `[{\"no\": 1, \"name\": \"...\", \"version\": \"...\", \"url\": \"...\"}]`.\r\n    * **Response (429):** Too Many Requests (Rate limit exceeded).\r\n\r\n### 3. Search Packages\r\n* **`GET /search`**\r\n    * **Description:** Search packages by name. Returns algorithmically ranked results.\r\n    * **Parameters:**\r\n        * `q` (string, required): The search query (min length: 2 chars).\r\n        * `limit` (int, default `250`): Max results to return (maximum 500).\r\n    * **Response (200):** Ranked list of package objects matching the query.\r\n    * **Response (422):** Validation Error if query is too short.\r\n    * **Response (429):** Too Many Requests (Rate limit exceeded).\r\n\r\n### 4. Utility Endpoints\r\n* **`GET /get-bypass`**: Generate a temporary rate limit bypass token (requires `secret` parameter). Send the result as the `X-Bypass-Token` header.\r\n* **`GET /debug-ip`**: View the IP and headers recorded by the API proxy (useful for debugging rate limits).\r\n* **`GET /rebuild`**: Trigger an asynchronous re-download and rebuild of the SQLite database (requires `secret` parameter).\r\n\r\n---\r\n\r\n## Additional Documentation\r\n\r\nPlease refer to the following documents for repository guidelines:\r\n* [Contributing](CONTRIBUTING.md)\r\n* [Security Policy](SECURITY.md)\r\n* [Code of Conduct](CODE_OF_CONDUCT.md)\r\n\r\n## License\r\n\r\nDistributed under the Apache 2.0 License. See `LICENSE` for more information.\r\n\r\n---\r\n*Maintained by [Amit Dutta](https://github.com/notamitgamer) | [amit.is-a.dev](https://amit.is-a.dev)*\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotamitgamer%2Fosma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnotamitgamer%2Fosma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotamitgamer%2Fosma/lists"}