{"id":20817887,"url":"https://github.com/vforteli/azuredatalakeindexer","last_synced_at":"2026-05-26T20:05:25.196Z","repository":{"id":198549916,"uuid":"686003402","full_name":"vforteli/AzureDataLakeIndexer","owner":"vforteli","description":"Azure Datalake gen2 search indexer fiddling","archived":false,"fork":false,"pushed_at":"2024-06-18T19:14:40.000Z","size":437,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-13T01:32:34.049Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vforteli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-09-01T14:01:51.000Z","updated_at":"2024-03-29T19:25:05.000Z","dependencies_parsed_at":"2025-12-12T00:09:38.663Z","dependency_job_id":null,"html_url":"https://github.com/vforteli/AzureDataLakeIndexer","commit_stats":null,"previous_names":["vforteli/searchindexertest"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vforteli/AzureDataLakeIndexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vforteli%2FAzureDataLakeIndexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vforteli%2FAzureDataLakeIndexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vforteli%2FAzureDataLakeIndexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vforteli%2FAzureDataLakeIndexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vforteli","download_url":"https://codeload.github.com/vforteli/AzureDataLakeIndexer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vforteli%2FAzureDataLakeIndexer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33536737,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"ssl_error","status_checked_at":"2026-05-26T15:22:15.568Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T21:44:39.141Z","updated_at":"2026-05-26T20:05:25.170Z","avatar_url":"https://github.com/vforteli.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AzureDataLakeIndexer\nAzure Datalake gen2 search indexer fiddling\nThe reason this exists, is because the built in datalake indexers in azure search are slow... mind numbingly slow. \nMost of the time the indexers seem to spend on listing paths in datalake and this project solves this by using a helper index for paths. Querying this index for modified files is much faster than listing paths in datalake\nThe built in indexers also has a habit of forgetting to renew the access tokens even when using RBAC causing indexing to fail.\n\n\n## Overview\n\n```mermaid\nflowchart LR\n    datalake[(DataLake)]\n\n    datalake--\u003e|BlobCreated|pathfunc\n    datalake--\u003e|BlobDeleted|pathfunc\n    \n    pathfunc--\u003e|UpsertPaths|pathindex\n    pathfunc--\u003e|UpsertPaths|deletedpathindex\n\n    subgraph FuncHost\n      pathfunc{{PathFunc}}\n      indexerfunc{{IndexerFunc}}\n    end\n\n    subgraph AzureSearch\n      pathindex[(Path index)]\n      deletedpathindex[(Deleted Path index)]\n      dataindex[(Data index)]\n    end\n\n    pathindex--\u003e|ListPaths|indexerfunc\n    datalake--\u003e|Readdocs|indexerfunc\n    indexerfunc--\u003e|UpsertData|dataindex  \n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvforteli%2Fazuredatalakeindexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvforteli%2Fazuredatalakeindexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvforteli%2Fazuredatalakeindexer/lists"}