{"id":34745907,"url":"https://github.com/dfpc-coe/etl-base","last_synced_at":"2026-03-04T20:00:42.905Z","repository":{"id":89946277,"uuid":"606603253","full_name":"dfpc-coe/etl-base","owner":"dfpc-coe","description":"Helper library for building NodeJS based ETL Lambdas","archived":false,"fork":false,"pushed_at":"2026-01-30T18:01:21.000Z","size":1213,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-18T04:14:34.741Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://dfpc-coe.github.io/etl-base/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dfpc-coe.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"ingalls"}},"created_at":"2023-02-26T01:20:17.000Z","updated_at":"2026-01-30T18:01:46.000Z","dependencies_parsed_at":"2023-07-22T07:16:05.128Z","dependency_job_id":"ce672e9f-48c2-4d1b-9441-ae4d9c532f4a","html_url":"https://github.com/dfpc-coe/etl-base","commit_stats":null,"previous_names":["dfpc-coe/etl-base","tak-ps/etl-base"],"tags_count":117,"template":false,"template_full_name":null,"purl":"pkg:github/dfpc-coe/etl-base","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfpc-coe%2Fetl-base","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfpc-coe%2Fetl-base/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfpc-coe%2Fetl-base/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfpc-coe%2Fetl-base/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dfpc-coe","download_url":"https://codeload.github.com/dfpc-coe/etl-base/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dfpc-coe%2Fetl-base/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30088351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T15:40:14.053Z","status":"ssl_error","status_checked_at":"2026-03-04T15:40:13.655Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T04:36:47.328Z","updated_at":"2026-03-04T20:00:42.884Z","avatar_url":"https://github.com/dfpc-coe.png","language":"TypeScript","funding_links":["https://github.com/sponsors/ingalls"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=center\u003eETL Base\u003c/h1\u003e\n\n\u003cp align=center\u003eHelper library for building NodeJS based ETL Lambdas\u003c/p\u003e\n\nThe [ETL Server](https://github.com/dfpc-coe/CloudTAK) is designed to manage and run\nLambda Container Images that perform various ETL functions to convert data into\nformats that are more easily ingestible via TAK.\n\nThe ETL server can run runtimes of any language but helper functions are primarily\nwritten in [NodeJS](https://nodejs.org/en). This library serves as both a usable\nbase for removing much of the boilerplate to interacting with the TAK ETL service,\nas well as serving as a (hopefully) straightforward and readable example for how\nservices in other languages could be written\n\n## API\n\nThe ETL Base Class is designed to be extended by classes performing ETL functions.\n\nThe following is an example of as simple an ETL service as possible that shows\nhow this extension is performed.\n\n```ts\nimport { Type } from '@sinclair/typebox';\nimport ETL, { TaskLayer, Event, SchemaType, handler as internal, local, DataFlowType, InvocationType } from '@tak-ps/etl';\n\nexport default class Task extends ETL {\n    static name = 'etl-example';\n    static flow = [ DataFlowType.Incoming, DataFlowType.Outgoing ];\n    static invocation = [ InvocationType.Schedule ];\n\n    schema(\n        type: SchemaType = SchemaType.Input,\n        flow: DataFlowType = DataFlowType.Incoming\n    ): Promise\u003cTSchema\u003e\n        if (flow === DataFlowType.Incoming \u0026\u0026 type === SchemaType.Input) {\n            return Type.Object({\n                password: Type.String({\n                    description: 'The password for the service'\n                })\n            });\n        } else if (flow === DataFlowType.Outgoing \u0026\u0026 type === SchemaType.Input) {\n            return Type.Object({\n                callSign: Type.String({\n                    description: 'The CallSign of the returned point'\n                })\n            });\n        } else {\n            return Type.Object({});\n        }\n    }\n\n    async control(): Promise\u003cvoid\u003e {\n        // Provided function to obtain all environment config as defined by a user in the UI\n        const layer = await this.fetchLayer();\n\n        // The Layer object contains all the properties as defined by the Get Layer API\n        const environment = layer.environent;\n\n        // Provided submit function to submit geospatial features to be converted to CoT\n        // See format overview: https://github.com/tak-ps/node-cot#geojson-spec\n        await this.submit({\n            \"type\": \"FeatureCollection\",\n            \"features\": [{\n                \"type\": \"Feature\",\n                \"properties\": {\n                    callsign: environment.CallSign\n                },\n                \"geometry\": {\n                    \"type\": \"Point\",\n                    \"coordinates\": [1.0, 2.0]\n                }\n            }]\n        });\n    }\n}\n\n// Optionally allow CLI calls\nawait local(new Task(), import.meta.url);\nexport async function handler(event: Event = {}, context?: unknown) {\n    return await internal(new Task(), event, context);\n}\n```\n\n### API\n\nThe API documentation is generated automatically by any commit to the `main` branch.\n\nDocumentation for the API can be found at [dfpc-coe.github.io/etl-base](https://dfpc-coe.github.io/etl-base/)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfpc-coe%2Fetl-base","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdfpc-coe%2Fetl-base","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdfpc-coe%2Fetl-base/lists"}