{"id":24228636,"url":"https://github.com/linz/elasticsearch-shipper","last_synced_at":"2025-10-24T06:11:00.336Z","repository":{"id":37826884,"uuid":"283042963","full_name":"linz/elasticsearch-shipper","owner":"linz","description":"AWS-CDK construct to ship logs to elastic search using Lambda","archived":false,"fork":false,"pushed_at":"2024-03-01T06:34:32.000Z","size":1156,"stargazers_count":2,"open_issues_count":13,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-03T14:41:09.350Z","etag":null,"topics":["aws-cdk","aws-cdk-construct","elasticsearch","lambda"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-27T23:35:24.000Z","updated_at":"2023-08-14T20:45:27.000Z","dependencies_parsed_at":"2024-12-23T15:50:00.795Z","dependency_job_id":null,"html_url":"https://github.com/linz/elasticsearch-shipper","commit_stats":{"total_commits":532,"total_committers":5,"mean_commits":106.4,"dds":"0.23872180451127822","last_synced_commit":"0840e9b3e7c697320909c7130f2420a6a731ac82"},"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linz%2Felasticsearch-shipper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linz%2Felasticsearch-shipper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linz%2Felasticsearch-shipper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linz%2Felasticsearch-shipper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linz","download_url":"https://codeload.github.com/linz/elasticsearch-shipper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233865139,"owners_count":18742471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-cdk","aws-cdk-construct","elasticsearch","lambda"],"created_at":"2025-01-14T11:17:54.737Z","updated_at":"2025-10-24T06:10:55.299Z","avatar_url":"https://github.com/linz.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elastic Log Shipper\n\n[![Build Status](https://github.com/linz/elasticsearch-shipper/workflows/Build/badge.svg)](https://github.com/linz/elasticsearch-shipper/actions)\n\nLambda function to ship logs from various AWS sources into a elastic search instance of your choosing.\n\n## Usage\n\nThis package exposes a Lambda Function that can be used in a CDK stack that can be used to configure logs to be automatically shipped into an elastic search of your choosing\n\n`./src/config.mjs`\n\n```typescript\nexport const Config = {\n  name: 'default',\n  accounts: [\n    {\n      id: '1234567890',\n      elastic: '/es-shipper-config/elastic-default',\n      name: 'linz',\n      tags: ['hello'],\n      prefix: 'account-prefix',\n      logGroups: [\n        {\n          filter: '**',\n          prefix: 'lg-prefix',\n        },\n      ],\n    },\n  ],\n};\n```\n\n`./src/index.mjs`\n\n```typescript\nimport { Config } from './config.js';\nimport { logHandler, LogShipper } from '@linzjs/cdk-elastic-shipper';\nLogShipper.configure(Config);\n\nexport const handler = logHandler;\n```\n\n```javascript\nimport * as cdk from 'aws-cdk-lib';\nimport { App, CfnOutput, Duration } from 'aws-cdk-lib';\nimport * as lambda from 'aws-cdk-lib/aws-lambda';\nimport * as lNjs from 'aws-cdk-lib/aws-lambda-nodejs'\nimport {Config} from './config.mjs'\nimport {Env} from '@linzjs/cdk-elastic-shipper';\n\nexport class YourStack extends cdk.Stack {\n\n  public constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {\n    super(scope, id, props);\n    const vpc = new ec2.Vpc(this, 'Vpc');\n\n    const cfg = validateConfig(Config)\n    // Validate all the connection references are valid\n    const allParameters: ssm.IStringParameter[] = [];\n    for (const elasticId of elasticIds) {\n      allParameters.push(ssm.StringParameter.fromStringParameterName(this, 'ElasticConfig' + elasticId, elasticId));\n    }\n\n    this.lambda = new NodejsFunction(this, 'Shipper', {\n        runtime: lambda.Runtime.NODEJS_16_X,\n        memorySize: 256,\n        timeout: Duration.seconds(30),\n        handler: 'handler',\n        entry: './src/index.mjs',\n        environment: {\n          // Deploy with a known hash and version\n          [Env.GitHash]: execFileSync('git', ['rev-parse', 'HEAD']).toString().trim(),\n          [Env.GitVersion]: execFileSync('git', ['describe', '--tags', '--always', '--match', 'v*']).toString().trim(),\n        },\n        logRetention: RetentionDays.ONE_MONTH,\n    });\n\n    for (const param of allParameters) param.grantRead(this.lambda);\n\n\n    /** Register a listener on a bucket, so when files are added they are to submitted to the log shipper */\n    const logBucket = new Bucket.fromName(this, 'LogBucket');\n    bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new LambdaDestination(this.lambda));\n    bucket.grantRead(this.lambda);\n  }\n}\n```\n\nThe elastic connection strings need to be stored inside of SSM before running the deployment, there are three options for elasticsearch connections\n\n1. AWS\n\n```\n{ url: 'https://node-name.eu-west-1.es.amazonaws.com' }\n```\n\n2. ElasticCloud\n\n```\n{\n  id: 'cloud:abc123',\n  username: 'foo',\n  password: 'bar'\n}\n```\n\n3. Http\n\n```\n{\n  url: 'https://fake.com'\n  username: 'foo'\n  password: 'bar'\n}\n```\n\n## Building\n\nThis repository requires [NodeJs](https://nodejs.org/en/) \u003e 12 \u0026 [Yarn](https://yarnpkg.com/en/)\n\nUse [n](https://github.com/tj/n) to manage nodeJs versions\n\n```bash\n# Download the latest nodejs \u0026 yarn\nn latest\nnpm install -g yarn\n\n# Install node deps\nyarn\n\n# Build everything into /build\nyarn run build\n\n# Run the unit tests\nyarn run test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinz%2Felasticsearch-shipper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinz%2Felasticsearch-shipper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinz%2Felasticsearch-shipper/lists"}