https://github.com/freeformz/logs

Some thoughts/tools for logs
https://github.com/freeformz/logs

Last synced: 7 days ago
JSON representation

Some thoughts/tools for logs

• Host: GitHub
• URL: https://github.com/freeformz/logs
• Owner: freeformz
• Created: 2018-10-18T03:57:10.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2018-10-18T04:48:22.000Z (about 6 years ago)
• Last Synced: 2024-12-24T06:47:42.665Z (10 days ago)
• Language: Go
• Size: 1.95 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Logs

## Ideas

- Detect json vs. new line

- Batching vs. Streaming

## Carriage

> A load-carrying device from which logs are suspended and which rides back and forth along the skyline on sheaves for yarding.

cli to move logs across a network to a destination

## Hooktender

>  The working foreman in charge of a yarding crew.

Orchestrates carriages

## Config Sources

### Amazon DynamoDB

???

### Amazon S3

Read object in a format.

Poll object every X for changes.

### Amazon SNS

### Hashicorp Consul

### etcd

### Zookeeper

## Destinations

### Amazon Cloudwatch Logs

Single Stream

A log stream is a sequence of log events that share the same source. Each separate source of logs into CloudWatch Logs makes up a separate log stream.

A log group is a group of log streams that share the same retention, monitoring, and access control settings. You can define log groups and specify which streams to put into each group. There is no limit on the number of log streams that can belong to one log group.

### Amazon Kinesis

Probably best to put in batches?

### Amazon Redshift

### Amazon S3 objects

Timestamped objects with a prefix

### Apache Kafka

Probably best to write batches for throughput

Can feed Google Cloud Dataflow (although not sure how)

### Google Cloud Pub/Sub

https://cloud.google.com/pubsub/

Can feed Google Cloud Dataflow

### Google Cloud Datastore

https://cloud.google.com/datastore/

Can feed Google Cloud Dataflow

### Google Stackdriver Logs

Both http REST and grpc endpoints

### Heroku Logplex format

One or more batches. Can't "stream" really as content length must be known in advance.