{"id":26103326,"url":"https://github.com/topfreegames/eventsgateway","last_synced_at":"2025-04-12T17:23:29.429Z","repository":{"id":43331326,"uuid":"119835372","full_name":"topfreegames/eventsgateway","owner":"topfreegames","description":"Receive events and send them encoded as avro to kafka","archived":false,"fork":false,"pushed_at":"2024-11-28T17:57:23.000Z","size":449,"stargazers_count":4,"open_issues_count":0,"forks_count":4,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-12T17:23:04.406Z","etag":null,"topics":["analytics","avro","data","events","golang","kafka"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/topfreegames.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-01T12:52:21.000Z","updated_at":"2024-11-27T18:14:43.000Z","dependencies_parsed_at":"2024-11-19T19:38:20.930Z","dependency_job_id":"58a5cc8a-6c57-496d-b443-820d6d5ede9c","html_url":"https://github.com/topfreegames/eventsgateway","commit_stats":null,"previous_names":[],"tags_count":83,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topfreegames%2Feventsgateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topfreegames%2Feventsgateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topfreegames%2Feventsgateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topfreegames%2Feventsgateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/topfreegames","download_url":"https://codeload.github.com/topfreegames/eventsgateway/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248602759,"owners_count":21131689,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","avro","data","events","golang","kafka"],"created_at":"2025-03-09T20:06:22.332Z","updated_at":"2025-04-12T17:23:29.399Z","avatar_url":"https://github.com/topfreegames.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"TFGCo EventsGateway\n===================\n\n## [Development](#development-readme)\n## [Client](#client-readme)\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n# Client\n\n## Important Notice\nThe Events Gateway service works following the At Least Once semantics, that means it will not prevent duplicates.\nIn case the Client loses its connection (because of networking instabilities) after sending the event to the Events Gateway server, the server will complete the \nrequest with success and send the data to Kafka, but won't be able to acknowledge the client. If the client has some retry logic (built in on async mode) it will\nretry to send the same event and will duplicate it.\n\nIf you really need distinct values guarantee, consider handling it on the downstream pipelines.\n\n## Configuration\n\nAbout client.NewClient(...) arguments:\n\n* configPrefix: whatever comes just before `client` portion of your config\n* config: a viper config with at least a `client` key holding Events Gateway settings\n* logger: a logger.Logger instance\n* client: should be nil for most cases, except for unit testing\n* opts: extra []grpc.DialOption objects\n\n`client` config format, with defaults:\n\n```yaml\nclient:\n  async: false # if you want to use the async or sync dispatch\n  channelBuffer: 500 # (async-only) size of the channel that holds events\n  lingerInterval: 500ms # (async-only) how long to wait before sending messages, in the hopes of filling the batch\n  batchSize: 10 # (async-only) maximum number of messages to send in a batch\n  maxRetries: 3 # (async-only) how many times to retry a dispatch if it fails\n  retryInterval: 1s # (async-only) first wait time before a retry, formula =\u003e 2^retryNumber * retryInterval\n  numRoutines: 2 # (async-only) number of go routines that read from events channel and send batches\n  kafkatopic: default-topic # default topic to send messages\n  grpc:\n    serverAddress: localhost:5000\n    timeout: 500ms\n```\n\nCode example:\n\n```go\nimport (\n  \"context\"\n\n  \"github.com/spf13/viper\"\n  \"github.com/topfreegames/eventsgateway/v4\"\n  \"github.com/topfreegames/eventsgateway/v4/logger\"\n)\n\nfunc ConfigureEventsGateway() (*eventsgateway.Client, error) {\n  config := viper.New() // empty Viper config\n  config.Set(\"eventsgateway.client.async\", true)\n  config.Set(\"eventsgateway.client.kafkatopic\", \"my-client-default-topic\")\n  logger := \u0026logger.NullLogger{} // Initialize you logger.Logger implementation here\n  client, err := eventsgateway.NewClient(\"eventsgateway\", config, logger, nil)\n  if err != nil {\n    return nil, err\n  }\n}\n\nfunc main() {\n  client, err := ConfigureEventsGateway()\n  if err != nil {\n    panic(err)\n  }\n  // here you pass along the context.Context you received,\n  // DON'T pass just a context.Background() if you have a previous context.Context\n  // Sync clients should handle errors accordingly\n  err := client.Send(context.Background(), \"event-name\", map[string]string{\"some\": \"value\"})\n  // Async clients error handling are transparent to the user \n  client.Send(context.Background(), \"event-name\", map[string]string{\"some\": \"value\"})\n}\n\n```\n\n# Development\n\n## Running locally\n\nAll dependencies required to produce and consume events locally are bundled in this project.\n\n1. `make build-dev` will build a development docker image for the client.\n2. `make deps-start` will start docker containers for `zookeeper` `kafka`, `eventsgateway-api`\n2. \n\nThese are the necessary dependencies for EventsGateway server.\n\n2. `make run` starts EventsGateway server.\n\n3. `make producer` executes a client that sends one dummy event.\n\n4. `make spark-notebook` runs a jupyter-notebook container with a mounted notebook to consume from Kafka and write ORC files to S3.\n\nCheckout the localhost address to access the Web UI over the container logs.\n\n5. `make hive-start` starts hive stack containers necessary to create tables in hive-metastore and to query from a presto client.\n\n### Bootstraping Localstack bucket and prefixes\n\nAfter `make deps-start` and before `make gobblin` you'll need to bootstrap localstack's s3 to transfer data from kafka.\n\n`aws --endpoint-url=http://localhost:4572 s3 mb s3://eventsgateway-local`\n\n`aws --endpoint-url=http://localhost:4572 s3api put-bucket-acl --bucket eventsgateway-local --acl public-read-write`\n\n`aws --endpoint-url=http://localhost:4572 s3api put-object --bucket eventsgateway-local --key output/sv-uploads-default-topic/daily/`\n\nNote that `default-topic` should be replaced by the topic you're using in your client, that's the one used by `make testclient`.\n\n### Creating topic table in Hive metastore\n\nRun it inside `docker exec -it hive_hive-server_1 sh -c \"/opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\"`.\n\nTo get the commands necessary to create database and table, run the respective cell at the end of `eventsgateway-streaming-orc` notebook.\n\nAfter creation, you need to run `msck repair table table_name;` from hive server container to be able to query recent data.\n\n### Querying with Presto\n\nInstall presto client, on mac `brew install presto`.\n\n`presto --catalog hive --schema default`\n\npresto:default \u003e\u003e `show tables;`\n\npresto:default \u003e\u003e `select * from defaulttopic;`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftopfreegames%2Feventsgateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftopfreegames%2Feventsgateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftopfreegames%2Feventsgateway/lists"}