{"id":20840858,"url":"https://github.com/kareem-emad/simple-apm","last_synced_at":"2026-04-17T10:08:52.561Z","repository":{"id":57550680,"uuid":"305141340","full_name":"Kareem-Emad/simple-apm","owner":"Kareem-Emad","description":"an open source application/in-house performance management server that monitors your services and allows you to  perform historical/real-time analysis for your endpoints","archived":false,"fork":false,"pushed_at":"2020-10-23T17:49:06.000Z","size":33,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-19T00:05:09.303Z","etag":null,"topics":["apm","apm-server","cassandra","elasticsearch","monitoring","monitoring-server","monitoring-tool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kareem-Emad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-18T16:06:22.000Z","updated_at":"2020-10-23T17:49:08.000Z","dependencies_parsed_at":"2022-09-04T07:31:46.244Z","dependency_job_id":null,"html_url":"https://github.com/Kareem-Emad/simple-apm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kareem-Emad%2Fsimple-apm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kareem-Emad%2Fsimple-apm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kareem-Emad%2Fsimple-apm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kareem-Emad%2Fsimple-apm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kareem-Emad","download_url":"https://codeload.github.com/Kareem-Emad/simple-apm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243196657,"owners_count":20251861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apm","apm-server","cassandra","elasticsearch","monitoring","monitoring-server","monitoring-tool"],"created_at":"2024-11-18T01:18:05.291Z","updated_at":"2026-04-17T10:08:52.468Z","avatar_url":"https://github.com/Kareem-Emad.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple APM\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Build Status:](https://github.com/Kareem-Emad/simple-apm/workflows/Build/badge.svg)](https://github.com/Kareem-Emad/simple-apm/actions)\n[![GoReportCard example](https://goreportcard.com/badge/github.com/Kareem-Emad/simple-apm)](https://goreportcard.com/report/Kareem-Emad/simple-apm)\n\nan open source/in-house application performance management server that monitors your services and allows you to perform historical/real-time analysis for your endpoints\n\n## setup\n\nFirst make sure to init schema in both cassendra and elastic search\n\n```sh\n./elastic_init_index.sh\n```\n\nin `cqlsh`\n\n```sql\ncreate keyspace simple_apm with replication = {'class': 'SimpleStrategy', 'replication_factor': 1};\nuse simple_apm;\n\ncreate table request_info (service_name text, url text, method text, status smallint, response_time int, created_at timestamp, primary key (service_name, created_at, method, url));\n```\n\nTo run server\n\n```shell\n# run one instance of sever to handle http request and push jobs in queues\nmake run_server\n```\n\nTo run worker\n\n```shell\n# you need two types of workers online at least\nmake run_worker job_type=ES_SYNC target_queue=sync_es batch_size=5 # elastic search sync worker\nmake run_worker job_type=DB_WRITE target_queue=cassendra_write batch_size=5 # cassendra DB write worker\n```\n\nparams:\n\n- `batch_size` number of jobs to handle by worker per one exec\n- `target_queue` the job queue worker will listen to\n- `job_type` the type of job the spawned worker will handle\n\n## Environment Variables\n\nList of envs needs to be setup before starting the service\n\n- `PRODUCTION_QUEUES` comma separated list of queues to push jobs into from server/producer\n- `REDIS_CONNECTION_URL` connection url to setup redis client\n- `JWT_SECRET` secret used to sign and verify tokens between sdk/server\n- `CASSENDRA_KEY_SPACE` name of the keyspace the tables are stored\n- `CASSENDRA_HOSTS` number of cassendra nodes that are under simple-apm\n- `SERVER_PORT` port used by http server\n- `ES_BULK_INSERTION_CONCURRENCY` max concurrency level use in bulk es index updates\n\n## SDK\n\n- JS: \u003cgithub.com/Kareem-Emad/simple-apm-express\u003e\n\n## custom SDK\n\nYou can build your own custom sdk in whatever lang/way you want, just make sure you\n\n- sign a token by the same secret set here in the envs\n- use the same request format\n\n```shell\ncurl --location --request POST 'http://localhost:5000/requests' \\\n--header 'Authorization: Bearer jwt_token_placeholder' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n    \"url\": \"https://google.com\",\n    \"http_method\": \"GET\",\n    \"response_time\": 3000,\n    \"service_name\": \"service_name\",\n    \"status_code\": 200,\n    \"created_at\": \"2020-04-02 02:10:01\"\n}'\n```\n\n- make sure you use the same date format as Elastic search is optimzied for this layout specificly as shown below\n\n## Cassandra DB Schema\n\nThe database contains one table called `request_info`, containing Fields:\n\n| Field | Data Type| Description |\n| --- | --- | --- |\n| `service_name` | `text/varchar` | name of the service that contain this endpoint|\n| `url` | `text/varchar` | full url of the route in this service |\n| `method` | `text/varchar` | type of the request handled by the endpoint |\n| `status` | `small_int` | code returned to the requestor |\n| `response_time` | `int` | time taken from request recieval in the server to responding to the client |\n| `created_at` | `timestamp` | the timestamp when this request was done |\n\n### Indexing\n\nWe have two types of indexes:\n\n- `partition index` set to  `service_name` column\n- `cluster_index` set to the (`created_at`, `method`, `url`) in the order respectively\n\nNote that accessing the data in cassendra should be optimized to use the index in its order to avoid latencies\nSo it's better to always start by specifying `service_name`, range of dates `created_at`, http method `method`, and finally the endpoints you want to include in the query result `url`.\n\n## Elastic Search mapping\n\n| Field | Data Type|\n| --- | --- |\n| `service_name` | `keyword` |\n| `url` | `text` |\n| `http_method` | `keyword` |\n| `status_code` | `short` |\n| `response_time` | `long` |\n| `created_at` | `date` in format `yyyy-MM-dd HH:mm:ss` |\n\n## Acessing analytics\n\nTo access data through elastic search server, there are already some ready to use queries to\nfetch important data like `throughput`, `average_response_time`, `min/max_response_time`, `x_percentile`\n\n### throughput\n\nTo calculate throughput, we need to use histograms in elastic search.\nFollowing on the assumption that `throughput` is the sucessfull number of requests handled per miute, we can achieve it with a query/aggregate like this one\n\n```shell\ncurl -X GET \"localhost:9200/request_info/_search?pretty\" -H 'Content-Type: application/json' -d'\n{\n    \"query\": {\n      \"bool\": {\n        \"must\": [ {\"match\": {\"service_name\": \"YOUR_SERVICE_NAME\"}} , {\"match\":{\"url\": \"SOME_URL\"}}, {\"match\":{\"http_method\": \"GET\"}}],\n        \"filter\": {\n          \"range\": {\n            \"created_at\": {\n              \"gte\": \"START_DATE IN FORMAT (2020-04-01 02:01:01)\",\n              \"lte\": \"END_DATE IN FORMAT (2020-04-10 02:01:01)\"\n            }\n          }\n        }\n      }\n    }, \"aggs\": {\n\t\t  \"throughput_over_time\": {\n                \"date_histogram\": {\n                \"field\": \"created_at\",\n                \"fixed_interval\": \"1m\"\n            }\n          }\n    }\n}\n'\n```\n\nNote that you can omit some of the matche queries in the `must` to get calculate the througput curve over your whole service or a specific set of endpoints\n\nAlso notice that the interval here is tunable through `fixed_interval` field, meaning you can calculate the throughput of total successfull requests per minute/hour/day/month/etc\n\n### stats(average/min/max/x_percent response time)\n\nTo get the full stats for a certain endpoint(s)/service, we can use this query/aggregate:\n\n```shell\ncurl -X GET \"localhost:9200/request_info/_search?pretty\" -H 'Content-Type: application/json' -d'\n{\n    \"query\": {\n      \"bool\": {\n        \"must\": [ {\"match\": {\"service_name\": \"YOUR_SERVICE_NAME\"}} , {\"match\":{\"url\": \"SOME_URL\"}}, {\"match\":{\"http_method\": \"POST\"}}],\n        \"filter\": {\n          \"range\": {\n            \"created_at\": {\n              \"gte\": \"START_DATE\",\n              \"lte\": \"END_DATE\"\n            }\n          }\n        }\n      }\n    }, \"aggs\": {\n          \"request_percentiles\": {\n              \"percentiles\": {\n                  \"field\": \"response_time\" \n              }\n          },\n          \"request_stats\":{\n              \"stats\":{\n                  \"field\": \"response_time\"\n              }\n          }\n    }\n}\n'\n```\n\nNote that you need to specify the period of your search in both curls in the `range` filter to get your stats within a certain timeline (last week/month/...)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkareem-emad%2Fsimple-apm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkareem-emad%2Fsimple-apm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkareem-emad%2Fsimple-apm/lists"}