{"id":15161784,"url":"https://github.com/mycroft/bookish-couscous","last_synced_at":"2026-02-02T02:02:57.362Z","repository":{"id":136947687,"uuid":"219931563","full_name":"mycroft/bookish-couscous","owner":"mycroft","description":"My own attempt to zen.ly hiring challenge.","archived":false,"fork":false,"pushed_at":"2019-11-06T06:56:40.000Z","size":32,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-23T19:48:37.780Z","etag":null,"topics":["docker","docker-compose","golang","kafka","microservices","protobuf","redis","scylladb","zenly"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mycroft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-06T06:56:11.000Z","updated_at":"2023-11-08T20:08:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"fae06351-d30a-43ac-a1cb-a92567f072b1","html_url":"https://github.com/mycroft/bookish-couscous","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mycroft/bookish-couscous","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mycroft%2Fbookish-couscous","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mycroft%2Fbookish-couscous/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mycroft%2Fbookish-couscous/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mycroft%2Fbookish-couscous/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mycroft","download_url":"https://codeload.github.com/mycroft/bookish-couscous/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mycroft%2Fbookish-couscous/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29001513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-02T01:32:03.847Z","status":"online","status_checked_at":"2026-02-02T02:00:07.448Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","golang","kafka","microservices","protobuf","redis","scylladb","zenly"],"created_at":"2024-09-27T00:45:10.197Z","updated_at":"2026-02-02T02:02:57.344Z","avatar_url":"https://github.com/mycroft.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"bookish-couscous\n================\n\n## Introduction\n\nMy own attempt to zen.ly hiring challenge (~2017/11/15).\n\nThe description is in ChallengeZenly.md.\n\nLanguage used:\n\n- golang\n\nSoftware component used:\n\n- kafka \u0026 zookeeper (wurstmeister/kafka, wurstmeister/zookeeper)\n- scylladb (scylladb/scylla)\n- redis (redis:alpine)\n- nginx (as a tcp load balancer for frontends)\n\nLibs used:\n\n- https://github.com/golang/geo\n- https://github.com/gocql/gocql\n- https://github.com/bsm/sarama-cluster\n- https://github.com/golang/protobuf\n- https://godoc.org/google.golang.org/grpc\n- https://github.com/Shopify/sarama\n- https://github.com/garyburd/redigo/redis\n- https://github.com/golang/protobuf/proto\n\nPlease note this is my first use of following technologies, so I mostly\nfollowed basic tutorials and I'm maybe not using them perfectly:\n\n- S2 geo library;\n- scylla \u0026 gocql;\n- kafka \u0026 shopify's sarama;\n- protobuf\n- grpc\n\n\n## Up \u0026 running\n\n```shell\n# Build an initial image, with all deps, to make things faster\n$ docker build -t bookish-couscous-base .\n\n# Start the stack:\n$ docker-compose up -d\n[...]\n\n$ docker-compose ps\n            Name                           Command              State              Ports\n----------------------------------------------------------------------------------------------------\nbookishcouscous_client_1        /bin/sh -c /bin/sh /go/src      Up\n                                ...\nbookishcouscous_fo01_1          /bin/sh -c /go/bin/fo           Up\nbookishcouscous_fo02_1          /bin/sh -c /go/bin/fo           Up\nbookishcouscous_fo_1            /opt/nginx/sbin/nginx -g d      Up      443/tcp, 80/tcp\n                                ...\nbookishcouscous_generator_1     /bin/sh -c /bin/sh /go/src      Up\n                                ...\nbookishcouscous_kafka_1         start-kafka.sh                  Up      0.0.0.0:32794-\u003e9092/tcp\nbookishcouscous_processor01_1   /go/bin/processor -init         Up\nbookishcouscous_processor02_1   /bin/sh -c /go/bin/processor    Up\nbookishcouscous_processor03_1   /bin/sh -c /go/bin/processor    Up\nbookishcouscous_processor04_1   /bin/sh -c /go/bin/processor    Up\nbookishcouscous_redis_1         docker-entrypoint.sh redis      Up      6379/tcp\n                                ...\nbookishcouscous_scylla_1        /docker-entrypoint.py           Up      10000/tcp, 7000/tcp,\n                                                                        7001/tcp, 9042/tcp,\n                                                                        9160/tcp, 9180/tcp\nbookishcouscous_zookeeper_1     /bin/sh -c /usr/sbin/sshd       Up      0.0.0.0:2181-\u003e2181/tcp,\n                                ...                                     22/tcp, 2888/tcp, 3888/tcp\n\n# To send events:\n$ docker-compose exec generator generator -h\nUsage of generator:\n  -days int\n        Number of days (default 15)\n  -event int\n        Number of event to inject per day (default 5000)\n  -friends int\n        Number of friends per user (default 10)\n  -kafka string\n        kafka host port (default \"kafka:9092\")\n  -redis string\n        redis host port (default \"redis:6379\")\n  -users int\n        Number of users (default 100)\n\n# Generate events...\n$ docker-compose exec generator generator -days 7 -event 1000 -friends 5 -users 10\n[...]\n\n# Check in db...\n$ docker-compose exec scylla cqlsh -e 'select * from zenly.kyf;'\n[...]\n\n# To query fo:\n$ docker-compose exec client client -uid 42\n2017/11/16 10:25:27 My uid is 42\n2017/11/16 10:25:27 Best friend:          1\n2017/11/16 10:25:27 Crush:                42\n2017/11/16 10:25:27 Most seen:            1\n2017/11/16 10:25:27 Mutual love:          1\n2017/11/16 10:25:27 Mutual love all time: 1\n\n# (note: if given uid = user id, then it is like it was not found;\n# just a lazy thing from me :) )\n```\n\n\n## Components description\n\n### generator\n\nGenerate friends, relationships, and sessions; It writes friends' relationships in redis,\nand sessions in a kafka queue. It also generate \u0026 stores users' SP in redis (only one, home).\nIt will generate days * events number of events, over 100 users having each 10 friends (pick randomnly).\n\n### processor\n\nFetch sessions from queue, and ran processing over it. It will then store in DB valuable information,\nready to be handled by fo on request.\n\nIts main algo is:\n\nfor each session in queue:\n\n- fetch session from queue;\n- if session invalid (not friends), return;\n- grab metadata from both user 1 \u0026 user 2 relationships with each others;\n- compute time passed together on session, store it in metadata struct according of session's day (\"most seen\" feature);\n- grab both users SP from redis;\n- if relevant, store time passed together for \"best friend\" feature;\n- store time passed for \"most seen, all time\" feature\n- if night \u0026 position requirements are fullfilled, store night in structure as well (\"crush feature\");\n- save both structure in database;\n- done!\n\nNote that \"mutual love\" feature is not computed here. It'll be computed when information will\nbe necessary by FO (The reason is that when the FO call occurs, this information may be\nalready outdated by relationships, and thus will be needed to be recomputed...)\n\nIt runs only 4 db queries: 2 read \u0026 2 writes (1 for each users of the session).\nIt also runs 2 read on redis.\nIt will invalidate also data computed \u0026 stored by fo component (see next section).\n\nIt can be ran multiple instance of processors. Each are independent and keeps no persistant information\nin memory.\n\n### fo\n\nListens for client's requests through a grpc socket.\n\nOn a client request, it will fetch user's friends data from scylla and compute through a single iteration\nover its friend's relationships metadata for best friends, most seens, etc.\n\nAt the end of the process, we'll do the same over its \"most seen\" (7 days \u0026 all time) friend,\nand if uids are the same on each sides of relationship, they we can conclude \"mutual love\".\n\nResult is stored in redis (cached 12h - because daily data become obsolete at a point) \u0026 returned by grpc socket.\n\n### client\n\nOpens a grpc socket, queries the fo.\n\n\n## DB Schema\n\nI've used only 1 scylla table for this:\n\n```sql\nCREATE TABLE IF NOT EXISTS zenly.kyf (\n    user_id bigint,\n    rel_user_id bigint,\n    PRIMARY KEY(user_id, rel_user_id),\n    duration bigint,\n    week_most bigint,\n    week_friends bigint,\n    nights list\u003ctimestamp\u003e,\n    week_most_list map\u003ctimestamp, int\u003e,\n    week_friends_list map\u003ctimestamp, int\u003e,\n);\n```\n\nNote that the primary key is over (userid, reluserid). I do not know if this is\nefficient enough to do scan queries over all userid records (that I do on the\nfo part). I'm pretty sure this is something to investigate, perform some\nperfomance tests \u0026 read docs about it.\n\nSample queries ran over this schema:\n\n```sql\nSELECT rel_user_id, week_friends_list, week_most_list, nights, duration\n    FROM kyf\n    WHERE user_id = ?\n```\n\n```\nSELECT duration, nights, week_most_list, week_friends_list\n    FROM kyf WHERE user_id = ? AND rel_user_id = ?\n```\n\nData sample:\n\n```sql\ncqlsh\u003e select * from zenly.kyf;\n\n user_id | rel_user_id | duration | nights                       | week_friends_list                                                                                                                                                                                                                              | week_most_list\n---------+-------------+----------+------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n       0 |           1 |    15440 | ['2017-11-14 06:00:35+0000'] | {'2017-11-11 00:00:00+0000': 1973, '2017-11-12 00:00:00+0000': 2127, '2017-11-13 00:00:00+0000': 1569, '2017-11-14 00:00:00+0000': 1818, '2017-11-15 00:00:00+0000': 2516, '2017-11-16 00:00:00+0000': 2348, '2017-11-17 00:00:00+0000': 1131} | {'2017-11-11 00:00:00+0000': 2167, '2017-11-12 00:00:00+0000': 2127, '2017-11-13 00:00:00+0000': 1569, '2017-11-14 00:00:00+0000': 2289, '2017-11-15 00:00:00+0000': 2516, '2017-11-16 00:00:00+0000': 2348, '2017-11-17 00:00:00+0000': 1236}\n       1 |           0 |    15440 | ['2017-11-14 06:00:35+0000'] |  {'2017-11-11 00:00:00+0000': 2167, '2017-11-12 00:00:00+0000': 2127, '2017-11-13 00:00:00+0000': 1569, '2017-11-14 00:00:00+0000': 1913, '2017-11-15 00:00:00+0000': 1916, '2017-11-16 00:00:00+0000': 2065, '2017-11-17 00:00:00+0000': 788} | {'2017-11-11 00:00:00+0000': 2167, '2017-11-12 00:00:00+0000': 2127, '2017-11-13 00:00:00+0000': 1569, '2017-11-14 00:00:00+0000': 2289, '2017-11-15 00:00:00+0000': 2516, '2017-11-16 00:00:00+0000': 2348, '2017-11-17 00:00:00+0000': 1236}\n```\n\n## Known drawbacks \u0026 possible enhancements\n\n- timezones are not managed;\n- long sessions are not managed either (multiple nights);\n- data generator could be enhanced to simulate day periods in days \u0026 nights, create similar\n  patterns, and make a better use of SPs\n- May require better integration testing\n- database records locking issue (2 processors working on same users will lead to data overwrite.)\n  A solution would be not to write data in scylla from processors, but send computations done by\n  processors in a kafka topic \u0026 have a last unique process to store everything in cassandra. We\n  would not store values, but add them to stored (update kyf set duration = duration + nn where uid = 42)\n  Another would be to lock row records, but scylla doesn't allow this. Doing it elsewhere seems dangerous.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmycroft%2Fbookish-couscous","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmycroft%2Fbookish-couscous","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmycroft%2Fbookish-couscous/lists"}