{"id":20850467,"url":"https://github.com/noteed/euranova","last_synced_at":"2025-08-03T14:38:40.920Z","repository":{"id":66654548,"uuid":"45737935","full_name":"noteed/euranova","owner":"noteed","description":"Storm Case for EURA NOVA recruitment process","archived":false,"fork":false,"pushed_at":"2015-11-07T13:59:08.000Z","size":0,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-19T06:11:35.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noteed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-07T13:40:18.000Z","updated_at":"2016-01-31T20:55:16.000Z","dependencies_parsed_at":"2023-03-01T13:16:59.678Z","dependency_job_id":null,"html_url":"https://github.com/noteed/euranova","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteed%2Feuranova","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteed%2Feuranova/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteed%2Feuranova/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteed%2Feuranova/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noteed","download_url":"https://codeload.github.com/noteed/euranova/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243216137,"owners_count":20255322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T03:09:40.992Z","updated_at":"2025-03-12T12:25:49.186Z","avatar_url":"https://github.com/noteed.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is the code I have produced for EURA NOVA's recruitment process last year\n(i.e. october 2014). This was fun as I learned about Storm. This was also a lot\nof work since\n\n- Storm or Kafka were new to me\n- I haven't work with the JVM for a long time\n- parts of Storm are in Clojure\n- in addition to implementing the core elements (the Storm topology), they also\n  asked for some real-time UI, that the result could be deployed on their own\n  laptops easily, ...\n\nI also felt it was quite abusive since in addition to the large amount of work,\nthe process was quite obscure and lack any concrete feedback.\n\n\n# Storm case\n\nA case for Eura Nova's recruitment process using Apache Storm and the JVM.\n\nThe original case document is `storm-case-v02.odt` (the PDF version is saved\nfrom LibreOffice).\n\nThis repository has its history rewritten so it doesn't contain the above\ndocument (which were added in the first commit).\n\nCase received on Monday 13th, October ~ 03:15 pm.\n\nThis implementation of the case uses Kafka as a messaged queue and a\nWebSocket-based web interface to display the top-selling models and to simulate\ncash register tickets. The different parts of the project are built and\ndeployed using Docker images. One of the image, using Nginx, is not JVM-based\nbut is only used to serve two HTML pages.\n\nA Makefile is provided to build the different images specific to this project.\nOther images are available from Docker's public registry.\n\nThe topology is linear as follow:\n\n    Spout  \"from_kafka\"\n    Bolt \"models\"\n    Bolt \"sums\"\n    Bolt \"rolling\"\n    Bolt \"best_intermediate\"\n    Bolt \"best\"\n    Bolt \"to_kafka\"\n\n- \"from_kafka\" reads messages from Kafka,\n- \"models\" extract models and counts information from the Spout tuple\n- \"sums\" sums counts of the same models. It is not strictly necessary as the\n  next bolt also performs the sums but is present as a mean to rate-limit\n  messages at beginning of the topology.\n- \"rolling\" performs sums of the same model over a sliding window.\n- \"best_intermediate\" keeps the 10 best-selling models. It is similar to the\n  next bolt and serves to lower the load on that next bolt.\n- \"best\" does the same computation as \"best_intermediate\" but its parallelism\n  must be set to one.\n- \"to_kafka\" is the final bolt in the topology and writes its tuples to Kafka.\n\n## Docker images\n\n`images/storm-starter` is not necessary to build or run the project. It was\nused at the beginning to create the solution which was moved out of the\nstorm-starter example directory later.\n\n`images/kafka-websocket` is simply used to build the WebSocket server (as a\njar) serving Kafka messages. The jar is needed to build\n`images/websocket-server`.\n\n`images/websocket-server` containes the above jar. It connects to Kafka to send\nthe simulated cash register messages and also to receive the best-selling\nmodels.\n\n`images/maven` is a Docker image with the JDK and Maven to build the Storm\ntopolgy.\n\nRun the image with:\n\n    \u003e docker run -t -i \\\n        -v `pwd`/topologies:/home/storm/topologies \\\n        noteed/maven bash\n\nThis makes the local `topologies` directory available within the home\ndirectory.\n\nWithin the container, compile the solution with:\n\n    \u003e cd topologies\n    \u003e mvn compile\n\nTo run the solution:\n\n    \u003e mvn exec:java -Dstorm.topology=com.noteed.stormcase.Topology \\\n        -Dexec.args=\"172.17.0.2 172.17.0.3\"\n\nTo save a `jar` that can be submitted to a cluster:\n\n    \u003e mvn package\n    \u003e # target/storm-starter-0.9.2-incubating-jar-with-dependencies.jar can be saved\n\n`images/storm` containes a binary release of Storm. In particular it provides\nthe `storm` executable to submit a topology to a Storm cluster.\n\nTo run the jar built above against a cluster (using the noteed/storm image):\n\n    \u003e mkdir .storm\n    \u003e echo 'nimbus.host: \"172.17.0.4\"' \u003e .storm/storm.yaml\n    \u003e ./release/bin/storm jar \\\n        /source/storm-starter-0.9.2-incubating-jar-with-dependencies.jar \\\n        com.noteed.stormcase.Topology 172.17.0.2 172.17.0.3 stormcase\n\nNote: the above IP addresses are the Nimbus, ZooKeeper, and Kafka container\naddresses.\n\n## Running\n\nRunning the Storm case is done in two steps. The first step launch a set of\ncontainers to run Storm and Kafka. The second stop submits the Storm case\ntopology to Storm.\n\nThe first step is done by the `launch.sh` script:\n\n    \u003e ./launch.sh\n\nThis will spawn a few Docker containers:\n\n- a ZooKeeper container, used for Kafka and for Storm,\n- a Kafka container (the script also creates the two topics used by the Storm\n  case),\n- a Storm Nimbus container (its IP address will be used in the second step),\n- a Storm supervisor container,\n- a Storm UI container, accessible on \u003ccontainer-ip\u003e:8080,\n- a Webscocket server,\n- and a Nginx server, accessible directly from the host on port 80.\n\nWhile the containers are spawned, the script outputs the container IDs and\nassociated IP addresses.\n\nThe second step is done by:\n\n    \u003e docker run -v `pwd`:/source noteed/storm /submit.sh \u003cnimbus\u003e \u003czk\u003e \u003ckafka\u003e \n\nNote: `\u003cnimbus\u003e` is Nimbus address, `\u003czk\u003e` is ZooKeeper address, and `\u003ckafka\u003e`\nis Kafka address, as displayed by `launch.sh`.\n\nThe kafka image can also be used to generate messages:\n\n    \u003e docker run --rm --link zookeeper:zk -i -t wurstmeister/kafka:0.8.1.1-1 bash\n\nThen within the container:\n\n    \u003e $KAFKA_HOME/bin/kafka-console-producer.sh --topic=tickets --broker-list=172.17.0.3:9092\n\nOr to consume the messages:\n\n    \u003e $KAFKA_HOME/bin/kafka-console-consumer.sh --topic=tickets --zookeeper=$ZK_PORT_2181_TCP_ADDR\n\n## TODOs\n\n- Use something like https://github.com/joewalnes/reconnecting-websocket.\n- Make the websocket-server Docker image based on the JRE only.\n- Add missing Bootstrap files.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoteed%2Feuranova","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoteed%2Feuranova","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoteed%2Feuranova/lists"}