{"id":27248853,"url":"https://github.com/bakdata/hpi-ii-project-2022","last_synced_at":"2025-04-10T23:48:18.487Z","repository":{"id":38009585,"uuid":"483159124","full_name":"bakdata/hpi-ii-project-2022","owner":"bakdata","description":"Repository containing the code for the assignment of the Information Integration course in SoSe 2022","archived":false,"fork":false,"pushed_at":"2022-10-14T13:17:13.000Z","size":190,"stargazers_count":2,"open_issues_count":0,"forks_count":16,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-10T23:48:14.056Z","etag":null,"topics":["elasticsearch","kafka","kafka-connect"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-19T08:24:17.000Z","updated_at":"2022-05-16T09:55:00.000Z","dependencies_parsed_at":"2023-01-19T13:38:24.996Z","dependency_job_id":null,"html_url":"https://github.com/bakdata/hpi-ii-project-2022","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fhpi-ii-project-2022","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fhpi-ii-project-2022/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fhpi-ii-project-2022/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fhpi-ii-project-2022/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/hpi-ii-project-2022/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","kafka","kafka-connect"],"created_at":"2025-04-10T23:48:17.925Z","updated_at":"2025-04-10T23:48:18.478Z","avatar_url":"https://github.com/bakdata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HPI information integration project SoSe 2022\n\nThis repository provides a code base for the information integration course in the summer semester of 2022. Below you\ncan find the documentation for setting up the project.\n\n## Prerequisites\n\n- Install [Python 3.10.5](https://www.python.org/downloads/release/python-3105/)\n- Install [Poetry](https://python-poetry.org/docs/#installation)\n- Install [Docker](https://docs.docker.com/get-docker/) and [docker-compose](https://docs.docker.com/compose/install/)\n- Install [Protobuf compiler (protoc)](https://grpc.io/docs/protoc-installation/). If you are using windows you can\n  use [this guide](https://www.geeksforgeeks.org/how-to-install-protocol-buffers-on-windows/)\n- Install [jq](https://stedolan.github.io/jq/download/)\n\n## Architecture\n\n![](architecture-overview.png)\n\n## Components\n\n### RB website\n\nThe [Registerbekanntmachung website](https://www.handelsregisterbekanntmachungen.de/index.php?aktion=suche) contains\nannouncements concerning entries made into the companies, cooperatives, and\npartnerships registers within the electronic information and communication system. You can search for the announcements.\nEach announcement can be requested through the link below. You only need to pass the query parameters `rb_id`\nand `land_abk`. For instance, we chose the state Rheinland-Pfalz `rp` with an announcement id of `56267`, the\nnew entry of the company BioNTech.\n\n```shell\nexport STATE=\"rp\" \nexport RB_ID=\"56267\"\ncurl -X GET  \"https://www.handelsregisterbekanntmachungen.de/skripte/hrb.php?rb_id=$RB_ID\u0026land_abk=$STATE\"\n```\n\n### RB crawler\n\nThe Registerbekanntmachung crawler (rb_crawler) sends a get request to the link above with parameters (`rb_id`\nand `land_abk`) passed to it and extracts the information from the response.\n\nWe use [Protocol buffers](https://developers.google.com/protocol-buffers)\nto define our [schema](proto/bakdata/rb/announcement/v1/announcement.proto).\nThe crawler uses the generated model class (i.e., `Announcement` class) from\nthe [protobuf schema](proto/bakdata/rb/announcement/v1/announcement.proto).\nWe will explain further how you can generate this class using the protobuf compiler.\n\nThe compiler creates an `Announcement` object with the fields defined in the schema.\nThe crawler fills the object fields with the extracted data from the website.\nIt then serializes the `Announcement` object to bytes so that Kafka can read it and produces it to\nthe `rb-announcements`topic.\n\nAfter that, it increments the `rb_id` value and sends another GET request.\nThis process continues until the end of the announcements is reached, and the crawler will stop automatically.\n\n### rb-announcements topic\n\nThe `rb-annourncements` topic holds all the announcements produced by the `rb_crawler`. Each message in a Kafka topic\nconsist of a key and value.\n\nThe key type of this topic is `String`. The `rb_crawler generates the key`. The key\nis a combination of the `land_abk` and the `rb_id`. If we consider the `rb_id` and `land_abk` from the example above,\nthe key will look like this: `rp_56267`.\n\nThe value of the message contains more information like `event_name`, `event_date`, and more. Therefore, the value type\nis complex and needs a schema definition.\n\n### rb-corporates topic\n\nThis topic contains the extracted information about the corporates of the Registerbekanntmachung. The key is a self\ngenerated hash from the corporate name and the value is a\n[complex schema type](proto/bakdata/rb/corporate/v1/corporate.proto).\n\n### rb-persons topic\n\nThis topic contains the extracted information about the persons of the Registerbekanntmachung. The key is a self\ngenerated hash from the source(i.e., rb), firstname, lastname, and corporate name. The value is a\n[complex schema type](proto/bakdata/rb/person/v1/person.proto).\n\n### BaFin website\n\nThe Federal Financial Supervisory Authority [(BaFin)](https://www.bafin.de/EN/Homepage/homepage_node.html) brings\ntogether under one roof the supervision of banks and\nfinancial services providers, insurance undertakings and securities trading.\nThe website also contains registered announcements of managers’ transactions pursuant to Article 19 of the MAR.\nThese announcements describe the transaction details that an executive director (manager) of a company did in the\nstocks. The website holds the transaction information in a one-year time window.\nThe first announcement has a message ID of `18794` at the time of this writing.\nYou can find the message ID on this [page](https://portal.mvp.bafin.de/database/DealingsInfo/sucheForm.do).\n\n### BaFin crawler\n\nThis crawler extracts the information from the BaFin portal and fills the model objects with the extracted data.\nMoreover, it serializes the objects and produces each of them to the desired topic.\n\nThe crawler is initialized with a `message_id` at the beginning of the crawl and sends a request to the portal URL of\nBaFin. This process is demonstrated in the script below:\n\n```shell\nexport MESSAGE_ID=\"18794\"\ncurl -X GET  https://portal.mvp.bafin.de/database/DealingsInfo/ergebnisListe.do?cmd=loadEmittentenAction\u0026meldepflichtigerId=$MEESAGE_ID\n```\n\nAfter retrieving the HTML of the page, the crawler extracts the `BaFin-ID` in the table and\nsends another request to retrieve the detailed transaction information. This is demonstrated with a shell script:\n\n```shell\nexport MESSAGE_ID=\"18794\"\nexport BAFIN_ID=40002082\nhttps://portal.mvp.bafin.de/database/DealingsInfo/transaktionListe.do?cmd=loadTransaktionenAction\u0026emittentBafinId=$BAFIN_ID\u0026meldungId=$MESSAGE_ID\u0026KeepThis=true\u0026TB_iframe=true\u0026modal=true\n```\n\nThe crawler uses the HTML response and extracts the information, and produces them for each topic.\nIt then increases the `message_id` and repeats this process.\n\n### bafin_trades topic\n\nThis topic contains all the transaction information a person made.\nThe key is a self-generated string from the message-id and the BaFin id.\nThe value is a [complex schema type](proto/bakdata/bafin/trade/v1/trade.proto).\n\n### bafin_corporates topic\n\nThe `bafin_corporates` topic contains all the information about a corporate.\nThe key is a self-generated hash from the corporate name.\nThe value is a [complex schema type](proto/bakdata/bafin/corporate/v1/corporate.proto).\n\n### bafin_persons topic\n\nThis topic contains the extracted information about the persons who made a transaction. The key is a self-generated hash from the source (i.e., BaFin), first name, last name, and corporate name. The value is a\n[complex schema type](proto/bakdata/bafin/person/v1/person.proto).\n\n### Kafka Connect\n\n[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) is a tool to move large data sets into\n(source) and out (sink) of Kafka.\nHere we only use the Sink connector, which consumes data from a Kafka topic into a secondary index such as\nElasticsearch.\n\nWe use the [Elasticsearch Sink Connector](https://docs.confluent.io/kafka-connect-elasticsearch/current/overview.html)\nto move the data from the `coporate-events` topic into the Elasticsearch.\n\n## Setup\n\nThis project uses [Poetry](https://python-poetry.org/) as a build tool.\nTo install all the dependencies, just run `poetry install`.\n\nThis project uses Protobuf for serializing and deserializing objects.\nYou can find these schemas under the [`proto`](proto) folder.\nFurthermore, you must generate the Python code for the model class from the proto file.\nTo do so run the [`generate-proto.sh`](./generate-proto.sh) script.\nThis script uses the [Protobuf compiler (protoc)](https://grpc.io/docs/protoc-installation/) to generate the model class\nunder the [`bakdata`](bakdata) folder.\n\n## Run\n\n### Infrastructure\n\nUse `docker-compose up -d` to start all the services: [Zookeeper](https://zookeeper.apache.org/)\n, [Kafka](https://kafka.apache.org/), [Schema\nRegistry](https://docs.confluent.io/platform/current/schema-registry/index.html), [Redpanda Console](https://github.com/redpanda-data/console),\n[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html),\nand [Elasticsearch](https://www.elastic.co/elasticsearch/). Depending on your system, it takes a couple of minutes\nbefore the services are up and running. You can use a tool\nlike [lazydocker](https://github.com/jesseduffield/lazydocker) to check the status of the services.\n\n**NOTE:** Kafka Connect start time for the Apple silicon is more than 5 minutes!\nYou can start using Kafka Connect whenever the status of the container is `running (healthy)`.\n\n### Kafka Connect\n\nAfter all the services are up and running, you need to configure Kafka Connect to use\nthe Elasticsearch or the Neo4j sink connector.\nThe config file is a JSON formatted file. We provided the sink configuration for the different topics under the\n[`connect`](./connect) folder.\n\nYou can find more information about the configuration properties for the Elasticsearch sink on\nthe [official documentation page](https://docs.confluent.io/kafka-connect-elasticsearch/current/overview.html).\nDetails on configuring the Neo4j sink connector are available on the\n[official documentation page](https://neo4j.com/labs/kafka/4.0/kafka-connect/).\n\nTo start the connector, you must push the JSON config file to Kafka. You can use the UI dashboard in Redpanda Console or\nthe [bash script provided](./connect/push-config.sh). It is possible to remove a connector by deleting it\nthrough Redpanda's Console or calling the deletion API in the [bash script provided](./connect/delete-config.sh).\n\n### RB crawler\n\nYou can start the crawler with the command below:\n\n```shell\npoetry run python -m rb_crawler.main --id $RB_ID --state $STATE\n```\n\nThe `--id` option is an integer, which determines the initial event in the Handelsregisterbekanntmachungen to be\ncrawled.\n\nThe `--state` option takes a string (only the ones listed above). This string defines the state where the crawler should\nstart from.\n\nYou can use the `--help` option to see the usage:\n\n```\nUsage: main.py [OPTIONS]\n\nOptions:\n  -i, --id INTEGER                The rb_id to initialize the crawl from\n  -s, --state [bw|by|be|br|hb|hh|he|mv|ni|nw|rp|sl|sn|st|sh|th]\n                                  The state ISO code\n  --help                          Show this message and exit.\n```\n\n### BaFin crawler\n\nYou can start the crawler with the command below:\n\n```shell\npoetry run python -m bafin_crawler.main --id $MESSAGE_ID\n```\n\nThe `--id` option is an integer, which determines the initial event in the BaFin portal to be\ncrawled.\n\nYou can use the `--help` option to see the usage:\n\n```\nUsage: main.py [OPTIONS]\n\nOptions:\n  -i, --id INTEGER  The message_id to initialize the crawl from\n  --help            Show this message and exit.\n```\n\n## Query data\n\n### Redpanda Console\n\n[Redpanda Console](https://github.com/redpanda-data/console) is a web application that helps you manage and debug your Kafka workloads\neffortlessly. You can create, update, and delete Kafka resources like Topics and Kafka Connect configs.\nYou can open Redpanda Console in your browser under http://localhost:8080.\n\n### Elasticsearch\n\nTo query the data from Elasticsearch, you can use\nthe [query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/query-dsl.html) of elastic. For example:\n\n```shell\ncurl -X GET \"localhost:9200/_search?pretty\" -H 'Content-Type: application/json' -d'\n{\n    \"query\": {\n        \"match\": {\n            \u003cfield\u003e\n        }\n    }\n}\n'\n```\n\n`\u003cfield\u003e` is the field you wish to search. For example:\n\n```\n\"first_name\":\"Sussane\"\n```\n\n## Teardown\n\nYou can stop and remove all the resources by running:\n\n```shell\ndocker-compose down\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fhpi-ii-project-2022","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fhpi-ii-project-2022","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fhpi-ii-project-2022/lists"}