{"id":20885379,"url":"https://github.com/lab11/gatd","last_synced_at":"2025-05-12T19:30:51.960Z","repository":{"id":145294040,"uuid":"8766817","full_name":"lab11/gatd","owner":"lab11","description":"Versatile data collection service that supports real time streaming and data archival.","archived":false,"fork":false,"pushed_at":"2015-10-06T19:38:28.000Z","size":5072,"stargazers_count":5,"open_issues_count":13,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-14T04:45:08.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lab11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-14T02:58:30.000Z","updated_at":"2024-04-14T04:45:08.823Z","dependencies_parsed_at":"2023-04-09T03:47:04.746Z","dependency_job_id":null,"html_url":"https://github.com/lab11/gatd","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab11%2Fgatd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab11%2Fgatd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab11%2Fgatd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab11%2Fgatd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lab11","download_url":"https://codeload.github.com/lab11/gatd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225148872,"owners_count":17428430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T08:13:19.960Z","updated_at":"2024-11-18T08:13:20.813Z","avatar_url":"https://github.com/lab11.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"GATD: Get All The Data\n======================\n\nGATD is a cloud based system for managing and storing data streams. It was\nborn out of a need to handle data generated by disparate sensors with varying\ndata types, transmission protocols, and end-use goals.\n\nGATD has three major design goals:\n\n1. *Modularity*. GATD is a relatively loose collection of modules connected with\ninfinite length queues and a database layer. Each module is part of a certain\nblock of the system and many modules can exist for the same block. For example,\nin the receiver block there is one module that listens for UDP packets and\nanother module that listens for HTTP requests. This allows GATD to be trivially\nextended as functionality changes and new sensors come online.\n\n2. *Flexibility*. GATD makes virtually no assumptions about the format, type,\nor content of any data coming into the system. The exclusive requirement is\nthat a sensor must be able to identify its data stream to the system so it\ncan be processed properly. Each data stream has a custom parser that knows\nhow to make sense of its own data. The parser simply returns key,value pairs\nwith no restrictions on the key names or value types. GATD is designed to adapt\nto the sensors, and not vice-versa.\n\n3. *Timeliness*. GATD is specifically designed to support real-time streaming\napplications where data comes in as it is generated and is sent out\nto interested clients immediately. Every component is optimized for this\nworkflow. Additionally, all data is stored and can be retrieved and processed\nlater if necessary.\n\n\nStructure\n---------\n\ndiagram\n\nThe major blocks of GATD are as follows:\n\n- *Receiver*. Responsible for accepting data from any sensors. Records\nall relevant metadata with the data before passing it all to the formatter.\n\n- *Formatter*. The formatter is a stateless block that converts raw data\nfrom sensors into key,value pairs. The formatter calls the appropriate\nparser to interpret the raw data before storing them in a database and\npassing them on to any streamers.\n\n- *Streamer*. The streamer block sends data to any interested clients.\nClients register a query with a streamer and any matching packets are sent\nto the client.\n\n\nImplementation\n--------------\n\nThe current version of GATD is a research oriented implementation designed\nfor speed of development and experimentability rather than performance.\nMost modules are written in Python, although due to the loose, modular approach\nsome are written in Node.js and C as well.\n\nGATD uses RabbitMQ for the inter-module queues and MongoDB for data storage.\n\n### Requirements\n\n- Python 2.7.*\n- MongoDB\n- RabbitMQ\n- Node.js\n- tup\n\n\n\nInstallation\n------------\n\n### Ubuntu / RHEL\n\n1. Install [MongoDB](http://docs.mongodb.org/manual/installation/) and\n[RabbitMQ Server](http://www.rabbitmq.com/download.html).\n\n2. Install dependencies\n\n    ```\n    sudo apt-get install python-pip git python-dev screen\n       --- or ---\n    sudo yum install python-pip git python-devel screen\n       --- or ---\n    sudo port install py27-pip git-core\n    ```\n    \n2. Setup user and checkout gatd. You will also want to add yourself to the\n`gatd` group and then log out and back in. Probably can skip this step on Mac.\n\n    ```\n    sudo adduser gatd\n    cd /opt\n    sudo git clone https://github.com/lab11/gatd.git\n    sudo chown gatd:gatd gatd -R\n    sudo chmod -R g+w gatd\n    sudo usermod -a -G gatd \u003cusername\u003e\n    ```\n    \n2. Copy the example GATD config file and set the necessary values.\nYou will want to make sure any passwords set\nin the next steps are reflected in this file.\n\n    ```\n    cd /opt/gatd/config\n    cp gatd.config.example gatd.config\n    ```\n\n2. Configure MongoDB using the template config file in the `mongo` folder.\n  1. Copy the config file to `/etc/mongodb.conf`.\n\n        ```\n        sudo cp /opt/gatd/mongo/mongodb.conf /etc/mongodb.conf\n        ```\n\n  2. Edit the config file with the port you want to use.\n  3. Create a directory for the database.\n\n        ```\n        sudo mkdir -p /data/mongodb\n        sudo chown mongodb:mongodb /data/mongodb\n        ```\n        \n  4. Restart the MongoDB daemon.\n\n        ```\n        sudo service mongod restart\n        ```\n\n  5. Add the gatd user to the Mongo database\n\n        ```\n        mongo --port \u003cmongo db port\u003e\n        use getallthedata\n        db.createUser({\n            user: \"reportsUser\",\n            pwd: \"12345678\",\n            roles: [\n                     { role: \"dbAdmin\", db: \"getallthedata\" }\n                   ]\n            }\n        )\n        ```\n\n3. Configure RabbitMQ using the config files in the `rabbitmq` folder.\n  1. Copy the config files to `/etc/rabbitmq`.\n\n        ```\n        sudo cp /opt/gatd/rabbitmq/rabbitmq* /etc/rabbitmq\n        ```\n  \n  2. Edit `rabbitmq-gatd.config` with the port you want to use.\n  3. Restart the rabbitmq server.\n  \n        ```\n        sudo rabbitmqctl stop\n        sudo service rabbitmq-server start\n        ```\n\n  4. Delete the default rabbitmq user, create a GATD user, and set permissions.\n  \n        ```\n        sudo rabbitmqctl delete_user guest\n        sudo rabbitmqctl add_user gatd \u003cpassword\u003e\n        sudo rabbitmqctl set_user_tags gatd administrator\n        sudo rabbitmqctl set_permissions -p / gatd \".*\" \".*\" \".*\"\n        ```\n\n\n4. Set up Python environment.\n\n    ```bash\n    sudo pip2 install virtualenv\n    cd /opt/gatd\n    virtualenv .\n    source ./bin/activate\n    pip2 install -r requirements.pip\n    ```\n\n9. Setup the database in MongoDB.\n\n    ```\n    cd /opt/gatd/mongo\n    ./init_mongo.py\n    ````\n    \n10. Run GATD\n  1. Start the receivers.\n  \n        ```\n        cd /opt/gatd/receiver\n        ./run_receiver.sh\n        ```\n\n  2. Run the formatter.\n    \n        ```\n        cd /opt/gatd/formatter\n        ./run_formatter.sh\n        ```\n\n  3. Run the streamers.\n   \n        ```\n        cd /opt/gatd/streamer\n        ./run_streamer.sh\n        ```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flab11%2Fgatd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flab11%2Fgatd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flab11%2Fgatd/lists"}