{"id":13488864,"url":"https://github.com/hashmapinc/TempusDevEnvironment","last_synced_at":"2025-03-28T02:31:13.391Z","repository":{"id":87033790,"uuid":"105096160","full_name":"hashmapinc/TempusDevEnvironment","owner":"hashmapinc","description":"The Tempus Development Environment","archived":false,"fork":false,"pushed_at":"2019-03-13T11:28:27.000Z","size":476208,"stargazers_count":2,"open_issues_count":9,"forks_count":1,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-10-31T01:34:51.397Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hashmapinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-09-28T03:07:13.000Z","updated_at":"2019-10-22T12:21:10.000Z","dependencies_parsed_at":"2023-05-30T07:30:48.829Z","dependency_job_id":null,"html_url":"https://github.com/hashmapinc/TempusDevEnvironment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashmapinc%2FTempusDevEnvironment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashmapinc%2FTempusDevEnvironment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashmapinc%2FTempusDevEnvironment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hashmapinc%2FTempusDevEnvironment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hashmapinc","download_url":"https://codeload.github.com/hashmapinc/TempusDevEnvironment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245957652,"owners_count":20700312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:01:23.173Z","updated_at":"2025-03-28T02:31:13.068Z","avatar_url":"https://github.com/hashmapinc.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/images/tempus/Tempus_Logo_Black_with_TagLine.png\" width=\"950\" height=\"245\" alt=\"Hashmap, Inc Tempus\"/\u003e\n[![License](http://img.shields.io/:license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n\n# Tempus Development Environment\n\nThis environment is meant to be used to develop and test Tempus applications and the Tempus framework.\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/images/tempus/Tempus_High_Level.png\" alt=\"Tempus\"/\u003e\n\nThe intention is to provide an IoT, Streaming Analytics, and Storage Platform on top of Big Data. The solution is based on the following components.\n- Storage\n    * HBase\n    * Phoenix\n- Processing\n    * Spark\n- Ingest\n    * NiFi\n    * Thingsboard\n    * Kafka\n- Visualization\n    * Thingsboard\n- Security\n    * LDAP (Optional)\n\nThis readme will take you through how to setup the development environment for development, and run a sample end-to-end application.\n\n## Table of Contents\n\n- [Requirements](#requirements)\n- [Getting Started](#getting-started)\n- [Usage](#usage)\n\n## Requirements\n\n* JDK 1.8 at a minimum\n* Maven 3.1 or newer\n* Git 2.7 or higher\n* Docker (with compose)\n* A machine with 16 GB of RAM and preferably quad core\n* Preferred OS is MacOS or Linux\n\n### Configuring Docker\n\nMake sure Docker has 8 GB of RAM and 4 cores allocated to it. Docker commands require use of 'sudo'. To give yourself unrestricted access to docker, use this command:\n\n    sudo usermod -a -G docker $USER\n\n## Getting Started\n\nTo get started we must clone the TempusDevEnvironment repo from github:\n\n        git clone https://github.com/hashmapinc/TempusDevEnvionment\n\nIn adiition, we also need following 3 github repositories to setup our dev environment:\n   \n        git clone https://github.com/hashmapinc/HashmapAnalyticsFramework\n        git clone https://github.com/hashmapinc/Tempus -b dev --clone dev branch\n        git clone https://github.com/hashmapinc/nifi-simulator-bundle\n\n*Note: On a slow connection, this might take a while*\n\nCreate a directory say 'data' in your local machine and create following subdirectories under it: \n\n- cassandra\n- hsqldb\n- kafka\n- postgres\n- spark\n- ldap\n- nifi\n\nUnder nifi create following sub-directories:\n    \n- content\n- db\n- flowfile\n- logs\n- provenance\n\nThese directories will hold the data from various Tempus processes. However, these processes will execute within docker containers, so we must point tell docker to point them to these physical directories. These locations will persist data even when the docker containers are destroyed (i.e. persistent storage).\n\nOnce the clones are completed change directory into the **TempusDevEnvironment** directory\n\n        cd TempusDevEnvironment\n\n### Configure the environment\n\nAll Tempus environment variables are stored in .env file.\n\n\nEdit the .env file and point the environment variables to their respective directories using full path. For example:\n\nNIFI_DATA_DIR=/home/ubuntu/data/nifi\n\nNext you will want to configure the Makefile so that the PROJECT_DIR variables are pointing to the location of the parent POM file in the cloned local Tempus \u0026 nifi-simulator-bundle repositories. For example\n\nPROJECT_DIR := /home/ubuntu/Tempus\n\nSIM_PROJECT_DIR := /home/ubuntu/nifi-simulator-bundle\n\nYou also need to point following variables to HashmapAnalyticsFrameowrk directory on your machine. For example:\n\nAPI_DISCOVERY_DIR := /home/ubuntu/HashmapAnalyticsFramework/api-discovery\n\nIDENTITY_SERVICE_DIR := /home/ubuntu/HashmapAnalyticsFramework/identity-service\n\nGo to Tempus root directory, where pom.xml is present and run 'mvn validate'. After you see the 'BUILD SUCCESS' message return to\nTempusDevEnvironment root directory\n\nCompile Tempus and nifi-simulator bundle source and build the docker images (*Note: This WILL take a long time, as NiFi is almost 1 GB*)\n\n        make all\n\nAt this point there will be a lot of information scrolling across the screen as the logs from each container will be displayed. The\ncontainer creation process may take between 1-2.5 hours depending upon the wifi speed. Once up the following containers will have been created:\n\n- NiFi (http://localhost:9090/nifi/)\n- Thingsboard (tb) (http://localhost:8080/)\n- Zookeeper (zk)\n- Kafka\n- Spark (http://localhost:8181)\n- Postgres (storage for data in development environments, not for production use)\n- cassandra\n- identity_service\n- api_discovery\n\nYou can use 'docker ps' command to verify this anytime.\n\nTempus Development Environment is automatically built with Test Data. So you can run demos directly without further setup.\n\n### Enabling LDAP Security - (Please skip this Step for now.)\n\nThe default installation with 'make all' doesn't use LDAP security. However, It can be changed to use LDAP server for authentication and thingsboard to authorize the user based on the authentication.\n\nTo enable LDAP authentication change the value of flag 'LDAP_AUTHENTICATION_ENABLED' to value 'true' in 'tb.env' and run the command 'make all-ldap' instead of 'make all' for installation.\n\nThis will also bring up the docker containers with 'openldap' and a web interface to access the LDAP server.\n\nThe openldap server can be accessed via a web interface in browser on url 'http://localhost:9080' admin credentials are -\n\nLogin DN: cn=admin,dc=example,dc=org\nPassword: admin\n\nWhen the environment is Up, The user which should be authenticated needs to be created in LDAP server. It can be done by importing a ldif file in following format on web interface at \"http://localhost:9080\".\n\ndn: uid=tenant@thingsboard.org,dc=example,dc=org\nobjectclass: account\nobjectclass: simpleSecurityObject\nobjectclass: top\nuid: tenant@thingsboard.org\nuserpassword: tenant\n\n\n### Docker Compose Controls\n\nNote: if you would like to bring up the environment in non-interactive mode use the following command from TempusDevEnvironment root directory:\n\n    docker-compose up -d\n\nTo stop the containers execute the following command\n\n    docker-compose stop\n\nIf no changes have been made you can executed the following command to bring up containers that were already created with either\nthe up or the build command\n\n    docker-compose start\n\nThe NiFi installation will also come preloaded wtih the Hashmap Nifi-Simulator-Bundle and the associated flow to Thingsboard.\n\nAt this point the environment is up and running. Now we will build a sample flow.\n\n## Usage\n\n### Add a device in Tempus\nOpen Tempus UI by navigating to http://localhost:8080 using your browser. The default user name and password is as follows:\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/login.png\" alt=\"Tempus\"/\u003e\n\n- System Admin User/Pass: sysadmin@hashmapinc.com/sysadmin)\n- Tenant Admin User/Pass: demo@hashmapinc.com/tenant\n- Tempus User User/Pass: bob.jones@hashmapinc.com/driller\n\nOnce logged in click on DEVICES in the left-hand menu and select Tank 123.'Tank 123' and 'Tank 456' test devices have already been created for demo purpose and you should use them.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/Tank123.png\" alt=\"Tempus\"/\u003e\n\n Under Details, click on Manage Credentials, and copy the access token.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/CopyAccessToken.png\" alt=\"Tempus\"/\u003e\n\n### Setup the flow in NiFi\n\nNavigate to NiFi (http://localhost:9090/nifi). A flow called nifi should already be created. Make sure all processors are stopped by clicking the stop icon in the Operate panel. You should see three flows - Tank 123, Tank 456 and OPC.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/Nifi-Processes.png\" alt=\"Tempus\"/\u003e\n\nDouble-click on Publish MQTT that is connected to the GenerateTimeSeriesFlowFile processor. Click on Configure. Click on the properties tab, and for User name enter the access token that was copied above. All of the other options should remain the same. (Note that in order to pass the validation of the processor properties, the password is simply a space character. It is not actually used).\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/processor%20configuration.png\" alt=\"Tempus\"/\u003e\n\nClick Apply to close the properties window. Do the same thing for the PublishMQTT process connected to the GenerateFlowFile processor.\n\nStart all the processors by clicking the play button in the Operate Panel.\n\nGo back to Tempus UI and go to the devices again by clicking on Devices in the left hand menu and clicking on Test Device. Click on\nLatest Telemetry to ensure you are receiving data (it should be refreshing approximately once per second). This data is coming from the flow that contains the GenerateTimeSeriesFlowFile processor.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/Test%20Device%20Telemetry.png\" alt=\"Tempus\"/\u003e\n\nClick on Attributes and ensure that there are 2 attributes (Attn and waterTankLevel). This data comes from the JSON message that is in the GenerateFlowFile processor.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/TelemetryData.png\" alt=\"Tempus\"/\u003e\n\nAt this point we have now successfully started transmitting data to Tempus from NiFi via MQTT.\n\nRepeat the same steps for device 'Tank 456'.\n\n### Setup the connection to Kafka - (This has already been done and you should see Kafka Plug In)\n\nThe connection to Kafka is done via activating a plugin in thingsboard. Click on Plugins in the left-hand menu bar and click on the orange + in the lower right-hand corner to add a plugin. Click on the up arrow to import a plugin. In the /tb directory from where this repo was cloned, drag the file called kafka_plugin_for_spark_streaming_sample.json into the dashed box. Once it is done you should see the file name below the dashed box. Click Import.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/KafkaPlugIn.png\" alt=\"Tempus\"/\u003e\n\nYou should now see the Kafka Plugin for Spark Streaming Sample in a Suspended state. This plugin is where the Kafka producer would be confgured, but the configuration is already done. To view it, you can click on the plugin. On the main Plugin page click the play icon on the Kafka Plugin for Spark Streaming Sample to start the plugin. It should now say active. At this point all we have done is configured a kafka publisher from Thingsboard.\n\nKafka Plug In will publish the data to kafka cluster running in Kafka docker image under a topic called water-tank-level-data.\n\nTo see the data in Kafka, run\n\n    docker ps\n\nYou should see a list of containers running on your machine. Locate the Kafka container and copy its id to clipboard. Then rn following command to enter the docker image:\n\n    docker exec -t -i \u003cimage\u003e bash\n\nYou should now see the # prompt of Kafka image. Go to /opt/kafka/bin directory and run following command to see the topic list:\n\n    ./kafka-topics.sh --zookeeper zk:2181 --list\n\nThis should list water-tank-level-data topic. To see the data being published under this topic, run a Kafka console consumer with following command\n\n    ./kafka-console-consumer.sh --zookeeper zk:2181 --topic water-tank-level-data\n\nYou should see data being displayed as it is published. If you stop the flow in Nifi, this data will stop.\n\nRepeat the same steps for Tank 456. At this stage, you have successfully setup a MQTT device to monitor water level of Tanks, publish it to Thingsboard gateway, apply a rule (value is not null), which in turn calls Kafka Plug In action, which starts a Kafka Publisher and publishes the data to Kafka under specified topic.\n\nYou can view the details of Water Tank Telemetry Rule to understand functionality of Thingsboard Rule Engine.\n\n\u003cimg src=\"https://github.com/hashmapinc/hashmap.github.io/blob/master/devenv/WaterTankTelemetryRule.png\" alt=\"Tempus\"/\u003e\n\n### Building and Provisioning an App\n\nAt this point we will build a simple spark application that will read data from kafka and perform some computations on it and send the results back to thingsboard. For this we have created a set of demo applications and bundled them into project tempus-extensions\n\nTo explore these modules further, we will need to perform the following steps.\n\nGo to a directory where you can clone the Spark application and clone it with the following command:\n\n    git clone https://github.com/hashmapinc/tempus-extensions\n\nEnter the directory that was just cloned and build the application\n\n    cd tempus-extensions\n    mvn clean package\n\nBuild should create multiple jar files - one of each project, which is located within the target of the respective project. For example, we will run following two applications and their jars are located at\n\n    tempus-rateofchange/ratechange/target/ratechange-0.0.1-SNAPSHOT.jar\n    tempus-WaterLevelAggregator/target/WaterLevelAggregator-0.0.1-SNAPSHOT.jar\n\nTo run a spark application, list all running containers with\n\n    docker ps\n\nLocate Spark container and copy its id to clipboard. Then run following command to copy the jar files to Spark container:\n\n    docker cp tempus-rateofchange/ratechange/target/ratechange-0.0.1-SNAPSHOT.jar \u003cimage\u003e:/usr/livy-server-0.3.0\n    docker cp tempus-WaterLevelAggregator/target/WaterLevelAggregator-0.0.1-SNAPSHOT.jar \u003cimage\u003e:/usr/livy-server-0.3.0\n\nNow got to Spark container\n\n    docker exec -t -i \u003cimage\u003e bash\n\n#### Submitting the Spark App\n\nNow we are ready to run the Apps. To run WaterLevelAggregator, use following command:\n\nspark-submit --class com.hashmapinc.tempus.spark.WaterLevelAggregator WaterLevelAggregator-0.0.1-SNAPSHOT.jar -kafka kafka:9092 -mqttbroker tcp://tb:1883 -token GATEWAY_ACCESS_TOKEN -topic water-tank-level-data -window 10000\n\nThe parameters are as follows:\n- Kafka - the location of the broker\n- mqttbroker - the thingsboard MQTT broker\n- token the token copied from above from the Spark Gateway\n- topic - the topic to read from on kafka (configured in the Rule)\n- window - batching period, the window to perform aggregations\n\nthe --class parameter is the entrypoint to the spark application.\n\nThis App reads the Water Tank level from Kafka, averages it for a period for each tank and pushes the average level for each tank back in Thingsboard under the same device. You should be able to see the average values under Device Telemetry tab. (As an exercise, try to change this to publish avearge values under a different device name).\n\nSimilarly to run Rate of Change App, use following command:\n\nspark-submit --class com.hashmap.tempus.WaterLevelPredictor ratechange-0.0.1-SNAPSHOT.jar tcp://tb:1883 kafka:9092 water-tank-level-data 70.0 10000 GATEWAY_ACCESS_TOKEN DEBUG=INFO\n\nThis application reads the water tank level from Kafka, computes the time it will take it to reach specified level (70% in our case) and publishes that back to device telemetry in Thingsboard, which can be mapped to a Dashboard.\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashmapinc%2FTempusDevEnvironment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhashmapinc%2FTempusDevEnvironment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhashmapinc%2FTempusDevEnvironment/lists"}