{"id":21482858,"url":"https://github.com/ibmstreams/sample.netflow","last_synced_at":"2025-03-17T09:22:54.374Z","repository":{"id":74839420,"uuid":"281410410","full_name":"IBMStreams/sample.netflow","owner":"IBMStreams","description":"Netflow sample","archived":false,"fork":false,"pushed_at":"2020-11-25T13:27:03.000Z","size":42246,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-01-23T18:50:34.786Z","etag":null,"topics":["edge-computing","samples","streaming-processing"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IBMStreams.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-21T13:48:42.000Z","updated_at":"2023-12-05T03:09:52.000Z","dependencies_parsed_at":"2023-07-22T16:30:29.283Z","dependency_job_id":null,"html_url":"https://github.com/IBMStreams/sample.netflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fsample.netflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fsample.netflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fsample.netflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBMStreams%2Fsample.netflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IBMStreams","download_url":"https://codeload.github.com/IBMStreams/sample.netflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244006303,"owners_count":20382443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["edge-computing","samples","streaming-processing"],"created_at":"2024-11-23T12:38:18.216Z","updated_at":"2025-03-17T09:22:54.369Z","avatar_url":"https://github.com/IBMStreams.png","language":"JavaScript","readme":"# sample.netflow\n\nThis repository contains artifacts for 2 different applications [Netflow Map Viewer Application](README.md#netflow-map-viewer-application) and the [Netflow To Database Application](README.md#netflow-to-database-application).\n\nBoth applications simulate the collection of network traffic data from IP routers or L3 switches. \nThe Netflow messages are parsed and the data volume is aggregated. The transferred data volumes per source/destination/protocol-group is transferred \nfrom the collector job to the viewer or store job. \nThe aggregation time interval can be configured. \nThe Netflow Map Viewer Application visualizes the aggregated data, including the geographical locations of source and destination. \nThe Netflow To Database Application stores the aggregated data into a database.\n\n## Netflow Map Viewer Application\n\nThe Netflow Map Viewer Application is a is simple out of the box running demonstration for an application that simulates the collection of network \nstatistics and presents them centrally in a web display. The Netflow Map Viewer Application is composed of 2 jobs; the `NetflowViewerCollector` and the `NetflowViewer`.\n\n### NetflowViewerCollector\n\n* The `NetflowViewerCollector` simulates the collection of Netflow data from Edge devices, makes the first data aggregation and transmits the data to `NetflowViewer`.\n* The connection from `NetflowViewerCollector` to `NetflowViewer` is currently a HTTP Post and requires the Endpoint-Monitor application in Cloud Pak for Data.\n\n### NetflowViewer\n\n* The `NetflowViewer` translates the IP addresses into location data, makes the final data aggregation and provides the “Views” for a Web-Browser.\n* The `NetflowViewer` provides a raw data view, the Location Stream View and the Netflow View.\n* Due to connection limitations (no websocket connections), the Netflow View is currently not available in Cloud Pak for Data.\n\n## Netflow To Database Application\n\nThe Netflow To Database Application is a demonstration for an application that collects network statistics and puts them in \na central database, the Netflow To Database Application is composed of 2 Streams Jobs: `NetflowStore` and `NetflowStoreCollector`.\n\n**Note:** The Netflow To Database Application is still under construction.\n\n### NetflowStoreCollector\n\n* The `NetflowStoreCollector` collects the Netflow data from the router and makes the data aggregation and transmits the data to `NetflowStore`.\n* One or more  `NetflowStoreCollector` job are located in every datacenter.\n* The connection from `NetflowStoreCollector` to `NetflowStore` is currently a HTTP Post and requires the Endpoint-Monitor application on Cloud Pak for Data.\n* The application logic is able to compensate network outages up to a certain amount of time.\n\n### NetflowStore\n\n* The `NetflowStore` pushes the netflow information into a database.\n* The `NetflowStore` provides a raw data view of the pushed data and a web page with some statistics information.\n\n## Requirements\n\nThis application requires the following Streams toolkits:\n\n* com.ibm.streamsx.network version 3.2.1 or higher\n* com.ibm.streamsx.inet version 3.4.0 or higher\n* com.ibm.streamsx.inetserver version 4.3.2 or higher\n\nAdditionally the Netflow To Database Application requires toolkit:\n* com.ibm.streamsx.jdbc version 1.6.0\n\n## How to build\n\n### Build from Command Line\n\nClone the repository or download the source archive.\nMove to the project directories NetflowViewerCollector, NetflowViewer, NetflowStore or NetflowStoreCollector and execute command `make`.\n\nThe build scripts assume that the toolkits are installed in ${STREAMS_INSTALL}/toolkits/\nYou can overwrite the default toolkit locations by setting environments STREAMS_NETWORK_TOOLKIT, STREAMS_INET_TOOLKIT, STREAMS_INETSERVER_TOOLKIT and STREAMS_JDBC_TOOLKIT\n\n### Build with Streams Studio\n\nClone the repository or download the source archive.\nImport all existing projects to your workspace.\nMake sure that all required toolkits are listed in *Streams Explorer*\nRight click the project and select `Build`\n\n### Build the NetflowViewer with VsCode and IBM Streams extension\n\n* Clone the repository or download the source archive.\n* Open the `Explorer View` and right click the workspace and select `Add Folder to Workspace`. Navigate to your project \ndirectory, go to sub-directory `NetflowMapViewer/NetflowViewer` and select `add`.\n* In the `Streams Explorer` connect to a Streams Instance, enter the credentials and refresh the toolkit list.\n* Go back to the `Explorer View`.\n* In the project directory right click the `Makefile` and select `build`.\n\nNow the sab-file is built and loaded to the output directory. If the request times out open the `Extension View`, select \nthe *IBM Streams* extension and open the `Extension Settings`. Increase the variable `Timeout For Requests`.\n\n### Build the NetflowViewerCollector with VsCode and IBM Streams extension\n\nIf you want to run this job in an Micro Edge System omit this step and goto \n[Build the Edge Application Image NetflowViewerCollector with VsCode and IBM Streams extension](README.md#build-the-edge-application-image-netflowviewercollector-with-vscode-and-ibm-streams-extension)\n\n* Clone the repository or download the source archive.\n* Open the `Explorer View` and right click the workspace and select `Add Folder to Workspace`. Navigate to your project \ndirectory, go to sub-directory `NetflowMapViewer/NetflowViewerCollector` and select `add`.\n* Go back to the `Explorer View`.\n* In the project directory right click the `Makefile` and select `build`.\n\nNow the sab-file is built and loaded to the output directory. If the request times out open the `Extension View`, select \nthe *IBM Streams* extension and open the `Extension Settings`. Increase the variable `Timeout For Requests`.\n\n### Build the Edge Application Image NetflowViewerCollector with VsCode and IBM Streams extension\n\nIf you want to run this job in a on premise Streams instance omit this step.\n\n* Clone the repository or download the source archive.\n* Open the `Explorer View` and right click the workspace and select `Add Folder to Workspace`. Navigate to your project \ndirectory, go to sub-directory `NetflowMapViewer/NetflowViewerCollector` and select `add`.\n* Open the file `NetflowViewerCollector` and replace the default value of the `url` parameter according to your needs. see [Url Configuration](README.md#url-configuration)\n* In the `Streams Explorer` connect to a Cloud Pak for Data Streams instances with image build enabled, enter the credentials and refresh the toolkit list.\n* Go back to the `Explorer View`.\n* In the project directory right click the `Makefile` and select `Build Edge Application Image`. Now a editor window `Configure edge application image build for sample.netflow.viewer.NetflowViewerCollector.sab` opens.\n* Select the checkbox `Create a sample file`.\n* Select a base image with a name like *streams-base-edge-application..*\n* Enter the image name: *streams-sample-netflow-collector* and a version tag: *v0.0.1*\n* Select the `Configuration type` advanced and click `Create sample configuration file` and save the configuration file *buildconfig.json*.\n* Open *buildconfig.json* file in an editor and append the line *\"rpms\": [\"libpcap\"]* (don't forget to add the comma in the previous line) and save the changes.\n* Click `Build Image`\n\nNow the edge application image is build an pushed into the Openshift Image registry. \n\nNow you must package the Edge application Image see [Packaging and edge app](https://www.ibm.com/support/knowledgecenter/SSQNUZ_3.0.1/svc-edge/usage-register-app.html)\n\n## Run Netflow Map Viewer Application\n\n### Run the NetflowViewer job\n\nIf you run this sample in an Clod Pack for Data Streams Instance you must install the [Streams Endpoint Monitor](https://github.com/IBMStreams/endpoint-monitor) \nThe [README.md](https://github.com/IBMStreams/endpoint-monitor/blob/develop/README.md) contains the installation instructions.  \n\nThe `NetflowViewer` requires a certain amount of memory for the ip-location database. If you are going to run this job on a Clod Pack for Data Streams instance, you must increase \nthe default Memory limit to 4GiB for the Applocation pod of this instance.  \n\nSubmit first the `NetflowViewer` job and provide the webserver port as submission time parameter. If the port is not provided, the default 6060 is used.  \nIf this application is submitted to a Cloud Pak for Data instance, you should enter a job name for this job. The job name \nis used from the streams-endpoint-monitor to build the path to the web server.\n\n### Run the NetflowViewerCollector job\n\nNow you can start the NetflowViewerCollector job.\n\nIf you are using an on premise Streams instance, submit the `NetflowViewerCollector` job and provide the url of your `NetflowViewer` job as submission time parameter.\n\nTo start the NetflowViewerCollector on an Micro Edge System, use the packaged edge application image from \n[Build the Edge Application Image NetflowViewerCollector with VsCode and IBM Streams extension](README.md#Build-the-edge-application-image-netflowViewerCollector-with-vscode-and-ibm-streams-extension) and \nfollow the steps in [Deploying and edge app](https://www.ibm.com/support/knowledgecenter/SSQNUZ_3.0.1/svc-edge/usage-deploy.html)\n\n### Url Configuration\n\nThe parameter url of the `NetflowViewerCollector` is:\n\n* OnPrem              http://\\\u003chostname\\\u003e:\\\u003cwebserver port\\\u003e/InjectedTuples/ports/output/0/inject\n* Cloud Pak for Data  https://\\\u003cexposed route of streams-endpoint-monitor\\\u003e/\\\u003cjob name\\\u003e/InjectedTuples/ports/output/0/inject\n\nIf the `NetflowCollector` job has no job name you must substitute \\\u003cjob name\\\u003e with streams/jobs/\\\u003cjobid\\\u003e\n\nTo view the Location Stream View open with your browser the url:\n\n* OnPrem             http://\\\u003chostname\\\u003e:\\\u003cwebserver port\\\u003e/LocationStreamView/ports/input/0/tuples\n* Cloud Pak for Data https://\\\u003cexposed route of streams-endpoint-monitor\\\u003e/\\\u003cjob name\\\u003e/LocationStreamView/ports/input/0/tuples\n\nTo view the Netflow visualization open with your browser the url:\n\n* OnPrem             http://\\\u003chostname\\\u003e:\\\u003cwebserver port\\\u003e/NetflowViewer\n* Cloud Pak for Data not available\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibmstreams%2Fsample.netflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibmstreams%2Fsample.netflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibmstreams%2Fsample.netflow/lists"}