{"id":13815995,"url":"https://github.com/apache/metron","last_synced_at":"2025-04-28T10:05:36.000Z","repository":{"id":49742460,"uuid":"47607561","full_name":"apache/metron","owner":"apache","description":"Apache Metron","archived":true,"fork":false,"pushed_at":"2020-08-08T01:43:32.000Z","size":97572,"stargazers_count":843,"open_issues_count":24,"forks_count":511,"subscribers_count":103,"default_branch":"master","last_synced_at":"2024-05-22T08:13:01.028Z","etag":null,"topics":["metron"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-08T08:00:05.000Z","updated_at":"2024-05-16T17:13:21.000Z","dependencies_parsed_at":"2022-08-12T20:21:56.437Z","dependency_job_id":null,"html_url":"https://github.com/apache/metron","commit_stats":null,"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fmetron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fmetron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fmetron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fmetron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/metron/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225349549,"owners_count":17460402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["metron"],"created_at":"2024-08-04T05:00:30.437Z","updated_at":"2024-11-19T12:31:15.574Z","avatar_url":"https://github.com/apache.png","language":"Java","funding_links":[],"categories":["Java","大数据"],"sub_categories":[],"readme":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n[![Build Status](https://travis-ci.org/apache/metron.svg?branch=master)](https://travis-ci.org/apache/metron)\n\n# Apache Metron\n\nMetron integrates a variety of open source big data technologies in order\nto offer a centralized tool for security monitoring and analysis. Metron\nprovides capabilities for log aggregation, full packet capture indexing,\nstorage, advanced behavioral analytics and data enrichment, while applying\nthe most current threat intelligence information to security telemetry\nwithin a single platform.\n\nFor the latest information, please visit our website at http://metron.apache.org/\n\nMetron can be divided into 4 areas:\n\n1. **A mechanism to capture, store, and normalize any type of security\ntelemetry at extremely high rates.**  Because security telemetry is constantly\nbeing generated, it requires a method for ingesting the data at high speeds\nand pushing it to various processing units for advanced computation and analytics.  \n\n2. **Real time processing and application of enrichments** such as threat\nintelligence, geolocation, and DNS information to telemetry being collected.\nThe immediate application of this information to incoming telemetry provides\nthe context and situational awareness, as well as the who and where\ninformation critical for investigation\n\n3. **Efficient information storage** based on how the information will be used:\n   - Logs and telemetry are stored such that they can be efficiently mined and\nanalyzed for concise security visibility\n   - The ability to extract and reconstruct full packets helps an analyst answer\nquestions such as who the true attacker was, what data was leaked, and where\nthat data was sent\n   - Long-term storage not only increases visibility over time, but also enables\nadvanced analytics such as machine learning techniques to be used to create\nmodels on the information.  Incoming data can then be scored against these\nstored models for advanced anomaly detection.  \n\n4. **An interface that gives a security investigator a centralized view of data\nand alerts passed through the system.**  Metron’s interface presents alert\nsummaries with threat intelligence and enrichment data specific to that alert\non one single page.  Furthermore, advanced search capabilities and full packet\nextraction tools are presented to the analyst for investigation without the\nneed to pivot into additional tools.   \n\nBig data is a natural fit for powerful security analytics. The Metron\nframework integrates a number of elements from the Hadoop ecosystem to provide\na scalable platform for security analytics, incorporating such functionality as\nfull-packet capture, stream processing, batch processing, real-time search, and\ntelemetry aggregation.  With Metron, our goal is to tie big data into security\nanalytics and drive towards an extensible centralized platform to effectively\nenable rapid detection and rapid response for advanced security threats.  \n\n# Obtaining Metron\n\nTo obtain a release of Metron, please visit http://metron.apache.org/documentation/#releases\n\nThis repository is a collection of submodules for convenience which is regularly\nupdated to point to the latest versions. Github provides multiple ways to obtain\nMetron's code:\n\n1. git clone --recursive https://github.com/apache/metron\n2. [Download ZIP](https://github.com/apache/metron/archive/master.zip)\n3. Clone or download each repository individually\n\nOption 3 is more likely to have the latest code.\n\n# Getting Started\n\nTo start exploring the capabilities of Apache Metron [follow these instructions to launch Metron in a single-node VM](metron-deployment/development/centos6) on your own hardware.  \n\n# Building Metron\n\nBuild the full project and run tests:\n```\n$ mvn clean install\n```\n\nBuild without tests:\n```\n$ mvn clean install -DskipTests\n```\n\nBuild with the HDP profile:\n```\n$ mvn clean install -PHDP-2.5.0.0\n```\n\nYou can swap \"install\" for \"package\" in the commands above if you don't want to deploy the artifacts to your local .m2 repo.\n\n\n# Build Metron Reporting\n\nTo build and run reporting with code coverage:\n```\n$ mvn clean install\n$ mvn site site:stage-deploy site:deploy\n```\n\nCode coverage can be skipped by skipping tests:\n```\n$ mvn clean install -DskipTests site site:stage-deploy site:deploy\n```\n\nThe staged site is deployed to /tmp/metron/site/index.html, and can be viewed by opening the file in a browser.\n\n## Building with Docker\n\nA Docker container with all the required software, with the proper versions, is available to be used as well.\nsee [ansible-docker](metron-deployment/packaging/docker/ansible-docker)\n\n# Navigating the Architecture\n\nMetron is at its core a Kappa architecture with Apache Storm as the processing\ncomponent and Apache Kafka as the unified data bus.\n\nSome high level links to the relevant subparts of the architecture, for\nmore information:\n* [Parsers](metron-platform/metron-parsing) : Parsing data from kafka into the Metron data model and passing it downstream to Enrichment.\n* [Enrichment](metron-platform/metron-enrichment) : Enriching data post-parsing and providing the ability to tag a message as an alert and assign a risk triage level via a custom rule language.\n* [Indexing](metron-platform/metron-indexing) : Indexing the data post-enrichment into HDFS, Elasticsearch or Solr.\n\nSome useful utilities that cross all of these parts of the architecture:\n* [Stellar](metron-platform/metron-common) : A custom data transformation language that is used throughout metron from simple field transformation to expressing triage rules.\n* [Model as a Service](metron-analytics/metron-maas-service) : A Yarn application which can deploy machine learning and statistical models onto the cluster along with the associated Stellar functions to be able to call out to them in a scalable manner.\n* [Data management](metron-platform/metron-data-management) : A set of data management utilities aimed at getting data into HBase in a format which will allow data flowing through metron to be enriched with the results.  Contains integrations with threat intelligence feeds exposed via TAXII as well as simple flat file structures.\n* [Profiler](metron-analytics/metron-profiler-common) : A feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.\n\n# Notes on Adding a New Sensor\nIn order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x,\nit is necessary to add an additional field to the templates and mapping for existing sensors.\n\nPlease see a description of the steps necessary to make this change in metron-elasticsearch [Using Metron with Elasticsearch 5.6](./metron-platform/metron-elasticsearch/metron-elasticsearch-common#using-metron-with-elasticsearch-56)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fmetron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fmetron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fmetron/lists"}