{"id":18927772,"url":"https://github.com/observertc/observer-service","last_synced_at":"2025-04-15T13:34:19.090Z","repository":{"id":37423067,"uuid":"266331336","full_name":"ObserveRTC/observer-service","owner":"ObserveRTC","description":"Microservice to monitor and analyze WebRTC stacks","archived":false,"fork":false,"pushed_at":"2023-03-23T10:24:04.000Z","size":5231,"stargazers_count":36,"open_issues_count":6,"forks_count":5,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-28T22:11:37.777Z","etag":null,"topics":["analytics","webrtc","webrtc-observer"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ObserveRTC.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-23T12:24:29.000Z","updated_at":"2024-12-10T09:48:11.000Z","dependencies_parsed_at":"2024-11-08T11:33:23.893Z","dependency_job_id":null,"html_url":"https://github.com/ObserveRTC/observer-service","commit_stats":null,"previous_names":["observertc/observer-service","observertc/observer"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObserveRTC%2Fobserver-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObserveRTC%2Fobserver-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObserveRTC%2Fobserver-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ObserveRTC%2Fobserver-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ObserveRTC","download_url":"https://codeload.github.com/ObserveRTC/observer-service/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249080663,"owners_count":21209558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","webrtc","webrtc-observer"],"created_at":"2024-11-08T11:20:27.991Z","updated_at":"2025-04-15T13:34:19.079Z","avatar_url":"https://github.com/ObserveRTC.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"WebRTC-Observer\n==\n\n![Build](https://github.com/ObserveRTC/observer/actions/workflows/build.yml/badge.svg)\n![Test](https://github.com/ObserveRTC/observer/actions/workflows/test.yml/badge.svg)\n\nObserver is a server-side service for monitoring WebRTC applications. You can run it in a docker container with a command `docker run observertc/observer`. \nThe service by default is listening on the port `7080`.\n\nTo see ObserveRTC integrations and examples, check out the [full stack examples repository](https://github.com/ObserveRTC/full-stack-examples).\n\n### Table of Contents:\n * [Overview](#overview)\n * [Scalability](#scalability)\n * [Configurations](#configurations)\n   * [Micronaut configs](#micronaut-configs)\n   * [Management endpoints configs](#management-endpoints-configs)\n   * [Observer configs](#observer-configs)\n   * [Hamok configs](#hamok-configs)\n * [Sinks](#sinks)\n   * [LoggerSink](#loggersink)\n   * [Apache Kafka](#apache-kafka)\n   * [Mongo Database](#mongo-database)\n   * [AWS Firehose](#aws-firehose-sink)\n   * [AWS S3 Bucket](#aws-s3-bucket-sink)\n   * [AWS Credential Settings](#aws-credential-settings)\n * [Performance considerations](#performance-considerations)\n * [Compatibility](#compatibility)\n * [Getting Involved](#getting-involved)\n * [Changelog](#changelog)\n * [License](#license)\n\n\n## Overview\n\n![Overview](docs/images/superficial-overview.png)\n\nObserver is a server-side service for monitoring WebRTC applications. The service receives [Samples](https://observertc.org/docs/overview/schemas/#samples)\nfrom [Monitors](https://observertc.org/docs/overview/monitors/). Based on the received samples, the observer:\n\n * Identify calls\n * Create events (call started, call ended, client joined, client left, etc.)\n * Match clients' inbound-rtp sessions with remote clients' outbound-rtp sessions\n * Match SFUs RTP sessions with browser-side clients' in-, and outbound-rtp sessions.\n\n\n### Receiving Samples\n\nThe observer accepts Samples on the following endpoints:\n\n* **REST**: `http(s)://${OBSERVER_ADDRESS}:{PORT}/rest/samples/{serviceId}/{mediaUnitId}`\n* **Websocket**: `ws(s)://${OBSERVER_ADDRESS}:{PORT}/samples/{serviceId}/{mediaUnitId}`\n\n### Sending Reports\n\nThe observer create [Reports](https://observertc.org/docs/overview/schemas/#reports) based on the obtained events, matched sessions and received samples.\nThe reports are forwarded to [Sinks](#sinks).  Currently, the following type of sinks are supported:\n* [KafkaSink](#kafkasink): Apache Kafka Integration\n* [MongoSink](#mongosink): Mongo Database integration\n* [FirehoseSink](#firehosesink): AWS Firehose integration\n* [AwsS3Sink](#awss3sink): AWS S3 integration\n* [WebsocketSink](#websocketsink): Websocket Sink\n\n### Scalability\n\nThe service uses a distributed in-memory object storage [hamok](https://github.com/balazskreith/hamok) as its base to store information about calls while \nclients and SFUs are sending samples. If the service need to be scaled horizontally you need to setup\nhamok in your configuration. Additionally, you can take a look at the section   \n[hamok configurations](#hamok-configs) to have more information about the built in discovery and communication endpoint features.\n\nSome useful tips you can find how to design your service in terms of traffic at the [performance consideration](#performance-considerations) section.\n\n## Configurations\n\nAt startup time the application fetches the configuration and sets up the service. \nThe default configuration the observer starts with \ncan be found [here](https://github.com/ObserveRTC/observer/blob/master/src/main/resources/application.yaml).\n\nThe configuration file is divided to the following main parts:\n * **micronaut**: defines the server base properties (listening port, REST authentication methods, etc...)\n * **endpoints**: defines the management endpoints of the service.\n * **observer**:  defines the observer functionalities\n\n### Micronaut configs\n\nThe application is written by using [micronaut](https://micronaut.io) framework. Micronaut defines the base \nproperties of the application including which port it listens on for REST or websocket requests.\nThe complete guide to configure a micronaut application can be found [here](https://docs.micronaut.io/3.4.1/guide/configurationreference.html). \n\nHere is an example for the micronaut configuration part:\n\n```yaml\nmicronaut:\n  server:\n    # defines the port the service is listening on \n    port: 7080\n  metrics:\n    # enable metric collection of the service (cpu usage, memory usage, etc.)\n    enabled: True\n    export:\n      # Sets an exporting methods of the collected metrics\n      prometheus:\n        enabled: true\n        descriptions: true\n        # Sets the frequency to collect the metrics\n        step: PT1M\n  # Sets authentication methods to REST API endpoints\n  # for complete guideline to setup the security for REST API requests check the\n  # documentations https://micronaut-projects.github.io/micronaut-security/3.4.1/guide/configurationreference.html\n  security:\n    enabled: false\n```\n\n#### Management endpoints configs\n\nThe application have management endpoints. Management endpoints can be reached on a different port number than \nother micronaut endpoints. This allows for example to integrate the service into a container orchestration systems like \nkubernetes. \n\nHere is an example for the endpoints configuration part:\n\n```yaml\nendpoints:\n  all:\n    # defines the port number for the management endpoints\n    port: 7081\n  prometheus:\n    # indicate if /metrics endpoint is provided or not\n    enabled: true\n    # indicate if the provided prometheus endpoint requires authentication or not\n    # the authentication method is defined in the micronaut.security part\n    sensitive: false\n  health:\n    # indicate if the service provide /heath endpoint.\n    enabled: true\n```\n\n### Observer configs\n\nThe observer configuration can be divided into the following components:\n * **security**: define properties of security related processes (hash algorithm, hash salt, allowed serviceIds, etc.). \n * **sources**: define properties of endpoints collecting Samples (for example enable or disable collecting for REST or Websocket) \n * **sinks**: define connections observer generated Reports are forwarded to (mongoDB, kafka, etc.).\n * **repository** define properties of the repository the observer stores data in (media track expiration time, peer connection expiration time).\n * **buffers** defines the properties of the internal buffers (samples buffering time, reports buffering time, etc.)\n * **hamok**:  defines the settings of the [hamok](https://github.com/balazskreith/hamok) distributed object storage the service uses.\n\nHere is an example for the observer configuration part:\n\n```yaml\nobserver:\n   \n   # configuration to expose metrics\n   metrics:\n      # the prefix for metrics exposed by the observer processes\n      prefix: \"observertc\"\n      # the name of the tag for serviceId used where it is exposed\n      serviceIdTagName: \"service\"\n      # the name of the tag for mediaUnitId used where it is exposed\n      mediaUnitIdTagName: \"mediaUnit\"\n      # metrics exposed from evaluators\n      evaluatorMetrics:\n         # flag indicate if evaluator metrics are exposed or not\n         enabled: false\n      reportMetrics:\n         # flag indicate if report metrics are exposed or not\n         enabled: false\n      sinkMetrics:\n         # flag indicate if sink metrics are exposed or not\n         enabled: false\n      sourceMetrics:\n         # flag indicate if source metrics are exposed or not\n         enabled: false\n      repositoryMetrics:\n         # flag indicate if repository metrics are exposed or not\n         enabled: false\n         # sets the period for exposing metrics about the number of entries the repository holds\n         exposePeriodInMins: 5\n         \n  sources:\n     # flag indicate if sfu sampels are accepted or not\n    acceptSfuSamples: true\n    # flag indicate if client samples are accepted or not\n    acceptClientSamples: true\n    # if it sets to true then observer overrides the samples provided timestamp to its own, to simplify the problem \n    # of different clients reporting timestamp in different time zones.\n    useServerTimestamp: false\n    # settings related to REST API accepting Samples\n    rest:\n      # indicate if Samples are accepted through REST or not\n      enabled: true\n    # settings related to Websockets accepting Samples\n    websocket:\n      # indicate if Samples are accepted through websockets or not\n      enabled: true\n      # this defines how many remote observer peers must be connected to the grid before \n      # the websocket starts accepting sessions\n      # this configuration is useful to avoid a huge amount of session directed to the first \n      # available observer instance\n      minRemotePeers: 0\n\n  repository:\n    # indicate if the repositories storages in hamok should use distributed backup service or not.\n    # NOTE: Distributed backup service increases traffic inside the cluster and volume of data stored in the memory\n    useBackups: false\n    # the maximum idle time for a call object after which it is removed from the repository\n    callMaxIdleTimeInS: 300\n    # the maximum idle time for an SFU object after which it is removed from the repository\n    sfuMaxIdleTimeInS: 300\n    # the maximum idle time for a Sfu Transport object after which it is removed from the repository\n    sfuTransportMaxIdleTimeInS: 600\n    # the maximum idle time for a client object after which it is removed from the repository\n    clientsMaxIdle: 300\n    # the maximum idle time for a peer connection object after which it is removed from the repository\n    peerConnectionsMaxIdle: 300\n    # the maximum idle time for an inbound track object after which it is removed from the repository\n    inboundTracksMaxIdle: 300\n    # the maximum idle time for an outbound track object after which it is removed from the repository\n    outboundTracksMaxIdle: 300\n    # the maximum idle time for a sfu inbound rtp pad object after which it is removed from the repository\n    sfuInboundRtpPadMaxIdleTimeInS: 300\n    # the maximum idle time for a sfu outbound rtp pad object after which it is removed from the repository\n    sfuOutboundRtpPadMaxIdleTimeInS: 300\n    # the maximum idle time for a sfu sctp channel object after which it is removed from the repository\n    sfuSctpChannelMaxIdleTimeInS: 300\n\n  buffers:\n    # settings of the buffer collect samples from the sources\n    samplesCollector:\n      # the maximum number of items of the buffer\n      maxItems: 10000\n      # the max time the buffer can hold an item in milliseconds\n      maxTimeInMs: 10000\n    # settings of the buffer holds item between evaluators\n    debouncers:\n      # the maximum number of items of the buffer\n      maxItems: 10000\n      # the max time the buffer can hold an item in milliseconds\n      maxTimeInMs: 1000\n    # settings of the buffer collect reports from evaluators\n    reportsCollector:\n      # the maximum number of items of the buffer\n      maxItems: 10000\n      # the max time the buffer can hold an item in milliseconds\n      maxTimeInMs: 10000\n\n  # sets up the security methods and values for the observer analysis\n  security:\n    # flag indicating if fetched configurations should be written to the console or not\n    printConfigs: True\n    # obfuscation methods related settings\n    obfuscations:\n      # the used hash algorithm for anonymization \n      hashAlgorithm: SHA-256\n      # the added extra salt for the hash in anonymization process\n      hashSalt: \"MySalt\"\n\n  # settings for analysing Samples\n  evaluators:\n     # settings for component responsible for updating call entity objects \n    callUpdater:\n       # defines the strategy to assign callId to clients.\n       # possible values:\n       #  - MASTER: the Observer responsible to assign new callId to clients, \n       #            and only the observer can end a call\n       # - SLAVE: the client is responsible to provide callId, and whenever a new callId\n       #          is provided for a room, the previous call is closed\n      callIdAssignMode: master\n     # settings for component analyse the samples from \n     # client applications\n    clientSamplesAnalyser:\n       # drop reports created from inbound-audio and video if \n       # their outbound pair has not been found  \n       dropUnmatchedReports: false\n\n    # settings for component analyse the samples from SFUs\n    sfuSamplesAnalyser:\n       # drop reports created from inboundRtpPad if their sfuStreamId\n       # been not registered by a client\n       dropUnmatchedInboundReports: false\n       # drop reports created from outboundRtpPad if their sfuSInkId\n       # been not registered by a client\n       dropUnmatchedOutboundReports: false\n       # drop reports created from internal inboundRtpPad samples if their counterpart\n       # internal outbound rtp pads are not matched\n       dropUnmatchedInternalInboundReports: false\n    \n    # obfuscator component settings\n    obfuscator:\n      # indicate if it is enabled or not\n      enabled: false\n      # indicate how the ICE addresses has been obfuscated.\n      # possible values are: none, anonymization\n      iceAddresses: none\n      # indicate how the room ids are obfuscated.\n      roomId: none\n      # indicate how the user ids are obfuscated.\n      userId: none\n\n  # Setup the sinks to forward Reports.\n  # Detailed description is in the Sinks section\n  sinks: {}\n\n  # settings related to hamok distributed object storage \n  # and discovery of other observer instances to create a grid\n  hamok:\n    # settings related to the storage grid the observer shares objects through\n    storageGrid:\n       # the retention time in minutes the Raft keeps logs for\n       # this determines how long a new peer can join to the cluster without \n       # issuing a storage sync. If a peer joins after a retention time,\n       # it issues a storage sync which requires the leader to synchronize all \n       # storage with the new follower\n       raftMaxLogEntriesRetentionTimeInMinutes: 5\n       \n       # the period of the heartbeat for the leader.\n       # this determines for example how fast the replicated storages get updates\n       heartbeatInMs: 150\n       \n       # the timeout for a follower when it does not get a heartbeat message \n       # from a leader and start an election\n       # rule of thumb to make it at least 3 or 5 times higher than the heartbeat + network delay\n       followerMaxIdleInMs: 1000\n       \n       # maximum time for a request between storage grids to exchange\n       # information. note that finding a balance for this value needs consideration \n       # of the network you put the instance in, and the resource capacity your running instance \n       # acquires. if it is too small and response time in the network too big it can cause false \n       # interpretation that the remote peer is not responding. However if it is too large \n       # then becasue of the blocking nature of storage operations this can cause an observer \n       # to throttle samples and reports\n       requestTimeoutInMs: 3000\n    \n    # For other type of discovery settings, please take a look at hamok configuration section\n    discovery:\n       type: StaticDiscovery\n       config:\n          peers:\n             my-other-peer:\n                port: 5603\n                host: \"localhost\"\n                \n    # For other type of endpoints for hamok to communicate please take a look at hamok cofiguration section\n    endpoint: \n       # The type of the endpoint hamok uses to communicate\n       type: WebsocketEndpoint\n       config:\n          # the hostname\n          serverHost: \"localhost\"\n          # the port number to offer a websocket connection\n          serverPort: 5600\n          \n          # Discovery strategy to discover another endpoint\n          \n```\n\n### Hamok configs\n\nHamok is a distributed in-memory object storage. The configuration in observer to setup the property \nof the storageGrid, providing local-, and to discover remote endpoints. \n```yaml\nobserver:\n  hamok:\n    storageGrid: {}\n    discovery: {}\n    endpoint: {}\n```\nCurrently the following endpoints can be made in observer for hamok:\n * [WebsocketEndpoint](#websocket-endpoint-settings)\n\nCurrently the following discovery settings can be made in observer for hamok:\n * [StaticDiscovery](#static-discovery-settings)\n * [KubernetesDiscovery](#kubernetes-discovery-settings)\n \n#### Websocket Endpoint Settings\n\n```yaml\nendpoint: {}\n    # The type of the endpoint hamok uses to communicate\n    type: WebsocketEndpoint\n    config:\n       # the hostname\n       serverHost: \"localhost\"\n       # the port number to offer a websocket connection\n       serverPort: 5600\n```\n\n#### Static Discovery Settings\n\n```yaml\n discovery:\n    type: StaticDiscovery\n    config:\n       peers:\n          my-other-wonderful-peer:\n             port: 5603\n             host: \"localhost\"\n```\n#### Kubernetes Discovery Settings\n\n```yaml\n discovery:\n    type: K8sPodsDiscovery\n    config:\n       # The namespace where the instance is runninng in\n       namespace: \"default\"\n       \n       # prefix for the service the pods are installed by\n       prefix: \"observertc\"\n       \n       # the port number on which the remote peer accepts a connection to\n       port: 5601# \n```\n\n## Sinks\n\nThe observer forward the generated reports through sinks.\nThere are different type of sinks implemented to integrate different type of services.\nA sink has the following configuration structure:\n\n\n```yaml\nsinks:\n  # a unique key for a sink added to the configuration\n  MyUniqueSinkKey:\n    # The type of the sink. \n    # Each integration has its own sink type. \n    # for example for kafka the type of the sink KafkaSink\n    type: SinkType # REQUIRED\n    # A config passed to the instantiated SinkType\n    config: {}\n    # determines which type of reports are allowed to forward on this sinks\n    # if it is not set then it allows all type of reports to send\n    reports:\n        sendObserverEvents: true\n        sendCallEvents: true\n        sendCallMeta: true\n        sendClientExtensions: true\n        sendInboundAudioTracks: true\n        sendInboundVideoTracks: true\n        sendOutboundAudioTracks: true\n        sendOutboundVideoTracks: true\n        sendPeerConnectionTransports: true\n        sendIceCanddiatePairs: true\n        sendClientDataChannels: true\n        sendSfuEvents: true\n        sendSfuMeta: true\n        sendSfuTransports: true\n        sendSfuSctpStreams: true\n        sendSfuOutboundRtpStreams: true\n        sendSfuInboundRtpStreams: true\n        sendSfuExtensions: true\n    \n```\n\n### LoggerSink\n\nObserver can print reports to the console or to the configured logback output by using `LoggerSink`\n\n```yaml\nsinks:\n  MyLoggerSink:\n    type: LoggerSink\n    config:\n      # the level of the logger used to print information to the console\n      logLevel: INFO\n      # prints the received reports to the console \n      printReports: True # default is False\n      # prints the summary of the type of reports received by the sink\n      printTypeSummary: False\n```\n\n### Apache Kafka\n\nObserver can send reports to [Apache Kafka](https://kafka.apache.org/). by using `KafkaSink`\n\n```yaml\nsinks:\n    MyKafkaSink:\n        type: KafkaSink\n        config:\n          # indicate if the Sink tries to reconnect in case of failure or not\n          tryReconnectOnFailure: True\n          # the name of the kafka topic reports are sent to \n          topic: \"reports\"\n          # the strategy to assign keys for Kafka entries. \n          # This is important if you have multiple brokers and consumers from the same topic \n          # possible values are:\n          #  * RANDOM - keys are randomly distributed\n          #  * INSTANCE_BASED - reports from the same observer will have the same key\n          #  * OBJECT_HIERACHY_BASED - reports belongs to the same call having the same key. \n          #                           if callId is not available then clientId or sfuId is used \n          keyAssignmentStrategy: INSTANCE_BASED\n          # properties of the created kafka producer.\n          # More info on settings: (https://kafka.apache.org/documentation/#producerconfigs).\n          properties:\n            bootstrap.servers: localhost:9092\n```\n\n### Mongo Database\n\nObserver can send reports to  [Mongo Database](https://www.mongodb.com/). by using `MongoSink`\n\n```yaml\nsinks:\n  MyMongoSink:\n    type: MongoSink\n    config:\n      # the URI used to connect to mongo. either this or the connection must be provided\n      uri: \"mongodb://root:password@localhost:27017\"\n      # a config object describes the uri. either this or the uri must be given\n      connection:\n        # the username for mongodb\n        username: \"admin\"\n        # the password for mongodb\n        password: \"password\"\n          # list of mongodb servers\n          servers:\n            - \"localhost:27017\"\n      \n      # the name of the database the Reports are written to\n      database: \"reports\"\n      # the strategy to save the reports\n      # Possible values are:\n      # * REPORT_TYPE_BASED - different type of reports are written to different collections\n      # * SERVICE_ID_BASED - reports have different serviceId are written to different collections\n      # * ONE_COLLECTION - every report is written into one collection\n      savingStrategy: ONE_COLLECTION\n      # print the summary to the console each time a batch of writing has been finished\n      printSummary: True\n```\n\n### AWS Firehose Sink\n\nObserver can send reports to [Aws Firehose](https://aws.amazon.com/kinesis/data-firehose/) via DIRECT PUT method.\n\n```yaml\nsinks:\n   MyFireHoseSink:\n      type: FirehoseSink\n      config:\n         # The encoding format of the forwarded data. Possible values are: JSON, CSV\n         # For csv format, please check the schemas repository for the headers\n         encodingType: CSV\n         # The AWS region the firehose has been configured\n         regionId: eu-west-1\n         # the name of the delivery stream the records are forwarded by default\n         defaultDeliveryStreamId: observertc-reports-csv\n         # map report types to delivery streams records are forwarded to\n         # Optional. \n         # if both streams and the defaultDeliveryStreamId are provided than the report forwarded to the default if it is not mapped\n         streams:\n            observer-event: observertc-observer-event-reports-csv\n            call-event: observertc-call-event-reports-csv\n            call-meta-data: observertc-call-meta-data-reports-csv\n            client-extension-data: observertc-client-extension-data-reports-csv\n            peer-connection-transport: observertc-peer-connection-transport-reports-csv\n            ice-candidate-pair: observertc-ice-candidate-pair-reports-csv\n            peer-connection-data-channel: observertc-peer-connection-data-channel-reports-csv\n            inbound-audio-track: observertc-inbound-audio-track-reports-csv\n            inbound-video-track: observertc-inbound-video-track-reports-csv\n            outbound-autio-track: observertc-outbound-autio-track-reports-csv\n            outbound-video-track: observertc-outbound-video-track-reports-csv\n            sfu-event: observertc-sfu-event-reports-csv\n            sfu-meta-data: observertc-sfu-meta-data-reports-csv\n            sfu-extension-data: observertc-sfu-extension-data-reports-csv\n            sfu-transport: observertc-sfu-transport-reports-csv\n            sfu-inbound-rtp-pad: observertc-sfu-inbound-rtp-pad-reports-csv\n            sfu-outbound-rtp-pad: observertc-sfu-outbound-rtp-pad-reports-csv\n            sfu-sctp-stream: observertc-sfu-sctp-stream-reports-csv\n\n         # A provided credential to access AWS resources\n         # see detailed description in Credentials section\n         # To configure credentials for AWS services, \n         # please look at the AWS Credential Settings section in the readme.\n         credentials:\n            type: AwsStatic\n            config:\n               accessKeyId: \u003cACCESS_KEY\u003e\n               secretAccessKey: \u003cSECRET\u003e\n\n         # in case CSV encoding is used, this instructs the CSV format written into the records\n         # possible values are: DEFAULT, EXCEL, INFORMIX_UNLOAD, INFORMIX_UNLOAD_CSV, MONGODB_CSV, MONGODB_TSV, MYSQL, ORACLE,\n         # POSTGRESQL_CSV, POSTGRESQL_TEXT, RFC4180\n         # by default it is DEFAULT\n         csvFormat: DEFAULT\n         # The number of reports put into one Firehose PUT request.\n         # default is 100\n         csvChunkSize: 100\n```\n\nTo configure credentials for AWS services, please look at the [AwsCredentials Configuration](#aws-credentials-configuration) section.\n\n\n### AWS S3 Bucket Sink\n\nObserver can send reports to [Aws S3](https://aws.amazon.com/s3/) via REST requests\n\n```yaml\nsinks:\n  MyAwsS3Sink:\n    type: AwsS3Sink\n    config:\n       # The encoding format of the forwarded data. Possible values are: JSON, CSV\n       # For csv format, please check the schemas repository for the headers\n       encodingType: CSV\n       # the name of the bucket observer upload the reports to\n       bucketName: \"my-bucket\"\n       # The AWS region the firehose has been configured\n       regionId: eu-west-1\n       # the name of the default prefix used if no other prefix defined\n       defaultPrefix: observertc-reports-csv\n       # flag indicate to add serviceId prefix for the uploaded reports\n       addServiceIdPrefix: false\n       # flag indicate to add reports category (call-reports / sfu-reports) prefix for the uploaded reports\n       addReportCategoryPrefix: false\n       # flag indicate to add sfu or call id prefix for the uploaded reports\n       addSfuOrCallIdPrefix: false\n       # prefix used to upload objects to s3\n       prefixes:\n          observer-event: observertc-observer-event-reports-csv\n          call-event: observertc-call-event-reports-csv\n          call-meta-data: observertc-call-meta-data-reports-csv\n          client-extension-data: observertc-client-extension-data-reports-csv\n          peer-connection-transport: observertc-peer-connection-transport-reports-csv\n          ice-candidate-pair: observertc-ice-candidate-pair-reports-csv\n          peer-connection-data-channel: observertc-peer-connection-data-channel-reports-csv\n          inbound-audio-track: observertc-inbound-audio-track-reports-csv\n          inbound-video-track: observertc-inbound-video-track-reports-csv\n          outbound-autio-track: observertc-outbound-autio-track-reports-csv\n          outbound-video-track: observertc-outbound-video-track-reports-csv\n          sfu-event: observertc-sfu-event-reports-csv\n          sfu-meta-data: observertc-sfu-meta-data-reports-csv\n          sfu-extension-data: observertc-sfu-extension-data-reports-csv\n          sfu-transport: observertc-sfu-transport-reports-csv\n          sfu-inbound-rtp-pad: observertc-sfu-inbound-rtp-pad-reports-csv\n          sfu-outbound-rtp-pad: observertc-sfu-outbound-rtp-pad-reports-csv\n          sfu-sctp-stream: observertc-sfu-sctp-stream-reports-csv\n\n       # A provided credential to access AWS resources\n       # see detailed description in Credentials section\n       # To configure credentials for AWS services, \n       # please look at the AWS Credential Settings section in the readme.\n       credentials:\n          type: AwsStatic\n          config:\n             accessKeyId: \u003cACCESS_KEY\u003e\n             secretAccessKey: \u003cSECRET\u003e\n          \n       # in case CSV encoding is used, this instructs the CSV format written into the records\n       # possible values are: DEFAULT, EXCEL, INFORMIX_UNLOAD, INFORMIX_UNLOAD_CSV, MONGODB_CSV, MONGODB_TSV, MYSQL, ORACLE,\n       # POSTGRESQL_CSV, POSTGRESQL_TEXT, RFC4180\n       # by default it is DEFAULT\n       csvFormat: DEFAULT\n```\n\nTo configure credentials for AWS services, please look at the [AwsCredentials Configuration](#aws-credentials-configuration) section.\n\n\n### AWS Credential Settings\n\nAWS services require providing credentials. The credential configuration is part of the actual sink configuration\nsetting up to connect to an AWS service.\nFor example Firehose or AwsS3 sink have the `credentials` property in their configurations.\n\nPossible credential types and configuration structures are the followings:\n * [AWS Profile credential settings](#aws-profile-credential-settings) \n * [AWS Environment credential settings](#aws-environment-credential-settings)\n * [AWS STS Assume Role Session Settings](#aws-sts-assume-role-session-settings)\n * [AWS Static Credential Settings](#aws-static-credential-settings)\n * [AWS System Credential Settings](#aws-system-credential-settings)\n * [AWS Web Identity Token Credential Settings](#aws-web-identity-token-credential-settings)\n\n#### AWS Profile credential settings\n\nFile based profile credentials\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      credentials:\n         type: AwsProfile\n         config:\n           profileFilePath: /my/path/to/credential/file\n           # possible values are CONFIGURATION, and CREDENTIALS. default CREDENTIALS\n           profileFileType: CREDENTIALS\n           # the name of the profile fetched for credentials\n           profileName: myprofileName\n```\n\n#### AWS Environment credential settings\n\nLoads credentials from the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      credentials:\n         type: AwsEnvironmentVariable\n```\n\n#### AWS STS Assume Role Session Settings\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      credentials:\n         type: AwsStsAssumeRoleSession\n         config:\n            # the region id of the assumed role\n            regionId: eu-west-1\n            # the ARN of the role\n            roleArn: \"THE_ARN_OF_THE_ROLE\"\n            roleSessionName: \"observer\"\n         \n            # give credentials to the STS client accessing to the assumed role\n            # if the client requires different credentials than the currently provided one.\n            # the settings of the credential here is the same as to any AWS credential listed here\n            credentials: # optional \n               type: AwsStatic\n               config:\n                  accessKeyId: accessKey\n                  secretAccessKey: secret\n            serialNumber: \"1234-5678\" # optional\n            policy: \"policy\" # optional\n            externalId: \"external\" # optional\n```\n\n#### AWS Static Credential Settings\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      credentials:\n         type: AwsStatic\n         config:\n            accessKeyId: accessKey\n            secretAccessKey: secret\n```\n\n#### AWS System Credential Settings\n\nLoads credentials from the aws.accessKeyId, aws.secretAccessKey and aws.sessionToken system properties.\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      credentials:\n        type: AwsSystemProperty\n```\n\n#### AWS Web Identity token credential settings\n\nA credential provider that will read web identity token file path, \naws role arn and aws session name from system \nproperties or environment variables for using web \nidentity token credentials with STS. Use of this credentials \nprovider requires the 'sts' module to be on the classpath.\n\n```yaml\nMyAwsS3Sink:\n   type: AwsS3Sink\n   config:\n      type: AwsWebIdentityTokenFile\n      config:\n        roleArn: roleArn\n        roleSessionName: roleSessionName\n```\n\n## Performance considerations\n\nObserver performance is highly influenced by two factors: how many clients providing samples at the same time, \nand how much time it takes to forward the reports to the configured sink(s). The performance can be tuned by settings the buffers.\nFor example higher threshold for samples making larger batches to process and send to the sinks, giving more time to the sinks \nto forward the previous batch, but also it can make larger spikes in memory as tens or hundreds of thousands of reports \nare going to be throttled to the sinks waiting to forward. \n\nThough it is very hard to give an exact number of how many clients can be connected to one observer instance and \nhow much memory and CPU should be given to one instance, to start with you can use the following may help in your calculations:\n\n![Performance Box](docs/images/performance-box.png)\n\nAssuming the majority of calls (80%) are less than 4 participants, then 1000 active session can produce around 30 Samples per sec\n(if the sampling period is around 10s for each client) then 2GB of memory should / could handle 1000 active session at the same time assuming, \nproducing 1000 reports per sec to the sinks to handle.\n\nAnothe case if you design a service for 5000 active sessions at any time, then either you scale it vertically to have bigger CPU and around 10GB memory,\nor you can scale horizontally and start running 5 instances with the same configurations.  \n\nIt must be noted that this estimation is superficial and based on experiments and experience we have so far.\n\nIn terms of horizontal scalability it must be noted too, that since the service uses in-memory distributed storage memory consumption is \ncrucial and an instance cannot / should not run out of memory. \n\n\n## Compatibility\n\nObserver versions and compatible [schema](https://github.com/ObserveRTC/schemas) versions\n\n| schemas → \u003cbr/\u003eobserver ↓ | 2.1.y              | 2.2.y              |   |   |\n|---------------------------|--------------------|--------------------|---|---|\n| 1.0.x                     | :white_check_mark: | :white_check_mark: |   |   |\n\n\n## Getting involved\n\nObserver is made with the intention to provide a scalable open-source monitoring solution for \nWebRTC developers. We develop new features and maintaining the current \nproduct with the help of the community. If you are interested in getting involved \nplease read our [contribution](CONTRIBUTING.md) guideline.\n\n## Changelog\n\n[Changelog](CHANGELOG.md)\n\n## License\n\n[Apache 2.0](LICENSE)\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobservertc%2Fobserver-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobservertc%2Fobserver-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobservertc%2Fobserver-service/lists"}