Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/alibaba-archive/aliyun-log-producer-java

Last synced: 6 days ago
JSON representation
Host: GitHub
URL: https://github.com/alibaba-archive/aliyun-log-producer-java
Owner: alibaba-archive
License: apache-2.0
Archived: true
Created: 2018-02-06T09:54:49.000Z (almost 7 years ago)
Default Branch: master
Last Pushed: 2022-06-17T01:53:46.000Z (over 2 years ago)
Last Synced: 2024-08-02T16:47:45.833Z (3 months ago)
Language: Java
Size: 254 KB
Stars: 18
Watchers: 11
Forks: 12
Open Issues: 2
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

        # Aliyun LOG Java Producer (deprecated)

[![Build Status](https://travis-ci.org/aliyun/aliyun-log-producer-java.svg?branch=master)](https://travis-ci.org/aliyun/aliyun-log-producer-java)

[![License](https://img.shields.io/badge/license-Apache2.0-blue.svg)](/LICENSE)

[中文版README](/README_CN.md)

This project is no longer maintained. Please use the new version of [Aliyun LOG Java Producer](https://github.com/aliyun/aliyun-log-producer).

The `Aliyun LOG Java Producer` is a high concurrency writing library written for Java applications. The Producer Library and [Consumer Library](https://yq.aliyun.com/articles/6817) is a reading and writing encapsulation for Loghub. It aims to reduce the threshold for data collection and consumption.

## Feature

* Provide an asynchronous sending interface, thread safety.

* You can configure more than one project.

* You can control the thread number for sending data.

* You can control the number and size of logs being merged into one package.

* You can control the memory usage. When the memory usage reach the threshold configured by user, the producer's send interface will be blocked until the free memory is available.

## Advantage

* `Disk Free`: the generation data will be send to AliCloud Log Service in real time through network.

* `High Throughput`: support more than 100 write operations per second.

* `The Computing and IO is separated`: recoding logs do not affect computing time.

In the above scenarios, the `Aliyun LOG Java Producer` will alleviate the burden of your development. You don't need to care about log collection implementation details. And there is no need to worry about that log collection will impact on the normal operation of your business. Reducing the threshold of data acquisition greatly.

The comparison of all kinds of access methods

Access Mode

Pros/Cons

Use Case

log file + Logtail

Log writing

 and log collection decoupling, there is no need to modify your code.

common case

syslog + Logtail

Good performance (80MB/S), disk free, the syslog protocol needs to be supported.

syslog

Aliyun Log Java SDK

Disk free, the data will be send to AliCloud Log Service directly, you need to care about the switch of network IO and program IO.

The log will not be written to a file

Aliyun LOG Java Producer

Disk free, asynchronous and high throughput.

The log will not be written to a file, high QPS.

## Configuration Steps

### 1. Adding the Dependencies in pom.xml

```

	com.google.protobuf

	protobuf-java

	2.5.0

	com.aliyun.openservices

 	aliyun-log

	0.6.27

	com.aliyun.openservices

	log-loghub-producer

	0.1.16

```

### 2. Configure ProducerConfig

The configuration format is as follows. Please refer to the parameter description part for parameter value.

```

public class ProducerConfig {

    // Specify the timeout for sending package, in milliseconds, default is 3000

    public int packageTimeoutInMS = 3000;

    // Specify the maximum log count per package, the upper limit is 4096

    public int logsCountPerPackage = 4096;

    // Specify the maximum cache size per package, the upper limit is 3MB, in bytes

    public int logsBytesPerPackage = 3 * 1024 * 1024;

    // The upper limit of the memory that can be used by each producer instance, in bytes, default is 100MB

    public int memPoolSizeInByte = 100 * 1024 * 1024;

    // If shardHash is specified when you send data, you should care about this parameter, otherwise there is no need to care about it.

    // The backend thread will merge the data being sent to the same shard together, and shard is associated with a hash interval.

    // The producer will pull the hash interval information for each shard from AliCloud Log Service regularly and update the local value, this parameter stands for the time interval.

    public int shardHashUpdateIntervalInMS = 10 * 60 * 1000;

    // Specify the retry times, default is 10. If exceeds this value, the exception will pass to the callback.

    public int retryTimes = 10;

    // protobuf

    public String logsFormat = "protobuf";

    // Specify the I/O thread pool's maximum pool size, the main function of the I/O thread pool is to send data, default is 8

    public int maxIOThreadSizeInPool = 8;

    // userAgent

    public String userAgent = "loghub-producer-java";

}

```

### 3. Extends ILogCallback

The main function of callback is to handle the results of sending data. The results include successful status or exception. If you don't care about the result, there is no need to extends ILogCallback.

## Parameter Description

Parameter

Description

Value

packageTimeoutInMS

Specify the timeout for sending package.

Integer, in milliseconds

logsCountPerPackage

Specify the maximum log count per package.

Integer, 1~4096

logsBytesPerPackage

Specify the maximum cache size per package.

Integer, 1~3145728(3M),in bytes

memPoolSizeInByte

The upper limit of the memory that can be used by each producer instance.

Integer, in bytes

maxIOThreadSizeInPool

Specify the I/O thread pool's maximum pool size, the main function of the I/O thread pool is to send data.

Integer, default is 8

shardHashUpdateIntervalInMS

If shardHash is specified when you send data, you should care about this parameter, otherwise there is no need to care about it. The backend thread will merge the data being sent to the same shard together, and shard is associated with a hash interval. The producer will pull the hash interval information for each shard from AliCloud Log Service regularly and update the local value, this parameter stands for the time interval.

Integer, in milliseconds

retryTimes

Specify the retry times, default is 10. If exceeds this value, the exception will pass to the callback.

Integer

## Sample Code

[ProducerSample.java](/src/main/java/com/aliyun/openservices/log/producer/sample/ProducerSample.java)

[CallbackSample.java](/src/main/java/com/aliyun/openservices/log/producer/sample/CallbackSample.java)

## RAM

If you want to write data to logstore through Aliyun LOG Java Producer, you should configure the following permissions for the account you use. [RAM doc](https://www.alibabacloud.com/help/doc-detail/57445.htm)

Action

Resource

log:PostLogStoreLogs

acs:log:${regionName}:${projectOwnerAliUid}:project/${projectName}/logstore/${logstoreName}

## Aliyun Log Java SDK

If the interface provided by producer can't meet your log collection requirements, you can develop your own log collection API based on [Aliyun Log Java SDK](https://github.com/aliyun/aliyun-log-java-sdk).

## Contact Us

- [Alicloud Log Service homepage](https://www.alibabacloud.com/product/log-service)

- [Alicloud Log Service doc](https://www.alibabacloud.com/help/product/28958.htm)

- [Alicloud Log Servic official forum](https://yq.aliyun.com/groups/50)

- Alicloud Log Servic official technical support: [submit tickets](https://workorder.console.aliyun.com/#/ticket/createIndex)

## Contributors

[@zzboy](https://github.com/zzboy) made a great contribution to this project.

Thanks for the excellent work by [@zzboy](https://github.com/zzboy)