https://github.com/pinterest/secor

Secor is a service implementing Kafka log persistence
https://github.com/pinterest/secor

kafka

Last synced: 4 days ago
JSON representation

Secor is a service implementing Kafka log persistence

• Host: GitHub
• URL: https://github.com/pinterest/secor
• Owner: pinterest
• License: apache-2.0
• Created: 2014-04-15T22:26:44.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2024-10-19T01:58:06.000Z (3 months ago)
• Last Synced: 2024-10-29T15:35:06.303Z (2 months ago)
• Topics: kafka
• Language: Java
• Homepage:
• Size: 3.32 MB
• Stars: 1,846
• Watchers: 68
• Forks: 540
• Open Issues: 248
• Metadata Files:
  • Readme: README.dev.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-kafka - secor - Secor is a service implementing Kafka log persistence. (Libraries / Kafka)
• awesome-kafka - Secor - service for persisting Kafka logs
• awesome-data-engineering - Secor - Pinterest's Kafka to S3 distributed consumer. (Data Ingestion)

README

        
# Secor developer notes

## Run tests (optional)

```sh

cd ${SECOR_INSTALL_DIR}

./scripts/run_tests.sh

# OR:

MVN_PROFILE= ./scripts/run_tests.sh

```

## Dependencies

### Bumping to new default kafka version 

* Make new profile if it doesn't exist, matching kafka version as id tag.

* Ensure the new profile is the only one active pr default with `activation->activeByDefault`

* Change release profile to have the same stuff in it's profile. (used for

  buildservers who signs artifacts with GPG and attach javadocs and pushes to

public M2 repository..)

* Update MVN_PROFILE in `Makefile`

* Update .travis and add it into the matrix, so CI runs tests for your kafka

  version profile.

Profit.

### scala versions

Ensure you check `mvn dependency:tree` and ensure you don't pull in libraries

compiled with different scala versions. This is bad.