https://github.com/harelba/mail2kafka
Service that converts incoming emails to kafka messages
https://github.com/harelba/mail2kafka
Last synced: 7 months ago
JSON representation
Service that converts incoming emails to kafka messages
- Host: GitHub
- URL: https://github.com/harelba/mail2kafka
- Owner: harelba
- Created: 2016-03-16T11:45:14.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2016-03-19T10:34:54.000Z (over 9 years ago)
- Last Synced: 2025-01-30T03:12:55.846Z (8 months ago)
- Language: JavaScript
- Homepage:
- Size: 78.1 KB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# mail2kafka
This is a simple service which gets emails through SMTP, parses them (using the great [mailin](http://mailin.io/) module) and outputs a JSON containing the parsed data to a kafka topic.
# Running it
Run ````npm install```` and ````grunt serve````.
"Production" deployment will be available later on.
Only kafka 0.8+ is supported.
# Generate emails from the command line
You can use the ````generate-incoming-email```` script to send an email from the command line:
````
./generate-incoming-email -s "my subject" -f origin@email.com -t "destination@blah.com"
or
echo "this is my content" | ./generate-incoming-email ...
Defaults exists for everything.
````
# Configuration
Lies currently in ````server/config/environment/index.js````. Will be moved soon to a separate location.
Configuration:
````
// SMTP port to listen to
smtpPort : 2526,
// Kafka configuration
kafka : {
zk : {
// Zookeeper connection path
connectionString : "localhost:2181",
// Zookeeper session timeout
sessionTimeout : 30000,
// Delay between reconnections
spinDelay : 1000 ,
// Number of retries to reconnect
retries : 0
},
// Topic to publish email details
outputTopic : 'incoming-emails',
// Kafka client id - If null, will be autogenerated
clientId : null,
// Mode of publishing to kafka
requireAcks : {
// Are acks required? (safe vs fast), either 1 or 0
acksRequired : 1,
// Timeout for acks if required
ackTimeoutMs : 200,
// If no acks required, then batch size of messages before sending (otherwise null)
noAckBatchSize : null,
// If no acks required, then max age of messages before sending (otherwise null)
noAckBatchAge : null
},
// Type of partitioner - should be 0, will be used in the future perhaps
partitionerType : 0
}
````
## Output format
Kafka output messages are in JSON format (string encoded topic). Format is as follows:
````
{ emailDetails : { ... } }
````
emailDetails contains a very elaborate object that contains everything related to the email :) Just read it and do whatever you want.