https://github.com/nodefluent/salesforce-kafka-connect
Salesforce connector for node kafka connect
https://github.com/nodefluent/salesforce-kafka-connect
Last synced: about 1 year ago
JSON representation
Salesforce connector for node kafka connect
- Host: GitHub
- URL: https://github.com/nodefluent/salesforce-kafka-connect
- Owner: nodefluent
- License: mit
- Created: 2017-06-30T09:02:53.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2022-12-03T15:48:29.000Z (over 3 years ago)
- Last Synced: 2024-04-14T07:54:12.064Z (about 2 years ago)
- Language: JavaScript
- Size: 1.11 MB
- Stars: 14
- Watchers: 4
- Forks: 6
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-kafka-connect - nodefluent/salesforce-kafka-connect - Node.js Salesforce source and sink connector (CRM & Marketing / Salesforce)
README
# salesforce-kafka-connect
[](https://greenkeeper.io/)
Kafka Connect connector for Salesforce with bulk options configuration support.
[](https://travis-ci.org/nodefluent/salesforce-kafka-connect)
## Use API
```
npm install --save salesforce-kafka-connect
```
### salesforce -> kafka
```es6
const { runSourceConnector } = require("salesforce-kafka-connect");
runSourceConnector(config, [], onError).then(config => {
//runs forever until: config.stop();
});
```
### kafka -> salesforce
```es6
const { runSinkConnector } = require("salesforce-kafka-connect");
runSinkConnector(config, [], onError).then(config => {
//runs forever until: config.stop();
});
```
### kafka -> salesforce (with custom topic (no source-task topic))
```es6
const { runSinkConnector, ConverterFactory } = require("salesforce-kafka-connect");
const etlFunc = (messageValue, callback) => {
//type is an example json format field
if (messageValue.type === "publish") {
return callback(null, {
id: messageValue.payload.id,
name__c: messageValue.payload.name,
info__c: messageValue.payload.info
});
}
if (messageValue.type === "unpublish") {
return callback(null, null); //null value will cause deletion
}
callback(new Error("unknown messageValue.type"));
};
const converter = ConverterFactory.createSinkSchemaConverter({}, etlFunc);
runSinkConnector(config, [converter], onError).then(config => {
//runs forever until: config.stop();
});
/*
this example would be able to store kafka message values
that look like this (so completely unrelated to messages created by a default SourceTask)
{
payload: {
id: 1,
name__c: "first item",
info__c: "some info"
},
type: "publish"
}
*/
```
## Use CLI
note: in BETA :seedling:
```
npm install -g salesforce-kafka-connect
```
```
# run source etl: salesforce -> kafka
nkc-salesforce-source --help
```
```
# run sink etl: kafka -> salesforce
nkc-salesforce-sink --help
```
## Config(uration)
```es6
const config = {
kafka: {
zkConStr: "localhost:2181/",
logger: null,
groupId: "kc-salesforce-test",
clientName: "kc-salesforce-test-name",
workerPerPartition: 1,
options: {
sessionTimeout: 8000,
protocol: ["roundrobin"],
fromOffset: "earliest", //latest
fetchMaxBytes: 1024 * 100,
fetchMinBytes: 1,
fetchMaxWaitMs: 10,
heartbeatInterval: 250,
retryMinTimeout: 250,
requireAcks: 1,
//ackTimeoutMs: 100,
//partitionerType: 3
}
},
topic: "sc-test-topic",
partitions: 1,
maxTasks: 1,
pollInterval: 2000,
produceKeyed: true,
produceCompressionType: 0,
connector: {
tokenAuth: {
oauth2: {
clientId: "client-id",
clientSecret: "client-secret",
redirectUri: "redirect-uri"
},
accessToken: "access-token",
refreshToken: "refresh-token"
}
// username: "salesforce",
// password: "salesforce",
loginUrl: "https://test.salesforce.com",
streamingSource: {
batchSize: 5,
topic: "StreamingTopic",
kafkaTopic: "sf-test-topic",
idProperty: "id"
},
restSink: {
sObject: "sobject",
idProperty: "id",
bulkOptions: {
concurrencyMode: 'Serial',
},
batchSize: 500
}
},
http: {
port: 3149,
middlewares: []
},
enableMetrics: true,
batch: {
batchSize: 100,
commitEveryNBatch: 1,
concurrency: 1,
commitSync: true
}
};
```