{"id":16728749,"url":"https://github.com/demian85/gnip","last_synced_at":"2025-03-21T21:31:10.677Z","repository":{"id":3406809,"uuid":"4456996","full_name":"demian85/gnip","owner":"demian85","description":"Connect to Gnip streaming API and manage rules","archived":false,"fork":false,"pushed_at":"2022-12-07T17:31:52.000Z","size":246,"stargazers_count":27,"open_issues_count":10,"forks_count":29,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-14T22:12:58.645Z","etag":null,"topics":["gnip","gnip-stream","javascript","json","nodejs","stream","tweets","twitter","twitter-streaming-api"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"textoo/BackgroundLocationEnabler","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/demian85.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-05-26T18:45:52.000Z","updated_at":"2024-03-18T10:26:32.000Z","dependencies_parsed_at":"2023-01-13T12:29:26.322Z","dependency_job_id":null,"html_url":"https://github.com/demian85/gnip","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/demian85%2Fgnip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/demian85%2Fgnip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/demian85%2Fgnip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/demian85%2Fgnip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/demian85","download_url":"https://codeload.github.com/demian85/gnip/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244874156,"owners_count":20524576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gnip","gnip-stream","javascript","json","nodejs","stream","tweets","twitter","twitter-streaming-api"],"created_at":"2024-10-12T23:11:34.580Z","updated_at":"2025-03-21T21:31:10.078Z","avatar_url":"https://github.com/demian85.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"NodeJS Gnip module\n====\n\nConnect to Gnip streaming API and manage rules. \nYou must have a Gnip account with any data source available, like Twitter Power Track.\n\nCurrenly, this module only supports JSON activity stream format, so you must enable data normalization in your admin panel.\n\n# Gnip.Stream\nThis class is an EventEmitter and allows you to connect to the stream and start receiving data.\n\n## Constructor options\n\n- `timeout`\nAs requested in the Gnip docs (http://support.gnip.com/apis/powertrack/api_reference.html), this option in the constructor allows us to set a read timeout in the client. The recommended value is \u003e=30 seconds, so the constructor will throw an error if a smaller timeout is provided. The default value for this option is 35 seconds.\n- `backfillMinutes`\nNumber of minutes to backfill after connecting to the stream. Optional. Value should be 0 - 5.\n- `partition`\nPartition of the Firehose stream you want to connect to. Only required for Firehose streams.\n- `parser`\nParser library for incoming JSON data. Optional, but defaults to the excellent [json-bigint](https://www.npmjs.com/package/json-bigint) library.  \nMatching tag IDs are sent to us as big integers which can't be reliably parsed by the native JSON library in Node.js. More info on this issue can be found at [StackOverflow](http://stackoverflow.com/questions/8663298/json-transfer-of-bigint-12000000000002539-is-converted-to-12000000000002540)\n\n## API methods\n\n- `start()`\nConnect to the stream and start receiving data. At this point you should have registered at least one event listener for any of these events: 'data', 'object' or 'tweet'.\n\n- `end()`\nTerminates the connection.\n\n## Events\n\n- `ready`\nEmitted when the connection has been successfully established\n- `data`\nEmitted for each data chunk (decompressed)\n- `error`\nEmitted when any type of error occurs. An error is raised if the response status code is not 20x. {error: String} objects are also checked here.\n- `object`\nEmitted for each JSON object.\n- `tweet`\nEmitted for each tweet.\n- `delete`\nEmitted for each deleted tweet.\n- `end`\nEmitted when the connection is terminated. This event is always emitted when an error occurs and the connection is closed.\n\n# Gnip.Rules\nThis class allows you to manage an unlimited number of tracking rules.\n\n## Constructor options\n\n- `user`\nGNIP account username.\n- `password`\nGNIP account password.\n- `url`\nGNIP Rules endpoint url e.g. `https://gnip-api.twitter.com/rules/${streamType}/accounts/${account}/publishers/twitter/${label}.json`\n- `batchSize`\nThe batch size used when adding/deleting rules in bulk. (Defaults to 5000)\n- `parser`\nMuch like the `parser` option allowed in the [Gnip Stream](https://github.com/demian85/gnip#gnipstream) constructor, you can pass a custom parser handler/library for incoming JSON data. This is optional, and defaults to the [json-bigint](https://www.npmjs.com/package/json-bigint) library. [More details](https://github.com/demian85/gnip#optionsparser).\n- `cacheFile`\nInternally `Gnip.Rules` uses a file for caching the current state of the rules configuration, the default path is in the directory of the package. This optional configuration allows you to change the path as the default one may cause problems in applications where `node_modules` is in a read-only filesystems (e.g. AWS Lambda).\n\n## API methods\n\n- `getAll(callback)`\nGet cached rules.\n\n- `update(rules: Array, callback)`\nCreates or replaces the live tracking rules.  \nRules are sent in batches of `options.batchSize`, so you can pass an unlimited number of rules.  \nThe current tracking rules are stored in a local JSON file so you can update the existing rules efficiently without having to remove them all.\nThe callback receives an object as the 2nd argument and contains the number of added and deleted rules.\n\n- `clearCache(callback)`\nClears cached rules.\n\n### The following methods uses Gnip API directly and ignores the local cache. Avoid usage if you are working with too many rules!\n- `live.update(rules: Array, callback)`\n- `live.add(rules: Array, callback)`\n- `live.remove(rules: Array, callback)`\n- `live.getAll(callback)`\n- `live.getByIds(ids: Array, callback)`\n- `live.removeAll(callback)`\n\n# Gnip.Search\nThis class is an EventEmitter and allows you to connect to either the 30 day or full archive search API and start receiving data.\n\n## Constructor options\n\n- `user`\nGNIP account username.\n- `password`\nGNIP account password.\n- `url`\nGNIP Search endpoint url e.g. ```https://gnip-api.twitter.com/search/30day/accounts/{ACCOUNT_NAME}/{LABEL}.json```\n- `query`\nRule to match tweets.\n- `fromDate`\nThe oldest date from which tweets will be gathered. Date given in the format 'YYYYMMDDHHMM'. Optional.\n- `toDate`\nThe most recent date to which tweets will be gathered. Date given in the format 'YYYYMMDDHHMM'. Optional.\n- `maxResults`\nThe maximum number of search results to be returned by a request. A number between 10 and 500. Optional.  \n- `tag`\nUsed to segregate rules and their matching data into different logical groups. Optional.\n- `bucket`\nThe unit of time for which count data will be provided. Options: \"day\", \"hour\", \"minute\". Optional, for /counts calls.\n- `rateLimiter`\nA \u003ca href=\"https://www.npmjs.com/package/limiter\"\u003elimiter\u003c/a\u003e object, used to control the rate of collection. Optional. If unspecified, a rate limit of 30 requests a minute will be shared between Search streams. If you have a non-standard rate limit, you should pass this parameter.\n\n```js\nconst RateLimiter = require('limiter').RateLimiter;\n// Allow 60 requests per minute\nconst limiter = new RateLimiter(60, 'minute');\nconst stream = new Gnip.Search({\n\trateLimiter : limiter,\n  ...\n});\n```\n## API methods\n\n- `start()`\nStart receiving data. At this point you should have registered at least one event listener for 'object' or 'tweet'.\n\n- `end()`\nTerminates the connection.\n\n## Events\n\n- `ready`\nEmitted when tweets have started to be collected.\n- `error`\nEmitted when a recoverable (non fatal) error occurs. \n- `object`\nEmitted for each JSON object.\n- `tweet`\nEmitted for each tweet.\n- `end`\nEmitted when the connection is terminated. If the stream has ended due to a fatal error, the error object will be passed.\n\n# Gnip.Usage\nThis class allows you to track activity consumption across Gnip products.\n\n## Constructor options\n\n- `url`\nGNIP API url, e.g: https://gnip-api.twitter.com/metrics/usage/accounts/{ACCOUNT_NAME}.json\n- `user`\nUsername for authentication.\n- `password`\nPassword for authentication.\n\n```js\nconst usage = new Gnip.Usage({\n\turl : 'https://gnip-api.twitter.com/metrics/usage/accounts/{ACCOUNT_NAME}.json',\n\tuser : 'xxx',\n\tpassword : 'xxx'\n});\n```\n\n## API Methods\n\n- `get(callback)`\nError passed as first parameter to callback, result as second.\n\n- `get(parameters: object, callback)`\n\u003ca href=\"http://support.gnip.com/apis/usage_api/api_reference.html#GETData\"\u003eInformation on request parameters can be found here.\u003c/a\u003e\n\n```js\nusage.get({ bucket:'day', fromDate:'201612010000', toDate:'201612100000' },function( err, body )\n{\n\t...\n});\n````\n\nInstallation\n====\n\tnpm install gnip\n\nExample Usage\n====\n```js\nconst Gnip = require('gnip');\n\nconst stream = new Gnip.Stream({\n  url : 'https://gnip-stream.twitter.com/stream/powertrack/accounts/xxx/publishers/twitter/prod.json',\n  user : 'xxx',\n  password : 'xxx',\n  backfillMinutes: 5 // optional\n});\nstream.on('ready', function() {\n  console.log('Stream ready!');\n});\nstream.on('tweet', function(tweet) {\n  console.log(tweet);\n});\nstream.on('error', function(err) {\n  console.error(err);\n});\n\nconst rules = new Gnip.Rules({\n  url : 'https://gnip-api.twitter.com/rules/powertrack/accounts/xxx/publishers/twitter/prod.json',\n  user : 'xxx',\n  password : 'xxx',\n  batchSize: 1234 // not required, defaults to 5000\n});\n\nconst newRules = [\n  '#hashtag', \n  'keyword', \n  '@user',\n  {value: 'keyword as object'},\n  {value: '@demianr85', tag: 'rule tag'}\n];\n\nrules.update(newRules, function(err) {\n  if (err) throw err;\n  stream.start();\n});\n\nconst search = new Gnip.Search({\n  url : 'https://gnip-stream.twitter.com/stream/powertrack/accounts/xxx/publishers/twitter/prod.json',\n  user : 'xxx',\n  password : 'xxx',\n  query : '@user'\n});\n\nsearch.on('tweet', function(tweet) {\n  console.log(tweet);\n});\n\nsearch.on('error', function(err) {\n  console.error(err);\n});\n\nsearch.on('end', function(err) {\n  if( err ) \n    console.error(err);\n});\n\n// search counts usage\nconst counts = new Gnip.Search({\n  url : 'https://gnip-stream.twitter.com/stream/powertrack/accounts/xxx/publishers/twitter/prod/counts.json',\n  user : 'xxx',\n  password : 'xxx',\n  query : '@user',\n  bucket: 'day'\n});\n\ncounts.on('object', function(object) {\n  console.log(object.results);\n  counts.end();\n});\n ```\n\t\nMore details and tests soon...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdemian85%2Fgnip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdemian85%2Fgnip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdemian85%2Fgnip/lists"}