{"id":22828687,"url":"https://github.com/busterc/uniqueue","last_synced_at":"2025-03-31T01:17:53.271Z","repository":{"id":57386357,"uuid":"143066326","full_name":"busterc/uniqueue","owner":"busterc","description":":snowflake: only queue unique items and retrieve them only once","archived":false,"fork":false,"pushed_at":"2018-08-25T22:01:56.000Z","size":75,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T15:44:52.672Z","etag":null,"topics":["crawl","crawling","queue","queues","set","unique","uniqueness","walking"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/busterc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-31T20:47:46.000Z","updated_at":"2023-09-08T17:43:21.000Z","dependencies_parsed_at":"2022-09-05T05:00:10.478Z","dependency_job_id":null,"html_url":"https://github.com/busterc/uniqueue","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/busterc%2Funiqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/busterc%2Funiqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/busterc%2Funiqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/busterc%2Funiqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/busterc","download_url":"https://codeload.github.com/busterc/uniqueue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246399803,"owners_count":20770909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawl","crawling","queue","queues","set","unique","uniqueness","walking"],"created_at":"2024-12-12T19:11:31.850Z","updated_at":"2025-03-31T01:17:53.250Z","avatar_url":"https://github.com/busterc.png","language":"JavaScript","readme":"# uniqueue [![NPM version][npm-image]][npm-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][daviddm-image]][daviddm-url] [![Coverage percentage][coveralls-image]][coveralls-url]\n\n\u003e only queue unique items and retrieve them only once\n\n## Installation\n\n```sh\n$ npm install --save uniqueue\n```\n\n## Usage\n\n```js\n'use strict';\n\nconst Uniqueue = require('uniqueue');\n\nconst { push, pop, remaining, queued, processed, clear } = new Uniqueue();\n\nconst urls = [\n  'https://google.com',\n  'https://google.com',\n  'https://google.com/',\n  'https://www.google.com'\n];\n\n// Only add unique items to the queue\nurls.forEach(url =\u003e {\n  push(url);\n});\nconsole.log('=\u003e', queued.size, processed.size, remaining());\n// =\u003e 3 0 3\n\n// Extract items from queue\nwhile (remaining() \u003e 0) {\n  console.log('=\u003e remaining:', remaining());\n  console.log('=\u003e', pop());\n  console.log('=\u003e processed:', processed.size);\n  console.log();\n}\n// =\u003e remaining: 3\n// =\u003e https://google.com\n// =\u003e processed: 1\n//\n// =\u003e remaining: 2\n// =\u003e https://google.com/\n// =\u003e processed: 2\n//\n// =\u003e remaining: 1\n// =\u003e https://www.google.com\n// =\u003e processed: 3\n\n// Clear the queue\nclear();\nconsole.log('=\u003e', queued.size, processed.size, remaining());\n// =\u003e 0 0 0\n\n// Use a custom matcher function to prevent similar duplicates\nconst matcher = (a, b) =\u003e {\n  return (\n    a.replace(/[^a-z]/gi, '') === b.replace(/[^a-z]/gi, '') ||\n    a.replace(/:\\/\\/www\\./i, '://') === b.replace(/:\\/\\/www\\./i, '://')\n  );\n};\nurls.forEach(url =\u003e {\n  push(url, matcher);\n});\nconsole.log('=\u003e', [...queued]);\n// =\u003e [ 'https://google.com' ]\n\n// * Use a custom matcher function during initialization\nconst queue = new Uniqueue(matcher);\n// ...\n```\n\n## API\n\n### Uniqueue([matcher])\n\nReturns a new `queue` instance.\n\n- #### matcher\n\n  - `Optional` : `Function` is evaluated on every attempt to add an item to the queue.\n\n    - Overrides the default `Set` [value equality check](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set#Value_equality)\n\n    - e.g. `(a, b) =\u003e a.name.toUpperCase() === b.name.toUpperCase()` will prevent `{ name: \"Bob\" }` from being added if `{ name: \"BOB\" }` is already in the queue.\n\n### queue\n\n`Uniqueue` instance.\n\n- #### .push(item [, matcher])\n\n  Adds an item of `Any` type to the queue.\n\n  - ##### matcher\n\n    `Optional` : `Function` overrides the `matcher` function provided during initialization (if it was provided).\n\n- #### .pop()\n\n  Returns the next available item from the queue or `null` if no items are available.\n\n- #### .remaining()\n\n  Returns the `Number` of available items from the queue.\n\n- #### .queued\n\n  The `Set` holding all unique items added to the queue.\n\n- #### .processed\n\n  The `Set` holding all items that have been `pop()`'d from the queue.\n\n- #### .clear()\n\n  Clears the queue, including `.queued` and `.processed`.\n\n## License\n\nISC © [Buster Collings]()\n\n[npm-image]: https://badge.fury.io/js/uniqueue.svg\n[npm-url]: https://npmjs.org/package/uniqueue\n[travis-image]: https://travis-ci.org/busterc/uniqueue.svg?branch=master\n[travis-url]: https://travis-ci.org/busterc/uniqueue\n[daviddm-image]: https://david-dm.org/busterc/uniqueue.svg?theme=shields.io\n[daviddm-url]: https://david-dm.org/busterc/uniqueue\n[coveralls-image]: https://coveralls.io/repos/busterc/uniqueue/badge.svg\n[coveralls-url]: https://coveralls.io/r/busterc/uniqueue\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbusterc%2Funiqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbusterc%2Funiqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbusterc%2Funiqueue/lists"}