{"id":21201678,"url":"https://github.com/oleander/wire","last_synced_at":"2025-03-14T22:26:06.037Z","repository":{"id":56898298,"uuid":"1532901","full_name":"oleander/Wire","owner":"oleander","description":"Run a strict amount of threads during a time interval, primarily used for web scraping.","archived":false,"fork":false,"pushed_at":"2011-04-02T17:02:52.000Z","size":100,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-03-15T01:52:16.399Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/oleander/wire","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oleander.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-03-27T15:03:00.000Z","updated_at":"2012-12-31T16:28:06.000Z","dependencies_parsed_at":"2022-08-21T02:20:36.138Z","dependency_job_id":null,"html_url":"https://github.com/oleander/Wire","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oleander%2FWire","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oleander%2FWire/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oleander%2FWire/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oleander%2FWire/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oleander","download_url":"https://codeload.github.com/oleander/Wire/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243654401,"owners_count":20325893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T20:10:28.733Z","updated_at":"2025-03-14T22:26:06.011Z","avatar_url":"https://github.com/oleander.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wire\n\nRun a strict amount of threads during a time interval, primarily used for [web scraping](http://en.wikipedia.org/wiki/Web_scraping).\n\n## How to use\n\n### Example 1 - Basic\n\nStart 100 threads, only run 10 at the same time, with a 3 second delay between each new thread, except the first 10.\n\n    100.times do\n      Wire.new(max: 10, wait: 3) do\n        # Do stuff\n      end\n    end\n\n### Example 2 - Timer\n\n    11.times do\n      Wire.new(max: 10, wait: 1) do\n        sleep 0.1\n      end\n    end\n\nTime to run: ~ *1.2 seconds*.\n\nThis is how it works.\n\n- 11 threads is created, done at time 0.\n- Running 10 threads, done at time 0.1\n- Wait 1 second, done at time 1.1\n- Start the 11th thread, done at time 1.2\n    \n### Example 3 - Pass arguments\n\n    Wire.new(max: 10, wait: 1, vars: [\"A\", \"B\"]) do |first, last|\n      puts first # =\u003e \"A\"\n      puts last # =\u003e \"B\"\n    end\n\n    100.times do |n|\n      Wire.new(max: 10, wait: 1, vars: [n]) do |counter|\n        puts counter\n      end\n    end\n    \n    # =\u003e 1 2 3 4 5 ...\n\n### Example 4 - Scraping\n\nThis project was originally build to solve the request limit problem when using [Spotify´s Meta API](http://developer.spotify.com/en/metadata-api/overview/).\n\n\u003e In order to make the Metadata API snappy and open for everyone to use, rate limiting rules apply. If you make too many requests too fast, you’ll start getting 403 Forbidden responses. When rate limiting has kicked in, you’ll have to wait 10 seconds before making more requests. The rate limit is currently 10 request per second per ip. This may change.\n\nWe wanted to make as many request as possible without being banned due to the rate limit.\n\n    require \"rest-client\"\n    require \"wire\"\n    require \"uri\"\n\n    a_very_large_list_of_songs = [\"Sweet Home Alabama\", ...]\n    \n    a_very_large_list_of_songs.each do |s|\n      Wire.new(max: 10, wait: 1, vars: [s]) do |song|\n        data = RestClient.get \"http://ws.spotify.com/search/1/track.json?q=#{URI.encode(song)}\"\n        # Do something with the data\n      end\n    end\n\n### Tip\n\nDon't forget to join your threads using `Thread#join`.\n \n    list = []\n    10.times do |n|\n      list \u003c\u003c Thread.new do\n        # Do stuff\n      end\n    end\n    list.map(\u0026:join)\n    \nRead more about [#join](http://corelib.rubyonrails.org/classes/Thread.html#M001145) here.\n\n## Arguments to pass\n\nIngoing arguments to `new`.\n\n- **max** (Integer) The maximum amount of threads to run a the same time. The value 10 will be used if `max` is nil or zero.\n- **wait** (Integer) The time to wait before starting a new thread.\n- **vars** (Array) A list of arguments to the block.\n- **silent** (Boolean) The given block will not raise error if set to true. Default is false.\n- **timeout** (Integer) The maximum time to run *one* thread, default is *no limit*.\n- **retries** (Integer) How many times should we retry? Default is 0.\n- **delay** (Float) Time between each retry. Default is 0.\n\n## How do install\n\n    [sudo] gem install wire\n    \n## Requirements\n\nWire is tested on OS X 10.6.7 using Ruby 1.9.2.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foleander%2Fwire","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foleander%2Fwire","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foleander%2Fwire/lists"}