{"id":15546574,"url":"https://github.com/philou/storexplore","last_synced_at":"2025-10-12T00:20:15.437Z","repository":{"id":12575712,"uuid":"15246299","full_name":"philou/storexplore","owner":"philou","description":"Transform online stores into APIs !","archived":false,"fork":false,"pushed_at":"2019-01-04T06:13:08.000Z","size":132,"stargazers_count":21,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-08T22:29:23.679Z","etag":null,"topics":["api","scrapper"],"latest_commit_sha":null,"homepage":"http://philou.github.io/storexplore","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-12-17T05:48:55.000Z","updated_at":"2025-05-09T20:19:32.000Z","dependencies_parsed_at":"2022-09-11T16:40:44.048Z","dependency_job_id":null,"html_url":"https://github.com/philou/storexplore","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/philou/storexplore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philou%2Fstorexplore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philou%2Fstorexplore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philou%2Fstorexplore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philou%2Fstorexplore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philou","download_url":"https://codeload.github.com/philou/storexplore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philou%2Fstorexplore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002105,"owners_count":26083307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","scrapper"],"created_at":"2024-10-02T13:02:49.007Z","updated_at":"2025-10-12T00:20:15.420Z","avatar_url":"https://github.com/philou.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/philou/storexplore.svg?branch=master)](https://travis-ci.org/philou/storexplore) [![Test Coverage](https://codeclimate.com/github/philou/storexplore/badges/coverage.svg)](https://codeclimate.com/github/philou/storexplore) [![Code Climate](https://codeclimate.com/github/philou/storexplore/badges/gpa.svg)](https://codeclimate.com/github/philou/storexplore)\n\n# Storexplore\n\nA ruby gem that simplifies the declaration of APIs on online stores through scraping.\n\n## Why\nOnce upon a time, I wanted to create online groceries with great user experience ! That's how I started [mes-courses.fr](https://github.com/philou/mes-courses). Unfortunately, most online groceries don't have APIs, so I resorted to scrapping. Scrapping comes with its (long) list of problems as well !\n\n* Scrapping code is a mess\n* The scrapped html can change at any time\n* Scrappers are difficult to test\n\nRefactoring by refactoring, I extracted this library which defines scrappers for any online store in a straightforward way (check [auchandirect-scrAPI](https://github.com/philou/auchandirect-scrAPI) for my real world usage). A scrapper definition consists of :\n\n* a scrapper definition file\n* the selectors for the links\n* the selectors for the content you want to capture\n\nAs a result of using storexplore for mes-courses, the scrapping code was split between the storexplore gem and my special scrapper definition :\n\n* This made the whole overall code cleaner\n* I could write simple and reliable tests\n* Most importantly, I could easily keep pace with the changes in the online store html\n\n## Tutorial\n\n[![How to define an API on ikea.us in 10 minutes using storexplore](https://raw.githubusercontent.com/philou/storexplore/master/ikea.jpg)](http://www.youtube.com/watch?v=O30xReGgdVU)\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n    gem 'storexplore'\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install storexplore\n\nIn order to be able to enumerate all items of a store in constant memory,\nStorexplore requires ruby 2.0 for its lazy enumerators.\n\n## Usage\n\nOnline stores are typically organized as hierarchies. For example [Ikea (US)](http://www.ikea.com/us/en) is organized as follows :\n\n    Ikea\n    |-\u003e Living room\n    |   |-\u003e Sofas \u0026 armchairs\n    |   |   |-\u003e Fabric Sofas\n    |   |   |   |-\u003e Norsborg Sofa\n    |   |   |   |-\u003e Norborg Loveseat\n    |   |   |   |-\u003e ...\n    |   |   |   |-\u003e Pöang Footstool cushion\n    |   |   |-\u003e Leather Sofas\n    |   |   |-\u003e ...\n    |   |   |-\u003e Armchairs\n    |   |-\u003e TV \u0026 media furniture\n    |   |-\u003e ...\n    |   |-\u003e Living room textiles \u0026 rugs\n    |-\u003e Bedroom\n    |-\u003e ...\n    |-\u003e Dining\n\nStorexplore builds hierarchical APIs on the following pattern :\n\n    Store\n    |-\u003e Category 1\n    |   |-\u003e Sub Category 1\n    |   |   |-\u003e Item 1\n    |   |   |-\u003e ...\n    |   |   |-\u003e Item n\n    |   |-\u003e Sub Category 2\n    |   |-\u003e ...\n    |   |-\u003e Sub Category n\n    |-\u003e Category 2\n    |-\u003e ...\n    |-\u003e Category n\n\nThe store is like a root category. Any level of depth is allowed. Any category, at any depth level can have both children categories and items. Items cannot have children of any kind. Both categories and items can have attributes.\n\nAll searching of children and attributes is done through mechanize/nokogiri selectors (css or xpath).\n\nHere is a sample store api declaration for [Ikea](http://www.ikea.com/us/en) again:\n\n```ruby\nStorexplore::Api.define 'ikea.com/us' do\n\n  categories '.departmentLinkBlock a' do\n    attributes do\n      { :name =\u003e page.get_one(\"#breadCrumbNew .activeLink a\").content.strip }\n    end\n\n    categories '.departmentLinks a' do\n      attributes do\n        { :name =\u003e page.get_one(\"#breadCrumbNew .activeLink a\").content.strip }\n      end\n\n      categories 'a.categoryName' do\n        attributes do\n          { :name =\u003e page.get_one(\"#breadCrumbNew .activeLink a\").content.strip }\n        end\n\n        items '.productDetails \u003e a' do\n          attributes do\n            {\n              :name =\u003e page.get_one('#name').content.strip,\n              :type =\u003e page.get_one('#type').content.strip,\n              :price =\u003e page.get_one('#price1').content.strip.sub('$','').to_f,\n              :salesArgs =\u003e page.get_one('#salesArg').content.strip,\n              :image =\u003e page.get_one('#productImg').attributes['src'].content,\n              :ikea_id =\u003e page.uri.to_s.match(\"^.*\\/([0-9]+)\\/?$\").captures.first\n            }\n          end\n        end\n      end\n    end\n  end\nend\n```\n\nThis defines a hierarchical API on the IKEA store that will be used to browse any store which URI contains `ikea.com/us`.\n\nNow here is how this API can be accessed to pretty print all its content:\n\n```ruby\nStorexplore::Api.browse('http://www.ikea.com/us/en').categories.each do |category|\n\n  puts \"category: #{category.title.strip}\"\n  puts \"attributes: #{category.attributes}\"\n\n  category.categories.each do |sub_category|\n\n    puts \"  category: #{sub_category.title.strip}\"\n    puts \"  attributes: #{sub_category.attributes}\"\n\n    sub_category.categories.each do |sub_sub_category|\n\n      puts \"    category: #{sub_sub_category.title.strip}\"\n      puts \"    attributes: #{sub_sub_category.attributes}\"\n\n      sub_sub_category.items.each do |item|\n\n        puts \"      item: #{item.title.strip}\"\n        puts \"      attributes: #{item.attributes}\"\n\n      end\n    end\n  end\nend\n```\n\n(This sample can be found in samples/ikea.rb)\n\n## Testing\n\nNOTE : please keep in mind that these testing utilities have been extracted from my first real use case ([auchandirect-scrAPI](https://github.com/philou/auchandirect-scrAPI)) and might still rely on assumptions coming from there. Any help cleaning this up is welcome.\n\n### Testing Code Relying On A Scrapped Thirdparty\n\nThis can be quite a challenge. Storexplore can help you with that :\n\n* it provides a customizable offline (disk) dummy store generator\n* it provides an API for this store\n* As long as your dummy store provides the same attributes than the real store, you can use it in your tests\n\nDummy stores can be generated to the file system using the Storexplore::Testing::DummyStore and Storexplore::Testing::DummyStoreGenerator classes.\n\nTo use it, add the following to your spec_helper.rb for example :\n\n```ruby\nrequire 'storexplore/testing'\n\nStorexplore::Testing.config do |config|\n  config.dummy_store_generation_dir= File.join(Rails.root, '../tmp')\nend\n```\n\nIt is then possible to generate a store with the following :\n\n```ruby\nDummyStore.wipe_out_store(store_name)\n@store_generator = DummyStore.open(store_name)\n@store_generator.generate(3).categories.and(3).categories.and(item_count).items\n```\n\nYou can add custom elements with explicit values :\n\n```ruby\n@store_generator.\n  category(cat_name = \"extra long category name\").\n  category(sub_cat_name = \"extra long sub category name\").\n  item(item_name = \"super extra long item name\").generate().\n    attributes(price: 12.3)\n```\n\nStorexplore provides an api definition for dummy stores in\n'storexplore/testing/dummy_store_api'. It can be required independently if\nneeded.\n\n### Testing Your Own Scrapper\n\nStorexplore also ships with an rspec shared examples macro. It guarantees basic scrapper well behavior such as the presence of many categories, of item names and prices\n\n```ruby\nrequire 'storexplore/testing'\n\ndescribe \"MyStoreApi\" do\n  include Storexplore::Testing::ApiSpecMacros\n\n  it_should_behave_like_any_store_items_api\n\n  ...\n\nend\n```\n\n### Summary Testing Files To Require\n\n* To only get the api definition for a previously generated dummy store, it is enough to require 'storexplore/testing/dummy_store_api'\n* To be able to generate and scrap dummy stores, it's needed to require 'storexplore/testing/dummy_store_generator'\n* To do all the previous and to use rspec utilities, require 'storexplore/testing'\n\n## Contributing\n\n1. Fork it\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new Pull Request\n\n## Authors\n\n* [Philippe Bourgau](http://philippe.bourgau.net)\n\n\u003ca href=\"https://github.com/philou/storexplore\"\u003e\u003cimg style=\"position: absolute; top: 0; right: 0; border: 0;\" src=\"https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png\" alt=\"Fork me on GitHub\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilou%2Fstorexplore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilou%2Fstorexplore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilou%2Fstorexplore/lists"}