{"id":13858546,"url":"https://github.com/molybdenum-99/infoboxer","last_synced_at":"2025-04-05T01:07:16.890Z","repository":{"id":33796571,"uuid":"37488972","full_name":"molybdenum-99/infoboxer","owner":"molybdenum-99","description":"Wikipedia information extraction library","archived":false,"fork":false,"pushed_at":"2024-03-01T16:54:22.000Z","size":8563,"stargazers_count":175,"open_issues_count":54,"forks_count":13,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-29T00:08:21.658Z","etag":null,"topics":["data-extraction","mediawiki","wikipedia"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/molybdenum-99.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-06-15T20:16:55.000Z","updated_at":"2025-02-05T11:11:46.000Z","dependencies_parsed_at":"2023-02-12T10:31:12.521Z","dependency_job_id":"3f1a0c88-f884-4320-b85a-ff28af6746e2","html_url":"https://github.com/molybdenum-99/infoboxer","commit_stats":{"total_commits":332,"total_committers":2,"mean_commits":166.0,"dds":0.003012048192771122,"last_synced_commit":"b1770d0d85a4c6db315a3fb8c6b20cae31e08738"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molybdenum-99%2Finfoboxer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molybdenum-99%2Finfoboxer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molybdenum-99%2Finfoboxer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molybdenum-99%2Finfoboxer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/molybdenum-99","download_url":"https://codeload.github.com/molybdenum-99/infoboxer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247271530,"owners_count":20911587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-extraction","mediawiki","wikipedia"],"created_at":"2024-08-05T03:02:13.028Z","updated_at":"2025-04-05T01:07:16.869Z","avatar_url":"https://github.com/molybdenum-99.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Infoboxer\n\n[![Gem Version](https://badge.fury.io/rb/infoboxer.svg)](http://badge.fury.io/rb/infoboxer)\n![Build Status](https://github.com/molybdenum-99/infoboxer/workflows/CI/badge.svg?branch=master)\n[![Coverage Status](https://coveralls.io/repos/molybdenum-99/infoboxer/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/molybdenum-99/infoboxer?branch=master)\n[![Code Climate](https://codeclimate.com/github/molybdenum-99/infoboxer/badges/gpa.svg)](https://codeclimate.com/github/molybdenum-99/infoboxer)\n[![Infoboxer Gitter](https://badges.gitter.im/molybdenum-99/infoboxer.svg)](https://gitter.im/molybdenum-99/infoboxer)\n\n**Infoboxer** is pure-Ruby Wikipedia (and generic MediaWiki) client and\nparser, targeting information extraction (hence the name).\n\nIt can be useful in tasks like:\n\n* get a plaintext abstract of an article (paragraphs before first heading);\n* get structured data variables from page's **infobox**;\n* list page's sections and count paragraphs, images and tables in them;\n* convert some huge \"comparison table\" to data;\n* and much, much more!\n\nThe whole idea is: you can have any Wikipedia page as a parsed tree with\nobvious structure, you can navigate that tree easily, and you have a\nbunch of hi-level helpers method, so typical information extraction\ntasks should be super-easy, one-liners in best cases.\n\n_(For those already thinking \"Why should you do this, we already have\nDBPedia?\" -- please, read \"[Reasons](https://github.com/molybdenum-99/infoboxer/wiki/Reasons)\"\npage in our wiki.)_\n\n## Showcase\n\n```ruby\nInfoboxer.wikipedia.\n  get('Breaking Bad (season 1)').\n  sections('Episodes').templates(name: 'Episode table').\n  fetch('episodes').templates(name: /^Episode list/).\n  fetch_hashes('EpisodeNumber', 'EpisodeNumber2', 'Title', 'ShortSummary')\n# =\u003e [{\"EpisodeNumber\"=\u003e#\u003cVar(EpisodeNumber): 1\u003e, \"EpisodeNumber2\"=\u003e#\u003cVar(EpisodeNumber2): 1\u003e, \"Title\"=\u003e#\u003cVar(Title): Pilot\u003e, \"ShortSummary\"=\u003e#\u003cVar(ShortSummary): Walter White, a 50-year old che...\u003e},\n#     {\"EpisodeNumber\"=\u003e#\u003cVar(EpisodeNumber): 2\u003e, \"EpisodeNumber2\"=\u003e#\u003cVar(EpisodeNumber2): 2\u003e, \"Title\"=\u003e#\u003cVar(Title): Cat's in the Bag...\u003e, \"ShortSummary\"=\u003e#\u003cVar(ShortSummary): Walt and Jesse try to dispose o...\u003e},\n#     ...and so on\n```\n\nDo you _feel_ it now?\n\nYou also can take a look at [Showcase](https://github.com/molybdenum-99/infoboxer/wiki/Showcase).\n\n## Usage\n\n### Install gem\n\nInstall it as usual: `gem 'infoboxer'` in your Gemfile, then `bundle install`.\n\nOr just `[sudo] gem install infoboxer` if you prefer.\n\n### Grab the page\n\n```ruby\n# From English Wikipedia\npage = Infoboxer.wikipedia.get('Argentina')\n# or\npage = Infoboxer.wp.get('Argentina')\n\n# From other language Wikipedia:\npage = Infoboxer.wikipedia('fr').get('Argentina')\n\n# From any wiki with the same engine:\npage = Infoboxer.wiki('http://companywiki.com').get('Our Product')\n```\n\nSee more examples and options at [Retrieving pages](https://github.com/molybdenum-99/infoboxer/wiki/Retrieving%20pages)\n\n### Play with page\n\nBasically, page is a tree of [Nodes](https://github.com/molybdenum-99/infoboxer/wiki/Nodes), you can think of it as some kind of\n[DOM](https://en.wikipedia.org/wiki/Document_Object_Model).\n\nSo, you can navigate it:\n\n```ruby\n# Simple traversing and inspect\nnode = page.children.first.children.first\nnode.to_tree\nnode.to_text\n\n# Various lookups\npage.lookup(:Template, name: /^Infobox/)\n```\n\nSee [Tree navigation basics](https://github.com/molybdenum-99/infoboxer/wiki/Tree-navigation-basics).\n\nOn the top of the basic navigation Infoboxer adds some useful shortcuts\nfor convenience and brevity, which allows things like this:\n\n```ruby\npage.section('Episodes').tables.first\n```\n\nSee [Navigation shortcuts](https://github.com/molybdenum-99/infoboxer/wiki/Navigation-shortcuts)\n\nTo put it all in one piece, also take a look at [Data extraction tips and tricks](https://github.com/molybdenum-99/infoboxer/wiki/Tips-and-tricks).\n\n### infoboxer executable\n\nJust try `infoboxer` command.\n\nWithout any options, it starts IRB session with infoboxer required and\nincluded into main namespace.\n\nWith `-w` option, it provides a shortcut to MediaWiki instance you want.\nLike this:\n\n```\n$ infoboxer -w https://en.wikipedia.org/w/api.php\n\u003e get('Argentina')\n =\u003e #\u003cPage(title: \"Argentina\", url: \"https://en.wikipedia.org/wiki/Argentina\"): ....\n```\n\nYou can also use shortcuts like `infoboxer -w wikipedia` for common\nwikies (and, just for fun, `infoboxer -wikipedia` also).\n\n## Advanced topics\n\n* [Reasons](https://github.com/molybdenum-99/infoboxer/wiki/Reasons) for\n  Infoboxer creation;\n* [Parsing quality](https://github.com/molybdenum-99/infoboxer/wiki/Parsing-quality)\n  (TL;DR: very good, but not ideal);\n* [Performance](https://github.com/molybdenum-99/infoboxer/wiki/Performance)\n  (TL;DR: 0.1-0.4 sec for parsing hugest pages);\n* [Localization](https://github.com/molybdenum-99/infoboxer/wiki/Localization)\n  (TL;DR: For now, you'll need some work to use Infoboxer's\n  most advanced features with non-English or non-WikiMedia wikis; basic\n  and mid-level features work always);\n* If you plan to use Wikipedia or sister projects data in production,\n  please consider [Wikipedia terms and conditions](https://github.com/molybdenum-99/infoboxer/wiki/Wikipedia-terms-and-conditions).\n\n## Compatibility\n\nAs of now, Infoboxer reported to be compatible with any MRI Ruby since 2.0.0\n(1.9.3 previously, dropped since Infoboxer 0.2.0). In Travis-CI tests,\nJRuby is failing due to bug in old Java 7/Java 8 SSL certificate support\n([see here](https://github.com/jruby/jruby/issues/2599)),\nand Rubinius failing 3 specs of 500 by mystery, which is uninvestigated yet.\n\nTherefore, those Ruby versions are excluded from Travis config, though,\nthey may still work for you.\n\n## Links\n\n* [Wiki](https://github.com/molybdenum-99/infoboxer/wiki)\n* [API Docs](http://www.rubydoc.info/gems/infoboxer)\n* [Contributing](https://github.com/molybdenum-99/infoboxer/wiki/Contributing)\n* [Roadmap](https://github.com/molybdenum-99/infoboxer/wiki/Roadmap)\n\n## License\n\n[MIT](https://github.com/molybdenum-99/infoboxer/blob/master/LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolybdenum-99%2Finfoboxer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmolybdenum-99%2Finfoboxer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolybdenum-99%2Finfoboxer/lists"}