{"id":16540996,"url":"https://github.com/linki/thief","last_synced_at":"2025-09-13T06:32:57.612Z","repository":{"id":899765,"uuid":"655276","full_name":"linki/thief","owner":"linki","description":null,"archived":false,"fork":false,"pushed_at":"2010-08-01T20:01:02.000Z","size":9244,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-04T04:41:33.282Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linki.png","metadata":{"files":{"readme":"README.textile","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-05-07T16:34:18.000Z","updated_at":"2021-01-13T19:28:56.000Z","dependencies_parsed_at":"2022-08-16T11:20:50.544Z","dependency_job_id":null,"html_url":"https://github.com/linki/thief","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/linki/thief","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linki%2Fthief","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linki%2Fthief/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linki%2Fthief/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linki%2Fthief/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linki","download_url":"https://codeload.github.com/linki/thief/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linki%2Fthief/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272131792,"owners_count":24878986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T18:53:50.117Z","updated_at":"2025-08-25T21:05:07.402Z","avatar_url":"https://github.com/linki.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"h1. Thief\n\nh2. Installation\n\n\u003cpre\u003e\n  gem install bundler\n  bundle install\n  rake thief:create_tables\n\u003c/pre\u003e\n\nh2. Usage\n\n\u003cpre\u003e\n  bin/thief\n  bin/server\n\u003c/pre\u003e\n\nThen browse to http://localhost:4567\n\n\u003cpre\u003e\n  bin/console\n\u003c/pre\u003e\n\nUse the interactive console\n\nh2. Internals\n\n\u003cpre\u003e\n  Thief.sources \u003c\u003c Thief::Source::SomeSource.new\n  Thief.fetch\n\u003c/pre\u003e\n\nWill go through all assigned sources (must be located in /thief/sources/*.rb) and call their fetch method.\nEach Source has its own table for \"Person\" objects (dapi_people, dapi_wikipedia) and will store all the information in there.\n\n\u003cpre\u003e\n  Thief.integrate\n\u003c/pre\u003e\n\nWill go through all assigned sources (same as above) and call their integrate method.\nEach Integrator will convert its own Person-schema to the integrated schema (Thief::DAPI::Person -\u003e Thief::Person)\n\n\u003cpre\u003e\n  Thief::Person.all.each { |person| puts person.name }\n\u003c/pre\u003e\n\nprints out the stored and integrated people.\n\nh2. Wikipedia\n\nSpecial download mechanism in Thief::Wikipedia::ETL loads the necessary txt file which contains the person data if the file doesn't exist.\n\n\u003cpre\u003e\n  Thief::Wikipedia:ETL.download_file!\n\u003c/pre\u003e\n\nThis method is called by the fetch method if the necessary file containing the person data doesn't exist. It will download the zip file, unzip it and rename and move the contained text file to the\nwikipedia directory.\n\nh2. Extending\n\nYou can copy the dapi.rb and dapi-folder to create a new source with ETL, Integrator and Person objects.\nThe defaults will look under the specific namespace for the etl and integrator classes. (Thief::YourLibrary::ETL)\nYou can change that by overwriting the etl and integrator methods.\n\n\u003cpre\u003e\n  require 'somewhere/over/the/rainbow'\n\n  module Thief\n    module Sources\n      class YourAwesomeSource\n        def etl\n          ::OneETLToRuleThemAll.new\n        end\n      end\n    end\n  end\n\u003c/pre\u003e\n\nh2. Another way to start the server\n\nStart the app on any rack-compatible webserver with the config.ru file. (in production mode in the examples)\n\nThin:\n\n\u003cpre\u003e\n  thin --rackup config.ru --environment production start\n\u003c/pre\u003e\n\nor Unicorn:\n\n\u003cpre\u003e\n  unicorn --env production\n\u003c/pre\u003e\n\nor WEBrick through rackup:\n\n\u003cpre\u003e\n  THIEF_ENV=production rackup --server webrick config.ru\n\u003c/pre\u003e\n\nor with shotgun:\n\n\u003cpre\u003e\n  shotgun --env production\n\u003c/pre\u003e\n\nand so on..\n\nh2. Special\n\nThief::ETL now has inheritable mechanisms to download and extract files as well as store them in their specific temporary directories.\n\nThief::Integrator now has inheritable mechanisms for defining a mapping from a source specific schema to the global schema (experimental)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinki%2Fthief","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinki%2Fthief","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinki%2Fthief/lists"}