{"id":22286971,"url":"https://github.com/koraa/growler","last_synced_at":"2025-03-25T20:45:37.610Z","repository":{"id":6196282,"uuid":"7426904","full_name":"koraa/growler","owner":"koraa","description":"Growler is an educational web crawler. It's main purpose is to help me learn rails.","archived":false,"fork":false,"pushed_at":"2018-11-27T12:30:40.000Z","size":27,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-30T18:22:49.683Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koraa.png","metadata":{"files":{"readme":"README.rdoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-03T17:22:17.000Z","updated_at":"2018-11-27T12:30:43.000Z","dependencies_parsed_at":"2022-08-26T02:23:54.478Z","dependency_job_id":null,"html_url":"https://github.com/koraa/growler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fgrowler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fgrowler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fgrowler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fgrowler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koraa","download_url":"https://codeload.github.com/koraa/growler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245542794,"owners_count":20632566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-03T16:58:47.445Z","updated_at":"2025-03-25T20:45:37.581Z","avatar_url":"https://github.com/koraa.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Growler\n\nGrowler is an educational web crawler.\nIt's main purpose is to help me learn rails.\n\n== Desired Features\n\n* A web frontend for administrators\n* A Crawler\n* An analyzer\n\n=== Wishlist\n    \n* A web frontend for users\n* A REST api\n** Makeing the frontends frontends for _this_ api\n\n== Model/Outline\n\n\n* *DB* Data (Structures) stored in the DB\n* *Key* Data (Actual keys) stored in DB\n* *Jobs* jobs to be exectuted\n* *Code* executable code\n* *User* User Interface\n* *API* Application Programmer Interface\n* *List:\u003ctype\u003e* List of elements \n\n=== Schema\n\n* Data *SETTINGS*\n    Key       :: *max-dns* How often do we allow dns requests for the same url\n---------------------------------------\n* Data *Snapshot*\n    Key       :: *DOM*\n    Key       :: *timestamp*\n---------------------------------------\n* Data *URL*\n    Key       :: *URL*\n    Reference :: *Domain*\n* Data *Domain*\n    Key       :: *Domain*\n    Reference :: *IP*\n    Key       :: *Last Request* (timestamp)\n* Data *IP*\n    Key       :: *IP-Addr*\n    Key       :: *Last Request* (timestamp)\n----------------------------------------\n* *Jobs* (in general)\n    Key       :: *Priority* (float - 0 \u003c= x \u003c= 1 - Where to put this task in the Queue; 0 is front; 1 is last)\n    Key       :: *Repeat*   (bool - Should this job be repeated?)\n    Code      :: *invoke*\n* Jobs *Crawler*\n    # There is a crawler for every url\n    Reference :: *URL*\n* Jobs *Processing*\n    # There is a processing job for every snapshot for every data-mining plugin\n    Reference :: *Timestamp*\n\n=== Timeline\n\n1. Initialize database\n2. Create basic fetcher\n3. Extract urls and generate recoursive crawler jobs\n4. Add no-auth admin interface\n5. Add auth to admin interface\n6. Initiate crawler jobs from admin iface\n7. Add possibility to stop crawler\n8. Add index/search index\n\n=== Remarks\n\n* How do we organize the data-mining plugins? Their DB schemata?\n\n* Instad of handling dns in rails, use decent dns caching server?\n    * Multiple dns server? Loadbalancing?\n    -\u003e Ignore DNS optimizations in the beginning\n* Support multiple IPs? IPv4/IPv6?\n* Time based jobs for dns (and timeouts) so all the timed events dont block the queue\n* Ignore non html? non text?\n* Proxys?\n* Crawling maxdepth?\n* Parameterized URLs?\n* Multithreading?\n* Non Blocking IO?\n\n== Links\n\n* https://github.com/defunkt/resque\n* https://github.com/jeffkreeftmeijer/navvy\n* https://github.com/collectiveidea/delayed_job\n\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoraa%2Fgrowler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoraa%2Fgrowler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoraa%2Fgrowler/lists"}