{"id":15713507,"url":"https://github.com/traven-b/mymodern","last_synced_at":"2026-08-12T16:31:26.730Z","repository":{"id":246587049,"uuid":"821181980","full_name":"Traven-B/mymodern","owner":"Traven-B","description":"Crystal app lists checked-out and ready-hold books at multiple library websites. Uses CSS selectors, CSP concurrency. Beginner-friendly with mock data and easy inclusion of custom scraping logic.","archived":false,"fork":false,"pushed_at":"2025-12-11T06:33:50.000Z","size":629,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-12T02:23:40.920Z","etag":null,"topics":["beginner-friendly","book-tracking","cli-application","concurrency","crystal","crystal-lang","csp","css-selectors","fibers","html-parsing","http-client","library-tools","mock-data","modular-design","public-libraries","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Crystal","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Traven-B.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-06-28T01:51:39.000Z","updated_at":"2025-12-11T06:33:54.000Z","dependencies_parsed_at":"2024-06-28T21:47:51.948Z","dependency_job_id":"91e9ca90-7b26-492a-a337-221bd356b3fa","html_url":"https://github.com/Traven-B/mymodern","commit_stats":null,"previous_names":["traven-b/mymodern"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Traven-B/mymodern","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmymodern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmymodern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmymodern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmymodern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Traven-B","download_url":"https://codeload.github.com/Traven-B/mymodern/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmymodern/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36572155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-06T04:43:03.162Z","status":"online","status_checked_at":"2026-08-12T02:00:07.134Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beginner-friendly","book-tracking","cli-application","concurrency","crystal","crystal-lang","csp","css-selectors","fibers","html-parsing","http-client","library-tools","mock-data","modular-design","public-libraries","web-scraping"],"created_at":"2024-10-03T21:31:50.794Z","updated_at":"2026-08-12T16:31:26.724Z","avatar_url":"https://github.com/Traven-B.png","language":"Crystal","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mymodern\n\nmymodern is a command line program written in [Crystal][] that scrapes web\npages from public library websites. It uses CSS selector rules to parse the\npages and lists checked-out books and books on hold ready for pickup.\n\nThe program uses Crystal's concurrency support to handle multiple HTTP requests\nsimultaneously. It is modular, allowing you to define modules for different\nlibrary websites and customize the scraping logic for each.\n\nWe encourage you to explore Crystal, which features Ruby-like syntax and a\nconcurrency model similar to Go's. Additionally, we have published Ruby and Go\nversions of this program, all of which provide a working demo out of the box.\n\n## Installation\n\nTo set up mymodern on your machine:\n\n1. Clone the repository.\n2. Navigate to the project directory.\n3. Run the following commands:\n\n```\nshards install\nshards build\n```\n\nThe library [kostya/lexbor](https://github.com/kostya/lexbor) parses HTML using\nCSS selectors. It wraps native C code and the compilation step for this\ndependency happens automatically when you run `shards install`\n\nOnce installed, the library will be available locally within your project directory.\n\nFor more information about the project and how to customize it for your needs,\nplease refer to our:\n\n- [Detailed README](project_docs/DETAILED_README.md) for **detailed\n  installation instructions**, and subsequent setup and usage notes.\n\n- [Project Structure Documentation](project_docs/PROJECT_STRUCTURE.md) which\n  outlines the specific parts you'll need to adapt or modify to work with your\n  library's website.\n\n## Usage\n\n```terminal\n~/at_the_prompt/crystal/mymodern$ bin/mymodern --help\nUsage: mymodern [OPTIONS]\nScrape pages at public libraries' web sites.\n    -m, --mock                       this option mocks everything\n    -t, --trace                      trace where it's all happening\n    -h, --help                       show this message\n```\n\nExample program output:\n\n```terminal\nHennepin Books Out\n\nThe Astronomer\nTuesday December 04, 2018\n\nSubterranean Twin Cities\nWednesday December 05, 2018\nRenewed: 1 time\n2 people waiting\n\nHennepin Books on Hold\n\nA History of America in Ten Strikes\nWednesday November 14, 2018\n\nSt. Paul Books Out\n\nSt. Paul Books on Hold\n\nThe mysterious flame of Queen Loana\nSaturday November 17, 2018\n\n```\n\n## Development\n\nThe program calls upon identically named methods provided by one or another\nlibrary's module.\n\n```\nmodule Hennepin\n  def self.lib_data\n    # returns data in the form of a NamedTuple\n\n  def self.parse_checkedout_page(page)\n    # return a sorted array of CheckedOutRecord\n    # uses already existing record and comparison definitions\n\n  def self.parse_on_hold_page(page)\n    # return a sorted array of OnHoldRecord\n```\n\nCode for another library where you're a regular patron would do the same, and\nparts of the program that invoke these methods iterate over an array of the\nmodule names, and don't hard code the number or names of the modules.\n\nYou can adapt the modules that define these few methods, name them Springfield\nand Shelbyville, and require them from a single file which also lists them in\nan array of module names.\n\n```\nrequire \"./hennepin\"\nrequire \"./stpaul\"\n\nMODULE_NAMES = [Hennepin, StPaul]\n```\n\nSo this can be used to scrape web pages at 1, 2, or 3 libraries, and all things\nbeing equal, runs in constant time no matter how many you're doing. *\n\n\\* Because concurrency.\n\n1 web site, 1.5 times faster. All things being equal, 2 websites, could be 3\ntimes faster, 3 web sites, 4.5 times faster. Conceivably.\n\n## Contributing\n\n1. Fork the repository (\u003chttps://github.com/Traven-B/mymodern/fork\u003e)\n2. Create a new feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n## Contributors\n\n- [Traven-B](https://github.com/Traven-B) Michael Kamb - creator, maintainer\n\n[Crystal]: https://crystal-lang.org\n[kostya/lexbor]: https://github.com/kostya/lexbor\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftraven-b%2Fmymodern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftraven-b%2Fmymodern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftraven-b%2Fmymodern/lists"}