{"id":28325430,"url":"https://github.com/traven-b/myasync","last_synced_at":"2025-06-10T20:04:20.172Z","repository":{"id":291412765,"uuid":"834330660","full_name":"Traven-B/myasync","owner":"Traven-B","description":"Ruby app lists checked-out and ready-hold books at multiple library websites. Uses CSS selectors, async http concurrency. Beginner-friendly with mock data and easy inclusion of custom scraping logic.","archived":false,"fork":false,"pushed_at":"2025-05-06T12:28:51.000Z","size":128,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-02T06:18:16.398Z","etag":null,"topics":["async","beginner-friendly","book-tracking","cli-application","concurrency","css-selectors","fibers","html-parsing","http-client","library-tools","mock-data","modular-design","public-libraries","ruby","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Traven-B.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-27T00:56:05.000Z","updated_at":"2025-05-06T12:28:54.000Z","dependencies_parsed_at":"2025-05-04T14:22:58.477Z","dependency_job_id":"b4e6ffce-95d0-4b2a-bafc-a4f650564a5b","html_url":"https://github.com/Traven-B/myasync","commit_stats":null,"previous_names":["traven-b/myasync"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmyasync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmyasync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmyasync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmyasync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Traven-B","download_url":"https://codeload.github.com/Traven-B/myasync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Traven-B%2Fmyasync/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259143542,"owners_count":22811903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","beginner-friendly","book-tracking","cli-application","concurrency","css-selectors","fibers","html-parsing","http-client","library-tools","mock-data","modular-design","public-libraries","ruby","web-scraping"],"created_at":"2025-05-25T21:13:29.608Z","updated_at":"2025-06-10T20:04:20.161Z","avatar_url":"https://github.com/Traven-B.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# myasync\n\nmyasync is a command line program written in [Ruby][] that scrapes web\npages from public library websites. It uses CSS selector rules to parse the\npages and lists checked-out books and books on hold ready for pickup.\n\nThe program uses Ruby's concurrency support to handle multiple HTTP requests\nsimultaneously. It is modular, allowing you to define modules for different\nlibrary websites and customize the scraping logic for each.\n\n## Usage\n\nThe program works out of the box as a demo when given the --mock option.\n\n```sh\napp/myasync -h\nUsage: myasync [OPTIONS]\nScrape pages at public libraries' web sites.\n    -m, --mock                       this option mocks everything\n    -s, --sleep-range 2,2.2          1 (or 2) comma separated numbers specifying\n                                     (range of) seconds that mocked requests sleep\n    -l, --local                      use server on localhost\n    -t, --trace                      trace where it's all happening\n    -h, --help, --usage              Show this message\n```\n\nExample program output:\n\n```sh\nHennepin Books Out\n\nThe Astronomer\nTuesday December 04, 2018\n\nSubterranean Twin Cities\nWednesday December 05, 2018\nRenewed: 1 time\n2 people waiting\n\nHennepin Books on Hold\n\nA History of America in Ten Strikes\nWednesday November 14, 2018\n\nSt. Paul Books Out\n\nSt. Paul Books on Hold\n\nThe mysterious flame of Queen Loana\nSaturday November 17, 2018\n```\n\n## Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone --depth 1 https://github.com/Traven-B/myasync.git\n   cd myasync\n   ```\n\n2. **Install Dependencies**:\n\n   Ensure you have Bundler installed:\n   ```bash\n   which bundle\n   ```\n\n   If not found:\n   ```bash\n   gem install bundler\n   ```\n\n   Configure Bundler to install gems locally:\n   ```bash\n   bundle config set --local path vendor/bundle\n   ```\n\n   to create a `.bundle/config` file with instruction to install gems in project directory.\n   ```\n   ---\n   BUNDLE_PATH: \"vendor/bundle\"\n   ```\n\n   Install the required gems:\n   ```bash\n   bundle install\n   ```\n\n3. **Run the Application**:\n   Use `bundle exec` to ensure the correct gem versions are used:\n\n   ```bash\n   bundle exec app/mymodest --mock\n   ```\n\n## Quick Start\n\n**When you first run the program, use the `--mock` option.**\nThis lets you see example output immediately, even if you haven’t set up credentials,\nURLs, and scraping routines.\n\nThe following will work also:\n\n```sh\nmyasync --help\nmyasync --mock                 # See simulated output right away\nmyasync --mock --trace         # With trace/debug output\nmyasync --mock --sleep-range 2 --trace\nmyasync --mock --sleep-range 2,2.2 --trace\n```\n\n### Fake the Internet: Use a Local Server\n\nInstead of faking internet requests, you can run against a local server:\n\n```sh\n$ bundle exec app/myasync --local\nFailed to open TCP connection to localhost:3000\n\nStart a server on localhost. Run:\nbundle exec lib/local_server.rb\n\n```\n\nStart a server on localhost in a separate terminal:\n\n```sh\n$ bundle exec lib/local_server.rb\n== Sinatra (v4.1.1) has taken the stage on 3000 for development with backup from WEBrick\n(Ctrl-C to exit)\n```\n\nThen try your command again:\n\n```sh\n$ bundle exec app/myasync --local\n```\n\nThe --local option is for advanced development and concurrency testing, but\nlike --mock, it also works out of the box.\n\n## Project Structure\n\n- `lib/`: Contains the main code using the Faraday HTTP gem - a more adaptable but involved implementation.\n    - `application.rb` (symlink): Points to one of the two async implementations below.\n    - `_app_Async_do.rb`: Implementation using `Async do ... end` blocks within a method.\n    - `_app_async_def.rb`: Version using `async/await` with syntactic sugar.\n- `archive/`: Contains earlier code using the `async/http/internet` gem.\n    - A similar setup with two different async idioms is present there as well.\n- `use_async_method_or_block.sh`: Script to switch which async implementation is active by updating the `application.rb` symlink.\n- `lib/local_server.rb`: Sinatra server used when `myasync --local` is specified.\n\n**Note:** The active async implementation is controlled by the `application.rb`\nsymlink. Use the provided script to switch between versions as needed.\n\n## Switching Async Implementations\n\nTo choose which async implementation is active, use:\n\n```\n\n./use_async_method_or_block.sh async_def    # Use async/await version\n./use_async_method_or_block.sh Async_do     # Use Async do ... end version\n\n```\n\nRunning the script with no arguments reports which version is currently active.\n\n**Note on Git and symlinks:**  \nGit tracks the symlink’s name and target path (not the contents of the target file). If you switch the symlink to point to a different file, `git status` may show it as modified. You only need to `git add` and commit if you want to record the new target in version control.\n\n## Development\n\n\n### Modular Design\n\n**myasync** is designed to support multiple library websites through a\nmodule system. Adding a new library only requires:\n\n1. **Defining a module** with parsing logic and configuration.\n2. **Adding fixture HTML files** for mock/local modes.\n3. **No changes to the core CLI code** are needed.\n\n### Adding a New Library Module\n\n1. **Create a module** in `lib/` (e.g., `springfield.rb`):\n\n```ruby\n\nmodule Springfield\n\n  BASE_URL_ACTUAL = \"https://springfield.lib.example.com\"\n  BASE_URL_LOCAL = \"http://localhost:3000/springfield\"\n  BASE_URL = Local.local? ? BASE_URL_LOCAL : BASE_URL_ACTUAL\n\n  def self.lib_data\n    {\n      post_url: \"#{BASE_URL_LOCAL}login\",\n      checked_out_url:\"#{BASE_URL}/checkedout\",\n      checked_out_fixture: \"springfield_checked_out.html\",  # Your fixture file name\n      # ... other config\n    }\n  end\n\n  def self.parse_checkedout_page(page)\n    # Custom parsing logic for this library\n  end\nend\n```\n\n2. **Add the module** to `MODULE_NAMES` in `lib/module_names.rb`:\n\n```ruby\nMODULE_NAMES = [Spingfield, Shelbyville]\n```\n\n3. **Add fixture HTML files** to `mock_data/html_pages/` (e.g., `springfield_checked_out.html`).\n\n### How Discovery Works\n\nAutomatic Discovery: The program automatically discovers which library modules\nto use, their URLs, and related configuration. Just define your modules and add\nthem to MODULE_NAMES - no manual wiring or hardcoded lists needed.\n\nMinimal Configuration: Each module provides its own URLs and fixture filenames\nvia its lib_data method. The main program and local server use this info\ndirectly, so you don’t need to configure URLs or file paths elsewhere.\n\nMock Mode: The --mock option automatically uses your fixture HTML files for all\nintercepted network requests, with no need for real HTTP calls.\n\nLocal Server: The local_server.rb script dynamically maps URLs from all modules\nin MODULE_NAMES to serve their corresponding  fixture files for development and\nconcurrency testing. No manual URL setup is required - the server reads\neverything it needs from your modules.\n\nIn short: Whether running in normal, mock, or local server mode, the program\n“just works” as long as your modules and fixtures are in place. Add a new\nlibrary module, list it in MODULE_NAMES, and both the CLI and local server will\npick it up automatically.\n\n---\n\n### Example Workflow for a New Library\n\n1. Define `lib/libraries/seattle.rb` with parsing logic.\n2. Add `Seattle` to `MODULE_NAMES`.\n3. Add `seattle_checked_out.html`, `seattle_on_hold.html` to `mock_data/html_pages/`.\n4. Test immediately:\n\n```bash\nbundle exec app/mymodest --mock    # Uses your fixtures\nbundle exec lib/local_server.rb    # Serves them at localhost:3000/seattle/...\n```\n\nFor more information on how to customize this code for your use, please refer\nto our sister project in the Ruby-like language Crystal.\n\nPlease refer to the following documents from the Crystal project for detailed\ninformation:\n\n- [Detailed README](https://github.com/Traven-B/mymodern/blob/main/project_docs/DETAILED_README.md) for **detailed installation instructions**, and subsequent setup and usage notes.\n- [Project Structure Documentation](https://github.com/Traven-B/mymodern/blob/main/project_docs/PROJECT_STRUCTURE.md) which outlines the specific parts you'll need to adapt or modify to work with your library's website.\n\n## Contributing\n\n1. Fork the repository (\u003chttps://github.com/Traven-B/myasync/fork\u003e)\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n## Contributors\n\n- [Traven-B](https://github.com/Traven-B) Michael Kamb - creator, maintainer\n\n[Ruby]: https://www.ruby-lang.org\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftraven-b%2Fmyasync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftraven-b%2Fmyasync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftraven-b%2Fmyasync/lists"}