{"id":37007775,"url":"https://github.com/xmlsquad/capture-lookups","last_synced_at":"2026-01-14T00:49:19.090Z","repository":{"id":57084738,"uuid":"135282179","full_name":"xmlsquad/capture-lookups","owner":"xmlsquad","description":"A Symfony Console command. When given configuration file listing URLs of Google Sheets, grabs them and stores them locally as CSV files.","archived":false,"fork":false,"pushed_at":"2019-12-04T08:52:03.000Z","size":583,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-31T09:57:11.510Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xmlsquad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-29T10:51:34.000Z","updated_at":"2024-09-25T05:15:37.000Z","dependencies_parsed_at":"2022-08-24T14:56:38.667Z","dependency_job_id":null,"html_url":"https://github.com/xmlsquad/capture-lookups","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/xmlsquad/capture-lookups","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlsquad%2Fcapture-lookups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlsquad%2Fcapture-lookups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlsquad%2Fcapture-lookups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlsquad%2Fcapture-lookups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xmlsquad","download_url":"https://codeload.github.com/xmlsquad/capture-lookups/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlsquad%2Fcapture-lookups/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28406901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T00:40:43.272Z","status":"ssl_error","status_checked_at":"2026-01-14T00:40:42.636Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-14T00:49:18.336Z","updated_at":"2026-01-14T00:49:19.082Z","avatar_url":"https://github.com/xmlsquad.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# capture-lookups\n\nA Symfony Console command. Searches for configuration file that lists URLs of Google Sheets, grabs the Sheets and stores their data locally as CSV files.\n\nDesigned be used in the context of the Symfony Console application at https://github.com/xmlsquad/xml-authoring-tools which, in turn, is used in the context of a known directory structure which is based on [xml-authoring-project](https://github.com/xmlsquad/xml-authoring-project).\n\n# Usage instructions\n\n## Specifying the Lookup tables to collect\n\nWe assume this command is run in the context of an [xml-authoring-project](https://github.com/xmlsquad/xml-authoring-project). ie. the key aspects of the structure of the directory is known.\n\nUse the `mapping.yaml` configuration file which defines the locations of the Google Sheets we must collect.\n\n### Example mapping.yaml\n\n```yaml\nLookupTableA:\n  # (string) Specifies the URL of the sheet to look into\n  url: \"https://docs.google.com/spreadsheets/d/1jOfsClbTj15YUqE-X2Ai9cvyhP-GLvP8CGZPgD1TysI/edit#gid=0\"\n  # (int) Sets at what row number we'll start reading data - use if you want to skip the beginning of the sheet, for example a header\n  startingFromRow: 2\n  \n  # (bool) Enable or disable fetching data in a batch. Doing so is faster, but may fail if there is a lot of data to be fetched\n  batchGet: true\n  \nLookupTableB:\n  url: \"https://docs.google.com/spreadsheets/d/1jOfsClbTj15YUqE-X2Ai9cvyhP-GLvP8CGZPgD1TysI/edit#gid=0\"\n  startingFromRow: 2\n  batchGet: false\n```  \n\n## Using the command\n\n1. Checkout the repository\n1. Install dependencies with `composer install`\n1. Put a `gApiServiceAccountCredentials.json` file in the project root or anywhere in any of the parent directories accessible to PHP\n1. Issue `bin/capture-lookups` to see all available mappings\n1. Issue `bin/capture-lookups --sheet=LookupTableA` to run the command interactively\n1. Issue `bin/capture-lookups --sheet=LookupTableA --no-interaction` to run the command without any prompts, skipping risky file names or existing files\n1. Issue `bin/capture-lookups --sheet=LookupTableA --no-interaction --force` to run the command without any prompts, **overwriting existing files** and **using sanitised file names**\n \n## Unit testing\n\n1. Install dependencies\n1. Run `./vendor/bin/phpunit`\n\n## Skipped Tabs - Naming convention\n\nBy _Google Sheet tab_ I mean one of the sheets _within_ a workbook. \n\nAny Google Sheet tab which has a trailing underscore will be considered to be skipped. \n\n* `foo_` *is* skipped.\n* `foo` is not skipped.\n* `_foo` is *not* skipped either. \n\n## Connecting to GSuite\n\nThe file that Google Api uses to authenticate access to GSuite should be in the root of the [xml-authoring-project](https://github.com/xmlsquad/xml-authoring-project).\n\nThe [ping-drive project explains how to get set up to connect to GSuite](https://github.com/xmlsquad/ping-drive#usage).\n\n\n## Run the command\n\nWhen the command is run, it will:\n\n* Search for the XmlAuthoringProjectSettings.yaml in the current working directory, if not found it will look in the parent recursively until a file named XmlAuthoringProjectSettings.yaml is found.\n* Determine the `DestinationDirectory` to write-to:\n  * If `DestinationDirectory` option is passed to command, use that.\n  * If no `DestinationDirectory` option is passed to command, set it to the default `DestinationDirectory` (see below). \n    * The default `DestinationDirectory` is the working directory in which the command was invoked. \n* For each Lookup table specified in the configuration file:\n  * Go to the Google Sheet on GSuite\n  * Determine and note the name of the Google Sheet\n  * For each tab in that sheet:\n    * If the tab's name indicates it should be ignored (has a trailing underscore), ignore that tab, skip and move on to the next tab.\n    * Else, note the tab name\n    * Combine the Google Sheet name with the tab name to set the resulting CSV file's name: `\u003cGoogleSheetName\u003e-\u003cTabName\u003e.csv`. \n    * Check the name to ensure it is made of only alphanumeric characters, dot, hyphen or underscore. (i.e the name is less likely to cause issues if used as a filename on Windows or MacOS)  \n    * If the name contains invalid characters, write a meaningful error message to STD_OUT and STD_ERR and exit with an error code.  \n    * Check to see if a CSV file matching that name is already stored in the destination directory\n    * If it is already present and the `-f` (--force) flag  is NOT set, ask user \"Permission to overwrite the file y/n?\". With the suggested default prompt being no, `[n]`.\n    * If it is already present and the -f (--force) flag  is set, overwrite the existing file without prompting the user.\n    * Else, create a CSV file with the chosen name. \n    * Write the contents of the Google Sheet Tab as a CSV file. (comma delimeter, double quotes used to encapsulate strings)  \n\n# TODO\n\n* Code reuse with `AbstractCommand`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmlsquad%2Fcapture-lookups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxmlsquad%2Fcapture-lookups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmlsquad%2Fcapture-lookups/lists"}