{"id":23143741,"url":"https://github.com/digitalinteraction/catalyst-trello-scraper","last_synced_at":"2025-04-04T12:14:58.757Z","repository":{"id":49604969,"uuid":"173931143","full_name":"digitalinteraction/catalyst-trello-scraper","owner":"digitalinteraction","description":"A Dockerized CLI to scrape a Trello list, parses label-relationships and stores into redis","archived":false,"fork":false,"pushed_at":"2021-08-31T12:22:34.000Z","size":413,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-09T22:11:19.701Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/digitalinteraction.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-05T11:07:11.000Z","updated_at":"2024-11-08T12:31:38.000Z","dependencies_parsed_at":"2022-09-10T21:30:44.284Z","dependency_job_id":null,"html_url":"https://github.com/digitalinteraction/catalyst-trello-scraper","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalinteraction%2Fcatalyst-trello-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalinteraction%2Fcatalyst-trello-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalinteraction%2Fcatalyst-trello-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitalinteraction%2Fcatalyst-trello-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/digitalinteraction","download_url":"https://codeload.github.com/digitalinteraction/catalyst-trello-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174467,"owners_count":20896078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-17T15:14:08.512Z","updated_at":"2025-04-04T12:14:58.735Z","avatar_url":"https://github.com/digitalinteraction.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Catalyst | Trello Scraper\n\nThis is the repo for the Not-Equal Catalyst's Trello scraper.\nIt is a [node.js](https://nodejs.org)\nCLI written in [TypeScript](https://www.typescriptlang.org/)\nand deployed through [Docker](https://www.docker.com/).\nIt queries the Trello API for cards on a specific list,\nparses relationships from card labels, then writes into [redis](https://redis.io/).\n\n[What is Not-Equal Catalyst?](https://github.com/unplatform/catalyst-about)\n\n\u003c!-- toc-head --\u003e\n\n## Table of contents\n\n- [Why a CLI](#why-a-cli)\n- [Usage](#usage)\n  - [Environment variables](#environment-variables)\n  - [Commands](#commands)\n- [Development](#development)\n  - [Setup](#setup)\n  - [Regular use](#regular-use)\n  - [Irregular use](#irregular-use)\n  - [Commits](#commits)\n  - [Code Structure](#code-structure)\n  - [Code formatting](#code-formatting)\n  - [Testing](#testing)\n- [Deployment](#deployment)\n  - [Building the image](#building-the-image)\n- [Future work](#future-work)\n\n\u003c!-- toc-tail --\u003e\n\n## Why a CLI\n\nThis is a CLI that is designed to be run with Docker and docker-compose.\nThis means you can set environment variables and link in a redis container.\n\nIt uses Docker's `ENTRYPOINT` so that you can pass your CLI command via docker.\nThis means you can deploy and run through `docker-compose` and customise the command\nat the deployment level, not at the application level.\n\n## Usage\n\nThis repo has a Docker image for each version of the scraper.\n\n### Environment variables\n\nTheses are the envionment variables to set inside the docker container.\nSee [Setup](#setup) for more info about generating these.\n\n- `TRELLO_APP_KEY` - Your trello app key, from https://trello.com/app-key\n- `TRELLO_TOKEN` - Your generated app token\n- `TRELLO_BOARD_ID` - The board id to scrape from\n- `TRELLO_TARGET_LIST_ID` - The list id to pull cards from\n- `TRELLO_CONTENT_LIST_ID` - The list if to pull content cards from (optional)\n- `REDIS_URL` - The url to access redis from\n\n### Commands\n\nSet the container's command to what you want it to perform,\nhere is the output of the `--help` option for reference.\n\n```\nUsage: @openlab/catalyst-trello-scraper [options] [command]\n\nOptions:\n  -V, --version              output the version number\n  -h, --help                 output usage information\n\nCommands:\n  fetch [options]            Fetch the current projects and content and store them in redis\n  schedule [options] \u003ccron\u003e  Schedule a fetch based on a cron job, https://crontab.guru\n  show:labels                Show the trello labels in redis\n  show:cards                 Show the matched trello cards in redis\n  show:content               Show the content stored in redis\n```\n\n## Development\n\n### Setup\n\nTo develop on this repo you will need to have [Docker](https://www.docker.com/) and\n[node.js](https://nodejs.org) installed on your dev machine and have an understanding of them.\nThis guide assumes you have the repo checked out and are on macOS.\nYou will also need a Trello account which is used to pull the data from.\n\nYou'll only need to follow this setup once for your dev machine.\n\n```bash\n# Install dependancies\nnpm install\n\n# Start up a redis container\n# -\u003e Launches a redis:4-alpine docker container\n# -\u003e Remember to 'docker-compose stop' after developing\n# -\u003e Binds port 6379 to localhost:6379\ndocker-compose up -d\n\n# Setup your environment\ncp .env.example .env\n\n# Get your Trello App Key and put it into your .env\nopen https://trello.com/app-key\n\n# Generate a Trello token for development, then fill it into your .env\nsource .env\nopen \"https://trello.com/1/authorize?expiration=never\u0026scope=read\u0026response_type=token\u0026name=Not-Equal%20Catalyst\u0026key=$TRELLO_APP_KEY\"\n\n# To get your TRELLO_BOARD_ID \u0026 TRELLO_LIST_ID, visit the board you want to pull from on the trello.com\n# Then add .json to the end of the url and inspect the contents to find your values\nopen https://trello.com\n```\n\n### Regular use\n\nThese are the commands you'll regularly run to develop the CLI, in no particular order.\n\n```bash\n# Run the CLI in development mode\n# -\u003e Runs the TypeScript directly with `ts-node` and loads the .env\n# -\u003e Use the `--` to stop npm swallowing the dash-dash arguments\nnpm run dev -- --help\n\n# Execute redis commands to inspect the store\n# -\u003e Runs a command in the redis container (started by `npm run redis`)\n# -\u003e Attaches std in/output so it behaves like you've ssh'd into it\n# -\u003e Runs the internal redis-cli, so you can directly talk to redis\n# -\u003e For reference: https://redis.io/commands\nnpm run redis-cli\n\n# Useful redis commands\n127.0.0.1:6379\u003e keys *       # List all keys\n127.0.0.1:6379\u003e get cards    # Get cards\n127.0.0.1:6379\u003e get labels   # Get labels\n127.0.0.1:6379\u003e get content  # Get content\n\n# Run unit tests\n# -\u003e Looks for files named `*.spec.ts` in the src directory\nnpm run test\n```\n\n### Irregular use\n\nThese are commands you might need to run but probably won't, also in no particular order.\n\n```bash\n# Generate the table of contents for this readme\n# -\u003e It'll replace content between the toc-head and toc-tail HTML comments\nnpm run gen-readme-toc\n\n# Manually lint code with TypeScript's `tsc`\nnpm run lint\n\n# Manually format code\n# -\u003e This repo is setup to automatically format code on git-push\nnpm run prettier\n\n# Manually transpile TypeScript to JavaScript\n# -\u003e This is part of the docker build which is triggered when deploying\n# -\u003e Writes files to dist, which is git-ignored\nnpm run build\n\n# Manually start code from transpilled JavaScript\n# -\u003e It'll automatically load your local .env\nnpm run start\n```\n\n### Commits\n\nAll commits to this repo must follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/).\nThis ensures changes are structured and means the [CHANGELOG.md](/CHANGELOG.md) can be automatically generated.\n\nThis standard is enforced through a `commit-msg` hook using [yorkie](https://www.npmjs.com/package/yorkie).\n\n### Code Structure\n\n| Folder       | Contents                                     |\n| ------------ | -------------------------------------------- |\n| dist         | Where the transpilled JavaScript is built to |\n| node_modules | Where npm's modules get installed into       |\n| src          | Where the code of the app is                 |\n\n### Code formatting\n\nThis repo uses [Prettier](https://prettier.io/) to automatically format code to a consistent standard.\nThis works using the [husky](https://www.npmjs.com/package/husky)\nand [lint-staged](https://www.npmjs.com/package/lint-staged) packages to\nautomatically format code whenever you commit code.\nThis means that code that is pushed to the repo is always formatted to a consistent standard.\n\nYou can manually run the formatter with `npm run prettier` if you want.\n\nPrettier is slightly configured in [.prettierrc.yml](/.prettierrc.yml)\nand also ignores files using [.prettierignore](/.prettierignore).\n\n### Testing\n\n\u003e This CLI is currently quite simple and doesn't have any unit tests yet\n\nThis repo uses [unit tests](https://en.wikipedia.org/wiki/Unit_testing) to ensure that everything is working correctly, guide development, avoid bad code and reduce defects.\nThe [Jest](https://www.npmjs.com/package/jest) package is used to run unit tests.\nTests are any file in `src/` that ends with `.spec.ts`, by convention they are inline with the source code,\nin a parallel folder called `__tests__`.\n\n```bash\n# Run the tests\nnpm test -s\n\n# Generate code coverage\nnpm run coverage -s\n```\n\n## Deployment\n\n### Building the image\n\nThis repo uses a [GitLab CI](https://about.gitlab.com/product/continuous-integration/)\nto build a Docker image when you push a git tag.\nThis is designed to be used with the `npm version` command so all docker images are [semantically versioned](https://semver.org/).\nThe `:latest` docker tag is not used.\n\nThis job runs using the [.gitlab-ci.yml](/.gitlab-ci.yml) file which\nruns a docker build using the [Dockerfile](/Dockerfile)\nand **only** runs when you push a [tag](https://git-scm.com/book/en/v2/Git-Basics-Tagging).\n\nIt pushes these docker images to the [GitLab registry](https://openlab.ncl.ac.uk/gitlab/catalyst/trello-scraper/container_registry).\nA slight nuance is that it will replace a preceding `v` in tag names, formatting `v1.0.0` to `1.0.0`.\n\n```bash\n# Generate a new release\n# -\u003e Generates a new version based on the commits since the last version\n# -\u003e Generates the CHANGELOG.md based on those commits\n# -\u003e There is a \"preversion\" script to lint \u0026 run tests\nnpm run release\n\n# Push the new version\n# -\u003e The GitLab CI will build a new docker image for it\ngit push --follow-tags\n```\n\n## Future work\n\n- Add automated testing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitalinteraction%2Fcatalyst-trello-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdigitalinteraction%2Fcatalyst-trello-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitalinteraction%2Fcatalyst-trello-scraper/lists"}