{"id":29168354,"url":"https://github.com/pocket/content-monorepo","last_synced_at":"2025-07-01T10:07:20.848Z","repository":{"id":272839284,"uuid":"734119099","full_name":"Pocket/content-monorepo","owner":"Pocket","description":"This repo contains Content owned TS microservices structured in a monorepo","archived":false,"fork":false,"pushed_at":"2025-06-26T18:58:21.000Z","size":3054,"stargazers_count":2,"open_issues_count":7,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-26T19:38:41.170Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Pocket.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-20T23:11:56.000Z","updated_at":"2025-06-11T21:10:05.000Z","dependencies_parsed_at":"2025-02-03T23:23:25.230Z","dependency_job_id":"c6f22260-8acb-462e-bc23-f1d068e1b8ba","html_url":"https://github.com/Pocket/content-monorepo","commit_stats":null,"previous_names":["pocket/content-monorepo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Pocket/content-monorepo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pocket%2Fcontent-monorepo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pocket%2Fcontent-monorepo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pocket%2Fcontent-monorepo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pocket%2Fcontent-monorepo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Pocket","download_url":"https://codeload.github.com/Pocket/content-monorepo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pocket%2Fcontent-monorepo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262941557,"owners_count":23388150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-01T10:07:19.936Z","updated_at":"2025-07-01T10:07:20.824Z","avatar_url":"https://github.com/Pocket.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Content-Monorepo\n\nThis repo contains Content owned TS microservices structured in a monorepo.\n\nSome of the patterns such as Docker \u0026 Github Actions have been adopted from [Pocket's Monorepo](https://github.com/Pocket/pocket-monorepo) setup by Daniel \u0026 Kat.\n\n## Technology Used\n\n- [Turborepo](https://turbo.build/) for build manager\n- [pnpm](https://pnpm.io/) for package manager\n- Docker for running dependent services\n- Node TS services run with `pnpm`/`turbo`\n- [ESLint](https://eslint.org/) for code linting\n- [Prettier](https://prettier.io) for code formatting\n\n## Monorepo Structure\n\nThe following is an outline of how the monorepo is structured:\n\n```\nmonorepo /\n\t.docker /\n\t\tshared resources here\n\t.github /\n\t\tCI workflows for all services and packages\n\tinfrastructure /\n\t\tinfra code per service\n\tapps /\n\t\tTBD (front-end apps go here?)\n\tlambdas\n\t\tservice AWS lambdas go here\n\tpackages\n\t\tinternal packages shared between services\n\tservers /\n\t\tback-end services go here\n```\n\nPurpose of some important files in the root of the monorepo:\n`.npmrc` - pnpm configuration\n`.syncpackrc` - configuration for keeping dependencies in sync across all `package.json` files\n`pnpm-workspace.yml` - Defines the root of the `pnpm` workspace and enables to include / exclude directories from the workspace\n`pnpm-lock.yml` - Lock file generated by the `pnpm` package manager contains a complete, resolved list of all dependencies and their versions.\n`turbo.json` - Configure behavior for `turbo`\n\n## CI \u0026 CD\n\nWe use Github Actions to:\n\n- Build Docker images\n- Push Docker images to AWS ECR. For `main`/`dev` pushes, also push the image to AWS ECS.\n- Execute unit and integration tests for all pull requests\n  - The integration tests in Github use the same `docker-compose.yml` file that is used locally.\n- Build, plan (for pull requests), and apply (for `main`/`dev` pushes) terraform\n- Perform Apollo schema checks and updates\n- Notify Sentry of new releases for `main`/`dev` pushes.\n- Start AWS CodeDeploy for pushes to `main` and `dev`.\n\n**Note** - We currently are _not_ queueing deployments. While a deployment is in progress, any new deployments will fail.\nThis is a point of reslience we should build.\n\n### Monitoring Deployments\n\nBuild and deployment progress can be monitored in the \"Actions\" tab in the Github Repository.\n\nCodeDeploy progress can be monitored directly in AWS.\n\n## Build\n\nFirst, create a local `.env` file and copy the contents of `example.env`.\nDon't forget to use the `node` version configured for this monorepo (`v20`).\n\nTo build all services:\n\n```\ncd content-monorepo\nnvm use\npnpm install\npnpm build\n```\n\nSample build output:\n\n```\n Tasks:    8 successful, 8 total\nCached:    8 cached, 8 total\n  Time:    416ms \u003e\u003e\u003e FULL TURBO\n```\n\nIf there are no changes to the services, `turbo` uses the cached results of the previous build.\n\n## Dev / Local Testing\n\nTo spin up all services from the root of the monorepo:\n\n```\ncd content-monorepo\ndocker compose up --wait\npnpm dev\n```\n\nThis will spin up dependent docker services (detached mode), and run the node ts services.\n\nTo spin up a service individually, pass the `--filter={service_name}` to the `pnpm` command:\n\n```\ncd content-monorepo\ndocker compose up --wait\npnpm build\npnpm dev --filter=prospect-api\n```\n\nNote that the GraphQL endpoints can be reached on the ip:port shown at startup. For example, `curated-corpus-api` can be reached at:\nhttp://127.0.0.1:4025\nhttp://127.0.0.1:4025/admin\n\n### Unit Tests\n\nTo run all unit tests for all services \u0026 packages:\n\n```\ncd content-monorepo\npnpm build\npnpm test\n```\n\nTo run unit tests individually per service:\n\n```\ncd content-monorepo\npnpm build\npnpm test --filter=prospect-api\n```\n\n### Integration Tests\n\nWhile it is possible to run all integration tests at the same time, this isn't\nadvised as you'll likely get errors related to Snowplow. This is because\nthe integration tests will run concurrently, and all hit the same Snowplow endpoint.\nThis means there will likely be collisions between tests, resulting in unexpected data/failures.\n\nIf you are undeterred and want to run all integration tests at the same time:\n\n```\ncd content-monorepo\ndocker compose up --wait\npnpm build\npnpm test-integrations\n```\n\nTo run integration tests for a specific service:\n\n```\ncd content-monorepo\ndocker compose up --wait\npnpm build\npnpm test-integrations --filter=curated-corpus-api\n```\n\n### Debugging Tests in VSCode\n\nTo enable step-through debugging in VSCode:\n\n1. Install the [Jest Runner](https://marketplace.visualstudio.com/items?itemName=firsttris.vscode-jest-runner) extension. (Be sure it's this one! There are a couple other extensions with almost the same name.)\n2. Open up your settings and search for `jest-runner`.\n3. In the \"Jest-Runner Config\" section, make sure the \"Jestrunner: Config Path\" setting is empty. (It's probably got a default value.)\n4. Open up a test file and click the `Run` and `Debug` commands that float above the test definition. It should work!\n\nIf you have trouble with any of the steps above, try re-starting VSCode.\n\n## DynamoDB\n\nProspect-api uses `dynamodb` as the db system. When running `docker compose up`, the `localstack` container executes a `dynamodb.sh` script where the prospect-api table\nis created.\nTo seed the table with data, run the seeding script:\n\n```\ncd content-monorepo\npnpm build\npnpm db:dynamo-seed\n```\n\nIf you want to delete dynamodb database, restart the docker and re-run the seed\n\n```\ndocker restart content-monorepo-localstack-1\n```\n\n## Prisma\n\nCollection API \u0026 Curated Corpus API use `prisma` as their ORM, and to set up \u0026 seed the tables, some tasks need to be run separately from `docker compose`.\n\n### Generate Prisma Typescript Types\n\n```\ncd content-monorepo\npnpm db:generate-prisma-client\n```\n\n### Resetting \u0026 Seeding the Databases\n\nMake sure the `.env` under each service using `prisma` contains the appropriate `DATABASE_URL`.\n\nFor `curated-corpus-api`: `DATABASE_URL=mysql://root:@localhost:3306/curation_corpus?connect_timeout=300`\n\nFor `collection-api`: `DATABASE_URL=mysql://root:@localhost:3306/collections?connect_timeout=300`\n\nRun:\n\n```\ncd content-monorepo\npnpm prisma db seed\n```\n\n### Applying migration\n\nPlease refer to the specific README of the service for applying a prisma migration.\n\n### Adding a new dependency via pnpm\n\nNavigate to the individual project and use `npm install` to update the item\nThen rebuild from the root of the monorepo with `pnpm`:\n\n```\npnpm update\npnpm build\n```\n\n## Package Synchronization\n\nWe use [Syncpack](https://jamiemason.github.io/syncpack/guide/getting-started/) to keep package versions consistent across servers, lambdas, and shared packages. Outside of the consistent functional expecations of using the same package version in all places, it's important to keep some package groups in sync to mitigate cross-package bugs, e.g AWS and Prisma packages.\n\nThe syncpack config can be found in the `./syncpackrc` file.\n\nThere are two command line operations associated with Syncpack:\n\n1. `pnpm list-mismatches` will tell you if any packages are out of sync/in violation of the rules set in `.syncpackrc`. This should only happen if your current branch is changing packages/package versions. Our CI will error if the rules in `.syncpackrc` are in violation.\n\n2. `pnpm fix-mismatches` will automatically fix `package.json` files that are in violation of the rules set in `.syncpackrc` by changing package versions. This is a quick and easy way to perform an upgrade, but as with any operation that can change things at scale, be sure you check the result is what you expect.\n\n## Tracing (Servers Only - WIP)\n\nWe leverage [Pocket's tracing package](https://www.npmjs.com/package/@pocket-tools/tracing) to perform traces in our `server` applications:\n\n- Collection API\n\n  - [Unleash feature flag](https://featureflags.getpocket.dev/projects/default/features/perm.content.tracing.collections) (Dev)\n  - [Unleash feature flag](https://featureflags.readitlater.com/projects/default/features/perm.content.tracing.collections) (Prod)\n\n- Curated Corpus API (coming soon)\n\n  - [Unleash feature flag](https://featureflags.getpocket.dev/projects/default/features/perm.content.tracing.curated-corpus-api) (Dev)\n  - [Unleash feature flag](https://featureflags.readitlater.com/projects/default/features/perm.content.tracing.curated-corpus-api) (Prod)\n\n- Prospect API\n\n  - [Unleash feature flag](https://featureflags.getpocket.dev/projects/default/features/perm.content.tracing.prospect-api) (Dev)\n  - [Unleash feature flag](https://featureflags.readitlater.com/projects/default/features/perm.content.tracing.prospect-api) (Prod)\n\nTraces and logs for the above services can be found in GCP (filter by service name):\n\n- [GCP Logs Explorer](https://console.cloud.google.com/logs/query;cursorTimestamp=2024-11-20T16:35:39.086537379Z;customDuration=today?inv=1\u0026invt=AbiAGg\u0026project=moz-fx-pocket-prod-61fb)\n- [GCP Trace Explorer](https://console.cloud.google.com/traces/list?inv=1\u0026invt=AbiAGg\u0026project=moz-fx-pocket-prod-61fb)\n\nTracing is performed using Open Telemetry NPM packages that send trace data to a standalone collector ECS service in AWS, which in turn exports trace data to GCP. The Pocket tracing package also implements an\n[Open Telemetry package](https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node) that hooks into the Winston logger (which we implement via the Pocket `ts-logger` package) to auto-forward log data to GCP.\n\n### Enable/Disable Tracing in Prod \u0026 Dev\n\nTracing can be enabled and disabled using an Unleash feature flag, which exists per service implementing tracing.\n\nTo enable and configure the feature flag:\n\n1. Visit the feature flag URL for the environment in question (linked above)\n2. Toggle the `default` environment to \"On\" in the left hand \"Enabled in environments\" box\n3. Expand the `default` environment in the main, right-hand panel\n4. Click the ✎ pencil icon to edit the \"Gradual rollout\" strategy\n5. Move the \"Rollout\" slider to 100%\n6. Click the \"Variants\" tab and adjust the \"Payload\" number to the sample rate you'd like for your traces\n   - In production, this should usually be 1% (0.01) to begin with, and can be increased slowly if needed\n7. Click \"Save strategy\"\n8. After some requests have been made to the service, go look at traces in GCP (using links above)\n\nTo disable tracing on a service, simply toggle the `default` environment to \"Off\".\n\n### Local Tracing\n\nLocal tracing is enabled by default and sends trace data to a Grafana Docker image. To view traces locally:\n\n1. Make sure the local service you want to trace has activity, e.g. by running a query in the Apollo Server Playground\n2. Navigate to the Grafana docker image endpoint at `http://localhost:3000/explore`\n3. Click \"Explore\" in the left hand menu\n4. In the dropdown at the top left of the middle pane, select \"Tempo\"\n5. In the main panel, select the \"Service Graph\" for \"Query type\"\n6. Click the service you want to view traces for and select \"View traces\"\n7. Trace away!\n\n## CI Status\n\n### Servers\n\n[![Collection API](https://github.com/Pocket/content-monorepo/actions/workflows/collection-api.yml/badge.svg)](https://github.com/Pocket/content-monorepo/actions/workflows/collection-api.yml)  \n[![Curated Corpus API](https://github.com/Pocket/content-monorepo/actions/workflows/curated-corpus-api.yml/badge.svg)](https://github.com/Pocket/content-monorepo/actions/workflows/curated-corpus-api.yml)  \n[![Prospect API](https://github.com/Pocket/content-monorepo/actions/workflows/prospect-api.yml/badge.svg)](https://github.com/Pocket/content-monorepo/actions/workflows/prospect-api.yml)\n\n### Lambdas\n\n[![Prospect Translation Lambda](https://github.com/Pocket/content-monorepo/actions/workflows/prospect-translation-lambda.yml/badge.svg)](https://github.com/Pocket/content-monorepo/actions/workflows/prospect-translation-lambda.yml)  \n[![Corpus Scheduler Lambda](https://github.com/Pocket/content-monorepo/actions/workflows/corpus-scheduler-lambda.yml/badge.svg)](https://github.com/Pocket/content-monorepo/actions/workflows/corpus-scheduler-lambda.yml)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpocket%2Fcontent-monorepo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpocket%2Fcontent-monorepo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpocket%2Fcontent-monorepo/lists"}