{"id":18395823,"url":"https://github.com/deepset-ai/dc-custom-component-template","last_synced_at":"2025-04-07T03:35:17.044Z","repository":{"id":256230803,"uuid":"836133842","full_name":"deepset-ai/dc-custom-component-template","owner":"deepset-ai","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-16T16:39:25.000Z","size":43,"stargazers_count":1,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-09-17T14:15:10.171Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepset-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-31T08:11:50.000Z","updated_at":"2024-09-16T16:39:29.000Z","dependencies_parsed_at":"2024-10-25T10:14:04.339Z","dependency_job_id":"69582173-f71f-40e4-acd2-1d92c34ea4cf","html_url":"https://github.com/deepset-ai/dc-custom-component-template","commit_stats":null,"previous_names":["deepset-ai/dc-custom-component-template"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fdc-custom-component-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fdc-custom-component-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fdc-custom-component-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fdc-custom-component-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepset-ai","download_url":"https://codeload.github.com/deepset-ai/dc-custom-component-template/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223271116,"owners_count":17117303,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T02:12:01.152Z","updated_at":"2025-04-07T03:35:17.031Z","avatar_url":"https://github.com/deepset-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# deepset Cloud Custom Component Template\n\nThis repository contains a template for creating custom components for your deepset Cloud pipelines. Components are Python code snippets that perform specific tasks within your pipeline. This template will guide you through all the necessary elements your custom component must include.\nThis template contains two sample components which are ready to be used: \n  - `CharacterSplitter` implemented in `/src/dc_custom_component/example_components/preprocessors/character_splitter.py`: A component that splits documents into smaller chunks by the number of characters you set. You can use it in indexing pipelines.\n  - `KeywordBooster` implemented in `/src/dc_custom_component/example_components/rankers/keyword_booster.py`: A component that boosts the score of documents that contain specific keywords. You can use it in query pipelines.\n\nWe've created these examples to help you understand how to structure your components. When importing your custom components to deepset Cloud, you can remove or rename the `example_components` folder with the sample components, if you're not planning to use them. \n\nThis template serves as a custom components library for your organization. Only the components present in the most recently uploaded template are available for use in your pipelines. \n\n## Documentation\nFor more information about custom components, see [Custom Components](https://docs.cloud.deepset.ai/docs/custom-components). \nFor a step-by-step guide on creating custom components, see [Create a Custom Component](https://docs.cloud.deepset.ai/docs/create-a-custom-component).\nSee also our tutorial for [creating a custom RegexBooster component](https://docs.cloud.deepset.ai/docs/tutorial-creating-a-custom-component).\n\n## 1. Setting up your local dev environment\n\n### Prerequisites\n\n- Python v3.12 or v3.13\n- `hatch` package manager\n\n### Hatch: A Python package manager\n\nWe use `hatch` to manage our Python packages. Install it with pip:\n\nLinux and macOS:\n```bash\npip install hatch\n```\n\nWindows:\nFollow the instructions under https://hatch.pypa.io/1.12/install/#windows\n\nOnce installed, create a virtual environment by running:\n\n```bash\nhatch shell\n```\n\nThis installs all the necessary packages needed to create a custom component. You can reference this virtual environment in your IDE.\n\nFor more information on `hatch`, please refer to the [official Hatch documentation](https://hatch.pypa.io/).\n\n## 2. Developing your custom component\n\n### Structure\n\n| File | Description |\n|------|-------------|\n| `/src/dc_custom_component/components` | Directory for implementing custom components. You can logically group custom components in sub-directories. See how sample components are grouped by type. |\n| `/src/dc_custom_component/__about__.py` | Your custom components' version. Bump the version every time you update your component before uploading it to deepset Cloud. This is not needed if you are using the GitHub action workflow (in this case the version will be determined by the GitHub release tag). |\n| `/pyproject.toml` | Information about the project. If needed, add your components' dependencies in this file in the `dependencies` section. |\n\nThe directory where your custom component is stored determines the name of the component group in Pipeline Builder. For example, the `CharacterSplitter` component would appear in the `Preprocessors` group, while the `KeywordBooster` component would be listed in the `Rankers` group. You can drag these components onto the canvas to use them.\n\nWhen working with YAML, the location of your custom component implementation defines your component's `type`. For example, the sample components have the following types because of their location:\n  - `dc_custom_component.example_components.preprocessors.character_splitter.CharacterSplitter`\n  - `dc_custom_component.example_components.rankers.keyword_booster.KeywordBooster`\n\nHere is how you would add them to a pipeline:\n```yaml\ncomponents:\n  splitter:\n    type: dc_custom_component.example_components.preprocessors.character_splitter.CharacterSplitter\n    init_parameters: {}\n  ...\n    \n```\n### Working on your component\n\n1. Fork this repository.\n2. Navigate to the `/src/dc_custom_component/components/` folder.\n3. Add your custom components following the examples.\n4. Update the components' version in `/src/__about__.py`.\n\n   \u003e **NOTE**: This is not needed if you are using the GitHub action workflow (in this case the version will be determined by the GitHub release tag).\n5. Format your code using the `hatch run code-quality:all` command. (Note that hatch commands work from the project root directory only.)\n\n### Formatting\nWe defined a suite of formatting tools. To format your code, run:\n\n```bash\nhatch run code-quality:all\n```\n\n### Testing\n\nIt's crucial to thoroughly test your custom component before uploading it to deepset Cloud. Consider adding unit and integration tests to ensure your component functions correctly within a pipeline.\n- `pytest` is ready to be used with `hatch`\n- implement your tests under `/test`\n- run `hatch run tests`\n\n## 3. Uploading your custom component\n\nYou can upload in one of two ways:\n- By releasing your forked directory.\n- By zipping the forked repository and uploading it with commands.\n\n### Uploading by releasing your forked repository\n\nWe use GitHub Actions to build and push custom components to deepset Cloud. The action runs the tests and code quality checks before pushing the component code to deepset Cloud. Create a tag to trigger the build and the push job. This method helps you keep track of the changes and investigate the code deployed to deepset Cloud.\n\nAfter forking or cloning this repository:\n\n1. Push all your changes to the forked repository.\n2. Add the `DEEPSET_CLOUD_API_KEY` [secret to your repository](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions). This is your deepset Cloud API key.\n(To add a secret, go to your repository and choose _Settings \u003e Secrets and variables \u003e Actions \u003e New repository secret_.)\n3. Enable workflows for your repository by going to _Actions \u003e Enable workflows_.\n4. (Optional) Adjust the workflow file in `.github/workflows/publish_on_tag.yaml` as needed.\n5. (Optional) If you're not using the European deepset Cloud tenant, change the `API_URL` variable in `.github/workflows/publish_on_tag.yaml`\n6. Create a new release with a tag to trigger the GitHub Actions workflow. The workflow builds and pushes the custom component to deepset Cloud with the tag as version. For help, see [GitHub documentation](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository).\n  \n   \u003e **Warning:** When using this GitHub Actions workflow, the version specified in the `__about__` file will be overwritten by the tag value. Make sure your tag matches the desired version number. \n\n\n\nYou can check the upload status in the `Actions` tab of your forked repository. \n\n### Uploading a zipped repository with commands\n\nIn this method, you run commands to zip and push the repository to deepset Cloud.\n1. (Optional) If you're not using the European tenant, set the API URL:\n  - deepset Cloud Europe:\n    - On Linux and macOS: `export API_URL=\"https://api.cloud.deepset.ai\"`\n    - On Windows: `set API_URL=https://api.cloud.deepset.ai`\n  - deepset Cloud US:\n    - On Linux and macOS: `export API_URL=\"https://api.us.deepset.ai\"`\n    - On Windows: `set API_URL=https://api.us.deepset.ai`\n2. Set your [deepset Cloud API key](https://docs.cloud.deepset.ai/docs/generate-api-key).\n   - On Linux and macOS: `export API_KEY=\u003cTOKEN\u003e`\n   - On Windows: `set API_KEY=\u003cTOKEN\u003e`\n3. Upload your project by running the following command from inside of this project:\n   - On Linux and macOS: `hatch run dc:build-and-push`\n   - On Windows: `hatch run dc:build-windows` and `hatch run dc:push-windows`\n   This creates a ZIP file called `custom_component.zip` in the `dist` directory and uploads it to deepset Cloud.\n\n## 4. Debugging\n\nTo debug the installation of custom components in deepset Cloud, you can run:\n\n- On Linux and macOS: `hatch run dc:logs` \n- On Windows: `hatch run dc:logs-windows`\n\nThis will print the installation logs of the latest version of your custom components.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fdc-custom-component-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepset-ai%2Fdc-custom-component-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fdc-custom-component-template/lists"}