{"id":22115332,"url":"https://github.com/abdulrhmangoni/llms-tdm-server","last_synced_at":"2026-04-11T03:04:53.984Z","repository":{"id":253646462,"uuid":"844112374","full_name":"AbdulrhmanGoni/LLMs-TDM-server","owner":"AbdulrhmanGoni","description":"This is the server of LLMs TDM web application, Which is made for creating and managing training datasets easily with different formats for training LLMs or using them in RAG systems.","archived":false,"fork":false,"pushed_at":"2024-10-29T20:42:24.000Z","size":279,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-30T00:18:20.513Z","etag":null,"topics":["ai","api","bun","clerk-auth","datasets","docker","llms","mongodb","rag","replication","sharding","training","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbdulrhmanGoni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-18T12:26:37.000Z","updated_at":"2024-10-29T20:42:28.000Z","dependencies_parsed_at":"2024-08-29T10:04:42.467Z","dependency_job_id":"301e72d7-9066-49ce-9c87-c5f73091dd37","html_url":"https://github.com/AbdulrhmanGoni/LLMs-TDM-server","commit_stats":null,"previous_names":["abdulrhmangoni/llms-training-datasets-management"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbdulrhmanGoni%2FLLMs-TDM-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbdulrhmanGoni%2FLLMs-TDM-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbdulrhmanGoni%2FLLMs-TDM-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbdulrhmanGoni%2FLLMs-TDM-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbdulrhmanGoni","download_url":"https://codeload.github.com/AbdulrhmanGoni/LLMs-TDM-server/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245212894,"owners_count":20578538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api","bun","clerk-auth","datasets","docker","llms","mongodb","rag","replication","sharding","training","typescript"],"created_at":"2024-12-01T12:15:26.846Z","updated_at":"2025-12-30T20:35:35.947Z","avatar_url":"https://github.com/AbdulrhmanGoni.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLMs Training Datasets Manager\r\n\r\nLLMs Training Datasets Manager is a web application for creating and managing training datasets\r\nwith different formats for training Large Language Models (LLMs)\r\nor for using them in Retrieval Augmented Generation (RAG) systems.\r\n\r\n[![CI Workflow](https://github.com/AbdulrhmanGoni/LLMs-TDM-server/actions/workflows/ci.yaml/badge.svg?branch=main)](https://github.com/AbdulrhmanGoni/LLMs-TDM-server/actions/workflows/tests.yaml)\r\n[![Code Coverage](https://codecov.io/gh/AbdulrhmanGoni/LLMs-TDM-server/branch/main/graph/badge.svg?token=QVHBAOJBBO)](https://app.codecov.io/gh/AbdulrhmanGoni/LLMs-TDM-server)\r\n\r\n# Features :sparkles:\r\n\r\n- :card_index_dividers: Creating datasets and adding instructions into it.\r\n- :gear: Managing the instructions of the datasets (updating or deleting the instructions).\r\n- :bookmark_tabs: Browse the instructions of datasets esaly in pagination model.\r\n- :inbox_tray: Export datasets into your machine (Actually download datasets).\r\n- :hugs: [Huggingface](https://Huggingface.co) integration via **Huggingface OAuth flow**. And the gained features will be :point_down:\r\n  * :outbox_tray: Linking datasets with Huggingface dataset repository and pushing local datasets to.\r\n  * :new: Creating new Huggingface dataset repository if needed\r\n  * :trackball: Syncing local datasets with their linked repositories after updates.\r\n  * :electric_plug: Unlinking a local dataset with its linked repository ***(with options to delete the entire repository or just the dataset file)***\r\n\r\n# Tech Stack :hammer_and_wrench:\r\n\r\n- Bun ( The all in one JavaScript + TypeScript runtime ) :fire:\r\n- Typescript\r\n- Docker\r\n- MongoDB (With Replication \u0026 Sharding for availability and scalability)\r\n- [Clerk](https://clerk.com/) (For authentication and managing users)\r\n\r\n![Typescript](readme_file_icons/typescript.svg)\r\n![Bun](readme_file_icons/bun.svg)\r\n![Docker](readme_file_icons/docker.svg)\r\n![MongoDB](readme_file_icons/mongodb.svg)\r\n![Clerk](readme_file_icons/clerk.svg)\r\n\r\n# Installation :arrow_down:\r\n\r\n## Prerequisites :page_with_curl:\r\n\r\nBefore installing the application, make sure you have the following requirements:\r\n\r\n- [Bun](https://bun.sh/) (1.1.31 or later)\r\n- [Docker Engine or Docker Desktop](https://www.docker.com/)\r\n\r\n## Installation Steps\r\n\r\n1. Clone the repository:\r\n   ```\r\n   git clone https://github.com/AbdulrhmanGoni/LLMs-TDM-server.git\r\n   cd LLMs-TDM-server\r\n   ```\r\n\r\n2. Install dependencies:\r\n   ```\r\n   bun install\r\n   ```\r\n\r\n3. Set up the environment file:\r\n\r\nCopy `.env.example` to `.env.development`, `.env.test` and `.env.production` files and modify the variables in these files according to your settings\r\n\r\n```\r\ncp .env.example .env.development\r\ncp .env.example .env.test\r\ncp .env.example .env.production\r\n```\r\n\r\n  - `.env.development` file for development environment\r\n  - `.env.test` file for tests correctly\r\n  - `.env.production` file for production environment\r\n\r\n\u003e Note: You can see more details about the environment variables inside `.env.example` file\r\n\r\n4. Start the server:\r\n   - Starting the server in development environment:\r\n\r\n     ```\r\n     bun dev\r\n     ```\r\n\r\nThe server should now be available at `http://localhost:9000` (or the port you specified).\r\n\r\n\r\n# Tests :test_tube:\r\n\r\nI use [Bun](https://bun.sh/)'s built-in and Jest-compatible [test runner](https://bun.sh/docs/cli/test) for writing and running tests.\r\n\r\n\u003e Note: \r\n\u003e Don't forget to set the environment variables of .env.test` file before running the tests\r\n\r\nYou can copy the following template for quick start :point_down:\r\n\r\n```\r\nNODE_ENV=test\r\nPORT=9100\r\nDB_NAME=test\r\nDB_HOST=127.0.0.1 # Default host of testing database\r\nDB_PORT=270111 # Default port of testing database\r\nDB_URL=\"mongodb://$DB_HOST:$DB_PORT/$DB_NAME?directConnection=true\"\r\nTESTING_USER_ID=\"user_Xm3A5q9gd3ghR73oh975bA\" # Random user id for test\r\nMUTE_LOGS=true\r\n```\r\n\r\nTo run all tests use the following command:\r\n```\r\nbun test\r\n```\r\n\r\nTo run a specific type of tests (unit, integration or e2e) use the following command:\r\n```\r\nbun test tests/\u003cunit|integration|e2e\u003e\r\n```\r\n\r\nTo run a specific tests file just use the name of the file:\r\n```\r\nbun test \u003ctest-file-name\u003e\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulrhmangoni%2Fllms-tdm-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabdulrhmangoni%2Fllms-tdm-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulrhmangoni%2Fllms-tdm-server/lists"}