{"id":49585002,"url":"https://github.com/idank0/deepseek-dataset-generator","last_synced_at":"2026-05-03T22:05:04.127Z","repository":{"id":296775673,"uuid":"994141345","full_name":"IDanK0/Deepseek-Dataset-Generator","owner":"IDanK0","description":"Deepseek-Dataset-Generator crea dataset conversazionali per il fine-tuning di LLM tramite API DeepSeek. Supporta vari formati (ChatML, ShareGPT, Alpaca, JSON, CSV), configurazione semplice via YAML e log dettagliati. Ideale per generare dati realistici e personalizzati in modo rapido.","archived":false,"fork":false,"pushed_at":"2025-06-02T00:10:06.000Z","size":169,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-02T09:25:49.716Z","etag":null,"topics":["ai-tools","alpaca-format","chatml","conversational-ai","data-augmentation","dataset-generation","deepseek","llm-finetuning","machine-learning","nlp","open-source","prompt-engineering","python","sharegpt","synthetic-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IDanK0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-01T09:54:27.000Z","updated_at":"2025-06-02T00:10:10.000Z","dependencies_parsed_at":"2025-06-03T07:31:28.726Z","dependency_job_id":null,"html_url":"https://github.com/IDanK0/Deepseek-Dataset-Generator","commit_stats":null,"previous_names":["idank0/deepseek-dataset-generator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/IDanK0/Deepseek-Dataset-Generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDanK0%2FDeepseek-Dataset-Generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDanK0%2FDeepseek-Dataset-Generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDanK0%2FDeepseek-Dataset-Generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDanK0%2FDeepseek-Dataset-Generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IDanK0","download_url":"https://codeload.github.com/IDanK0/Deepseek-Dataset-Generator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDanK0%2FDeepseek-Dataset-Generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32586189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-tools","alpaca-format","chatml","conversational-ai","data-augmentation","dataset-generation","deepseek","llm-finetuning","machine-learning","nlp","open-source","prompt-engineering","python","sharegpt","synthetic-data"],"created_at":"2026-05-03T22:05:02.187Z","updated_at":"2026-05-03T22:05:04.118Z","avatar_url":"https://github.com/IDanK0.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deepseek-Dataset-Generator\n\nConversational dataset generator for LLM fine-tuning, optimized for the DeepSeek API (chosen for its low cost and efficiency). It allows you to create datasets in various formats (ChatML, ShareGPT, Alpaca, JSON, CSV) by simulating realistic conversations in English (or any language) on customizable domains.\n\n## Main Features\n- **DeepSeek API**: uses DeepSeek to generate questions and answers, ideal for those looking for an affordable and scalable solution.\n- **Advanced prompts**: generates multi-turn conversations, chain-of-thought questions, and covers realistic and varied use cases.\n- **Supported formats**: exports to ChatML, ShareGPT, Alpaca, JSON, JSONL, CSV.\n- **Simple configuration**: everything managed via `config.yaml`.\n- **Detailed logging**: every generation is tracked in a log file.\n\n## Requirements\n- Python 3.8+\n- Dependencies installable via `pip install -r requirements.txt`\n\n## Installation\n```powershell\npip install -r requirements.txt\n```\n\n## Configuration\nEdit `config.yaml` to customize:\n- Number of examples/conversations\n- Model temperature\n- Number of turns per conversation\n- Domain/context of conversations\n- Output format and file path\n- DeepSeek API key\n\nExample `config.yaml`:\n```yaml\noutput_format: chatml\nnum_examples: 500\ntemperature: 0.7\nturns_per_conversation: 2\ndomain: \"Medical assistance\"\noutput_file: \"datasets/dataset.jsonl\"\nextend_existing: false\napi_key: \"DEEPSEEK-API-KEY\"\nmax_retries: 5\nretry_backoff: 2\nlog_file: \"generation.log\"\nchain_of_thought: false\ninclude_id: false\n```\n\n## Usage\nRun the generator with:\n```powershell\npython main.py --config config.yaml\n```\nOptions:\n- `--config`: path to the configuration file (default: `config.yaml`)\n- `--num_examples`: overrides the number of examples specified in config\n\n## Project Structure\n- `main.py`: entrypoint, handles arguments and logging\n- `data_generator.py`: logic for generating conversations and datasets\n- `deepseek_api.py`: wrapper for DeepSeek API calls with retry management\n- `formats.py`: conversion between formats (ChatML, ShareGPT, Alpaca)\n- `utils.py`: utilities for logging, file saving, config loading\n- `config.yaml`: main configuration\n- `requirements.txt`: Python dependencies\n- `generation.log`: detailed generation log\n- `datasets/`: output folder for datasets\n\n## DeepSeek API Notes\n- You must enter a valid API key in `config.yaml`.\n- In case of rate limit or errors, the system automatically retries.\n\n## Output Example (ChatML)\n```json\n{\n  \"messages\": [\n    {\"role\": \"user\", \"content\": \"Generated question...\"},\n    {\"role\": \"assistant\", \"content\": \"Generated answer...\"}\n  ]\n}\n```\n\n## License\nSee LICENSE file.\n\n---\n\n**Deepseek-Dataset-Generator** is designed for those who want to quickly generate high-quality datasets for LLM fine-tuning, leveraging the convenience of the DeepSeek API.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidank0%2Fdeepseek-dataset-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidank0%2Fdeepseek-dataset-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidank0%2Fdeepseek-dataset-generator/lists"}