{"id":22695668,"url":"https://github.com/tina4stack/thought","last_synced_at":"2025-03-29T18:09:41.795Z","repository":{"id":252925470,"uuid":"841873838","full_name":"tina4stack/thought","owner":"tina4stack","description":"Thought AI light framework for LLAMA based LLMS","archived":false,"fork":false,"pushed_at":"2024-08-21T12:38:22.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-04T18:50:49.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tina4stack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-13T08:07:33.000Z","updated_at":"2024-08-21T12:38:25.000Z","dependencies_parsed_at":"2024-12-10T04:21:35.835Z","dependency_job_id":null,"html_url":"https://github.com/tina4stack/thought","commit_stats":null,"previous_names":["tina4stack/thought"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tina4stack%2Fthought","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tina4stack%2Fthought/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tina4stack%2Fthought/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tina4stack%2Fthought/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tina4stack","download_url":"https://codeload.github.com/tina4stack/thought/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246223319,"owners_count":20743165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-10T04:11:27.996Z","updated_at":"2025-03-29T18:09:41.772Z","avatar_url":"https://github.com/tina4stack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# thought\nThought is a light wrapper for llama cpp, the building block for implementing and managing an efficient LLM integration\n\n## Installation\n\n```bash\npip install thought\n```\n\n## Get started quickly\n\n### Imports\n\nThe following imports cover all the functionality of loading, embedding and generating\n```python\n# loader\nfrom thought.model_loader import load_model\nfrom thought.model_loader import generate\nfrom thought.model_loader import embed\n\n# downloader\nfrom thought.model_downloader import get_models\nfrom thought.model_downloader import download_model\nfrom thought.model_downloader import add_model\n```\n\n### Downloading a model to play with\n\nThe following code shows you how to download models, especially useful when hot deploying to dockers and you need your code to download the models for runtime.\n\n```python\n# gets a bunch of models, set to true to download a bunch of models - not recommended!\nget_models(use_verified=False)\n# add a specific model - right click copy link of GGUF from hugging face\nadd_model(\"https://huggingface.co/mradermacher/DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored-i1-GGUF/resolve/main/DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored.i1-Q4_K_M.gguf\")\nadd_model(\"https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/blob/main/nomic-embed-text-v1.5.Q4_K_M.gguf\")\n# download the added models\ndownload_model(\"DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored\", \"i1-Q4_K_M\")\ndownload_model(\"nomic-embed-text-v1.5\", \"Q4_K_M\")\n```\n\n### Embedding text with a model\n\nThe flag `embedding=True` needs to be on!\n\n```python\n\nllm = load_model(\"./model_db/nomic-embed-text-v1.5.Q4_K_M.gguf\", verbose=True, embedding=True)\ntext = \"I think therefore I am\"\n\nvectors = embed(llm, text)\n# or\nvectors = llm.thought_embed(text)\n\n\nprint(vectors)\n```\n\n### Generating text from the LLM\n\nPlease note we do not do any weird prompt templating or black boxing on your input prompt. You need to look at the model \nand provide the prompt with the correct \"tokens\" in your prompt.\n\n```python\n\ndef token_stream(token):\n    print(token)\n\nllm = load_model(\"./model_db/DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored-Q4_K_S-imat.gguf\", verbose=True, embedding=False)\n\ntext = \"I think therefore I am\"\n\nresponse = generate(llm, \"Explain this text\u003c|eot_id|\u003eUser:\"+text+\"\u003c|eot_id|\u003eAI:\", stop=[\"\\n\"], seed=123456, call_back=token_stream)\n#or\nresponse = llm.thought_generate(\"Explain this text\u003c|eot_id|\u003eUser:\"+text+\"\u003c|eot_id|\u003eAI:\", stop=[\"\\n\"], seed=123456, call_back=token_stream)\n\nprint(response)\n```\n\n## Developing\n\nFirst create a virtual environment on Python 11 or higher\n\n### Windows\n```\npython -m venv .venv\n.\\.venv\\Scripts\\activate\npip install poetry\npoetry install\n```\n\n### GPU support (Cuda)\nllama-cpp-python seems not to build with CUDA support on Windows or Linux by default. Here are the basic commands we end up running each time we install.\n\n#### Windows\nPlease replace the CUDA version with your version you have on your disk.\n```powershell\n$env:CMAKE_ARGS=\"-DGGML_CUDA=on\"   \n$env:CUDACXX=\"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.6\\bin\\nvcc.exe\"\npoetry run pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade --verbose\n```\n\n#### Linux\n```bash\nCUDACXX=/usr/local/cuda-12/bin/nvcc CMAKE_ARGS=\"-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=all-major\" poetry run pip install llama-cpp-python --no-cache --force-reinstall --upgrade --verbose\n```\n\n## Build\n```\npoetry build\npython -m twine upload --repository pypi dist/*\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftina4stack%2Fthought","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftina4stack%2Fthought","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftina4stack%2Fthought/lists"}