{"id":15049228,"url":"https://github.com/grauho/sdc","last_synced_at":"2026-01-02T06:06:10.030Z","repository":{"id":229231455,"uuid":"776166641","full_name":"grauho/sdc","owner":"grauho","description":"A simple tool to convert .safetensor files that include certain large dtypes into ones that don't","archived":false,"fork":false,"pushed_at":"2024-05-04T13:26:12.000Z","size":59,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-20T20:40:57.127Z","etag":null,"topics":["ansi-c","c","c89","c90","c99","cli","conversion","converter","dtype","dtypes","json","safetensors","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grauho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-22T20:18:01.000Z","updated_at":"2025-01-14T03:19:44.000Z","dependencies_parsed_at":"2024-05-04T14:31:19.308Z","dependency_job_id":"d3f42479-9b87-4f49-a7a5-744c5b946628","html_url":"https://github.com/grauho/sdc","commit_stats":null,"previous_names":["grauho/sdc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grauho%2Fsdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grauho%2Fsdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grauho%2Fsdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grauho%2Fsdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grauho","download_url":"https://codeload.github.com/grauho/sdc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243505960,"owners_count":20301619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansi-c","c","c89","c90","c99","cli","conversion","converter","dtype","dtypes","json","safetensors","stable-diffusion"],"created_at":"2024-09-24T21:19:11.163Z","updated_at":"2026-01-02T06:06:09.953Z","avatar_url":"https://github.com/grauho.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SDC, Safetensors Dtype Converter\n\nThis is a simple utility for converting a .safetensors file that contains\ntypes not handled by GGML, eg: 64-bit floats or 64-bit integers, into a file\nwhere those same values are converted and re-encoded to a smaller float type.\nBy default a 32-bit \"full precision\" floating point type is used but IEEE \n16-bit \"half precision\" floats or 16-bit \"Brain\" floats may be used instead.\n\nThis is a quick and dirty solution to the problem and shouldn't be relied upon\nfor critical applications.\n\n## Building\n\nAn at least C99 compliant compiler is required to build this software, or for \nthe adventurous the program can be compiled in C90/C89 compliant mode if \nalternative definitions for the stdint types are provided along with other\nminor adjustments.\n\nCurrently only a POSIX makefile is included for automated building:\n\n``` shell\nmake\n```\n\nShould all else fail one may simply build the object files and link them \nmanually like in the bad old days:\n\n``` shell\ncc -Wall -pedantic -O2 -Wno-unused-function -c -o main.o main.c\ncc -Wall -pedantic -O2 -Wno-unused-function -c -o cJSON.o cJSON.c\ncc -Wall -pedantic -O2 -Wno-unused-function -c -o fileLoading.o fileLoading.c\ncc -Wall -pedantic -O2 -Wno-unused-function -c -o converting.o converting.c\ncc -Wall -pedantic -O2 -Wno-unused-function -o sdc main.o cJSON.o fileLoading.o converting.o\n```\n\nNotes:\n\nShould you find you get the \"Big-Endian\" warning despite knowing you are on a\nlittle-endian system you may force the portegg header library to recognize this\nby either defining PORTEGG\\_LITTLE\\_ENDIAN\\_SYSTEM either through a compile \ntime argument or simply in the source code before portegg.h is included in \nmain.h.\n\n## Command Line Options\n\n    -R, --replace                    : Converts file in-place, -o is ignored\n    -f, --float-out {F32, F16, BF16} : Desired float dtype output type\n    -i, --input  \u003cFILE PATH\u003e         : The safetensors file to be converted\n    -o, --output \u003cFILE PATH\u003e         : The desired output file \n    -v, --verbose                    : Prints more logging information\n    -h, --help                       : Prints a help message much like this one\n\nNotes: \n\n* Should no output file be specified the results will be outputted to a new\nfiled named output.safetensors\n\n* be sure to include .safetensors at the end of your desired output name\nif you want it to be automatically recognized by other programs as such\n\n* When using the replace option, -R, there is a possibility that if the \nprogram were to be terminated or crash during the overwriting of the input file \ndata could be lost. One could write out to a .swp file and then rename it to \nthe original but this would depend upon platform specific code\n\n* Non-C language float data types, F16 and BF16, rely on some bit fiddling to \nconvert down into, as such if running on a system that does not use the IEEE \nstandardized number of bits for the fraction, mantissa, and exponent the \nresulting values may be meaningless. \n\n## Example Invocation\n\n``` shell\n./sdc -i ../../foo.safetensors -o bar.safetensors --verbose -f F32\n```\n\n## Bugs\n\n* There is handling for endianess via the portegg header but this program was\ndeveloped and tested on a little-endian system so there is no guarantee that\nit will work as expected.\n\n* Especially large files may run into issues with fseek and it's historical\nlimitations, although in the authors experience it seems to manage on my \nLinux machine alright. The program will warn you if this is a possibility.\n\n* If the same file is given for both --input and --output using different \nrelative paths bad things may happen, but it should just die with a fread\nfailure.\n\n* It would seem some models just cannot handle the loss of precision when\nconverting down to F16 and BF16, for this reason it is strongly recommended\nnot to use the -R, --replace command line switch if converting to these \ndtypes.\n\n## License\n\nLicense information can be found in detail in LICENSE.txt but in short this\nprogram is licensed by me, grauho, under the BSD 4-Clause license while cJSON \nis licensed by it's authors under the MIT license, copies of both in their \nentirety may be found in the aforementioned file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrauho%2Fsdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrauho%2Fsdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrauho%2Fsdc/lists"}