{"id":22140878,"url":"https://github.com/stealth/tensor-pwn","last_synced_at":"2026-03-07T03:01:52.761Z","repository":{"id":213294960,"uuid":"733497936","full_name":"stealth/tensor-pwn","owner":"stealth","description":"AI fun","archived":false,"fork":false,"pushed_at":"2025-02-27T11:12:22.000Z","size":1593,"stargazers_count":24,"open_issues_count":1,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-02T16:53:21.503Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stealth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-19T13:12:55.000Z","updated_at":"2025-03-17T06:55:29.000Z","dependencies_parsed_at":"2023-12-19T16:40:37.663Z","dependency_job_id":"69e77c0a-34a7-4097-8fbb-f87a0f7e23de","html_url":"https://github.com/stealth/tensor-pwn","commit_stats":null,"previous_names":["stealth/tensor-pwn"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stealth/tensor-pwn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stealth%2Ftensor-pwn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stealth%2Ftensor-pwn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stealth%2Ftensor-pwn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stealth%2Ftensor-pwn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stealth","download_url":"https://codeload.github.com/stealth/tensor-pwn/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stealth%2Ftensor-pwn/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30206338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"online","status_checked_at":"2026-03-07T02:00:06.765Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T21:08:27.260Z","updated_at":"2026-03-07T03:01:52.732Z","avatar_url":"https://github.com/stealth.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"The state of AI\n===============\n\nThis is a research-repo. I occasionally add/remove stuff as I go.\nSome text may be out-dated as you read it, as some issues might have\nbeen fixed after I published it here.\n\n~~I do know nothing about tensors, fancy AI, models and such but~~ I do know\nabout coding, HPC and storage formats, so I will populate or not this repo\nwith classic IT security related stuff that I find myself funny about AI.\nTime has passed, meanwhile I also do have a clue about tensors, training, inference and deployment\nof AI models.\n\nThe usual topic is code execution, which should not be possible by just\nloading data (models) since few people would expect that they could\nexecute malware when fetching pre-trained models from somewhere. If\n*you* do expect that, this repo might not be for you but it is still\nconsidered fun for me since I quite know how data is handled in scientific\nenvironments.\n\n\nI was pointed to [a link](https://splint.gitbook.io/cyberblog/security-research/tensorflow-remote-code-execution-with-malicious-model) so\nsomeone already had fun with *TensorFlow* in past and all credits should go there. Since the attack surface\nfor model loading is just too obvious it is also safe to assume that more people (the known\nunknowns) had their fun with it in past. So it is interesting to see that AI security is widely anticipated\nas prompt-bypassing or mis-training of models on conferences and media but only few people are doing it right.\nHowever when I hear *prompt*, I certainly think of something different. :)\n\nThere also exist [tables for a safety overview](https://github.com/huggingface/safetensors), but it has to be read with\ncaution, e.g. their safety assumption for h5 files is wrong (see *TensorFlow* section).\n\n\nTar\n---\n\nInside [tarpwn](https://github.com/stealth/tensor-pwn/tree/master/tarpwn) you will find how to overwrite local files and therefore remotely execute code when\nattackers inject data in your network path when you are fetching training data. Training data is usually considered to be safe, unlike pickled trained models.\n\n\nPyTorch\n-------\n\nOne of the big Python frameworks for AI. One issue that can be found\nin this repo can be exploited by passing model data as plain *Pickle*\nso that the legacy loader is transparently invoked. Usually *PyTorch*\nhas a much more sophisticated storage format consisting of a zip file\nwith a certain structure.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/stealth/tensor-pwn/blob/master/pytorch/model1.jpg\" /\u003e\n\u003c/p\u003e\n\n*Code Exec demo during model load, popping xeyes on the screen*\n\nThe version I tested was `2.1.2`.\n\nThe more recent model saving format consists of a zip file containing `__torch__.py`\nfiles and such but I wasn't able to exec arbitrary code with it as the loader seems to\nbe a stripped down python. But that doesn't say anything, it might just work later in time.\nIf you happen to find a code exec there, just open a ticket here.\n\n\nAlso [check](https://github.com/stealth/tensor-pwn/blob/master/pytorch/flamegraph.jpg) how it execs code from 3rd party repos when you debug memory\nvia `torch.cuda.memory` and adds LPE opportunity.\n\nTensorFlow\n----------\n\nThe other big Python framework for AI. ~~I haven't checked in depth yet\nbut it looks like it is also using *Pickle*. However, there are safe\nways to use *Pickle*, so its left to check for the details.~~\n\n*TensorFlow* is supporting various formats such as legacy TF, HDF5 (Hirarchical Data Format)\nand the new Keras v3 format. The latter has got a `safe_mode` for a reason (`False` by default)\nwhen loading saved models, since *TensorFlow* models can have Lambda functions attached which\ncan be executed upon load. However, depending on actual used version and local package config,\nthe Keras code integrated into their upstream repo might not yet profit from the `safe_mode`\n(seems to be quite new feature?).\nNo such safe-mode with the HDF5 (h5) format. IOW you can create and save\na model that contains Base64 encoded Python bytecode that gets executed during load.\nThe dedicated subdir contains PoC code but not the model itself (as with *PyTorch*) since\nit contains much more data, so you have to call `create-model.py` first. The version\nI tested was `2.15.0`.\n\nTo be noted, that if you rename a h5 file to `model.keras` it still loads by the\nsame function `tf.keras.models.load_model()` with the vulnerable h5 loader, similar with *PyTorch*\nthat tries to be as convenient as possible when detecting storage formats.\n\nKeras\n-----\n\nMulti-framework API. In some of their versions there is a `safe_mode` check (default set to disable\nLambda execution) but it is not found in the fork thats integrated in *TensorFlow* so it is unclear\nwhich versions benefit from it.\n\n\nPaddlePaddle\n------------\n\n*PArallel Distributed Deep LEarning* framework from China. Also using *Pickle* as storage format\nand making a note on its security implications in their docu. If you check the docu, it looks a lot\nlike *PyTorch*.\n\n\nTorch\n-----\n\nThe original one and only real *Torch* written in Lua but seems to be discontinued.\n\nNumPy\n-----\n\nNot an AI framework but the foundation for a lot of data science stuff.\nTheir *Pickle* issue was fixed quite a while ago by setting `allow_pickle=False` by default\nin the `load()` function. However this will strip off a lot of functionallity thats commonly\nrequired for the npy persistence.\n\nONNX\n----\n\nONNX is a data format used for exchanging model data. It is based on\n*Protobuf*, so a simple code smuggling as with *Pickle* is not possible.\nHowever, it is subject to research. A problem with ONNX related to\n*PyTorch* can arise, since *PyTorch* is filename-agnostic so that a file\nending in `.onnx` could still be loaded as *Pickle* data, if the scientist\nas per confusion is just using `torch.load()` instead of `onnx.load()`.\n\n\nSafeTensors\n-----------\n\nA storage format for Tensors but you can't store entire models with it, so its not quite suitable\nfor the discussion here when you want to load pre-trained models.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstealth%2Ftensor-pwn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstealth%2Ftensor-pwn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstealth%2Ftensor-pwn/lists"}