{"id":17187466,"url":"https://github.com/davidmchan/mysfire","last_synced_at":"2025-08-20T14:07:00.491Z","repository":{"id":57444514,"uuid":"435766600","full_name":"DavidMChan/mysfire","owner":"DavidMChan","description":"Faster pytorch data loading","archived":false,"fork":false,"pushed_at":"2022-08-10T19:53:02.000Z","size":2693,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-18T19:22:21.003Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DavidMChan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-07T06:20:14.000Z","updated_at":"2023-09-01T00:24:16.000Z","dependencies_parsed_at":"2022-09-26T17:21:25.340Z","dependency_job_id":null,"html_url":"https://github.com/DavidMChan/mysfire","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/DavidMChan/mysfire","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fmysfire","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fmysfire/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fmysfire/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fmysfire/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DavidMChan","download_url":"https://codeload.github.com/DavidMChan/mysfire/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DavidMChan%2Fmysfire/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271330292,"owners_count":24740815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T01:06:28.188Z","updated_at":"2025-08-20T14:07:00.472Z","avatar_url":"https://github.com/DavidMChan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mysfire - Load data faster than light :)\n\nMysfire takes the headache out of writing dataset and data loader code for pytorch (that you usually repeat time and\ntime again). Mysfire encourages code reuse between projects when possible, and allows for easy extensibility when code\nreuse is impossible. Not only this, mysfire makes it easy to scale your datasets to hundreds of nodes, without thinking:\ncloud storage support is built in (and easy to extend), making it a powerful tool when going from your local laptop to\nyour public or private cloud.\n\n## Installation\n\nInstall this library with pip - `pip install mysfire[all]`\n\nFor a restricted subset of the data loading types, you can use different options:\n\n```bash\npip install mysfire # Default options, only basic processors\npip install mysfire[s3] # Include options for S3 connection\npip install mysfire[image] # Include image processors\npip install mysfire[video] # Include video processors\npip install mysfire[h5py] # Include H5py processors\npip install mysfire[nlp] # Include NLP processors\n```\n\n## Tour\n\nEach mysfire dataset is composed of three components:\n\n1. A definition describing the types of data (and preprocessing steps) in each column of your tabular file. Usually,\n   this is just the header of your CSV or TSV file.\n2. A tabular data store (usually just a CSV or TSV file, but we can load tabular data from S3, SQL or any other\n   extensible columnular store)\n3. A set of processors for processing and loading the data. For most common data types, these processors are built in,\n   but we recognize that every dataset is different, so we make it as easy as possible to add new processors, or\n   download third party processors from the mysfire community hub.\n\nLet's look at a hello-world mysfire dataset:\n\n```tsv\n# simple_dataset.tsv\nclass:int   data:npy\n0   sample_0.npy\n1   sample_1.npy\n2   sample_2.npy\n```\n\nThat's it. Easy as defining the types of each of the objects and a name for each column as a header in a TSV file. The\ndata is then super easy to load to your normal PyTorch workflow:\n\n```py\nfrom mysfire import DataLoader\n# Returns a standard PyTorch DataLoader, just replace the dataset with the TSV file!\ntrain_dataloader = DataLoader('simple_dataset.tsv', batch_size = 3, num_workers=12)\nfor batch in train_dataloader:\n    print(batch)\n```\n\nThis dataset will produce a dictionary:\n\n```py\n{\n    'class': [0, 1, 2]\n    'data': np.ndarray # Array of shape [BS x ...]\n}\n```\n\nWe handle loading, collating, and batching the data, so you can focus on training models, and iterating on experiments.\nOnboarding new datasets is as easy as setting up the new TSV file, and changing the links. No more messing around with\nthe code to add a new dataset switch! No coding that numpy loading dataset for the 100th time either - we've already\nlearned to handle all kinds of numpy types (even ragged arrays!)\n\nNeed S3? That's as easy as configuring a column with your S3 details:\n\n```\n# simple_s3_dataset.tsv\nclass:int   data:npy(s3_access_key=\"XXX\",s3_secret_key=\"XXX\",s3_endpoint=\"XXX\")\n0   s3://data/sample_0.npy\n1   s3://data/sample_1.npy\n2   s3://data/sample_2.npy\n```\n\nMerging two S3 sources? Configure each column independently:\n\n```\n# multisource_s3_dataset.tsv\nclass:int  data_a:npy(s3_access_key=\"AAA\",s3_secret_key=\"AAA\",s3_endpoint=\"AAA\")    data_b:npy(s3_access_key=\"BBB\",s3_secret_key=\"BBB\",s3_endpoint=\"BBB\")\n0   s3://data/sample_0.npy   s3://data/sample_0.npy\n1   s3://data/sample_1.npy   s3://data/sample_1.npy\n2   s3://data/sample_2.npy   s3://data/sample_2.npy\n```\n\nWorried about putting your keys in a dataset file? Use `$S3_SECRET_KEY` (a `$` prefix) to load environment variables at\nruntime.\n\n```\n# simple_s3_dataset.tsv\nclass:int   data:npy(s3_access_key=$S3_ACCESS_KEY,s3_secret_key=$S3_SECRET_KEY,s3_endpoint=$S3_ENDPOINT)\n0   s3://data/sample_0.npy\n1   s3://data/sample_1.npy\n2   s3://data/sample_2.npy\n```\n\nLoading images or video?\n\n```\n# multimedia_s3_dataset.tsv\nclass:int   picture:img(resize=256)  frames:video(uniform_temporal_subsample=16)\n0   image_1.png     video_1.mp4\n1   image_2.jpg     video_2.mp4\n2   image_3.JPEG     video_3.mp4\n```\n\nNeed to do NLP? Huggingface Tokenizers is built in\n\n```\n# tokenization_s3_dataset.tsv\nclass:int   labels:nlp.huggingface_tokenization(tokenizer_json=\"./tokenizer.json\")\n0   Hello world!\n1   Welcome to the Mysfire data processors\n```\n\nWorking with PyTorch Lightning? LightningDataModules are built in:\n\n```py\nfrom mysfire import LightningDataModule\ndatamodule = LightningDataModule(\n    'train.tsv',\n    'validate.tsv',\n    'test.tsv'\n)\n```\n\nNeed to run something at test-time? All you need to do is build a OneShotLoader:\n\n```py\nfrom mysfire import OneShotLoader\n\nloader = OneShotLoader(filename='train.tsv') # Initialize from a TSV\nloader = OneShotLoader(columns=[\"class:int\", \"data:npy\"]) # or pass the columns directly!\n\n\ndata = loader([[\"field 1\", \"field 2\"],[\"field 1\", \"field 2\"]]) # Load data with a single method\n```\n\nNeed to load a custom datatype? Or extend the existing datatypes? It's super easy:\n\n```py\nfrom mysfire import register_processor, Processor\n\n# Register the processor with mysfire before creating a dataset\n@register_processor\nclass StringAppendProcessor(Processor):\n\n    # Setup an init function with any optional arguments that are parsed from the column. We handle all of the\n    # complicated parsing for you, just take all options as Optional[str] arguments!\n    def __init__(self, string_to_append: Optional[str] = None):\n        self._string_to_append = string_to_append\n\n    # Define a typestring that is matched against the TSV columns. Registered processors take precidence over\n    # processors that are loaded by default\n    @classmethod\n    def typestr(cls):\n        return \"str\"\n\n    # Define a collate function for your data type which handles batching. If this is missing, we use the standard\n    # torch collate function instead\n    def collate(self, batch: List[Optional[str]]) -\u003e List[str]:\n        return [b or \"\" for b in batch]\n\n    # Add a call function which transforms the string data in the TSV into a single data sample.\n    def __call__(self, value: str) -\u003e str:\n        return value + self._string_to_append if self._string_to_append else \"\"\n```\n\nWant to add remote data loading to your processor? It's as easy as:\n\n```py\nfrom mysfire import register_processor, S3Processor\n\n# Start by extending the S3 processor\n@register_processor\nclass S3FileProcessor(S3Processor):\n    def __init__(self,\n                 s3_endpoint: Optional[str] = None,\n                 s3_access_key: Optional[str] = None,\n                 s3_secret_key: Optional[str] = None,\n                 s3_region: Optional[str] = None,):\n\n        super().__init__(\n            s3_endpoint=s3_endpoint,\n            s3_access_key=s3_access_key,\n            s3_secret_key=s3_secret_key,\n            s3_region=s3_region,\n        )\n\n    @classmethod\n    def typestr(cls):\n        return \"str\"\n\n    def collate(self, batch: List[Optional[str]]) -\u003e List[str]:\n        return [b or \"\" for b in batch]\n\n    def __call__(self, value: str) -\u003e Optional[str]:\n        try:\n            # Use resolve_to_local to fetch any file in S3 to a local filepath (or use a local file path if it's local)\n            with self.resolve_to_local(value) as f:\n                with open(f, 'r') as fp:\n                    return f\n        except Exception as ex:\n            return None\n```\n\nFor full details, and to check out everything that we offer, check out our docs!\n\n## Useful?\n\nCite us!\n\n```\nBibtex\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmchan%2Fmysfire","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidmchan%2Fmysfire","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidmchan%2Fmysfire/lists"}