{"id":19054658,"url":"https://github.com/ssbuild/fastdatasets","last_synced_at":"2025-11-12T15:08:29.432Z","repository":{"id":58546662,"uuid":"465264944","full_name":"ssbuild/fastdatasets","owner":"ssbuild","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-03T14:41:27.000Z","size":334,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"dev","last_synced_at":"2025-09-25T15:51:28.365Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-02T10:49:24.000Z","updated_at":"2025-05-03T14:41:30.000Z","dependencies_parsed_at":"2022-09-11T19:32:48.274Z","dependency_job_id":"6a91740b-ce4d-425d-a5a3-ba3cf07a120a","html_url":"https://github.com/ssbuild/fastdatasets","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/ssbuild/fastdatasets","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Ffastdatasets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Ffastdatasets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Ffastdatasets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Ffastdatasets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssbuild","download_url":"https://codeload.github.com/ssbuild/fastdatasets/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbuild%2Ffastdatasets/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284054900,"owners_count":26939664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-12T02:00:06.336Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T23:39:16.637Z","updated_at":"2025-11-12T15:08:29.416Z","avatar_url":"https://github.com/ssbuild.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\r\n## The update statement \r\n\r\n```text\r\n2023-10-28  support more torch well known datatasets \r\n2023-07-08: support some nested case\r\n2023-07-02: support arrow parquet\r\n2023-04-28: fix lmdb mutiprocess\r\n2023-02-13: add TopDataset with iterable_dataset and patch\r\n2022-12-07: modify a bug for randomdataset for batch reminder\r\n2022-11-07: add numpy writer and parser,add memory writer and parser\r\n2022-10-29: add kv dataset \r\n```\r\n\r\n## usage\r\n  [numpy_io](https://github.com/ssbuild/numpy_io) \r\n\r\n## Install\r\n```commandline\r\npip install -U fastdatasets\r\n```\r\n\r\n\r\n### 1. Record Write\r\n\r\n```python\r\nimport data_serialize\r\nfrom fastdatasets.record import load_dataset, gfile,TFRecordOptions, TFRecordCompressionType, TFRecordWriter\r\n\r\n# Example Features结构兼容tensorflow.dataset\r\ndef test_write_featrue():\r\n    options = 'GZIP'\r\n\r\n    def test_write(filename, N=3, context='aaa'):\r\n        with TFRecordWriter(filename, options=options) as file_writer:\r\n            for _ in range(N):\r\n                val1 = data_serialize.Int64List(value=[1, 2, 3] * 20)\r\n                val2 = data_serialize.FloatList(value=[1, 2, 3] * 20)\r\n                val3 = data_serialize.BytesList(value=[b'The china', b'boy'])\r\n                featrue = data_serialize.Features(feature=\r\n                {\r\n                    \"item_0\": data_serialize.Feature(int64_list=val1),\r\n                    \"item_1\": data_serialize.Feature(float_list=val2),\r\n                    \"item_2\": data_serialize.Feature(bytes_list=val3)\r\n                }\r\n                )\r\n                example = data_serialize.Example(features=featrue)\r\n                file_writer.write(example.SerializeToString())\r\n\r\n    test_write('d:/example.tfrecords0', 3, 'file0')\r\n    test_write('d:/example.tfrecords1', 10, 'file1')\r\n    test_write('d:/example.tfrecords2', 12, 'file2')\r\n\r\n\r\n# 写任意字符串\r\ndef test_write_string():\r\n    options = 'GZIP'\r\n\r\n    def test_write(filename, N=3, context='aaa'):\r\n        with TFRecordWriter(filename, options=options) as file_writer:\r\n            for _ in range(N):\r\n                # x, y = np.random.random(), np.random.random()\r\n                file_writer.write(context + '____' + str(_))\r\n\r\n    test_write('d:/example.tfrecords0', 3, 'file0')\r\n    test_write('d:/example.tfrecords1', 10, 'file1')\r\n    test_write('d:/example.tfrecords2', 12, 'file2')\r\n\r\n\r\n\r\n```\r\n\r\n### 2. record Simple Writer Demo\r\n\r\n```python\r\n# @Time    : 2022/9/18 23:27\r\nimport pickle\r\nimport data_serialize\r\nimport numpy as np\r\nfrom fastdatasets.record import load_dataset\r\nfrom fastdatasets.record import RECORD, WriterObject,FeatureWriter,StringWriter,PickleWriter,DataType,NumpyWriter\r\n\r\nfilename= r'd:\\\\example_writer.record'\r\n\r\ndef test_writer(filename):\r\n    print('test_feature ...')\r\n    options = RECORD.TFRecordOptions(compression_type='GZIP')\r\n    f = NumpyWriter(filename,options=options)\r\n\r\n    values = []\r\n    n = 30\r\n    for i in range(n):\r\n        train_node = {\r\n            \"index\": np.asarray(i, dtype=np.int64),\r\n            'image': np.random.rand(3, 4),\r\n            'labels': np.random.randint(0, 21128, size=(10), dtype=np.int64),\r\n            'bdata': np.asarray(b'11111111asdadasdasdaa')\r\n        }\r\n\r\n        values.append(train_node)\r\n        if (i + 1) % 10000 == 0:\r\n            f.write_batch( values)\r\n            values.clear()\r\n    if len(values):\r\n        f.write_batch(values)\r\n    f.close()\r\n\r\ndef test_iterable(filename):\r\n    options = RECORD.TFRecordOptions(compression_type='GZIP')\r\n    datasets = load_dataset.IterableDataset(filename, options=options).parse_from_numpy_writer()\r\n    for i, d in enumerate(datasets):\r\n        print(i, d)\r\n\r\ndef test_random(filename):\r\n    options = RECORD.TFRecordOptions(compression_type='GZIP')\r\n    datasets = load_dataset.RandomDataset(filename, options=options).parse_from_numpy_writer()\r\n    print(len(datasets))\r\n    for i in range(len(datasets)):\r\n        d = datasets[i]\r\n        print(i, d)\r\n\r\ntest_writer(filename)\r\ntest_iterable(filename)\r\n```\r\n\r\n### 3. IterableDataset demo\r\n\r\n```python\r\nimport data_serialize\r\nfrom fastdatasets.record import load_dataset, gfile, RECORD\r\n\r\ndata_path = gfile.glob('d:/example.tfrecords*')\r\noptions = RECORD.TFRecordOptions(compression_type=None)\r\nbase_dataset = load_dataset.IterableDataset(data_path, cycle_length=1,\r\n                                            block_length=1,\r\n                                            buffer_size=128,\r\n                                            options=options,\r\n                                            with_share_memory=True)\r\n\r\n\r\ndef test_batch():\r\n    num = 0\r\n    for _ in base_dataset:\r\n        num += 1\r\n    print('base_dataset num', num)\r\n\r\n    base_dataset.reset()\r\n    ds = base_dataset.repeat(2).repeat(2).repeat(3).map(lambda x: x + bytes('_aaaaaaaaaaaaaa', encoding='utf-8'))\r\n    num = 0\r\n    for _ in ds:\r\n        num += 1\r\n\r\n    print('repeat(2).repeat(2).repeat(3) num ', num)\r\n\r\n\r\ndef test_torch():\r\n    def filter_fn(x):\r\n        if x == b'file2____2':\r\n            return True\r\n        return False\r\n\r\n    base_dataset.reset()\r\n    dataset = base_dataset.filter(filter_fn).interval(2, 0)\r\n    i = 0\r\n    for d in dataset:\r\n        i += 1\r\n        print(i, d)\r\n\r\n    base_dataset.reset()\r\n    dataset = base_dataset.batch(3)\r\n    i = 0\r\n    for d in dataset:\r\n        i += 1\r\n        print(i, d)\r\n\r\n    # torch.utils.data.IterableDataset\r\n    from fastdatasets.torch_dataset import IterableDataset\r\n    dataset.reset()\r\n    ds = IterableDataset(dataset=dataset)\r\n    for d in ds:\r\n        print(d)\r\n\r\n\r\ndef test_mutiprocess():\r\n    print('mutiprocess 0...')\r\n    base_dataset.reset()\r\n    dataset = base_dataset.shard(num_shards=3, index=0)\r\n    i = 0\r\n    for d in dataset:\r\n        i += 1\r\n        print(i, d)\r\n\r\n    print('mutiprocess 1...')\r\n    base_dataset.reset()\r\n    dataset = base_dataset.shard(num_shards=3, index=1)\r\n    i = 0\r\n    for d in dataset:\r\n        i += 1\r\n        print(i, d)\r\n\r\n    print('mutiprocess 2...')\r\n    base_dataset.reset()\r\n    dataset = base_dataset.shard(num_shards=3, index=2)\r\n    i = 0\r\n    for d in dataset:\r\n        i += 1\r\n        print(i, d)\r\n\r\n```\r\n\r\n\r\n\r\n### 4. RandomDataset demo\r\n\r\n```python\r\nfrom fastdatasets.record import load_dataset, gfile, RECORD\r\n\r\ndata_path = gfile.glob('d:/example.tfrecords*')\r\noptions = RECORD.TFRecordOptions(compression_type=None)\r\ndataset = load_dataset.RandomDataset(data_path, options=options,\r\n                                     with_share_memory=True)\r\n\r\ndataset = dataset.map(lambda x: x + b\"adasdasdasd\")\r\nprint(len(dataset))\r\n\r\nfor i in range(len(dataset)):\r\n    print(i + 1, dataset[i])\r\n\r\nprint('batch...')\r\ndataset = dataset.batch(7)\r\nfor i in range(len(dataset)):\r\n    print(i + 1, dataset[i])\r\n\r\nprint('unbatch...')\r\ndataset = dataset.unbatch()\r\nfor i in range(len(dataset)):\r\n    print(i + 1, dataset[i])\r\n\r\nprint('shuffle...')\r\ndataset = dataset.shuffle(10)\r\nfor i in range(len(dataset)):\r\n    print(i + 1, dataset[i])\r\n\r\nprint('map...')\r\ndataset = dataset.map(transform_fn=lambda x: x + b'aa22222222222222222222222222222')\r\nfor i in range(len(dataset)):\r\n    print(i + 1, dataset[i])\r\n\r\nprint('torch Dataset...')\r\nfrom fastdatasets.torch_dataset import Dataset\r\n\r\nd = Dataset(dataset)\r\nfor i in range(len(d)):\r\n    print(i + 1, d[i])\r\n\r\n\r\n```\r\n\r\n\r\n\r\n### 5. leveldb dataset\r\n\r\n```python\r\n# @Time    : 2022/10/27 20:37\r\n# @Author  : tk\r\nimport numpy as np\r\nfrom tqdm import tqdm\r\nfrom fastdatasets.leveldb import DB,load_dataset,WriterObject,DataType,StringWriter,JsonWriter,FeatureWriter,NumpyWriter\r\n\r\ndb_path = 'd:\\\\example_leveldb_numpy'\r\n\r\ndef test_write(db_path):\r\n    options = DB.LeveldbOptions(create_if_missing=True,error_if_exists=False)\r\n    f = NumpyWriter(db_path, options = options)\r\n    keys,values = [],[]\r\n    n = 30\r\n    for i in range(n):\r\n        train_node = {\r\n            \"index\":np.asarray(i,dtype=np.int64),\r\n            'image': np.random.rand(3,4),\r\n            'labels': np.random.randint(0,21128,size=(10),dtype=np.int64),\r\n            'bdata': np.asarray(b'11111111asdadasdasdaa')\r\n        }\r\n        keys.append('input{}'.format(i))\r\n        values.append(train_node)\r\n        if (i+1) % 10000 == 0:\r\n            f.put_batch(keys,values)\r\n            keys.clear()\r\n            values.clear()\r\n    if len(keys):\r\n        f.put_batch(keys, values)\r\n        \r\n    f.get_writer.put('total_num',str(n))\r\n    f.close()\r\n\r\n\r\n\r\ndef test_random(db_path):\r\n    options = DB.LeveldbOptions(create_if_missing=False, error_if_exists=False)\r\n    dataset = load_dataset.RandomDataset(db_path,\r\n                                        data_key_prefix_list=('input',),\r\n                                        num_key='total_num',\r\n                                        options = options)\r\n\r\n    dataset = dataset.parse_from_numpy_writer().shuffle(10)\r\n    print(len(dataset))\r\n    for i in tqdm(range(len(dataset)),total=len(dataset)):\r\n        d = dataset[i]\r\n        print(i,d)\r\n\r\ntest_write(db_path)\r\ntest_random(db_path)\r\n\r\n```\r\n\r\n\r\n### 6. lmdb dataset\r\n\r\n```python\r\n# @Time    : 2022/10/27 20:37\r\n# @Author  : tk\r\n\r\nimport numpy as np\r\nfrom tqdm import tqdm\r\nfrom fastdatasets.lmdb import DB,LMDB,load_dataset,WriterObject,DataType,StringWriter,JsonWriter,FeatureWriter,NumpyWriter\r\n\r\ndb_path = 'd:\\\\example_lmdb_numpy'\r\n\r\ndef test_write(db_path):\r\n    options = DB.LmdbOptions(env_open_flag = 0,\r\n                env_open_mode = 0o664, # 8进制表示\r\n                txn_flag = 0,\r\n                dbi_flag = 0,\r\n                put_flag = 0)\r\n\r\n    f = NumpyWriter(db_path, options = options,map_size=1024 * 1024 * 1024)\r\n\r\n    keys, values = [], []\r\n    n = 30\r\n    for i in range(n):\r\n        train_node = {\r\n            'image': np.random.rand(3, 4),\r\n            'labels': np.random.randint(0, 21128, size=(10), dtype=np.int64),\r\n            'bdata': np.asarray(b'11111111asdadasdasdaa')\r\n        }\r\n        keys.append('input{}'.format(i))\r\n        values.append(train_node)\r\n        if (i + 1) % 10000 == 0:\r\n            f.put_batch(keys, values)\r\n            keys.clear()\r\n            values.clear()\r\n    if len(keys):\r\n        f.put_batch(keys, values)\r\n\r\n    f.get_writer.put('total_num',str(n))\r\n    f.close()\r\n\r\n\r\n\r\ndef test_random(db_path):\r\n    options = DB.LmdbOptions(env_open_flag=DB.LmdbFlag.MDB_RDONLY,\r\n                               env_open_mode=0o664,  # 8进制表示\r\n                               txn_flag=LMDB.LmdbFlag.MDB_RDONLY,\r\n                               dbi_flag=0,\r\n                               put_flag=0)\r\n    dataset = load_dataset.RandomDataset(db_path,\r\n                                        data_key_prefix_list=('input',),\r\n                                        num_key='total_num',\r\n                                        options = options)\r\n\r\n    dataset = dataset.parse_from_numpy_writer().shuffle(10)\r\n    print(len(dataset))\r\n    for i in tqdm(range(len(dataset)), total=len(dataset)):\r\n        d = dataset[i]\r\n        print(d)\r\n\r\ntest_write(db_path)\r\ntest_random(db_path)\r\n```\r\n\r\n\r\n\r\n### 7. arrow dataset \r\n\r\n\r\n```python\r\n\r\n\r\nfrom fastdatasets.arrow.writer import PythonWriter\r\nfrom fastdatasets.arrow.dataset import load_dataset,arrow\r\n\r\n\r\npath_file = 'd:/tmp/data.arrow'\r\n\r\n\r\n\r\nwith_stream = True\r\ndef test_write():\r\n    fs = PythonWriter(path_file,\r\n                        schema={'id': 'int32',\r\n                                'text': 'str',\r\n                                'map': 'map',\r\n                                'map2': 'map_list'\r\n                                },\r\n                        with_stream=with_stream,\r\n                        options=None)\r\n    for i in range(2):\r\n        data = {\r\n            \"id\": list(range(i * 3,(i+ 1) * 3)),\r\n            'text': ['asdasdasdas' + str(i) for i in range(3)],\r\n            'map': [\r\n                {\"a\": \"aa1\" + str(i), \"b\": \"bb1\", \"c\": \"ccccccc\"},\r\n                {\"a\": \"aa2\", \"b\": \"bb2\", \"c\": \"ccccccc\"},\r\n                {\"a\": \"aa3\", \"b\": \"bb3\", \"c\": \"ccccccc\"},\r\n            ],\r\n            'map2': [\r\n\r\n                [\r\n                    {\"a\": \"11\" + str(i), \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"12\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"13\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                ],\r\n                [\r\n                    {\"a\": \"21\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"22\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                ],\r\n                [\r\n                    {\"a\": \"31\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"32\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"32\", \"b\": \"bb\", \"c\": \"ccccccc22222222222222\"},\r\n                ]\r\n            ]\r\n        }\r\n        # fs.write_batch(data.keys(),data.values())\r\n        status = fs.write_batch(data.keys(),data.values())\r\n        assert status.ok(),status.message()\r\n\r\n\r\n    fs.close()\r\n\r\ndef test_random():\r\n    dataset = load_dataset.RandomDataset(path_file,with_share_memory=not with_stream)\r\n    print('total', len(dataset))\r\n    for i in range(len(dataset)):\r\n        print(i,dataset[i])\r\n\r\n\r\n\r\ndef test_read_iter():\r\n    dataset = load_dataset.IterableDataset(path_file,with_share_memory=not with_stream,batch_size=1)\r\n    for d in dataset:\r\n        print('iter',d)\r\n\r\n\r\ntest_write()\r\n\r\ntest_random()\r\n\r\ntest_read_iter()\r\n\r\n```\r\n\r\n### 8. parquet dataset \r\n\r\n```python\r\n\r\nfrom fastdatasets.parquet.writer import PythonWriter\r\nfrom fastdatasets.parquet.dataset import load_dataset\r\nfrom tfrecords.python.io.arrow import ParquetReader,arrow\r\n\r\n\r\npath_file = 'd:/tmp/data.parquet'\r\n\r\n\r\n\r\ndef test_write():\r\n    fs = PythonWriter(path_file,\r\n                      schema={'id': 'int32',\r\n                              'text': 'str',\r\n                              'map': 'map',\r\n                              'map2': 'map_list'\r\n                              },\r\n                        parquet_options=dict(write_batch_size = 10))\r\n    for i in range(2):\r\n        data = {\r\n            \"id\": list(range(i * 3, (i + 1) * 3)),\r\n            'text': ['asdasdasdas' + str(i) for i in range(3)],\r\n            'map': [\r\n                {\"a\": \"aa1\", \"b\": \"bb1\", \"c\": \"ccccccc\"},\r\n                {\"a\": \"aa2\", \"b\": \"bb2\", \"c\": \"ccccccc\"},\r\n                {\"a\": \"aa3\", \"b\": \"bb3\", \"c\": \"ccccccc\"},\r\n            ],\r\n            'map2': [\r\n\r\n                [\r\n                    {\"a\": \"11\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"12\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"13\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                ],\r\n                [\r\n                    {\"a\": \"21\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"22\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                ],\r\n                [\r\n                    {\"a\": \"31\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"32\", \"b\": \"bb\", \"c\": \"ccccccc\"},\r\n                    {\"a\": \"32\", \"b\": \"bb\", \"c\": \"ccccccc22222222222222\"},\r\n                ]\r\n            ]\r\n        }\r\n        # fs.write_batch(data.keys(),data.values())\r\n        fs.write_table(data.keys(),data.values())\r\n\r\n\r\n    fs.close()\r\n\r\ndef test_random():\r\n    dataset = load_dataset.RandomDataset(path_file)\r\n    print('total', len(dataset))\r\n    for i in range(len(dataset)):\r\n        print(dataset[i])\r\n\r\n\r\n\r\ndef test_read_iter():\r\n    dataset = load_dataset.IterableDataset(path_file,batch_size=1)\r\n    for d in dataset:\r\n        print('iter',d)\r\n\r\n\r\ntest_write()\r\n\r\ntest_random()\r\n\r\ntest_read_iter()\r\n\r\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbuild%2Ffastdatasets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssbuild%2Ffastdatasets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbuild%2Ffastdatasets/lists"}