{"id":17980979,"url":"https://github.com/nvidia/go-tfdata","last_synced_at":"2025-03-25T18:31:00.299Z","repository":{"id":66147200,"uuid":"261531098","full_name":"NVIDIA/go-tfdata","owner":"NVIDIA","description":"Go library that provides easy-to-use interfaces and tools for TensorFlow users, in particular allowing to train existing TF models on .tar and .tgz datasets","archived":false,"fork":false,"pushed_at":"2024-03-13T21:02:03.000Z","size":3798,"stargazers_count":14,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-20T16:47:12.508Z","etag":null,"topics":["datapipeline","golang","tar","tensorflow","tfexample","tfrecord"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-05T16:51:49.000Z","updated_at":"2023-12-06T09:34:29.000Z","dependencies_parsed_at":"2024-06-18T21:39:40.936Z","dependency_job_id":null,"html_url":"https://github.com/NVIDIA/go-tfdata","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fgo-tfdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fgo-tfdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fgo-tfdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fgo-tfdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA","download_url":"https://codeload.github.com/NVIDIA/go-tfdata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245519906,"owners_count":20628791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datapipeline","golang","tar","tensorflow","tfexample","tfrecord"],"created_at":"2024-10-29T18:07:09.963Z","updated_at":"2025-03-25T18:30:59.033Z","avatar_url":"https://github.com/NVIDIA.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The `go-tfdata` library\n\nThe `go-tfdata` is a Go library helping to work with tar/tgz archives and files in \n[TFRecord and tf.Example formats](https://www.tensorflow.org/tutorials/load_data/tfrecord), including converting\nTAR files to TFRecord files.\nIt provides interfaces and their default implementations on each intermediate step between tar and TFRecord format.\nAdditionally, it includes easy to use utilities to convert and augment data in intermediate steps.  \n\nThe library is designed with simplicity, speed and extensibility in mind. The goal is not to support multiple, complicated\ncommunication protocols for remote data handling or complex algorithms implementations, it's rather giving ability for\nusers to extend it in any possible way.  \n\n[Full documentation](https://pkg.go.dev/github.com/NVIDIA/go-tfdata)\n\n## Available Commands\n\n`go-tfdata` provides default implementations for manipulating tar and TFRecord files. It includes:\n\n- `FromTar(io.Reader)` - read Samples from `io.Reader` in Tar format\n- `TransformSamples(transformations)` - transform each `Sample` according to provided transformations (either predeclared in `go-tfdata`\nor provided by a user)\n- `SampleToTFExample(reader, [typesMapping]` - default transformation from `Sample` to `TFExample` format. If typesMapping provided,\nmaps sample to TFExample accordingly to types.\n- `TransformTFExamples(transformations)` - transform each `TFExample` according to provided transformations\n- `ToTFRecord(io.Writer)` - write serialized TFExamples to `io.Writer` in TFRecord file format\n- `FilterEmptyExamples(reader)`, `FilterEmptySamples(reader)` - filter reader from empty TFExamples / Samples\n\n## Available transformations and selections\n\n`go-tfdata` provides basic Samples and TFExamples transformations and selections, which can be easily applied to the data\n\n### Selections\n\n- `ByKey(key)` - selects entry which key equals to `key`\n- `ByKeyValue(key, value)` - selects entry which key equals `key` and value equals `value`\n- `ByPrefix(name)`, `BySuffix(name)`, `BySubstring(name)` - selects entries which key is prefix, suffix or substring of `name`\n- `BySampleF(f)`, `ByExampleF(f)` - selects entries which keys are in subset returned by function `f`\n- TBA...\n\n### Transformations\n\n- `RenameTransformation(dest string, src []string)` - renames `src` fields into `dest` field\n- `SampleF(f func(core.Sample) core.Sample)` - transforms Sample based on specified function `f`\n- `TFExampleF(f func(*core.TFExample) *core.TFExample)` - transforms TFExample based on specified function `f`\n\n## Examples\n\n#### Convert Tar file to TFRecord\n\n```go\npipeline := NewPipeline().FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)\npipeline.Do()\n```\n\n\n#### Convert Tar file to TFRecord, save in TFExample \"cls\" as int64, \"jpeg\" as bytes\n\n```go\npipeline := NewPipeline().FromTar(inFile)\npipeline.SampleToTFExample(core.TypesMap{\n    \"cls\": core.FeatureType.INT64,\n    \"jpeg\": core.FeatureType.BYTES,\n})\npipeline.ToTFRecord(outFile).Do()\n```\n\n#### Convert Tar file to TFRecord, log every 10 TFExamples\n\n```go\ntype Logger struct {\n    reader TFExampleReader\n    cnt    int\n}\n\nfunc (l *Logger) Read() (*TFExample, bool) {\n    cnt++\n    if cnt % 10 == 0 { log.Infof(\"read %d examples\", cnt) }\n    return l.reader.Read()\n}\n\npipeline := NewPipeline().WithTFExampleStage(func(reader TFExampleReader) TFExampleReader {\n    return \u0026Logger{reader: reader}\n}).FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)\n\npipeline.Do()\n```\n\n#### Convert TarGz file to TFRecord, select only \"image\" entries from Samples\n\n```go\npipeline := NewPipeline().TransformSamples(\n    transform.ExampleSelections(selection.ByKey(\"image\"))\n).FromTarGz(inFile).SampleToTFExample().ToTFRecord(outFile)\npipeline.Do()\n```\n\n#### Convert Tar file to TFRecord, transform Samples in FAAS service\n\n```go\ntype FAASClient struct { \n    reader SamplesReader\n    ...\n}\n\nfunc (c *FAASClient) Read() (Sample, bool) {\n    sample, ok := c.reader.Read()\n    if !ok { return nil, false }\n    id := c.Send(sample)\n    c.Receive(id, \u0026sample)\n    return sample, true\n}\n\npipeline := NewPipeline().WithSamplesStage(func(reader SamplesReader) SamplesReader {\n    return FAASClient{reader: reader} \n}).FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)\npipeline.Do()\n```\n\nTo see fully working implementation of some examples see `go-tfdata/tests` package.\n\n## Internals\n\n### Pipeline\n\n`pipeline` is abstraction for TAR-to-TFRecord process. `pipeline` is made of `stages`. Default pipeline implementation has 5 stages:\n\n| Stage | Consumes | Produces | Required |  \n| --- | --- | --- | --- |  \n| `TarStage` | - | `SamplesReader` | Yes |  \n| `SamplesStage` | `SamplesReader` | `SamplesReader` | No |  \n| `Sample2TFExampleStage` | `SamplesReader` | `TFExampleReader` | Yes |  \n| `TFExamplesStage` | `TFExampleReader` | `TFExampleReader` | No |  \n| `TFRecordStage` | `TFExampleReader` | - | No |  \n\nWith this approach, evaluation can be (but doesn't have to be) lazy, meaning that each of the stages process the data when final consumer - `TFRecordStage` -\ndecides to consume a TFExample\n\nPipeline is high-level abstraction and can be replaced, extended or limited.\nFor each stage, default implementation can be used (or none at all for optional stages), or custom implementation can be provided by a user via `pipeline.With[STAGE]` method\n\n### Readers\n\nThere exists two types of readers interfaces - `SamplesReader`, `TFExamplesReader`. Their methods:\n```\nTFExampleReader interface {\n    Read() (ex *TFExample, ok bool)\n}\n```\n\n```\nSampleReader interface {\n    Read() (sample Sample, ok bool)\n}\n```\n\nIt's up to Reader implementation how it behaves on creation or `Read` calls. It might be executing a transformation only when\n`Read` method is called (lazy) or Reader can drain internal Reader and do transformations immediately. It can as well \nprefetch part of internal Reader data. Each of approaches has it's advantages and should be considered per use-case.\n\n### TFExample\n\nTFExample format is based on [TensorFlow](https://www.tensorflow.org/) [example.proto](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/example)\nfiles. Thanks to [Go Protobuf API v2](https://blog.golang.org/protobuf-apiv2), a structure of TFExamples in TFRecord files is determined\nautomatically. Learn more about [TFExample](https://www.tensorflow.org/tutorials/load_data/tfrecord#tfexample).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia%2Fgo-tfdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvidia%2Fgo-tfdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia%2Fgo-tfdata/lists"}