{"id":27786279,"url":"https://github.com/clever/optimus","last_synced_at":"2025-06-20T18:35:58.900Z","repository":{"id":16402800,"uuid":"19153709","full_name":"Clever/optimus","owner":"Clever","description":"Concurrently extract, transform, and load tables of data in Go","archived":false,"fork":false,"pushed_at":"2025-04-24T19:23:37.000Z","size":450,"stargazers_count":34,"open_issues_count":2,"forks_count":7,"subscribers_count":66,"default_branch":"master","last_synced_at":"2025-06-05T21:03:11.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Clever.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-04-25T17:02:24.000Z","updated_at":"2025-04-24T19:23:43.000Z","dependencies_parsed_at":"2022-07-26T01:16:24.668Z","dependency_job_id":"74990b44-0643-4f3c-847c-94b26ba40409","html_url":"https://github.com/Clever/optimus","commit_stats":{"total_commits":231,"total_committers":25,"mean_commits":9.24,"dds":0.696969696969697,"last_synced_commit":"49aa626b5d0426c04c4cc90b5f48cdb8e026dacf"},"previous_names":["azylman/optimus","azylman/getl"],"tags_count":34,"template":false,"template_full_name":null,"purl":"pkg:github/Clever/optimus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Clever%2Foptimus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Clever%2Foptimus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Clever%2Foptimus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Clever%2Foptimus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Clever","download_url":"https://codeload.github.com/Clever/optimus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Clever%2Foptimus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260997704,"owners_count":23094958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-30T15:58:54.678Z","updated_at":"2025-06-20T18:35:53.886Z","avatar_url":"https://github.com/Clever.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# optimus\n--\n    import \"github.com/Clever/optimus/v4\"\n\nPackage optimus provides interfaces and methods for lazily, concurrently\nmanipulating collections of data.\n\nThe Table interfaces is at the core of optimus. A Table is a lazy collection of\ndata. Several implementations of Tables are provided for extracting data\n(\"sources\"), such as from a CSV or Mongo.\n\nA TransformFunc is a function that can be applied to a Table to lazily,\nconcurrently modify the data. Several implementations of TransformFuncs are\nprovided for common transformations (e.g. Map).\n\nLastly, a set of Sink functions are provided that will \"sink\" a table into some\noutput, such as a CSV.\n\n\n### Example\n\nHere's an example program that performs a set of field and value mappings on a\nCSV file:\n\n    package optimus\n\n    import(\n    \tcsvSource \"github.com/Clever/optimus/v4/sources/csv\"\n    \tcsvSink \"github.com/Clever/optimus/v4/sinks/csv\"\n    \t\"github.com/Clever/optimus/v4\"\n    \t\"github.com/Clever/optimus/v4/transforms\"\n    \t\"os\"\n    )\n\n    func main() {\n    \t// Errors ignored for brevity\n    \tf, _ := os.Open(\"example1.csv\")\n    \tout, _ := os.Create(\"output.csv\")\n    \tdefer out.Close()\n    \tbegin := csvSource.New(f)\n    \tstep1 := optimus.Transform(begin, transforms.Fieldmap(fieldMappings))\n    \tstep2 := optimus.Transform(step1, transforms.Valuemap(valueMappings))\n    \tend := optimus.Transform(step2, transforms.Map(arbitraryTransformFunction))\n    \t_ = csvSink.New(out)(end)\n    }\n\nHere's one that uses chaining:\n\n    package optimus\n\n    import(\n    \tcsvSource \"github.com/Clever/optimus/v4/sources/csv\"\n    \tcsvSink \"github.com/Clever/optimus/v4/sinks/csv\"\n    \t\"github.com/Clever/optimus/v4\"\n    \t\"github.com/Clever/optimus/v4/transformer\"\n    \t\"os\"\n    )\n\n    func main() {\n    \t// Errors ignored for brevity\n    \tf, _ := os.Open(\"example1.csv\")\n    \tout, _ := os.Create(\"output.csv\")\n    \tdefer out.Close()\n    \ttransformer.New(csvSource.New(f)).Fieldmap(fieldMappings).Valuemap(\n    \t\tvalueMappings).Map(arbitraryTransformFunction).Sink(csvSink.New(out))\n    }\n\n## Usage\n\n#### type Row\n\n```go\ntype Row map[string]interface{}\n```\n\nRow is a representation of a line of data in a Table.\n\n#### type Sink\n\n```go\ntype Sink func(Table) error\n```\n\nA Sink function takes a Table and consumes all of its Rows.\n\n#### type Table\n\n```go\ntype Table interface {\n\t// Rows returns a channel that provides the Rows in the table.\n\tRows() \u003c-chan Row\n\t// Err returns the first non-EOF error that was encountered by the Table.\n\tErr() error\n\t// Stop signifies that a Table should stop sending Rows down its channel.\n\t// A Table is also responsible for calling Stop on any upstream Tables it knows about.\n\t// Stop should be idempotent. It's expected that Stop will never be called by a consumer of a\n\t// Table unless that consumer is also a Table. It can be used to Stop all upstream Tables in\n\t// the event of an error that needs to halt the pipeline.\n\tStop()\n}\n```\n\nTable is a representation of a table of data.\n\n#### func  Transform\n\n```go\nfunc Transform(source Table, transform TransformFunc) Table\n```\nTransform returns a new Table that provides all the Rows of the input Table\ntransformed with the TransformFunc.\n\n#### type TransformFunc\n\n```go\ntype TransformFunc func(in \u003c-chan Row, out chan\u003c- Row) error\n```\n\nTransformFunc is a function that can be applied to a Table to transform it. It\nshould receive the Rows from in and may send any number of Rows to out. It\nshould not return until it has finished all work (received all the Rows it's\ngoing to receive, sent all the Rows it's going to send).\n\n## Development\nYou should develop Go packages from inside your Go path.\nFor `optimus`, that means that you should be in `$GOPATH/src/github.com/Clever/optimus/v4`.\n```\n# Example\ncd $GOPATH/src/gopkg.in/Clever\ngit clone git@github.com:Clever/optimus.git optimus.v3\n```\n\n## Releasing\nTo create a new Optimus version do the following:\n\n0. If releasing a major version, update all references to old version\n```\ngit grep optimus.v\n```\n\n1. Download gitsem, a tool for semantic versioning with Git\n```\ngo get github.com/Clever/gitsem\n```\n\n2. Run gitsem in the root directory of the master Optimus branch\n```\ngitsem \u003cVERSION\u003e\n```\nVERSION can one of ```newversion | patch | minor | major``` as documented at (https://github.com/Clever/gitsem)\n\n3. Push the changes to git\n```\ngit push \u0026\u0026 git push --tags\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclever%2Foptimus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclever%2Foptimus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclever%2Foptimus/lists"}