{"id":13724026,"url":"https://github.com/segmentio/encoding","last_synced_at":"2025-08-07T10:09:36.441Z","repository":{"id":40504235,"uuid":"225696514","full_name":"segmentio/encoding","owner":"segmentio","description":"Go package containing implementations of efficient encoding, decoding, and validation APIs.","archived":false,"fork":false,"pushed_at":"2025-07-28T04:38:48.000Z","size":13863,"stargazers_count":1021,"open_issues_count":19,"forks_count":53,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-07-28T06:14:57.726Z","etag":null,"topics":["ascii","decoding","encoding","go","golang","hacktoberfest","iso8601","json","performance","protobuf","validation"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/segmentio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-03T19:14:28.000Z","updated_at":"2025-07-24T00:59:48.000Z","dependencies_parsed_at":"2025-07-18T00:24:55.896Z","dependency_job_id":"4ddbd3ce-e035-41c9-8146-3563b5f727aa","html_url":"https://github.com/segmentio/encoding","commit_stats":{"total_commits":119,"total_committers":21,"mean_commits":5.666666666666667,"dds":0.5126050420168067,"last_synced_commit":"3055897c1c5b74bc4c4ec5f702c22ed90020f410"},"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"purl":"pkg:github/segmentio/encoding","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fencoding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fencoding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fencoding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fencoding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/segmentio","download_url":"https://codeload.github.com/segmentio/encoding/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/segmentio%2Fencoding/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269238181,"owners_count":24383487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ascii","decoding","encoding","go","golang","hacktoberfest","iso8601","json","performance","protobuf","validation"],"created_at":"2024-08-03T01:01:48.723Z","updated_at":"2025-08-07T10:09:36.421Z","avatar_url":"https://github.com/segmentio.png","language":"Go","readme":"# encoding ![build status](https://github.com/segmentio/encoding/actions/workflows/test.yml/badge.svg) [![Go Report Card](https://goreportcard.com/badge/github.com/segmentio/encoding)](https://goreportcard.com/report/github.com/segmentio/encoding) [![GoDoc](https://godoc.org/github.com/segmentio/encoding?status.svg)](https://godoc.org/github.com/segmentio/encoding)\n\nGo package containing implementations of encoders and decoders for various data\nformats.\n\n## Motivation\n\nAt Segment, we do a lot of marshaling and unmarshaling of data when sending,\nqueuing, or storing messages. The resources we need to provision on the\ninfrastructure are directly related to the type and amount of data that we are\nprocessing. At the scale we operate at, the tools we choose to build programs\ncan have a large impact on the efficiency of our systems. It is important to\nexplore alternative approaches when we reach the limits of the code we use.\n\nThis repository includes experiments for Go packages for marshaling and\nunmarshaling data in various formats. While the focus is on providing a high\nperformance library, we also aim for very low development and maintenance overhead\nby implementing APIs that can be used as drop-in replacements for the default\nsolutions.\n\n## Requirements and Maintenance Schedule\n\nThis package has no dependencies outside of the core runtime of Go.  It\nrequires a recent version of Go.\n\nThis package follows the same maintenance schedule as the [Go\nproject](https://github.com/golang/go/wiki/Go-Release-Cycle#release-maintenance),\nmeaning that issues relating to versions of Go which aren't supported by the Go\nteam, or versions of this package which are older than 1 year, are unlikely to\nbe considered.\n\nAdditionally, we have fuzz tests which aren't a runtime required dependency but\nwill be pulled in when running `go mod tidy`.  Please don't include these go.mod\nupdates in change requests.\n\n## encoding/json [![GoDoc](https://godoc.org/github.com/segmentio/encoding/json?status.svg)](https://godoc.org/github.com/segmentio/encoding/json)\n\nMore details about _how_ this package achieves a lower CPU and memory footprint\ncan be found [in the package README](json/README.md).\n\nThe `json` sub-package provides a re-implementation of the functionalities\noffered by the standard library's [`encoding/json`](https://golang.org/pkg/encoding/json/)\npackage, with a focus on lowering the CPU and memory footprint of the code.\n\nThe exported API of this package mirrors the standard library's\n[`encoding/json`](https://golang.org/pkg/encoding/json/) package, the only\nchange needed to take advantage of the performance improvements is the import\npath of the `json` package, from:\n```go\nimport (\n    \"encoding/json\"\n)\n```\nto\n```go\nimport (\n    \"github.com/segmentio/encoding/json\"\n)\n```\n\nThe improvement can be significant for code that heavily relies on serializing\nand deserializing JSON payloads. The CI pipeline runs benchmarks to compare the\nperformance of the package with the standard library and other popular\nalternatives; here's an overview of the results:\n\n**Comparing to encoding/json (`v1.16.2`)**\n```\nname                           old time/op    new time/op     delta\nMarshal/*json.codeResponse2      6.40ms ± 2%     3.82ms ± 1%   -40.29%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2    28.1ms ± 3%      5.6ms ± 3%   -80.21%  (p=0.008 n=5+5)\n\nname                           old speed      new speed       delta\nMarshal/*json.codeResponse2     303MB/s ± 2%    507MB/s ± 1%   +67.47%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2  69.2MB/s ± 3%  349.6MB/s ± 3%  +405.42%  (p=0.008 n=5+5)\n\nname                           old alloc/op   new alloc/op    delta\nMarshal/*json.codeResponse2       0.00B           0.00B           ~     (all equal)\nUnmarshal/*json.codeResponse2    1.80MB ± 1%     0.02MB ± 0%   -99.14%  (p=0.016 n=5+4)\n\nname                           old allocs/op  new allocs/op   delta\nMarshal/*json.codeResponse2        0.00            0.00           ~     (all equal)\nUnmarshal/*json.codeResponse2     76.6k ± 0%       0.1k ± 3%   -99.92%  (p=0.008 n=5+5)\n```\n\n*Benchmarks were run on a Core i9-8950HK CPU @ 2.90GHz.*\n\n**Comparing to github.com/json-iterator/go (`v1.1.10`)**\n```\nname                           old time/op    new time/op    delta\nMarshal/*json.codeResponse2      6.19ms ± 3%    3.82ms ± 1%   -38.26%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2    8.52ms ± 3%    5.55ms ± 3%   -34.84%  (p=0.008 n=5+5)\n\nname                           old speed      new speed      delta\nMarshal/*json.codeResponse2     313MB/s ± 3%   507MB/s ± 1%   +61.91%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2   228MB/s ± 3%   350MB/s ± 3%   +53.50%  (p=0.008 n=5+5)\n\nname                           old alloc/op   new alloc/op   delta\nMarshal/*json.codeResponse2       8.00B ± 0%     0.00B       -100.00%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2    1.05MB ± 0%    0.02MB ± 0%   -98.53%  (p=0.000 n=5+4)\n\nname                           old allocs/op  new allocs/op  delta\nMarshal/*json.codeResponse2        1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)\nUnmarshal/*json.codeResponse2     37.2k ± 0%      0.1k ± 3%   -99.83%  (p=0.008 n=5+5)\n```\n\nAlthough this package aims to be a drop-in replacement of [`encoding/json`](https://golang.org/pkg/encoding/json/),\nit does not guarantee the same error messages. It will error in the same cases\nas the standard library, but the exact error message may be different.\n\n## encoding/iso8601 [![GoDoc](https://godoc.org/github.com/segmentio/encoding/iso8601?status.svg)](https://godoc.org/github.com/segmentio/encoding/iso8601)\n\nThe `iso8601` sub-package exposes APIs to efficiently deal with with string\nrepresentations of iso8601 dates.\n\nData formats like JSON have no syntaxes to represent dates, they are usually\nserialized and represented as a string value. In our experience, we often have\nto _check_ whether a string value looks like a date, and either construct a\n`time.Time` by parsing it or simply treat it as a `string`. This check can be\ndone by attempting to parse the value, and if it fails fallback to using the\nraw string. Unfortunately, while the _happy path_ for `time.Parse` is fairly\nefficient, constructing errors is much slower and has a much bigger memory\nfootprint.\n\nWe've developed fast iso8601 validation functions that cause no heap allocations\nto remediate this problem. We added a validation step to determine whether\nthe value is a date representation or a simple string. This reduced CPU and\nmemory usage by 5% in some programs that were doing `time.Parse` calls on very\nhot code paths.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Fencoding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsegmentio%2Fencoding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsegmentio%2Fencoding/lists"}