{"id":15064032,"url":"https://github.com/altaris/turbo-broccoli","last_synced_at":"2026-01-04T23:38:29.673Z","repository":{"id":40638658,"uuid":"500375951","full_name":"altaris/turbo-broccoli","owner":"altaris","description":"JSON (de)serialization extensions","archived":false,"fork":false,"pushed_at":"2024-09-21T03:17:47.000Z","size":2296,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-03T00:39:08.350Z","etag":null,"topics":["bokeh","json","keras","numpy","pandas","pytorch","serialization","sklearn","tensorflow"],"latest_commit_sha":null,"homepage":"http://altaris.github.io/turbo-broccoli/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/altaris.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-06T09:48:24.000Z","updated_at":"2024-09-19T05:44:55.000Z","dependencies_parsed_at":"2024-02-20T10:29:55.544Z","dependency_job_id":"36baf8d3-6a32-4ef6-8e0e-d213e03e5076","html_url":"https://github.com/altaris/turbo-broccoli","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altaris%2Fturbo-broccoli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altaris%2Fturbo-broccoli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altaris%2Fturbo-broccoli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/altaris%2Fturbo-broccoli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/altaris","download_url":"https://codeload.github.com/altaris/turbo-broccoli/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239236414,"owners_count":19604902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bokeh","json","keras","numpy","pandas","pytorch","serialization","sklearn","tensorflow"],"created_at":"2024-09-25T00:10:40.013Z","updated_at":"2025-10-31T22:30:20.314Z","avatar_url":"https://github.com/altaris.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TurboBroccoli 🥦\n\n[![Repository](https://img.shields.io/badge/repo-github-pink)](https://github.com/altaris/turbo-broccoli)\n[![PyPI](https://img.shields.io/pypi/v/turbo-broccoli)](https://pypi.org/project/turbo-broccoli/)\n![License](https://img.shields.io/github/license/altaris/turbo-broccoli)\n[![Code\nstyle](https://img.shields.io/badge/style-black-black)](https://pypi.org/project/black)\n![hehe](https://img.shields.io/badge/project%20name%20by-github-pink)\n[![Documentation](https://badgen.net/badge/documentation/here/green)](https://altaris.github.io/turbo-broccoli/turbo_broccoli.html)\n\nJSON (de)serialization extensions, originally aimed at `numpy` and `tensorflow`\nobjects, but now supporting a wide range of objects.\n\n## Installation\n\n```sh\npip install turbo-broccoli\n```\n\n## Usage\n\n### To/from string\n\n```py\nimport numpy as np\nimport turbo_broccoli as tb\n\nobj = {\"an_array\": np.array([[1, 2], [3, 4]], dtype=\"float32\")}\ntb.to_json(obj)\n```\n\nproduces the following string (modulo indentation and the value of\n`$.an_array.data.data`):\n\n```json\n{\n  \"an_array\": {\n    \"__type__\": \"numpy.ndarray\",\n    \"__version__\": 5,\n    \"data\": {\n      \"__type__\": \"bytes\",\n      \"__version__\": 3,\n      \"data\": \"QAAAAAAAAAB7ImRhd...\"\n    }\n  }\n}\n```\n\nFor deserialization, simply use\n\n```py\ntb.from_json(json_string)\n```\n\n### To/from file\n\nSimply replace\n[`turbo_broccoli.to_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#to_json)\nand\n[`turbo_broccoli.from_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#from_json)\nwith\n[`turbo_broccoli.save_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#save_json)\nand\n[`turbo_broccoli.load_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#load_json):\n\n```py\nimport numpy as np\nimport turbo_broccoli as tb\n\nobj = {\"an_array\": np.array([[1, 2], [3, 4]], dtype=\"float32\")}\ntb.save_json(obj, \"foo/bar/foobar.json\")\n\n...\n\nobj = tb.load_json(\"foo/bar/foobar.json\")\n```\n\nIt is also possible to read/write compressed (with\n[zlib](https://www.zlib.net/)) JSON files:\n\n```py\ntb.save_json(obj, \"foo/bar/foobar.json.gz\")\n\n...\n\nobj = tb.load_json(\"foo/bar/foobar.json.gz\")\n```\n\n### [Contexts](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context)\n\nThe behaviour of\n[`turbo_broccoli.to_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#to_json)\nand\n[`turbo_broccoli.from_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#from_json)\ncan be tweaked by using\n[contexts](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context).\nFor example, to set a encryption/decryption key for [secret\ntypes](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/secret.html):\n\n```py\nimport nacl.secret\nimport nacl.utils\nimport turbo_broccoli as tb\n\nkey = nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)\nctx = tb.Context(nacl_shared_key=key)\nobj = {\"user\": \"alice\", \"password\": tb.SecretStr(\"dolphin\")}\ndoc = tb.to_json(obj, ctx)\n\n...\n\nobj = tb.from_json(doc, ctx)\n```\n\nThe behaviour of\n[`turbo_broccoli.save_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#save_json)\nand\n[`turbo_broccoli.load_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#load_json)\ncan be tweaked in a similar manner. For convenience, the argument of the\ncontext can be passed directly to the method instead of creating a context\nobject manually:\n\n```py\nimport nacl.secret\nimport nacl.utils\nimport turbo_broccoli as tb\n\nkey = nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)\nobj = {\"user\": \"alice\", \"password\": tb.SecretStr(\"dolphin\")}\ntb.save_json(obj, \"foo/bar/foobar.json\", nacl_shared_key=key)\n```\n\nSee [the\ndocumentation](https://altaris.github.io/turbo_broccoli/context.html#Context).\n\n### [Guarded blocks](https://altaris.github.io/turbo-broccoli/turbo_broccoli/guard.html)\n\nA\n[`turbo_broccoli.GuardedBlockHandler`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/guard.html#GuardedBlockHandler)\n\"guards\" a block of code, meaning it prevents it from being executed if it has\nbeen in the past. Check out [the\ndocumentation](https://altaris.github.io/turbo-broccoli/turbo_broccoli/guard.html)\nfor some examples.\n\n### [Guarded-parallel executors](https://altaris.github.io/turbo-broccoli/turbo_broccoli/parallel.html)\n\nA mix of\n[`joblib.Parallel`](https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html)\nand\n[`turbo_broccoli.GuardedBlockHandler`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/guard.html#GuardedBlockHandler):\na\n[`turbo_broccoli.Parallel`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/parallel.html#Parallel)\nobject can be used to execute jobs in parallel, but those whose results have\nalready been obtained in the past are skipped. See [the\ndocumentation](https://altaris.github.io/turbo-broccoli/turbo_broccoli/parallel.html)\nfor some examples.\n\n### Custom encoders/decoders\n\nYou can register you own custom encoders and decoders using\n[`turbo_broccoli.register_encoder`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/user.html#register_encoder)\nand\n[`turbo_broccoli.register_decoder`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/user.html#register_decoder):\n\n```py\nimport turbo_broccoli as tb\n\nclass MyClass:\n    a: int\n    b: np.ndarray\n    c: np.ndarray\n    def __init__(self, a: int, b: np.ndarray):\n        self.a, self.b = a, b\n        self.c = a + b\n\ndef encoder_c(obj: MyClass, ctx: tb.Context) -\u003e dict:\n    # If you register a decoder, you must include the key \"__type__\" and it\n    # must have value \"user.\u003cname_of_type\u003e\"\n    #       ↓\n    return {\"__type__\": \"user.MyClass\", \"a\": obj.a, \"b\": obj.b}\n\ndef decoder_c(obj: dict, ctx: tb.Context) -\u003e MyClass:\n    return MyClass(obj[\"a\"], obj[\"b\"])\n\ntb.register_encoder(encoder_c, \"MyClass\")\ntb.register_decoder(decoder_c, \"MyClass\")\n```\n\nAn encoder (for `MyClass`) is a function that takes two arguments: an object of\ntype `MyClass` and a\n[`turbo_broccoli.Context`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context),\nand returns a `dict`. That dict must contain objects that can be further\nserialized using TurboBroccoli (which includes all [supported\ntypes](https://altaris.github.io/turbo-broccoli/turbo_broccoli.html#supported-types)\nand any other type for which you registered an encoder). The return dict needs\nnot be flat.\n\nIf you register a decoder for `MyClass` (as in the example above), the dict\nmust contain the key/value `\"__type__\": \"user.MyClass\"`.\n\nA decoder (for `MyClass`) is a function that takes two arguments: a dict and a\n[`turbo_broccoli.Context`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context),\nand returns an object of type `MyClass`. The dict's values have already been\ndeserialized.\n\n### Artifacts\n\nIf an object inside `obj` is too large to be embedded inside the JSON file\n(e.g. a large numpy array), then an *artifact* file is created:\n\n```py\nimport numpy as np\nimport turbo_broccoli as tb\n\nobj = {\"an_array\": np.random.rand(1000, 1000)}\ntb.save_json(obj, \"foo/bar/foobar.json\")\n```\n\nproduces the JSON file\n\n```json\n{\n  \"an_array\": {\n    \"__type__\": \"numpy.ndarray\",\n    \"__version__\": 5,\n    \"data\": {\n      \"__type__\": \"bytes\",\n      \"__version__\": 3,\n      \"id\": \"1e6dff28-5e26-44df-9e7a-75bc726ce9aa\"\n    }\n  }\n}\n```\n\nand a file `foo/bar/foobar.1e6dff28-5e26-44df-9e7a-75bc726ce9aa.tb` containing\nthe array data. The artifact directory can be explicitely specified by setting\nit in the [serialization\ncontext](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context)\nor by setting the `TB_ARTIFACT_PATH` environment variable (see below.). The\ncode for loading the JSON file does not change:\n\n```py\nobj = tb.load_json(\"foo/bar/foobar.json\")\n```\n\nIf using\n[`turbo_broccoli.to_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#to_json),\nsince there is no output file path specified, the artifacts are storied in a\ntemporary directory instead:\n\n```py\nimport numpy as np\nimport turbo_broccoli as tb\n\nobj = {\"an_array\": np.random.rand(1000, 1000)}\ndoc = tb.to_json(obj)\n# An artifact has been created somewhere in e.g. /tmp\n```\n\nSince no information about this directory is stored in the output JSON string,\nit is not possible to load `doc` using\n[`turbo_broccoli.from_json`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/turbo_broccoli.html#load_json).\nIf deserialization is necessary, instantiate a [context](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context):\n\n```py\nimport numpy as np\nimport turbo_broccoli as tb\n\nctx = tb.Context()\nobj = {\"an_array\": np.random.rand(1000, 1000)}\ndoc = tb.to_json(obj, ctx)\n# An artifact has been created in ctx.artifact_path\n\n...\n\nobj = tb.from_json(doc, ctx)\n```\n\n## Supported types\n\n### Basic types\n\n- [`bytes`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/bytes.html#to_json)\n\n- [Collections](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/collections.html#to_json):\n  `collections.deque`, `collections.namedtuple`\n\n- [Dataclasses](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/dataclass.html#to_json):\n  serialization is straightforward:\n\n  ```py\n  @dataclass\n  class C:\n      a: int\n      b: str\n\n  doc = tb.to_json({\"c\": C(a=1, b=\"Hello\")})\n  ```\n\n  For deserialization, first register the class:\n\n  ```py\n  ctx = tb.Context(dataclass_types=[C])\n  tb.from_json(doc, ctx)\n  ```\n\n- [`datetime.datetime`, `datetime.time`,\n  `datetime.timedelta`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/datetime.html#to_json)\n\n- Non JSON-able dicts, i.e. dicts whose keys are not all `str`, `int`, `float`,\n  `bool` or `None`\n\n- [UUID objects](https://docs.python.org/3/library/uuid.html)\n\n- [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)\n\n### Generic objects\n\n**Serialization only**. A generic object is an object that\nhas the `__turbo_broccoli__` attribute. This attribute is expected to be a list\nof attributes whose values will be serialized. For example,\n\n```py\nclass C:\n    __turbo_broccoli__ = [\"a\", \"b\"]\n    a: int\n    b: int\n    c: int\n\nx = C()\nx.a, x.b, x.c = 42, 43, 44\ntb.to_json(x)\n```\n\nproduces the following string:\n\n```json\n{\"a\": 42,\"b\": 43}\n```\n\nRegistered attributes can of course have any type supported by TurboBroccoli,\nsuch as numpy arrays. Registered attributes can be `@property` methods.\n\n### [Keras](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/keras.html#to_json)\n\n- [`keras.Model`](https://keras.io/api/models/model/);\n\n- standard subclasses of [`keras.layers.Layer`](https://keras.io/api/layers/),\n  [`keras.losses.Loss`](https://keras.io/api/losses/),\n  [`keras.metrics.Metric`](https://keras.io/api/metrics/), and\n  [`keras.optimizers.Optimizer`](https://keras.io/api/optimizers/).\n\n### [Numpy](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/numpy.html#to_json)\n\n`numpy.number`, `numpy.ndarray` with numerical dtype, and `numpy.dtype`.\n\n### [Pandas](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/pandas.html#to_json)\n\n`pandas.DataFrame` and `pandas.Series`, but with the following limitations:\n\n- the following dtypes are not supported: `complex`, `object`, `timedelta`;\n\n- the column / series names cannot be ints or int-strings: the following are\n  not acceptable\n\n  ```py\n  df = pd.DataFrame([[1, 2], [3, 4]])\n  df = pd.DataFrame([[1, 2], [3, 4]], columns=[\"0\", \"1\"])\n  ```\n\n### [Tensorflow](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/tensorflow.html#to_json)\n\n`tensorflow.Tensor` with numerical dtype, but not `tensorflow.RaggedTensor`.\n\n### [Pytorch](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/pytorch.html#to_json)\n\n- `torch.Tensor`, **Warning**: loaded tensors are automatically placed on the\n  CPU and gradients are lost;\n\n- `torch.nn.Module`, don't forget to register your module type using a\n  [`turbo_broccoli.Context`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context):\n\n  ```py\n  # Serialization\n  class MyModule(torch.nn.Module):\n    ...\n\n  module = MyModule()  # Must be instantiable without arguments\n  doc = tb.to_json({\"module\": module})\n\n  # Deserialization\n  ctx = tb.Context(pytorch_module_types=[MyModule])\n  module = tb.from_json(doc, ctx)\n  ```\n\n  **Warning**: It is not possible to register and deserialize [standard pytorch\n  module containers](https://pytorch.org/docs/stable/nn.html#containers)\n  directly. Wrap them in your own custom module class. For following is not\n  acceptable\n\n  ```py\n  import turbo_broccoli as tb\n  import torch\n\n  module = torch.nn.Sequential(\n      torch.nn.Linear(4, 2),\n      torch.nn.ReLU(),\n      torch.nn.Linear(2, 1),\n      torch.nn.ReLU(),\n  )\n  obj = {\"module\": module}\n  doc = tb.to_json(obj)  # works, but...\n  tb.from_json(a, ctx)  # does't work\n  ```\n\n  but the following works:\n\n  ```py\n  class MyModule(torch.nn.Module):\n    module: torch.nn.Sequential  # Wrapped sequential\n\n    def __init__(self):\n        super().__init__()\n        self.module = torch.nn.Sequential(\n            torch.nn.Linear(4, 2),\n            torch.nn.ReLU(),\n            torch.nn.Linear(2, 1),\n            torch.nn.ReLU(),\n        )\n\n    ...\n\n  module = MyModule()  # Must be instantiable without arguments\n  doc = tb.to_json({\"module\": module})\n\n  ctx = tb.Context(pytorch_module_types=[MyModule])\n  module = tb.from_json(doc, ctx)\n  ```\n\n  To circumvent all these limitations, use custom encoders / decoders.\n\n- `torch.utils.data.ConcatDataset`, `torch.utils.data.StackDataset`,\n  `torch.utils.data.Subset`, `torch.utils.data.TensorDataset`, as long as the\n  nested structure of datasets ultimately lead to\n  `torch.utils.data.TensorDataset`s (e.g. a subset of a stack of subsets of\n  tensor datasets is supported)\n\n### [Scipy](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/scipy.html#to_json)\n\nJust `scipy.sparse.csr_matrix`. ^^\"\n\n### [Scikit-learn](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/sklearn.html#to_json)\n\n`sklearn` estimators (i.e. that inherit from\n[`sklean.base.BaseEstimator`](https://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html)).\nSupported estimators are: `AdaBoostClassifier`, `AdaBoostRegressor`,\n`AdditiveChi2Sampler`, `AffinityPropagation`, `AgglomerativeClustering`,\n`ARDRegression`, `BayesianGaussianMixture`, `BayesianRidge`, `BernoulliNB`,\n`BernoulliRBM`, `Binarizer`, `CategoricalNB`, `CCA`, `ClassifierChain`,\n`ComplementNB`, `DBSCAN`, `DecisionTreeClassifier`, `DecisionTreeRegressor`,\n`DictionaryLearning`, `ElasticNet`, `EllipticEnvelope`, `EmpiricalCovariance`,\n`ExtraTreeClassifier`, `ExtraTreeRegressor`, `ExtraTreesClassifier`,\n`ExtraTreesRegressor`, `FactorAnalysis`, `FeatureUnion`, `GaussianMixture`,\n`GaussianNB`, `GaussianRandomProjection`, `GraphicalLasso`, `HuberRegressor`,\n`IncrementalPCA`, `IsolationForest`, `Isomap`, `KernelCenterer`,\n`KernelDensity`, `KernelPCA`, `KernelRidge`, `KMeans`, `KNeighborsClassifier`,\n`KNeighborsRegressor`, `KNNImputer`, `LabelBinarizer`, `LabelEncoder`,\n`LabelPropagation`, `LabelSpreading`, `Lars`, `Lasso`, `LassoLars`,\n`LassoLarsIC`, `LatentDirichletAllocation`, `LedoitWolf`,\n`LinearDiscriminantAnalysis`, `LinearRegression`, `LinearSVC`, `LinearSVR`,\n`LocallyLinearEmbedding`, `LocalOutlierFactor`, `LogisticRegression`,\n`MaxAbsScaler`, `MDS`, `MeanShift`, `MinCovDet`, `MiniBatchDictionaryLearning`,\n`MiniBatchKMeans`, `MiniBatchSparsePCA`, `MinMaxScaler`, `MissingIndicator`,\n`MLPClassifier`, `MLPRegressor`, `MultiLabelBinarizer`, `MultinomialNB`,\n`MultiOutputClassifier`, `MultiOutputRegressor`, `MultiTaskElasticNet`,\n`MultiTaskLasso`, `NearestCentroid`, `NearestNeighbors`,\n`NeighborhoodComponentsAnalysis`, `NMF`, `Normalizer`, `NuSVC`, `NuSVR`,\n`Nystroem`, `OAS`, `OneClassSVM`, `OneVsOneClassifier`, `OneVsRestClassifier`,\n`OPTICS`, `OrthogonalMatchingPursuit`, `PassiveAggressiveRegressor`, `PCA`,\n`Pipeline`, `PLSCanonical`, `PLSRegression`, `PLSSVD`, `PolynomialCountSketch`,\n`PolynomialFeatures`, `PowerTransformer`, `QuadraticDiscriminantAnalysis`,\n`QuantileRegressor`, `QuantileTransformer`, `RadiusNeighborsClassifier`,\n`RadiusNeighborsRegressor`, `RandomForestClassifier`, `RandomForestRegressor`,\n`RANSACRegressor`, `RBFSampler`, `RegressorChain`, `RFE`, `RFECV`, `Ridge`,\n`RidgeClassifier`, `RobustScaler`, `SelectFromModel`, `SelfTrainingClassifier`,\n`SGDRegressor`, `ShrunkCovariance`, `SimpleImputer`, `SkewedChi2Sampler`,\n`SparsePCA`, `SparseRandomProjection`, `SpectralBiclustering`,\n`SpectralClustering`, `SpectralCoclustering`, `SpectralEmbedding`,\n`StackingClassifier`, `StackingRegressor`, `StandardScaler`, `SVC`, `SVC`,\n`SVR`, `SVR`, `TheilSenRegressor`, `TruncatedSVD`, `TSNE`, `VarianceThreshold`,\n`VotingClassifier`, `VotingRegressor`.\n\nDoesn't work with:\n\n- All CV classes because the `score_` attribute is a dict indexed with\n  `np.int64`, which `json.JSONEncoder._iterencode_dict` rejects.\n\n- Everything that is parametrized by an arbitrary object/callable/estimator:\n  `FunctionTransformer`, `TransformedTargetRegressor`.\n\n- Other classes that have non JSON-serializable attributes:\n\n    | Class                       | Non-serializable attr.    |\n    | --------------------------- | ------------------------- |\n    | `Birch`                     | `_CFNode`                 |\n    | `BisectingKMeans`           | `function`                |\n    | `ColumnTransformer`         | `slice`                   |\n    | `GammaRegressor`            | `HalfGammaLoss`           |\n    | `GaussianProcessClassifier` | `Product`                 |\n    | `GaussianProcessRegressor`  | `Sum`                     |\n    | `IsotonicRegression`        | `interp1d`                |\n    | `OutputCodeClassifier`      | `_ConstantPredictor`      |\n    | `Perceptron`                | `Hinge`                   |\n    | `PoissonRegressor`          | `HalfPoissonLoss`         |\n    | `SGDClassifier`             | `Hinge`                   |\n    | `SGDOneClassSVM`            | `Hinge`                   |\n    | `SplineTransformer`         | `BSpline`                 |\n    | `TweedieRegressor`          | `HalfTweedieLossIdentity` |\n\n- Other errors:\n\n  - `FastICA`: I'm not sure why...\n\n  - `BaggingClassifier`: `IndexError: only integers, slices (:), ellipsis\n    (...), numpy.newaxis (None) and integer or boolean arrays are valid\n    indices`.\n\n  - `GradientBoostingClassifier`, `GradientBoostingRegressor`,\n    `RandomTreesEmbedding`, `KBinsDiscretizer`: `Exception:\n    dtype object is not covered`.\n\n  - `HistGradientBoostingClassifier`: Problems with deserialization of\n    `_BinMapper` object?\n\n  - `PassiveAggressiveClassifier`: some unknown label type error...\n\n  - `SequentialFeatureSelector`: Problem with the unit test itself ^^\"\n\n  - `KNeighborsTransformer`: A serialized-deserialized instance seems to\n    `fit_transform` an array to a sparse matrix whereas the original object\n    returns an array?\n\n  - `RadiusNeighborsTransformer`: Inverse problem from `KNeighborsTransformer`.\n\n### [Bokeh](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/bokeh.html#to_json)\n\nBokeh [figures](https://docs.bokeh.org/en/latest) and\n[models](https://docs.bokeh.org/en/latest/docs/reference/models.html).\n\n### [NetworkX](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/networkx.html#to_json)\n\nAll NetworkX [graph\nobjects](https://networkx.org/documentation/stable/reference/classes/index.html#which-graph-class-should-i-use).\n\n### [Secrets](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/secret.html#to_json)\n\nBasic Python types can be wrapped in their corresponding secret type according\nto the following table\n\n| Python type | Secret type                  |\n| ----------- | ---------------------------- |\n| `dict`      | `turbo_broccoli.SecretDict`  |\n| `float`     | `turbo_broccoli.SecretFloat` |\n| `int`       | `turbo_broccoli.SecretInt`   |\n| `list`      | `turbo_broccoli.SecretList`  |\n| `str`       | `turbo_broccoli.SecretStr`   |\n\nThe secret value can be recovered with the `get_secret_value` method. At\nserialization, the this value will be encrypted. For example,\n\n```py\n## See https://pynacl.readthedocs.io/en/latest/secret/#key\nimport nacl.secret\nimport nacl.utils\nimport turbo_broccoli as tb\n\nkey = nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)\nctx = tb.Context(nacl_shared_key=key)\nobj = {\"user\": \"alice\", \"password\": tb.SecretStr(\"dolphin\")}\ntb.to_json(obj, ctx)\n```\n\nproduces the following string (modulo indentation and modulo the encrypted\ncontent):\n\n```json\n{\n  \"user\": \"alice\",\n  \"password\": {\n    \"__type__\": \"secret\",\n    \"__version__\": 2,\n    \"data\": {\n      \"__type__\": \"bytes\",\n      \"__version__\": 3,\n      \"data\": \"gbRXF3hq9Q9hIQ9Xz+WdGKYP5meJ4eTmlFt0r0Ov3PV64065plk6RqsFUcynSOqHzA==\"\n    }\n  }\n}\n```\n\nDeserialization decrypts the secrets, but they stay wrapped inside the secret\ntypes above. If the wrong key is provided, an exception is raised. If no key is\nprovided, the secret values are replaced by a\n`turbo_broccoli.LockedSecret`. Internally, TurboBroccoli uses\n[`pynacl`](https://pynacl.readthedocs.io/en/latest/)'s\n[`SecretBox`](https://pynacl.readthedocs.io/en/latest/secret/#nacl.secret.SecretBox).\n**Warning**: In the case of `SecretDict` and `SecretList`, the values contained\nwithin must be JSON-serializable **without** TurboBroccoli. The following is\nnot acceptable:\n\n```py\nimport nacl.secret\nimport nacl.utils\nimport numpy as np\nimport turbo_broccoli as tb\n\nkey = nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)\nctx = tb.Context(nacl_shared_key=key)\nobj = {\"data\": tb.SecretList([np.array([1, 2, 3])])}\ntb.to_json(obj, ctx)\n```\n\nSee also the `TB_SHARED_KEY` environment variable below.\n\n### Embedded dict/lists\n\nSometimes, it may be useful to store part of a document in its own file and\nhave referrenced in the main file. This is possible using\n[`EmbeddedDict`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/embedded.html)\nand\n[`EmbeddedList`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/embedded.html).\nFor example,\n\n```py\nfrom turbo_broccoli import save_json, EmbeddedDict\n\ndata = {\"a\": 1, \"b\": EmbeddedDict({\"c\": 2, \"d\": 3})}\nsave_json(data, \"data.json\")\n```\n\nwill result in a `data.json` file containing\n\n```json\n{\n  \"a\": 1,\n  \"b\": {\n    \"__type__\": \"embedded.dict\",\n    \"__version__\": 1,\n    \"id\": \"4ea0b3f3-f3e4-42bd-9db9-1e4e0b9f4fae\"\n  }\n}\n```\n\n(modulo indentation and the id), and an artefact file\n`data.4ea0b3f3-f3e4-42bd-9db9-1e4e0b9f4fae.json` containing\n\n```json\n{\"c\": 2, \"d\": 3}\n```\n\n###  External data\n\nIf you are serializing/deserializing from a file, you can use\n[`turbo_broccoli.ExternalData`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/custom/external.html)\nto point to data contained in another file without integrating it to the\ncurrent document.\n\nFor example, let's say you want to create `foo/bar.json` where the `a` key\npoints to the data contained in `foo/foooooo/data.np`:\n\n```py\nimport turbo_broccoli as tb\n\ndocument = {\n  \"a\": tb.ExternalData(\"foooooo/data.np\"),\n  ...\n}\n\n# data.np is loaded when creating the ExternalData object\nprint(document[\"a\"].data)\n\n# Saving\ntb.save_json(document, \"foo/bar.json\")\n\n# Loading\ndocument2 = tb.load_json(\"foo/bar.json\")\n\nfrom numpy.testing import assert_array_equal\nassert_array_equal(document[\"a\"].data, document2[\"a\"].data)\n```\n\nWarnings:\n\n- `document[\"a\"].data` is read only, the following will have no effect on\n  `foo/foooooo/data.np`:\n\n  ```py\n  document[\"a\"].data += 1\n  tb.save_json(document, \"foo/bar.json\")\n  ```\n\n- When serializing/deserializing a `ExternalData` object, an actual JSON\n  document must be involved. In particular, using `tb.to_json` or\n  `tb.from_json` is not possible.\n\n- The external data file's path must be a subpath of the output/intput JSON\n  file, and provided either relative to the output/intput JSON file, or in\n  absolute form:\n\n  ```py\n  # OK, relative\n  document = {\"a\": tb.ExternalData(\"foooooo/data.np\")}\n  tb.save_json(document, \"foo/bar.json\")\n\n  # OK, absolute\n  document = {\"a\": tb.ExternalData(\"/home/alice/foo/foooooo/data.np\")}\n  tb.save_json(document, \"foo/bar.json\")\n\n  # ERROR, not subpath\n  document = {\"a\": tb.ExternalData(\"/home/alice/data.np\")}\n  tb.save_json(document, \"/home/alice/foo/bar.json\")\n  ```\n\n## Environment variables\n\nSome behaviors of TurboBroccoli can be tweaked by setting specific environment\nvariables. If you want to modify these parameters programatically, do not do so\nby modifying `os.environ`. Rather, use a\n[`turbo_broccoli.Context`](https://altaris.github.io/turbo-broccoli/turbo_broccoli/context.html#Context).\n\n- `TB_ARTIFACT_PATH` (default: output JSON file's parent directory): During\n  serialization, TurboBroccoli may create artifacts to which the JSON object\n  will point to. The artifacts will be stored in `TB_ARTIFACT_PATH` if\n  specified.\n\n- `TB_KERAS_FORMAT` (default: `tf`, valid values are `keras`, `tf`, and `h5`):\n  The serialization format for keras models. If `h5` or `tf` is used, an\n  artifact following said format will be created in `TB_ARTIFACT_PATH`. If\n  `json` is used, the model will be contained in the JSON document (anthough\n  the weights may be in artifacts if they are too large).\n\n- `TB_MAX_NBYTES` (default: `8000`):\n  The maximum byte size of a python object beyond which serialization will\n  produce an artifact instead of storing it in the JSON document. This does not\n  limit the size of the overall JSON document though. 8000 bytes should be\n  enough for a numpy array of 1000 `float64`s to be stored in-document.\n\n- `TB_NODECODE` (default: empty):\n  Comma-separated list of types to not deserialize, for example\n  `bytes,numpy.ndarray`. Excludable types are:\n\n  - `bokeh`, `bokeh.buffer`, `bokeh.generic`,\n\n  - `bytes`, **Warning** excluding `bytes` will also exclude `bokeh`,\n    `numpy.ndarray`, `pytorch.module`, `pytorch.tensor`, `secret`,\n    `tensorflow.tensor`,\n\n  - `collections`, `collections.deque`, `collections.namedtuple`,\n    `collections.set`,\n\n  - `dataclass`, `dataclass.\u003cdataclass_name\u003e` (case sensitive),\n\n  - `datetime`, `datetime.datetime`, `datetime.time`, `datetime.timedelta`,\n\n  - `dict` (this only prevents decoding dicts with non-string keys),\n\n  - `embedded`, `embedded.dict`, `embedded.list`,\n\n  - `external`,\n\n  - `generic`,\n\n  - `keras`, `keras.model`, `keras.layer`, `keras.loss`, `keras.metric`,\n    `keras.optimizer`,\n\n  - `networkx`, `networkx.graph`,\n\n  - `numpy`, `numpy.ndarray`, `numpy.number`, `numpy.dtype`,\n    `numpy.random_state`,\n\n  - `pandas`, `pandas.dataframe`, `pandas.series`, **Warning**: excluding\n    `pandas.dataframe` will also exclude `pandas.series`,\n\n  - `pathlib`, `pathlib.path`, **Warning**: excluding `pathlib.path` will also\n    exclude `external`,\n\n  - `pytorch`, `pytorch.tensor`, `pytorch.module`, `pytorch.concatdataset`,\n    `pytorch.stackdataset`, `pytorch.subset`, `pytorch.tensordataset`\n\n  - `scipy`, `scipy.csr_matrix`,\n\n  - `secret`,\n\n  - `sklearn`, `sklearn.estimator`, `sklearn.estimator.\u003cestimator name\u003e` (case\n    sensitive, see the list of supported sklearn estimators below),\n\n  - `tensorflow`, `tensorflow.sparse_tensor`, `tensorflow.tensor`,\n    `tensorflow.variable`.\n\n  - `uuid`\n\n  - `external`\n\n- `TB_SHARED_KEY` (default: empty):\n  Secret key used to encrypt/decrypt secrets. The encryption uses [`pynacl`'s\n  `SecretBox`](https://pynacl.readthedocs.io/en/latest/secret/#nacl.secret.SecretBox).\n  An exception is raised when attempting to serialize a secret type while no\n  key is set.\n\n## Contributing\n\n### Dependencies\n\n- `python3.10` or newer;\n\n- `requirements.txt` for runtime dependencies;\n\n- `requirements.dev.txt` for development dependencies.\n\n- `make` (optional);\n\nSimply run\n\n```sh\nvirtualenv venv -p python3.10\n. ./venv/bin/activate\npip install --upgrade pip\npip install -r requirements.txt\npip install -r requirements.dev.txt\n```\n\n### Documentation\n\nSimply run\n\n```sh\nmake docs\n```\n\nThis will generate the HTML doc of the project, and the index file should be at\n`docs/index.html`. To have it directly in your browser, run\n\n```sh\nmake docs-browser\n```\n\n### Code quality\n\nDon't forget to run\n\n```sh\nmake\n```\n\nto format the code following [black](https://pypi.org/project/black/),\ntypecheck it using [mypy](http://mypy-lang.org/), and check it against coding\nstandards using [pylint](https://pylint.org/).\n\n### Unit tests\n\nRun\n\n```sh\nmake test\n```\n\nto have [pytest](https://docs.pytest.org/) run the unit tests in `tests/`.\n\n## Credits\n\nThis project takes inspiration from\n[Crimson-Crow/json-numpy](https://github.com/Crimson-Crow/json-numpy).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltaris%2Fturbo-broccoli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltaris%2Fturbo-broccoli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltaris%2Fturbo-broccoli/lists"}