{"id":21739514,"url":"https://github.com/kodemartin/json-feature-matrix-processor","last_synced_at":"2026-05-10T05:42:58.240Z","repository":{"id":137906629,"uuid":"203593892","full_name":"kodemartin/json-feature-matrix-processor","owner":"kodemartin","description":"Process a heterogeneous feature matrix in JSON format through pandas","archived":false,"fork":false,"pushed_at":"2019-09-10T13:46:24.000Z","size":211,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-25T21:26:29.733Z","etag":null,"topics":["data-science","dimensionality-reduction","json"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kodemartin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-21T13:48:19.000Z","updated_at":"2019-09-10T13:46:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"7b709ba8-a32b-460d-9780-71e3eaeb1f3f","html_url":"https://github.com/kodemartin/json-feature-matrix-processor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kodemartin%2Fjson-feature-matrix-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kodemartin%2Fjson-feature-matrix-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kodemartin%2Fjson-feature-matrix-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kodemartin%2Fjson-feature-matrix-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kodemartin","download_url":"https://codeload.github.com/kodemartin/json-feature-matrix-processor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244717336,"owners_count":20498283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","dimensionality-reduction","json"],"created_at":"2024-11-26T06:09:18.352Z","updated_at":"2026-05-10T05:42:58.204Z","avatar_url":"https://github.com/kodemartin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Processing a heterogeneous feature matrix with pandas\n\nWe introduce a simple class `JsonData` that\nencapsulates methods for processing a heterogeneous feature matrix\nin JSON format with `pandas`.\n\nThe processing features can be outlined as follows:\n\n* Read data from JSON file.\n* Transform to `pandas` dataframe.\n* Save the data to `csv` format.\n* Homogenize the data.\n* Evaluate basic statistics on the data.\n* Evaluate and visualize outliers.\n* Apply dimensionality reduction on the data.\n\nIn order to facilitate parallel processing a script is provided under\n`bin/process_bulk_json_data.py` that currently wraps a simple\npipeline that converts data to `csv`.\n\n## On the data\n\nNote that the data are treated as a black box. They merely provide a use-case\nof an heterogeneous matrix, that facilitates the demonstration of the various\nutilities developed herein.\n\n\n## Setup\n\n### System requirements\n\n* `make`\n* `tkinter`\n* `python3 (\u003e=3.6)`\n\n### Prepare development environment\n\n1. Clone the repository and change directory inside.\n\n2. Create virtual environment\n\n   ```\n   $ make install\n   ```\n\n3. Activate virtual environment\n\n   ```\n   $ source venv/bin/activate\n   (venv) $\n   ```\n\n## Instantiating\n\nObjects of the `JsonData` class are instantiated by passing\nthe JSON filename to the constructor.\n\n```python\nfrom processors import JsonData\n\ndata = JsonData('data.json')\n```\n\n## Features\n\n\n### Read raw data\n\nThe raw data are read while trying to get the `JsonData.raw` property.\n\n### Transform to dataframe\n\nThe dataframe representation of the raw data is exposed through\n`JsonData.df` that caches the results of the `JsonData.to_df`\nmethod.\n\n### Save to csv\n\nThe dataframe is serialized to csv with the `JsonData.to_csv`\nmethod. To this end, multi-valued features are flattened\nthrough the `JsonData.transform_vector_features` method.\n\n### Homogenize the data\n\n`JsonData` supports dynamic transformation of multi-valued features\nof the matrix through the `transform_vector_features` instance method.\n\nBy default, the latter method reduces the vector features to their\nnorm. This is the case when `JsonData.homogeneous_df` is evaluated.\n\n### Evaluate basic statistics\n\nFeature statistics are exposed through the `JsonData.stats` property\nthat wraps the `pandas.DataFrame.describe` method.\n\n`JsonData.stats` shows the statistics of `JsonData.homogeneous_df`.\n\n### Evaluate and visualize outliers\n\nThe outliers are first evaluated explicitly through the\n`JsonData.evaluate_outliers` method.\n\nIn this context, null or infinite values in `JsonData.homogeneous_df`\nare filled with values that lie outside the outer fences of each\nfeature. This result is cached in the `JsonData.filled_df` property.\n\nThe `JsonData.outliers` property exposes the filled feature values\nof those features that actually have values outside the respective outer\nfences.\n\nFinally, the outliers for any subset of the features can be visualized\nin a boxplot through the `JsonData.plot_outliers` method.\n\n![Outliers sample](./data.png \"Outliers - overview\")\n\n### Processing a large set of files\n\nA script to perform a prescribed processing pipeline for a set of files\nin parallel is available in `bin/process_bulk_json_data.py` that can be invoked\nthrough\n\n```\n(venv) $ python -m bin.process_bulk_json_data [--dir \u003cdirectory-of-json-files\u003e] [--workers \u003cnumber-of-parallel-workers\u003e]\n```\n\nAll files to be processed should be placed in specific directory. For testing\npurposes you can generate any number of copies of the original `data.json` files\ninside a `data/` directory with\n\n```\n$ source generate_data.sh [\u003cnumber-of-copies\u003e]\n```\n\n### Dimensionality reduction\n\nFinally `JsonData` allows for the dimensionality reduction of the original\nfeature matrix through `sklearn.decomposition.PCA`. The feature matrix\nis first scaled through `sklearn.preprocessing.StandardScaler`.\n\nThe task can be performed through the `JsonData.reduce_dimensions` method,\nwhich updates `JsonData.pca` and `JsonData.scaler` to allow for future\npostprocessing (e.g. inverse transform).\n\n#### A side-note on the expected benefits\n\nDimensionality reduction effectively reduces the dimension of the feature\nspace, so that we expect:\n\n1. Better classification performance by mitigating the \"curse of dimensionality\"\n2. Time- and space-complexity improvements.\n\nThe dimension of the feature space for the example we examine (i.e. `data.json`) can be reduced\nfrom 37 to only 7:\n\n```python\n\u003e\u003e\u003e import numpy as np\n\u003e\u003e\u003e\n\u003e\u003e\u003e cumsum = np.cumsum(data.pca.explained_variance_ratio_)\n\u003e\u003e\u003e\n\u003e\u003e\u003e len(cumsum[cumsum \u003c= .96])\n7\n\u003e\u003e\u003e\n\u003e\u003e\u003e len(data.df.columns)\n37\n```\n\nThat is, seven principal components suffice to model about 95% of the variance of the original data. This implies about 80% less operations in any matrix calculation, whereas similar gains in memory usage are to be expected.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkodemartin%2Fjson-feature-matrix-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkodemartin%2Fjson-feature-matrix-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkodemartin%2Fjson-feature-matrix-processor/lists"}