{"id":22690005,"url":"https://github.com/yahoojapan/yosegi","last_synced_at":"2025-04-06T02:09:56.659Z","repository":{"id":34584212,"uuid":"165814788","full_name":"yahoojapan/yosegi","owner":"yahoojapan","description":"Yosegi is a Schema-less columnar storage format. Provide flexible representation like JSON and efficient reading similar to other columnar storage formats.","archived":false,"fork":false,"pushed_at":"2025-01-30T08:24:13.000Z","size":2064,"stargazers_count":71,"open_issues_count":2,"forks_count":11,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-30T01:09:49.816Z","etag":null,"topics":["columnar-format","file-format","schema-less","storage-forma","yosegi"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yahoojapan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-15T08:36:44.000Z","updated_at":"2025-02-18T05:24:47.000Z","dependencies_parsed_at":"2024-06-19T09:28:46.014Z","dependency_job_id":"48d6de2c-5e0f-4a22-ab3e-e62aab5f4ed6","html_url":"https://github.com/yahoojapan/yosegi","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoojapan%2Fyosegi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoojapan%2Fyosegi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoojapan%2Fyosegi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoojapan%2Fyosegi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yahoojapan","download_url":"https://codeload.github.com/yahoojapan/yosegi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247423515,"owners_count":20936626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["columnar-format","file-format","schema-less","storage-forma","yosegi"],"created_at":"2024-12-10T00:25:04.079Z","updated_at":"2025-04-06T02:09:56.633Z","avatar_url":"https://github.com/yahoojapan.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"\u003c!---\n  Licensed under the Apache License, Version 2.0 (the \"License\");\n  you may not use this file except in compliance with the License.\n  You may obtain a copy of the License at\n\n   http://www.apache.org/licenses/LICENSE-2.0\n\n  Unless required by applicable law or agreed to in writing, software\n  distributed under the License is distributed on an \"AS IS\" BASIS,\n  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n  See the License for the specific language governing permissions and\n  limitations under the License. See accompanying LICENSE file.\n--\u003e\n# Introduction to Yosegi\n## What does this project do?\nYosegi is a Schema-less columnar storage format.\nProvide flexible representation like JSON and efficient reading\nsimilar to other columnar storage formats.\n\n\n## Why is this project useful?\nThere was a problem that it is too large to compress\nand save the data as it is in the Big Data era.\nFrom the demand for improvement in compression ratio and read performance,\nseveral columnar data formats (for example, Apache ORC and Apache Parquet)\nwere proposed.\nThey achieve the high compression ratio from similar data in column\nand reading performance for grouping data by column when data is used.\n\nHowever, these data formats are required\nthe data structure in a row (or a record) should be defined\nbefore saving the data.\nIt was necessary to decide how to use it at the time of data storage,\nand it was often a problem that it was difficult to decide\nwhat kind of data to use.\n\nIn this project, we provide a new columnar format\nwhich does not require the schema at the time of data storage\nwith compression and read performance equal to (or higher in case)\nthan other formats.\n\n## Use cases\n### Data Analysis\nAnalyzing big data requires store data compactly and get data smoothly.\nYosegi as a columnar format is useful for this needs.\n\n### Data Lake\nData Lake is a data pool that is not required the data structure\n(as a schema) in the row at the time of data storage.\nAnd stored data can be used with defining its schema at the time of analyzing.\nSee [DataLake](https://en.wikipedia.org/wiki/Data_lake).\n\n# License\nThis project is on the [Apache License](https://www.apache.org/licenses/LICENSE-2.0).\nPlease treat this project under this license.\n\n# How do I get started?\n\n## Java\nFor easy usage please see the [quick start](docs/java/quickstart.md).\n\n## CLI\nPlease see the repository of [yosegi-tools](https://github.com/yahoojapan/yosegi-tools) for details.\n\nIf you want to know what kind of function it has, look at the [command list](https://github.com/yahoojapan/yosegi-tools/blob/master/docs/command_list.md).\n\n## Apache Hadoop\nYosegi supports Apache Hadoop.\nPlease see the repository of [yosegi-hadoop](https://github.com/yahoojapan/yosegi-hadoop) for details.\n\nFor easy usage please see [quick start](https://github.com/yahoojapan/yosegi-hadoop/blob/master/docs/quickstart.md).\n\n## Apache Hive\nYosegi supports Apache Hive.\nPlease see the repository of [yosegi-hive](https://github.com/yahoojapan/yosegi-hive) for details.\n\nFor easy usage please see [quick start](https://github.com/yahoojapan/yosegi-hive/blob/master/docs/quickstart.md).\n\n## Apache Spark\nYosegi supports Apache Spark.\nPlease see the repository of [yosegi-spark](https://github.com/yahoojapan/yosegi-spark) for details.\n\nFor easy usage please see [quick start](https://github.com/yahoojapan/yosegi-spark/blob/master/docs/quickstart.md).\n\n## Where can I get more help, if I need it?\nSupport and discussion of Yosegi are on the Mailing list.\n\n* Mailing list: yosegi@googlegroups.com\n* Bug trackter:[JIRA](https://yosegi.atlassian.net/projects/YOSEGI)\n\nWe plan to support and discussion of Yosegi on the Mailing list.\nHowever, please contact us via GitHub until ML is opened.\n\n# How to contribute\nWe welcome to join this project widely.\n\nFor information on how to start contributing to the project, please refer to the [Yosegi contribution guide](CONTRIBUTING.md).\n\n# Building\n\n## System requirement\nFollowing environments are required.\n\n* Mac OS X or Linux\n* Java 8 Update 92 or higher (8u92+), 64-bit\n* Maven 3.3.9 or later (for building)\n\n## Maven\nYosegi sources can get from the Maven repository.\n\n## Compile sources\nCompile each source following instructions.\n\n    $ mvn clean install\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoojapan%2Fyosegi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyahoojapan%2Fyosegi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoojapan%2Fyosegi/lists"}