{"id":13480880,"url":"https://github.com/mongoeye/mongoeye","last_synced_at":"2025-03-27T11:31:03.266Z","repository":{"id":21214997,"uuid":"87823770","full_name":"mongoeye/mongoeye","owner":"mongoeye","description":"Schema and data analyzer for MongoDB written in Go.","archived":false,"fork":false,"pushed_at":"2022-05-16T17:39:13.000Z","size":39673,"stargazers_count":167,"open_issues_count":2,"forks_count":8,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-17T00:34:56.976Z","etag":null,"topics":["aggregation","analysis","histogram","mongodb","nosql","schema","statistics","visualization"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongoeye.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-10T14:57:17.000Z","updated_at":"2025-02-26T18:19:00.000Z","dependencies_parsed_at":"2022-08-09T09:40:23.252Z","dependency_job_id":null,"html_url":"https://github.com/mongoeye/mongoeye","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongoeye%2Fmongoeye","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongoeye%2Fmongoeye/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongoeye%2Fmongoeye/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongoeye%2Fmongoeye/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongoeye","download_url":"https://codeload.github.com/mongoeye/mongoeye/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245836037,"owners_count":20680308,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregation","analysis","histogram","mongodb","nosql","schema","statistics","visualization"],"created_at":"2024-07-31T17:00:46.192Z","updated_at":"2025-03-27T11:31:02.981Z","avatar_url":"https://github.com/mongoeye.png","language":"Go","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=JEMPF6RQJP7XA"],"categories":["Go","Tools"],"sub_categories":["Development"],"readme":"\u003ca href=\"https://raw.githubusercontent.com/mongoeye/mongoeye/master/_misc/logo_name_small.png?v1\" title=\"logo\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/mongoeye/mongoeye/master/_misc/logo_name_small.png?v1\" width=\"300\"/\u003e\u003c/a\u003e\n\n\nSchema and data analyzer for [MongoDB](https://www.mongodb.com) written in [Go](https://golang.org).\n\n[![GoDoc](https://godoc.org/github.com/golang/gddo?status.svg)](https://godoc.org/github.com/mongoeye/mongoeye)\n[![Coverage Status](https://coveralls.io/repos/github/mongoeye/mongoeye/badge.svg?branch=master)](https://coveralls.io/github/mongoeye/mongoeye?branch=master)\n[![Build Status](https://travis-ci.com/mongoeye/mongoeye.svg?branch=master)](https://travis-ci.com/mongoeye/mongoeye)\n[![Go Report Card](https://goreportcard.com/badge/github.com/mongoeye/mongoeye)](https://goreportcard.com/report/github.com/mongoeye/mongoeye)\n\n## Overview\n\nMongoeye provides a quick overview of the data in your MongoDB database.\n\n### Key features\n\n* *Fast:*\u0026nbsp; [the fastest](https://github.com/mongoeye/mongoeye/blob/master/_misc/comparison.png) schema analyzer for MongoDB\n* *Single binary:*\u0026nbsp; pre-built [binaries](https://github.com/mongoeye/mongoeye/releases) for Windows, Linux, and MacOS (Darwin)\n* *Local analysis:*\u0026nbsp; quick local analysis using a parallel algorithm (MongoDB 2.0+)\n* *Remote analysis:*\u0026nbsp; distributed analysis in database using the aggregation framework (MongoDB 3.5.10+)\n* *Rich features:*\u0026nbsp; [histogram](#value-histogram) (value, length, weekday, hour), [most frequent values](#frequency-of-values), ... \n* *Integrable:*\u0026nbsp; [table](#table-output), [JSON or YAML output](#json-and-yaml-output)\n\n## Demo\n\n\u003ca href=\"https://asciinema.org/a/149444\" target=\"_blank\" title=\"Open in asciinema.org\"\u003e\n\u003cimg src=\"https://github.com/mongoeye/mongoeye/blob/master/_misc/demo.gif?raw=true\"\u003e\n\u003c/a\u003e\n\n## Table of Contents\n * [Installation](#installation)\n * [Compilation](#compilation)\n * [Usage](#usage)\n    * [Table output](#table-output)\n    * [JSON and YAML output](#json-and-yaml-output)\n * [Features](#features)\n    * [Value - min, max, avg](#value---min-max-avg)\n    * [Length - min, max, avg](#length---min-max-avg)\n    * [Number of unique values](#number-of-unique-values)\n    * [Frequency of values](#frequency-of-values)\n    * [Value histogram](#value-histogram)\n    * [Length histogram](#length-histogram)\n    * [Weekday histogram](#weekday-histogram)\n    * [Hour histogram](#hour-histogram)\n * [Scope of analysis](#scope-of-analysis)\n * [List of flags and options](#list-of-flags-and-options)\n * [License](#license)\n\n## Installation\n\nMongoeye is one executable binary file. \n\nYou can download the archive from [GitHub releases page](https://github.com/mongoeye/mongoeye/releases) and extract the binary file for your platform.\n\n## Compilation\n\nIt is required to have [Go 1.8](https://golang.org). All external dependencies are part of the repository in the [vendor](https://github.com/mongoeye/mongoeye/tree/master/vendor) directory.\n\nCompilation process:\n```\n$ go get github.com/mongoeye/mongoeye\n$ cd $GOPATH/src/github.com/mongoeye/mongoeye\n$ make build\n```\n \nFor development, you need additional dependencies that can be installed using `make get-deps`.\n\nThe test architecture uses the [Docker](https://www.docker.com) to create the testing MongoDB database.\n\nIf you want to contribute to this project, see the actions in [Makefile](https://github.com/mongoeye/mongoeye/blob/master/Makefile) and the [_contrib](https://github.com/mongoeye/mongoeye/tree/master/_contrib) directory.\n\n## Usage\n\n```\nmongoeye [host] database collection [flags]\n```\n\nThe command `mongoeye --help` lists all available options.\n\n### Table output\n\nDefault output format is table. It shows only schema without other analyzes.\n\nExample table output:\n```\n            KEY            │ COUNT  │   %    \n────────────────────────────────────────────\n  all documents            │ 2548   │        \n  analyzed documents       │ 1000   │  39.2  \n                           │        │        \n  _id - objectId           │ 1000   │ 100.0  \n  address                  │ 1000   │ 100.0  \n  │ - int                  │    1   │   0.1  \n  └╴- string               │  999   │  99.9  \n  address line 2 - string  │ 1000   │ 100.0  \n  name - string            │ 1000   │ 100.0  \n  outcode - string         │ 1000   │ 100.0  \n  postcode - string        │ 1000   │ 100.0  \n  rating                   │ 1000   │ 100.0  \n  │ - int                  │  523   │  52.3  \n  │ - double               │  451   │  45.1  \n  └╴- string               │   26   │   2.6  \n  type_of_food - string    │ 1000   │ 100.0  \n  URL - string             │ 1000   │ 100.0  \n\nOK  0.190s (local analysis)\n    1000/2548 docs (39.2%)\n    9 fields, depth 2\n```\n\n### JSON and YAML output\n\nUse `--format json` or `--format yaml` flags to set these formats.\n\nFor output to a file use the option `-F /path/to/file`.\n\n## Features\n\nThis chapter explains the features of Mongoeye and their various outputs.\n\nUse `--format json` or `--format yaml` to get detailed results, otherwise only the schema table will appear.\n\nThe output of the analysis always contains these basic keys:\n* **database**: database name\n* **collection**: collection name\n* **plan**: `local` for local analysis, `db` for analysis using aggregation framework\n* **duration**: duration of analysis\n* **allDocs**: number of all documents in collection\n* **analyzedDocs**: number of analyzed documents from collection\n* **fieldsCount**: number of found fields\n* **fields**:  result of the analysis for each field\n   * **name**: name of field\n   * **level**: level of nested field, `0` is root level\n   * **count**: number of occurrences\n   * **types**: result of the analysis for each type of field\n      * **type**: name of type\n      * **count**: number of occurrences of type\n\n**Example result:**\n```yaml\ndatabase: company\ncollection: users\nplan: local\nduration: 46.515331ms\nallDocs: 2548\nanalyzedDocs: 1000\nfieldsCount: 9\nfields:\n  - name: rating\n    level: 0\n    count: 1000\n    types:\n    - type: int\n      count: 549\n      \u003c other outputs according to settings \u003e\n```\n\n### Value - min, max, avg\n\nUse the flag `--value` or `-v` to enable calculation of minimum, maximum, and average values.\n\n**Supported types**:\n* Minimum and maximum: `objectId`, `double`, `string`, `bool`, `date`, `int`, `timestamp`, `long`, `decimal`\n* Average: `double`, `bool`, `int`, `long`, `decimal`\n\n**Example result:**\n```yaml\nvalue:\n  min: 11.565586\n  max: 60.206787\n  avg: 38.51128\n```\n\n### Length - min, max, avg\n\nUse the flag `--length` or `-l` to enable calculation of minimum, maximum, and average lengths.\n\n**Supported types**: `string`, `array`, `object`\n\n**Example result:**\n```yaml\nlength:\n  min: 29\n  max: 153\n  avg: 112\n```\n\n### Number of unique values\n\nUse the flag `--count-unique` to count all unique values.\n\n**Supported types**: `double`, `string`, `date`, `int`, `timestamp`, `long`, `decimal`\n\n**Example result:**\n```yaml\nunique: 894\n```\n\n### Frequency of values\n\nUse the flag `--most-freq N` or `--least-freq N` to get the most or least occurring values.\n\n**Supported types**: `double`, `string`, `date`, `int`, `timestamp`, `long`, `decimal`\n\n**Example result:**\n```yaml\nmostFrequent:\n- value: USD\n  count: 599\n- value: EUR\n  count: 21\n- value: GBP\n  count: 5\n- value: CAD\n  count: 4\nleastFrequent:\n- value: EUR\n  count: 21\n- value: GBP\n  count: 5\n- value: CAD\n  count: 4\n- value: JPY\n  count: 3\n```\n\n### Value histogram\n\nUse the flag `--value-hist` or `-V` to generate value histogram.\n\n**Supported types**: `objectId` *- processed as a date*, `double`, `date`, `int`, `long`, `decimal`\n\n#### Calculation of step\n\nFlag `--value-hist-steps` sets the maximum number of steps (default `100`).\n\n* Step of the `int` and `long` type is a whole number\n* Step of the `double` and `decimal` type is: \n   * the smallest possible multiplication of [`1`, `5` or `2.5`] and `10^n` so the max. number of steps is kept\n   * eg. ..., `100`, `50`, `25`, `10`, `5`, `2.5`, `1`, `0.5`, `0.25`, `0.1`, ... \n* Step of the `date` and `objectId` type is rounded to:\n  * 1, 2, 5, 10, 15, 30 `seconds`\n  * 1, 2, 5, 10, 15, 30 `minutes`\n  * 1, 2, 3, 6, 12 `hours`\n  * 1, 2, 3, 4, ... `days`\n\n**Example result:**\n```yaml\nvalueHistogram:\n  start: 2.5\n  end: 12\n  range: 9.5\n  step: 0.5\n  numOfSteps: 19\n  intervals: [36, 25, 14, 81, 95, 86, 59, 6, 82, 84, 62, 33, 19, 9, 1, 14, 67, 2, 45]\n```\n\n**Graphic representation:**\n\u003cdiv style=\"text-align:left\"\u003e\n\u003cimg src=\"https://github.com/mongoeye/mongoeye/blob/master/_misc/histogram.png?raw=true\"\u003e\n\u003c/div\u003e\n\n\n### Length histogram\n\nUse the flag `--length-hist` or `-L` to generate length histogram.\n\nFlag `--length-hist-steps` sets the maximum number of steps (default `100`).\n\n**Supported types**: `string`, `array`, `object`\n\n**Example result:**\n```yaml\nlengthHistogram:\n  start: 0\n  end: 300\n  range: 300\n  step: 50\n  numOfSteps: 6\n  intervals: [96, 78, 3, 1, 1, 0]\n```\n\n### Weekday histogram\n\nUse the flag `--weekday-hist` or `-W` to generate weekday histogram.\n\nTo determine the day of week it uses the time zone from the `--timezone` flag (default `local`).\n\nFirst day is `Sunday`.\n\n**Example result:**\n```yaml\nweekdayHistogram: [5, 48, 23, 124, 45, 15, 87]\n```\n\n### Hour histogram\n\nUse the flag `--hour-hist` or `-H` to generate weekday histogram.\n\nTo determine the hour it uses the time zone from the `--timezone` flag (default `local`).\n\nFirst value is for interval `[ 00, 01 )`, last for interval `[ 23, 24 )`.\n\n**Example result:**\n```yaml\nhourHistogram: [47, 73, 18, 26, 30, 46, 91, 13, 28, 11, 52, 99, 76, 25, 94, 51, 87, 86, 19, 22, 11, 62, 28, 47]\n```\n\n## Scope of analysis\n\n*The scope of analysis is defined by the following options.*\n\nThe **`--match`** option is applied as the first:\n  - it selects documents for the analysis using [$match aggregation](https://docs.mongodb.com/manual/reference/operator/aggregation/match/)\n  - value is a string in JSON format\n  - suitable for include/exclude documents from analysis\n  - by default, all documents are included (if the argument is not present)\n\nThe **`--sample`** option is applied as the second:\n  - determines the sampling method using [$sort](https://docs.mongodb.com/manual/reference/operator/aggregation/sort/), [$limit](https://docs.mongodb.com/manual/reference/operator/aggregation/limit/) and [$sample](https://docs.mongodb.com/manual/reference/operator/aggregation/sample/) aggregations\n  - valid values are: `all`, `first:N`, `last:N`, `random:N`, where `N \u003e 1`\n  - default value is `random:1000`\n\nThe **`--project`** option is applied as the third:\n  - before the analysis it modifies document using [$project aggregation](https://docs.mongodb.com/manual/reference/operator/aggregation/project/)\n  - value is a string in JSON format\n  - suitable for include/exclude fields from analysis\n  - default is not applied (if the argument is not present)\n  \n***Note:** Be sure to escape JSON options correctly, eg. `--project \"{\\\"Field\\\": 0}\"`.*\n\n## List of flags and options\n\n#### Connection options\n```\n--host                    mongodb host (default \"localhost:27017\")\n--connection-mode         connection mode (default \"SecondaryPreferred\")\n--connection-timeout      connection timeout (default 5)\n--socket-timeout          socket timeout (default 300)\n--sync-timeout            sync timeout (default 300)\n```\n\n#### Authentication\n```\n-u, --user                username for authentication (default \"admin\")\n-p, --password            password for authentication\n    --auth-db             auth database (default: same as the working db)\n    --auth-mech           auth mechanism\n```\n\n#### Input options\n```\n    --db                  database for analysis\n    --col                 collection for analysis\n    --match               filter documents before analysis (json, $match aggregation)\n-s, --sample              all, first:N, last:N, random:N (default \"random:1000\")\n    --project             filter/project fields before analysis (json, $project aggregation)\n-d, --depth               max depth in nested documents (default 2)\n```\n\n#### Output options\n```\n    --full                all available analyzes\n-v, --value               get min, max, avg value\n-l, --length              get min, max, avg length\n-V, --value-hist          get value histogram\n    --value-hist-steps    max steps of value histogram \u003e=3 (default 100)\n-L, --length-hist         get length histogram\n    --length-hist-steps   max steps of length histogram \u003e=3 (default 100)\n-W, --weekday-hist        get weekday histogram for dates\n-H, --hour-hist           get hour histogram for dates\n    --count-unique        get count of unique values\n    --most-freq           get the N most frequent values\n    --least-freq          get the N least frequent values\n-f, --format              output format: table, json, yaml (default \"table\")\n-F, --file                path to the output file\n```\n\n#### Other options\n```\n-t, --timezone            timezone, eg. UTC, Europe/Berlin (default \"local\")\n    --use-aggregation     analyze with aggregation framework (mongodb 3.5.10+)\n    --string-max-length   max string length (default 100)\n    --array-max-length    analyze only first N array elements (default 20)\n    --concurrency         number of local processes (default 0 = auto)\n    --buffer              size of the buffer between local stages (default 5000)\n    --batch               size of batch from database (default 500)\n    --no-color            disable color output\n    --version             show version\n-h, --help                show this help\n```\n\n### Environment variables\n\nEnvironment variables can also be used for configuration. \n\nThe names of the environment variables have the `MONGOEYE_` prefix and match the flags.\n\nInstead of the `--count-unique` flag, for example, you can use `export MONGOEYE_COUNT-UNIQUE=true`.\n\n## TODO\n\n* Create a shared library for integration into other languages (Python, Node.js, ...)\n* Selection of fields for analysis (include and exclude list)\n* TLS/SSL support\n* Create a [web interface](https://github.com/mongoeye/mongoeye-ui).\n\n## Donation\n\nIf is this tool useful to you, so feel free to support its further development.\n\n[![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=JEMPF6RQJP7XA)\n\n\n## License\n\nMongoeye is under the GPL-3.0 license. See the [LICENSE](LICENSE.md) file for details.\n\n\u003csub title=\"Ad maiorem Dei gloriam. To the greater glory of God.\"\u003e\u003csub\u003e\nAMDG\n\u003c/sub\u003e\u003c/sub\u003e\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongoeye%2Fmongoeye","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongoeye%2Fmongoeye","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongoeye%2Fmongoeye/lists"}