Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eecavanna/mongo-diff
Command-line program that compares two MongoDB collections
https://github.com/eecavanna/mongo-diff
mongodb pymongo python typer
Last synced: about 1 month ago
JSON representation
Command-line program that compares two MongoDB collections
- Host: GitHub
- URL: https://github.com/eecavanna/mongo-diff
- Owner: eecavanna
- Created: 2024-02-01T19:40:14.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-07-13T22:10:05.000Z (4 months ago)
- Last Synced: 2024-09-30T12:41:21.958Z (about 2 months ago)
- Topics: mongodb, pymongo, python, typer
- Language: Python
- Homepage:
- Size: 82 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# mongo-diff
`mongo-diff` is a command-line tool people can use to compare two MongoDB collections.
Those collections can reside in either a single database or two separate databases (even across servers).
```mermaid
%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
%% Reference: https://github.com/pypi/warehouse/issues/13083
graph LR
script[["mongo_diff.py"]]
result["List of
differences"]
subgraph s1 \[Server]
subgraph d1 \[Database]
collection_a[("Collection A")]
end
end
subgraph s2 \[Server]
subgraph d2 \[Database]
collection_b[("Collection B")]
end
end
collection_a --> script
collection_b --> script
script --> result
```
## Usage
### Installation
Assuming you have `pipx` installed, you can install the tool by running the following command:
```shell
pipx install mongo-diff
```
> [`pipx`](https://pipx.pypa.io/stable/) is a tool people can use to
> [download and install](https://pipx.pypa.io/stable/#where-does-pipx-install-apps-from)
> Python scripts that are hosted on PyPI.
> You can [install `pipx`](https://pipx.pypa.io/stable/installation/) by running `$ python -m pip install pipx`.
### Running
You can display the tool's `--help` snippet by running:
```shell
mongo-diff --help
```
At the time of this writing, the tool's `--help` snippet is:
```console
Usage: mongo-diff [OPTIONS]
Compare two MongoDB collections.
Those collections can reside in either a single database or two separate
databases (even across servers).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --include-id --no-include-id Includes the `_id` field when comparing │
│ documents. │
│ [default: no-include-id] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Collection A ───────────────────────────────────────────────────────────────╮
│ * --mongo-uri-a TEXT Connection string for accessing │
│ the MongoDB server containing │
│ collection A. │
│ [env var: MONGO_URI_A] │
│ [required] │
│ * --database-name-a TEXT Name of the database containing │
│ collection A. │
│ [required] │
│ * --collection-name-a TEXT Name of collection A. [required] │
│ --identifier-field-name-a TEXT Name of the field of each document │
│ in collection A to use to identify │
│ a corresponding document in │
│ collection B. │
│ [default: id] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Collection B ───────────────────────────────────────────────────────────────╮
│ --mongo-uri-b TEXT Connection string for accessing the │
│ MongoDB server containing collection │
│ B (if different from that specified │
│ for collection A). │
│ [env var: MONGO_URI_B] │
│ --database-name-b TEXT Name of the database containing │
│ collection B (if different from that │
│ specified for collection A). │
│ --collection-name-b TEXT Name of collection B (if different │
│ from that specified for collection │
│ A). │
│ --identifier-field-name-b TEXT Name of the field of each document in │
│ collection B to use to identify a │
│ corresponding document in collection │
│ A (if different from that specified │
│ for collection A). │
╰──────────────────────────────────────────────────────────────────────────────╯
```
> Note: The above snippet was captured from a terminal window whose width was 80 characters.
#### MongoDB connection strings
As documented in the `--help` snippet above, you can provide the MongoDB connection strings to the tool via either (a)
command-line options; or (b) environment variables named `MONGO_URI_A` and `MONGO_URI_B`. The latter can come in handy
for MongoDB connection strings that contain passwords.
Here's how you could create those environment variables:
```shell
export MONGO_URI_A='mongodb://localhost:27017'
export MONGO_URI_B='mongodb://username:[email protected]:22222'
```
> Note: That will only create those environment variables in the current shell process. You can persist them by adding
> those same commands to your shell initialization script (e.g. `~/.bashrc`, `~/.zshrc`).
#### Example output
As the tool compares the collections, it will display the **differences** it detects; like this:
```console
Documents differ between collections: id=1,id=1. Differences: [('change', 'name', ('Joe', 'Joseph'))]
Document exists in collection A only: id=2
Document exists in collection A only: id=4
Document exists in collection B only: id=5
```
When the tool finishes comparing the collections, it will display a **summary** of the result; like this:
```console
Result
╭───────────────────────────────────────────┬──────────╮
│ Description │ Quantity │
├───────────────────────────────────────────┼──────────┤
│ Documents in collection A │ 4 │
│ Documents in collection B │ 3 │
├───────────────────────────────────────────┼──────────┤
│ Documents in collection A only │ 2 │
│ Documents in collection B only │ 1 │
├───────────────────────────────────────────┼──────────┤
│ Documents that differ between collections │ 1 │
╰───────────────────────────────────────────┴──────────╯
```
### Updating
You can update the tool to [the latest version available on PyPI](https://pypi.org/project/mongo-diff/) by running:
```shell
pipx upgrade mongo-diff
```
### Uninstallation
You can uninstall the tool from your computer by running:
```shell
pipx uninstall mongo-diff
```
## Development
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
- `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
- `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
### Clone repository
```shell
git clone https://github.com/eecavanna/mongo-diff.git
cd mongo-diff
```
### Create virtual environment
Create a Poetry virtual environment and attach to its shell:
```shell
poetry shell
```
> You can see information about the Poetry virtual environment by running: `$ poetry env info`
> You can detach from the Poetry virtual environment's shell by running: `$ exit`
From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."
### Install dependencies
At the Poetry shell, install the project's dependencies:
```shell
poetry install
```
### Make changes
Edit the tool's source code and documentation however you want.
While editing the tool's source code, you can run the tool as you normally would in order to test things out.
```shell
mongo-diff --help
```
### Build package
#### Update package version
PyPI [doesn't allow](https://pypi.org/help/#file-name-reuse) people to publish the same "version" of a package multiple
times.
You can update the version identifier of the package by running:
```shell
poetry version {version_or_keyword}
```
> You can replace `{version_or_keyword}` with either a literal version identifier (e.g. `0.1.1`) or a keyword
> (e.g. `major`, `minor`, or `patch`). You can run `$ poetry version --help` to see the valid keywords.
Alternatively, you can manually edit a line in `pyproject.toml`:
```diff
- version = "0.1.0"
+ version = "0.1.1"
```
#### Build package
At the Poetry shell, build the package based upon the latest source code:
```shell
poetry build
```
> That will create both a
> [source distribution](https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#creating-a-source-distribution)
> file (whose name ends with `.tar.gz`) and a
> [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format)
> file (whose name ends with `.whl`) in the `dist` directory.
### Publish package
#### Set up PyPI credentials
At the Poetry shell, create the following
[environment variable](https://python-poetry.org/docs/repositories/#configuring-credentials),
which Poetry will check for if credentials aren't specified to it in another way.
```shell
export POETRY_PYPI_TOKEN_PYPI="{api_token}"
```
> Replace `{api_token}` with a [PyPI API token](https://pypi.org/help/#apitoken) whose scope includes
> the PyPI project to which you want to publish the package.
#### Publish package to PyPI
At the Poetry shell, publish the newly-built package to PyPI:
```shell
poetry publish
```
At this point, people will be able to download and install the package from
[PyPI](https://pypi.org/project/mongo-diff/).