https://github.com/labbots/firestore-export-json
Convert google firestore/datastore LevelDB exports to JSON data.
https://github.com/labbots/firestore-export-json
hacktoberfest hacktoberfest2021
Last synced: about 2 months ago
JSON representation
Convert google firestore/datastore LevelDB exports to JSON data.
- Host: GitHub
- URL: https://github.com/labbots/firestore-export-json
- Owner: labbots
- License: mit
- Created: 2021-04-26T19:11:26.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2023-10-18T01:51:46.000Z (over 1 year ago)
- Last Synced: 2023-10-20T21:06:34.539Z (over 1 year ago)
- Topics: hacktoberfest, hacktoberfest2021
- Language: Python
- Homepage:
- Size: 16.6 KB
- Stars: 71
- Watchers: 2
- Forks: 17
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Firestore Export JSON
Convert google firestore/datastore LevelDB exports to JSON data.
## Dependencies
* Python 3.9
* [Python Google Appengine](https://github.com/GoogleCloudPlatform/appengine-python-standard)
## Setup
### Method 1 - Direct Install
1. Clone the repository.
```shell
$ git clone [email protected]:labbots/firestore-export-json.git
$ cd firestore-export-json
```
2. Create a new virtual environment within the project directory.
```shell
$ python -m venv venv
```
3. Activate the virtual environment in the project.
```shell
$ source ./venv/bin/activate
```
4. Install framework dependencies.
```shell
$ pip install -e .
```
### Method 2 - Make
1. Clone the repository.
```shell
$ git clone [email protected]:labbots/firestore-export-json.git
$ cd firestore-export-json
```
2. Run the make command
```
$ make install
```
## Usage
1. Export firestore collections using firestore admin tools. Follow the [instructions here](https://firebase.google.com/docs/firestore/manage-data/export-import#export_data) to backup firestore.
2. The exports are stored in google cloud storage. Download the exported data directory from cloud storage.
3. Use the below command to convert the exported data to JSON.
4. By default, running the command will use (num_cpus - 1) processes to process the data. This can greatly speed up processing.
To set this to something different use the `-P`/`--processes` flags e.g. `./fs_to_json.py source/ dest/ -P 1`
5. By default, this will not clear out any files from the destination directory.
To force a clean before running use the `-C`/`--clean-dest` flag e.g. `./fs_to_json.py source/ dest/ -C`
6. By default, this will attempt to use a hardware-accelerated crc32c check to verify the integrity of the blocks of data. Without
hardware acceleration, this can be a costly process. If you don't have hardware acceleration, you can disable the checks with the `-c`/`--no-check-crc` flag.
The penalty for not running any crc32c checks is that you might have corrupt or missing data in the json output.
### Method 1
`fs_to_json.py` can be used to convert the firestore/datastore export files to JSON data.
```shell
$ python fs_to_json.py path_to_source_dir path_to_destination
```
1. Provide the path to the directory where files such as `output-0` exists as source directory usually this would be `kinds_*` directory.
2. Destination directory must be specified where the json files will be created. If not provided then the json files will be created using `json` folder in source directory.
e.g.
```shell
# this will look for all files in exports/all_namespaces/kind_collection-id that start with `output-` and put the resulting json files in out/collection-id
python fs_to_json.py exports/all_namespaces/kind_collection-id out/collection-id
```
### Method 2
The project exposes console script using setuptools to provide `fs_to_json` CLI command which can be used to convert exports to JSON.
```shell
$ fs_to_json path_to_source_dir path_to_destination
```
## Inspired By
* [Venryx/firestore-leveldb-tools](https://github.com/Venryx/firestore-leveldb-tools) - LevelDB files to JSON converter written in python 2.7.