google-firebase-firestore-json-export's Introduction

Google Firebase Firestore JSON Export

Exports a Google Firestore database recursively including all collections and subcollections into plain JSON files.

Uses the built-in recursive() flag of the Query class: https://cloud.google.com/python/docs/reference/firestore/latest/query#recursive

Allows you to analyze the data in your Firestore database locally without being constrained to Firestore or BigQuery limitations. For example:

Finding incomplete documents
Finding broken document references
Finding documents without fields
Querying subcollections of collections without documents

You can use custom scripts or the powerful jq command-line JSON processor to perform complex lookups, such as this example, or the following, which extracts all documents that do not have the key 'createdBy':

jq -r 'with_entries(select(.value.createdBy == null))' collection.json

⚠️ Warning: A complete export causes 1 read operation for every document in your Firestore database. Depending on the size of your database, this might induce costs for database reads.

Requirements

Python 3

Setup and Usage

Install firebase-admin via pip
```
$ pip3 install firebase-admin
```
Run export to JSON.
```
$ python3 main.py
```

Todos

Serialize TimestampWithNanoseconds (instead of converting to string) https://code.luasoftware.com/tutorials/google-cloud-firestore/python-firestore-query-documents-to-json

Serialize document references (e.g. as document path instead of address)

"ref": "<google.cloud.firestore_v1.document.DocumentReference object at 0x111ef1c50>"

Import into local Firestore emulator
Allow to export as line-delimited JSON (LDJSON) to allow streaming and avoid loading the whole db into memory when importing.

Recommend Projects