huan-mongo/mongosh-snippets

Toff: Tim's Oplog Filtering Functions

The toff() method helps printing oplog entries. Chain commands together to filter oplog entries then .show(). By default noops and the config db are ommitted. When filtering by namespace, db, command, _id, and op, we search both top-level operations and sub-operations in applyOps. That way you do not miss anything that is part of a transaction. When filtering by namespace we take into account commands on db.$cmd.

Installation

Run mongosh. Then in the mongosh shell run the following commands:

config.set('snippetIndexSourceURLs', 'https://github.com/tfogo/mongosh-snippets/raw/main/index.bson.br;' + config.get('snippetIndexSourceURLs'))

Install the Toff snippet:

snippet refresh
snippet install toff

I recommend setting inspectDepth to Infinity to make sure nested oplog entries are fully parsed. If this causes problems with parsing extremely nested documents you can always set it to a finite number.

config.set("inspectDepth", Infinity)

View Toff help:

toff().help()

Updating toff:

snippet update toff
snippet load-all

Tips

Want to look at results one at a time? Set batchSize to 1, then you can iterate through the results with it. After you're done you can reset the batch size:

config.set("displayBatchSize", 1)

toff().show()

// Iterate through results with it

config.reset("displayBatchSize")

Examples

Show the oplog. By default it's in timestamp order (oldest to newest):

toff().show()

Show the last 5 oplog entries:

toff().limit(5).show()

Show the oplog from newest to oldest:

toff().reverse().show()

Show the oplog from newest to oldest from timestamp { t: 1690828162, i: 786 }:

toff().reverse().after({ t: 1690828162, i: 786 }).show()

Show the oplog from newest to oldest before timestamp { t: 1690828162, i: 786 }:

toff().reverse().before({ t: 1690828162, i: 786 }).show()

Show the oplog between wall times "2023-07-31T18:29:19.037889997Z" and "2023-07-31T18:29:19.081624304Z":

Note: wall times from mongosync logs may not correspond directly to the wall times in the oplog.

toff().afterWall("2023-07-31T18:29:19.037889997Z").beforeWall("2023-07-31T18:29:19.081624304Z").show()

Show only inserts into the partitions collection:

toff().ns("mongosync_reserved_for_internal_use.partitions").op("i").show()

Show only createIndexes commands:

toff().command("createIndexes").show()

Show only the mongosync_reserved_for_internal_use database except for globalState:

toff().db("mongosync_reserved_for_internal_use").excludeNs("globalState").show()

Show all operations on the admin.$cmd namespace:

toff().ns("admin.$cmd").show()

Find all ops in transaction 6753 for lsid.id "d94483c0-5d07-4b05-8b9b-a0c18cc495fa":

toff().txn(6753, "d94483c0-5d07-4b05-8b9b-a0c18cc495fa").show()

Find operations on docs with _id "64c7f9f11a4c236a31f5c6c4":

toff().byID("64c7f9f11a4c236a31f5c6c4").show()

Find operations on docs with _id "64c7f9f11a4c236a31f5c6c4" or "648170059cedcc216ef1d6d8:

toff().byID("64c7f9f11a4c236a31f5c6c4", "648170059cedcc216ef1d6d8").show()

Count how many times the document with _id 8 was inserted into test.melon:

toff().op("i").ns("test.melon").byID(8).count()

Use a projection to show only the timestamps for ops where _id 8 was inserted into test.melon:

toff().op("i").ns("test.melon").byID(8).project({"ts":1}).show()

Use a custom $match query to find updates which set the field a to 10:

toff().op("u").match({"o.a": 10}).show()

Only print the value of `"o._id"`` for each insert on namespace foo.bar:

toff().op("i").ns("foo.bar").printField("o._id")

Reference

includeNoop()
Includes noop operations

includeConfig()
Includes operations from the config db

before(ts)
Only includes operations from a time less than or equal to the timestamp ts

after(ts)
Only includes operations from a time greater than or equal to the timestamp ts

beforeWall(ts)
Only includes operations from a time less than or equal to the wall time ts

afterWall(ts)
Only includes operations from a time greater than or equal to the wall time ts

reverse()
Entries are sorted in descending order (newest to oldest)

db(db)
Only includes operations where ns is equal to db

ns(...ns)
Only includes operations from the namespace ns. You can include multiple namespaces

excludeDB(db)
Excludes operations where ns is equal to db

excludeNs(...ns)
Excludes operations from the namespace ns. You can exclude multiple namespaces

op(opType)
Only includes ops of type opType. Valid types are 'n','c','i','u','d'. These are noops, commands, inserts, updates, deletes.

command(commandName)
Only includes commands of type commandName

txn(txnNumber, lsid)
Only includes operations with transaction number equal to txnNumber and lsid. lsid is the lsid.id UUID value in the oplog

byID(...id)
Shows operations on objects with _id equal to id. Can specify multiple ids. Includes 'i', 'u', and 'd' ops, and 'applyOps' commands

match(query)
Add a custom $match stage using query

compact()
When printing, omit most info so objects are smaller. This can omit useful information

limit(n)
Limit the output to n entries

project(projection)
Add a projection to the output

getPipeline()
Shows the pipeline which will be used to generate the aggregation. Useful for seeing what is happening under the hood

count()
Instead of showing results, print the count of results from the query

get() Returns the result object from the query. An alternative to show() which allows you to use the result in code if needed

printField(key) Prints the value of the given key for each matching object. Should be used as an alternative to show()

show()
Prints the output from the query. Should be the final method called. Can be replaced with .get(), .count(), printField() or .getPipeline()