nmeylan/json-table-editor

A tool to view JSON array as table, to ease visualisation and edition of large json array with performance as main objective

Important

Memory usage: While this tool allow to edit large json file smoothly, it consumes a lot of memory due to technical decision which tradeoff memory in favor of better performance for navigation, edition and serialization. Read below section "Performance and Memory usage" for more details

Web demo

Genesis

I wanted to edit a small json file of 1.5mb, using a table view, javascript based tools like http://json2table.com/# and https://jsongrid.com/json-grid were too slow to edit this file and barely usable, so I decided to attempt to do it myself

Features

Implemented

• lag-free visualisation of large json array: only visible rows and columns are rendered
• Select column to render
• Filter out rows with null value at given columns
• Scroll to column
• Pin columns to left
• Open nested array in sub-table
• Open nested Object in sub-table
• Select depth for nested object
• Edit cell
• Filter columns by values
• Go to row number
• Search for matching terms
• Copy/Paste cell
• Insert row above/below another row

Usage

• You can launch the tool without arguments by executing the executable
• You can launch the tool with arguments ./json-editor [PATH_TO_JSON] [JSON_POINTER]
• Right click on cells to display a contextual menu

Json in video below is 372mb with an array of 309_759 entries, containing nested objects, it runs at ~60fps on an intel graphics chipset (Mesa UHD 630) as only visible cell are rendered

out.mp4

Performance and Memory usage

This editor use a custom json parser to deserialize json into a flat data structure allowing O(n) data access where n is number of fields of an array element, containing k elements. In general n not bigger than 200 but k can easily be >100000. While it is not the fastest parser on the market, it is still faster to use our custom parser instead of parsing using serde or other library generating tree data structure and then convert it to flat data structure.

As this structure is flat, it also allows to parse only a subset of json files by defining a depth limit, the json files is still fully read but after a given depth, content is not deserialize we only keep raw content as String which then can be parse later, by changing depth.

This mechanism allow fast parsing of big json files, but consume more memory as for each depth level we store the full string and the parsed content. Additionally, this mechanism allow to serialize only row that have been changed, unchanged rows are already serialized, speeding up edition of big files.

Disclaimers

This tool has not yet reached a stable version, thus it has the following limitations:

• If you open a non-valid json file, tool will crash, there is currently no graceful handling for json parser error
• If you open a json file starting by an object, you will be able to edit an array of your choice at the given json pointer, When you do save, it will only save this array Original json file

{"skills": [], "count":  1000}

If you select pointer /skills, then save, it will produce this file

{"skills": []}

About

Licence: Apache 2

Credits: this project would not have been possible without egui and its community, and maintainers of dependencies used by this project

json-table-editor's Issues

Editing array contaning nested object is not working

Opening subtable of an array containing

[
        {
          "level": 1,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 24
          }
        },
        {
          "level": 2,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 28
          }
        },
        {
          "level": 3,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 32
          }
        },
        {
          "level": 4,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 36
          }
        },
        {
          "level": 5,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 40
          }
        },
        {
          "level": 6,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 44
          }
        },
        {
          "level": 7,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 48
          }
        },
        {
          "level": 8,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 52
          }
        },
        {
          "level": 9,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 56
          }
        },
        {
          "level": 10,
          "value": {
            "bonus": "ChanceToInflictStatusOnAttackPercentage",
            "value": "Stone",
            "value2": 60
          }
        }
      ]

Generate

[
1,
  {

  },
1,
  {

  },
2,
  {

  },
2,
  {

  },
3,
  {

  },
3,
  {

  },
4,
  {

  },
4,
  {

  },
5,
  {

  },
5,
  {

  },
6,
  {

  },
6,
  {

  },
7,
  {

  },
7,
  {

  },
8,
  {

  },
8,
  {

  },
9,
  {

  },
9,
  {

  },
10,
  {

  },
10,
  {

  }
]

while editing this kind of array is working fine

[
  {
    "amount": 15,
    "level": 11
  },
  {
    "amount": 17,
    "level": 2
  },
  {
    "amount": 19,
    "level": 3
  },
  {
    "amount": 21,
    "level": 4
  },
  {
    "amount": 23,
    "level": 5
  },
  {
    "amount": 25,
    "level": 6
  },
  {
    "amount": 27,
    "level": 7
  },
  {
    "amount": 29,
    "level": 8
  },
  {
    "amount": 31,
    "level": 9
  },
  {
    "amount": 33,
    "level": 10
  }
]

Context menu: paste option

display number of rows

Filtered column should not duplicate data

Replace panel: toggle replace only visible rows

If filters is active add the option to replace only visibile rows

On hover large text, we should be able to focus and scroll on tooltip

Context menu: filter with this value

Icon in context menu

Object table: add new entry

Not sure it is a good idea, because it would mean to handle complex like this:

Add new entry: /a/b/c/d/0/a1/0/b, we have to create all nested elements

Unless we limit to a new sibling entry only and not nested entries

Taskbar should wrap

On hover display long text

Reload file

When file has been changed with another editor, add the option to reload content of json-table-editor, while preserving pinned column, filter, etc...

Not sure if it is a good idea, if structure has been changed to much (column removal e.g) it should be handled aswell

Save file

Serialize changed rows

Search matching terms while rows are filtered

When we search using matching term to go to a row and if a filter is active, it fails to locate the matching rows
, probably an issue with index

Filter should be cleared when last value matching the filter is removed

Filter is listing all values present for the given column in all row of the json array.
If a filter with value "A" is active, but there is no value "A" anymore in the filtered column, the list of filtered option is not showing "A" value anymore, thus it is not possible to remove the active filter

Active filter should be removed automatically if the value does not exists anymore

Toggle visible column

Add panel to toggle visible column:

• 1 line with checkbox + label per column (hiding columns should reset filter on it)
• 1 main toggle all button

Also provide option to reorder column:

• Drag drop to reorder columns

Error handling when opened json is invalid

on enter in search by term, trigger scroll to next occurrence

Render boolean column with checkbox

Array subtable add new line

parser crash when clicking on row #4

Shortcut to delete cell value

Add/delete row

Adding new row above or below an existing row implies following:

• as we store index of json array entry, all index after the new row should be updated
• as json pointer contains index (e.g: /skills/10/description), all json pointer should be updated aswell

Short cut for save and save as

Scroll to matching term column

Currently when we search rows containing matching term, it scroll to the row when we click on next occurrence, it should also scroll to matching column and highlight it

CI to generate linux/windows/wasm build

sort column

faster search

bug when editing nested object when not already parsed

If parent is Object(false) meaning it has not been parsed yet, the update of the nested fields from subtable generate weird result

Add/delete column

wasm: save file

Edit cell value

Edit from subpanel should also update parent table.
Currently subpanel data are independent data

Performance issues

On big json following features are slow:

• Insert new row: due to data structure with json pointer, inserting a new row means we need to iterate over each entry after the new added row to update their json-pointer as the position has changed.

{"skills": [{"copyFlags": {"reproduce": true, "plagiarism": false}}]}

/skills/0 -> {"copyFlags": {"reproduce": true, "plagiarism": false}} 
/skills/0/copyFlags -> {"reproduce": true, "plagiarism": false}
/skills/0/copyFlags/reproduce -> true
/skills/0/copyFlags/plagiarism -> false

Then inserting a new row above row 0, any row after row 0 should be updated like this

/skills/1 -> {"copyFlags": {"reproduce": true, "plagiarism": false}} 
/skills/1/copyFlags -> {"reproduce": true, "plagiarism": false}
/skills/1/copyFlags/reproduce -> true
/skills/1/copyFlags/plagiarism -> false

• Replace in column when criteria match too many row: due to the capability to open multiple view on the same row at different depth of the json object, we have to serialize and parse again the updated row:
  • we need to serialize it so root object of the row can be viewed in "Object table"
  • we then parse it again to update nested objects serialized entry

{"skills": [{"copyFlags": {"reproduce": true, "plagiarism": false}}]}

/skills/0 -> {"copyFlags": {"reproduce": true, "plagiarism": false}}  <- we keep serialized object for depth 3 view 
/skills/0/copyFlags -> {"reproduce": true, "plagiarism": false}  <- we keep serialized object for depth 4 view
/skills/0/copyFlags/reproduce -> true
/skills/0/copyFlags/plagiarism -> false

If we update plagiarism to true /skills/0/copyFlags/plagiarism -> true

we also need to update 
/skills/0 <- require serialization of root object
/skills/0/copyFlags

Filter with wildcard

We should have a way to toggle all filter starting by AC_*

Fix icons rendering

Currently icons don't have same size or color, it is ugly

on enter on scroll to column should trigger scroll to next occurrence

Subwindow should fit content size

When opening json object and selecting a pointer we should save other object entries

Sort filter

code cleanup

Remove duplication.
Line by line cleanup

wasm + github page

Highlight filtered column

Replace does not trigger file change status

When opening a file, then doing a replace, the title of the application is not updated with (*) indicating file has been updated.

Scroll to column: offset mis calculation when there is pinned column

Offset should take table of pinned columns width.
In addition when columns are pinned, they still appear in the next/prev

in the screen shot we should see N/4 and not N/6 because there are 2 over 6 columns, containing bonus, pinned

Copy/paste array/object

Currently it is not working, it copies/paste empty value

Replace in column

Paste in context menu