planetis-m/eminim

JSON serialization framework for Nim, works from a Stream directly to any type and back. Depends only on stdlib.

Γεια 👋

My name is Antonis Geralis. I am an engineering student. I speak 🇬🇷 Greek (native), 🇺🇲 English and 🇩🇪 German. I code mostly in Nim. My programming interests include multithreading, fuzzing, traffic simulations, games and web-apps. I love solving hard problems and learning about Computer Science.

Posts

date	post	brief	links
14/4/23	Using NimScript for your build system	Welcome to this tutorial on using NimScript for your build system!	full text
13/4/23	Naylib Goes Mobile: Porting to Android in Just 3 Days!	In this post, I'll share how I managed to port my Nim raylib bindings to Android in just three days.	full text
23/12/20	An introduction to ECS by example	In this post I'm exploring the inner workings of an strict ECS implementation, discuss the...	full text discussion
7/6/18	Creating a simple macro	Hello, as you might know Nim is a powerful programming language that supports metaprogramming...	full text

Software

Some of the most interesting software projects I've authored are:

Name	Description
drchaos	A powerful and easy-to-use fuzzing framework in Nim for C/C++/Obj-C targets.
libfuzzer	Thin interface for libFuzzer, an in-process, coverage-guided, evolutionary fuzzing engine.
eminim	JSON serialization framework, works from a Stream directly to any type and back.
manu	A linear algebra library in pure Nim. Supports constructing and manipulating real, dense matrices.
sync	Useful synchronization primitives
bingo	Binary serialization framework
goodluck	A hackable template for creating small and fast games. Nim port
breakout	Breakout game implemented using strict ECS architecture. Used as a testbed.
naylib	Yet another raylib wrapper
jsonpak	Packed ASTs for compact and efficient JSON representation, with JSON Pointer, JSON Patch support.
patgraph	Efficient graph data structure library. The graph is a seq of nodes plus a seq of edges.
ssostrings	Small String Optimized (SSO) string implementation
cowstrings	Copy-On-Write string implementation according to nim-lang/RFCs#221
html2karax	Converts static HTML to Karax.
neuralnet-examples	Neural network examples
protocoled	Interface macro
fusion/astdsl	A DSL for convenient construction of Nim ASTs
looper	For loop macros, zip, enumerate, collect (merged upstream)

eminim's Issues

Support case objects

Write code like:

var pairs: seq[(string, string)]
var kind: ObjKind
result = Obj(kind: kind)
for key, value in pairs:
  case kind
  of variantA:
  of variantB:
     case key
     of "fieldA":
        result.field = value

First class support for object variants

Requires #12 to be implemented.

type
  Bar = object
    shared: int
    case kind: Fruit
    of Banana:
      bad: float
      banana: int
    of Apple: apple: string

let s = newStringStream("""{"kind":"Apple","apple":"world"}""")
let a = s.jsonTo(Bar)

proc initFromJson(dst: var Bar, p: var JsonParser) =
  eat(p, tkCurlyLe)
  while p.tok != tkCurlyRi:
    if p.tok != tkString:
      raiseParseErr(p, "string literal as key")
    case p.a
    of "shared":
      discard getTok(p)
      eat(p, tkColon)
      initFromJson(dst.apple, p)
    of "kind":
      discard getTok(p)
      eat(p, tkColon)
      var kindTmp`gensym0: Fruit
      initFromJson(kindTmp`gensym0, p)
      if dst.kind != kindTmp`gensym0:
        case dst.kind
        of Banana:
          if onceBad or onceBanana:
            raiseParseErr(p, "valid object field")
          if onceShared:
            dst = (typeof dst)(kind: kindTmp`gensym0, shared: dst.shared)
        of Apple:
          if onceApple:
            raiseParseErr(p, "valid object field")
          if onceShared:
            dst = (typeof dst)(kind: kindTmp`gensym0, shared: dst.shared)
        else: discard
    of "bad":
      if dst.kind != Banana: # not shared
        raiseParseErr(p, "valid object field")
      discard getTok(p)
      eat(p, tkColon)
      initFromJson(dst.bad, p)
    of "banana":
      if dst.kind != Banana:
        if onceApple:
          raiseParseErr(p, "valid object field")
        if onceShared:
          dst = (typeof dst)(kind: Banana, shared: dst.shared)
      discard getTok(p)
      eat(p, tkColon)
      initFromJson(dst.banana, p)
    of "apple":
      if dst.kind != Apple:
        if onceBad or onceBanana:
          raiseParseErr(p, "valid object field")
        if onceShared:
          dst = (typeof dst)(kind: Apple, shared: dst.shared)
      discard getTok(p)
      eat(p, tkColon)
      initFromJson(dst.apple, p)
    else:
      raiseParseErr(p, "valid object field")
    if p.tok != tkComma:
      break
    discard getTok(p)
  eat(p, tkCurlyRi)

Huge generated code size

Macro needs to split each type to procedures.

1. Make most templates output procs
2. Make the macro call said procs
3. Profit.

Support for anonymous tuples.

Needs changes to the macro, but is not really something that can not be worked-around. stdlib doesn't support them. Contributions welcome :P

This could be rewritten

See the discussion at:
nim-lang/Nim#12391

Some points are addressed, however the current design is hard to debug/ extend / fix. A combination of overloading and typed macros is the better solution.

Can not parse seq of Components

import eminim, streams

type Component = ref object of RootObj
type Move = ref object of Component
  speed: float32
let s = newStringStream("""{"speed":20}""")
let a = Move(s.jsonTo(Component))
echo a.speed

From the marshal module docs:

Restriction: For objects, their type is not serialized. This means essentially that it does not work if the object has some other runtime type than its compiletime type.

Gracefully fail, instead of a logic error when a custom parser encounters keys that are not in order

proc initFromJson*[T](dst: var Matrix[T]; p: var JsonParser) =
  eat(p, tkCurlyLe)
  var count = 0
  while p.tok != tkCurlyRi:
    if p.tok != tkString:
      raiseParseErr(p, "string literal as key")
    case p.a
    of "m":
      discard getTok(p)
      eat(p, tkColon)
      if count > 1:
        raiseParseErr(p, "JSON keys in order")
      count = 1
      initFromJson(dst.m, p)
    of "n":
      discard getTok(p)
      eat(p, tkColon)
      if count > 2:
        raiseParseErr(p, "JSON keys in order")
      count = 2
      initFromJson(dst.n, p)
    of "data":
      discard getTok(p)
      eat(p, tkColon)
      if count > 3:
        raiseParseErr(p, "JSON keys in order")
      count = 3
      eat(p, tkBracketLe)
      assert dst.m != 0 and dst.n == 0
      dst.data = createData[T](dst.m * dst.n)
      var i = 0
      while p.tok != tkBracketRi:
        initFromJson(dst.data[i], p)
        inc(i)
        if p.tok != tkComma: break
        discard getTok(p)
      eat(p, tkBracketRi)
    else:
      raiseParseErr(p, "valid object field")
    if p.tok != tkComma: break
    discard getTok(p)
  eat(p, tkCurlyRi)

doesn't even seem enough...

add ndjson support

A new iterator could be created for parsing ndjson a better alternative to CSV format. Problem is there is no tokNewline in parsejson, but maybe getLine can be abused for the purpose of expecting a '\n' or even ignored allowing both newline delimited and concatenated JSON.

Add support for JsonNodes

It would be trivial to copy parseJson logic from std and support outputting to JsonNodes.

Skip (or error) on duplicate keys

In JSON either action is valid for duplicate keys. Currently we process every duplicate key with a potential performance penalty. A better alternative would be to generate code like:

var
  onceBanana = false
eat(p, tkCurlyLe)
while p.tok != tkCurlyRi:
  if p.tok != tkString:
    raiseParseErr(p, "string literal as key")
  case p.a
  of "banana":
    discard getTok(p)
    eat(p, tkColon)
    if not onceBanana:
      onceBanana = true
      initFromJson(dst.banana, p)
    else:
      when defined(emiDuplicateKey):
        skipJson(p)
      else: raiseParseErr(p, "duplicate object field")
  else:
    raiseParseErr(p, "valid object field")
  if p.tok != tkComma:
    break
  discard getTok(p)
eat(p, tkCurlyRi)

...and leave it up to the user to specify whether it should error or skip them.

Add back the skipping feature

Rebrand as marshal replacement

Well on my defense I started implementing eminim as a marshal replacement. It was later called a JSON serialization library because "marshal replacement" seemed to restrictive/unknown. Add more warnings about the limitations of direct serializers and also a reference to https://einarwh.wordpress.com/2020/05/08/on-the-complexity-of-json-serialization/

simple cycle detector

use this