Giter Site home page Giter Site logo

tobozo / yamlduino Goto Github PK

View Code? Open in Web Editor NEW
36.0 2.0 2.0 661 KB

YAML <=> JSON converter for ESP32, ESP8266, RP2040 and possibly other devices

License: Other

C++ 13.59% C 86.41%
arduino arduinojson cjson esp32 esp8266 libyaml yaml2json yamltojson json yaml yml rp2040 rp2040-zero rp2040w samd

yamlduino's Introduction

ArduinoYaml A.K.A YAMLDuino

arduino-library-badge PlatformIO Registry

This arduino library is based on libyaml.

Supported platforms:

  • ESP32
  • RP2040
  • ESP8266
  • SAMD
  • TeensyDuino

Features:

  • YAML➔JSON and JSON➔YAML conversion
  • Accepts valid JSON or YAML as the input.
  • Standalone serializers/deserializers
  • ArduinoJson serializers/deserializers
  • cJSON serializers/deserializers
  • Node accessors
  • l10n style gettext()
  • i18n loader

Usage

#include <ArduinoYaml.h>

or

#include <YAMLDuino.h>

Pure libyaml implementation

YAML is a superset of JSON, so native conversion from/to JSON is possible without any additional JSON library.

// Available values for output format:
//   OUTPUT_YAML (default)
//   OUTPUT_JSON
//   OUTPUT_JSON_PRETTY
// JSON/YAML document to YAML/JSON string
size_t serializeYml( yaml_document_t* src_doc, String &dest_string, OutputFormat_t format=OUTPUT_YAML );
// JSON/YAML object to YAML/JSON stream
size_t serializeYml( yaml_document_t* src_doc, Stream &dest_stream, OutputFormat_t format=OUTPUT_YAML );

// YAML stream to YAML document
int deserializeYml( YAMLNode& dest_obj, const char* src_yaml_str );
// YAML string to YAML document
int deserializeYml( YAMLNode& dest_obj, Stream &src_stream );

Convert YAML to JSON

String yaml_str = "hello: world\nboolean: true\nfloat: 1.2345";
YAMLNode yamlnode = YAMLNode::loadString( yaml_str );
serializeYml( yamlnode.getDocument(), Serial, OUTPUT_JSON_PRETTY ); // pretty JSON
// serializeYml( yamlnode.getDocument(), Serial, OUTPUT_JSON ); // ugly JSON

Convert JSON to YAML

String json_str = "{\"hello\": \"world\", \"boolean\": true, \"float\":1.2345}";
YAMLNode yamlnode = YAMLNode::loadString( yaml_str );
serializeYml( yamlnode.getDocument(), Serial, OUTPUT_YAML );

Bindings

ArduinoJson and cJSON bindings operate differently depending on the platform.

ArduinoJson support cJSON support
ESP32 detected (*) implicit (built-in esp-idf)
ESP8266 implicit implicit (bundled)
RP2040 implicit implicit (bundled)
SAMD implicit implicit (bundled)

(*) On ESP32 platform, the detection depends on __has_include(<ArduinoJson.h>) macro. So all ArduinoJson functions will be disabled unless #include <ArduinoJson.h> is found before #include <ArduinoYaml.h>.

On ESP8266/RP2040/SAMD platforms it is assumed that ArduinoJson is already available as a dependency.

In order to save flash space and/or memory, the default bindings can be disabled independently by setting one or all of the following macros before including ArduinoYaml:

#define YAML_DISABLE_ARDUINOJSON // disable all ArduinoJson functions
#define YAML_DISABLE_CJSON       // disable all cJSON functions

Note to self: this should probably be the other way around e.g. explicitely enabled by user.

Note to readers: should ArduinoJson and/or cJSON be implicitely loaded? Feedback is welcome!


ArduinoJson bindings

See the motivational post for this implementation.

ArduinoJson support is implicitely enabled on most platforms except for ESP32 where dependencies can be detected.

ESP32 plaforms must include ArduinoJson.h before ArduinoYaml.h or bindings will be disabled!*

#include <ArduinoJson.h>
#include <ArduinoYaml.h>

Enabling support will expose the following functions:

// ArduinoJSON object to YAML string
size_t serializeYml( JsonVariant src_obj, String &dest_string );
// ArduinoJSON object to YAML stream
size_t serializeYml( JsonVariant src_obj, Stream &dest_stream );
// Deserialize YAML string to ArduinoJSON object
DeserializationError deserializeYml( JsonObject &dest_obj, const char* src_yaml_str );
// Deserialize YAML stream to ArduinoJSON object
DeserializationError deserializeYml( JsonObject &dest_obj, Stream &src_stream );
// Deserialize YAML string to ArduinoJSON document
DeserializationError deserializeYml( JsonDocument &dest_doc, Stream &src_stream );
// Deserialize YAML string to ArduinoJSON document
DeserializationError deserializeYml( JsonDocument &dest_doc, const char *src_yaml_str) ;

cJSON bindings

cJSON support is implicitely enabled on most platforms, and will use the bundled cJSON version unless ESP32 platform is detected. ESP32 will use the built-in cJSON version from esp-idf instead of the YAMLDuino bundled version.

⚠️ Both versions of cJSON have a memory leak with floats, the leak happens only once though, and may be avoided by quoting the float, which won't affect yaml output.

Enabling support will expose the following functions:

// cJSON object to YAML string
size_t serializeYml( cJSON* src_obj, String &dest_string );
// cJSON object to YAML stream
size_t serializeYml( cJSON* src_obj, Stream &dest_stream );
// YAML string to cJSON object
int deserializeYml( cJSON* dest_obj, const char* src_yaml_str );
// YAML stream to cJSON object
int deserializeYml( cJSON* dest_obj, Stream &src_stream );
// YAML document to cJSON object
int deserializeYml( cJSON** dest_obj, yaml_document_t* src_document );

String/Stream helper

Although const char* is an acceptable source type for conversion, using Stream is recommended as it is more memory efficient.

The StringStream class is provided with this library as a helper.

String my_json = "{\"blah\":true}";
StringStream json_input_stream(my_json);

String my_output;
StringStream output_stream(my_output);

The StringStream bundled class is based on Arduino String and can easily be replaced by any class inheriting from Stream.

class StringStream : public Stream
{
public:
  StringStream(String &s) : str(s), pos(0) {}
  virtual ~StringStream() {};
  virtual int available() { return str.length() - pos; }
  virtual int read() { return pos<str.length() ? str[pos++] : -1; }
  virtual int peek() { return pos<str.length() ? str[pos] : -1; }
  virtual size_t write(uint8_t c) { str += (char)c; return 1; }
  virtual void flush() {}
private:
  String &str;
  unsigned int pos;
};

See ArduinoStreamUtils for other types of streams (i.e. buffered).


Output decorators

JSON and YAML indentation levels can be customized:

void YAML::setYAMLIndent( int spaces_per_indent=2 ); // min=2, max=16
void YAML::setJSONIndent( const char* spaces_or_tabs="\t", int folding_depth=4 );

Set custom JSON indentation and folding depth:

// this set two spaces per indentation level, unfolds up to 8 nesting levels
YAML::setJSONIndent("  ", 8 ); // lame fact: folds on objects, not on arrays

Set custom YAML indentation (minimum=2, max=16):

// annoy your friends with 3 spaces indentation, totally valid in YAML
YAML::setYAMLIndent( 3 );

YAML gettext Module

The gettext module is a member of YAMLNode object.

class YAMLNode
{
  // (...)
public:
  const char* gettext( const char* path, char delimiter=':' );
  // YAMLNode objects also bring few interesting methods to scope:
  const char* scalar();
  size_t size();
  bool isScalar();
  bool isSequence();
  bool isMap();
  bool isNull();
  // (...)
}

Usage (persistent)

Load from string:

YAMLNode yamlnode = YAMLNode::loadString( yaml_or_json_string );

Load from stream:

YAMLNode yamlnode = YAMLNode::loadStream( yaml_or_json_stream );

Access a value:

const char* text = yamlnode.gettext( "path:to:property:name" );

Usage (non persistent)

YAMLNode supports chaining:

// load yaml and extract value from 'stuff'
YAMLNode::loadString("blah:\n  stuff:\n    true\n").gettext("blah:stuff");
// load json and extract value from 'stuff'
YAMLNode::loadString("{\"blah\":{\"stuff\":\"true\"}}").gettext("blah:stuff");

I18N/L10N with gettext Module

Note: i18n Support is disabled with WIO Terminal (platform needs a proper fs::FS filesystem implementation). WIO Terminal can still use the native YAMLNode::gettext() though.

Usage

  • Include ArduinoJson and a fs::FS filesystem first
  • Create an i18n instance and assign the filesystem i18n_t i18n( &LittleFS );.
  • Load en-GB locale with i18n.setLocale("en-GB").
  • Use i18n.gettext() to access localized strings.

Example

YAML Sample /lang/en-GB.yml stored in LittleFS:

en-GB:
  hello: world
  blah:
    my_array:
    - first
    - second
    - third

Load the language file and access translations:

#include <LittleFS.h>      // Mandatory filestem (can be SPIFFS, SD, SD_MMC, LittleFS)
#include <YAMLDuino.h>     // Load the library


i18n_t i18n( &LittleFS ); // Create an i18n instance attached to filesystem

void setup()
{
  Serial.begin(115200);
  LittleFS.begin();

  // i18n.setFS( &SD ); // change filesystem to SD
  i18n.setLocale("en-GB"); // This will look for "en-GB.yml" language file in "/lang/" folder and set "en-GB" as locale
  // i18n.setLocale("/lang/en-GB.yml"); // This will load "/lang/en-GB.yml" language file and set "en-GB" as locale
  // i18n.setLocale("en-GB", "/non-locale/file.yml"); // This will set "en-GB" as locale and load arbitrary "/non-locale/file.yml" language file

  Serial.println( i18n.gettext("hello" ) ); // prints "world"
  Serial.println( i18n.gettext("blah:my_array:2" ) ); // prints "third"
}


void loop()
{

  delay(1000);
}

Debug

The debug level can be changed at runtime:

void YAML::setLogLevel( LogLevel_t level );

Set library debug level:

//
// Accepted values:
//   LogLevelNone    : No logging
//   LogLevelError   : Errors
//   LogLevelWarning : Errors+Warnings
//   LogLevelInfo    : Errors+Warnings+Info
//   LogLevelDebug   : Errors+Warnings+Info+Debug
//   LogLevelVerbose : Errors+Warnings+Info+Debug+Verbose
YAML::setLogLevel( YAML::LogLevelDebug );

Support the Project

There are a few things you can do to support the project:

  • Star 🌟 this repository and/or follow me on GitHub
  • Share and upvote on sites like Twitter, Reddit, and Hacker News
  • Report any bugs, glitches, or errors that you find

These things motivate me to to keep sharing what I build, and they provide validation that my work is appreciated! They also help me improve the project. Thanks in advance!


Credits and special thanks to:

Additional resources:

yamlduino's People

Contributors

tobozo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

blynkgo linor

yamlduino's Issues

ArduinoJson binding issue with lists of objects

There seems to be a problem with the ArduinoJson binding specifically affecting lists of objects. Here is a minimal example that takes a simple YAML list of objects, deserializes it, then reserializes it to Serial. It does this twice, in two different ways:

  1. The manual way: use YAMLDuino to convert the YAML to JSON, then deserialize that JSON using ArduinoJSON, then reserialize using ArduinoJSON.

  2. Use YAMLDuino + ArduinoJson bindings to deserialize the YAML directly to an ArduinoJson document, then serialize to JSON.

(1) works, (2) loses the content of the objects.

Code:

// #1
YAMLNode yamlnode;
JsonDocument doc;
String buffer;
deserializeYml(yamlnode,"[{\"A\":\"1\"},{\"B\":\"1\"}]");
serializeYml(yamlnode.getDocument(),buffer,OUTPUT_JSON_PRETTY);
deserializeJson(doc,buffer);
Serial.println("YAML -> JSON -> deserializeJson -> serializeJson");
serializeJson(doc,Serial);
Serial.println("\n");

// #2
deserializeYml(doc,"[{\"A\":\"1\"},{\"B\":\"1\"}]");
Serial.println("YAML -> deserializeYml -> serializeJson");
serializeJson(doc,Serial);
Serial.println();

Output:

YAML -> JSON -> deserializeJson -> serializeJson
[{"A":1},{"B":1}]

YAML -> deserializeYml -> serializeJson
{"":[{},{}]}

I also tried #1 with OUTPUT_JSON instead of OUTPUT_JSON_PRETTY and it still worked fine, so the issue seems to be somewhere in the bindings themselves.

load external yaml file

Would be great to have an example that shows how to load a yaml file that is embedded on the device, e.g. a config.yaml.

would also like to know if it's possible to load a yaml file; modify structure (with a simple physical button to toggle a boolean value for example) and save back as a yaml file?

Undefined Reference to `deserializeYml`

I have the following code:

...

#include <SD.h>
#include <SPI.h>
#include <FS.h>
#include <ArduinoJson.h>
#include <YAMLDuino.h>

...

void SdHelpers::readInTagInfo(){
	File file = SD.open(tagInfoFileName);
	
	if( !file ) {
		LifeHelpers::unrecoverableError("Can't open config file for reading.");
	}

	DynamicJsonDocument jsonDoc(2048);
	auto err = deserializeYml(jsonDoc, file); // convert yaml to json
	file.close();
	
	if( err ) {
		LifeHelpers::unrecoverableError(String("Unable to deserialize YAML to JsonDocument: ") + String(err.c_str()));
	}
	
	JsonObject configJson = jsonDoc.as<JsonObject>();

	TagInfo newInfo(configJson);
	
	SdHelpers::tagInfoObj = newInfo;
}

Which seems to get past the compiler's syntax check, but when fails during the linking process:

~/.arduino15/packages/esp32/tools/xtensa-esp32s3-elf-gcc/gcc8_4_0-esp-2021r2-patch5/bin/../lib/gcc/xtensa-esp32s3-elf/8.4.0/../../../../xtensa-esp32s3-elf/bin/ld: sketch/SdHelpers.cpp.o:(.literal._ZN9SdHelpers13readInTagInfoEv+0x20): undefined reference to `YAML::libyaml_arduinojson::deserializeYml(ArduinoJson6200_F1::JsonDocument&, Stream&)'
~/.arduino15/packages/esp32/tools/xtensa-esp32s3-elf-gcc/gcc8_4_0-esp-2021r2-patch5/bin/../lib/gcc/xtensa-esp32s3-elf/8.4.0/../../../../xtensa-esp32s3-elf/bin/ld: sketch/SdHelpers.cpp.o: in function `SdHelpers::readInTagInfo()':
~/gitRepos/NameTag/arduinoSketch/NameTag/SdHelpers.cpp:14: undefined reference to `YAML::libyaml_arduinojson::deserializeYml(ArduinoJson6200_F1::JsonDocument&, Stream&)'
collect2: error: ld returned 1 exit status
exit status 1
Error compiling for board UM FeatherS3.

Any ideas?

serialize comments to yaml

Hello,

The reason why I prefer to use YAML is the ability to use comments. So my question is :
Since JSON doesn't support comments, how can I serializeYml() with comments?

I know you can deserialize a YAML with comments to a JsonObject, but I guess you loose the comments.

I want to :

  1. read a YAML file (with comments in it)
  2. change the content
  3. write back the YAML with the changes and the comments

Excessive memory use loading large yaml stream?

We are developing a very large project base on ESP32 and currently in the process of switching from json based config file to YAML format because of readability. We are using a YAML file from around 9000 characters.

The YAML file is read from SPIFFS and provided as a Stream, The code that parses the YAML file, allocates a DynamicJsonDocument of 20000 characters and then calls deserializeYml(jsonDocument, stream). We also have ArduinoJson in our project.
Right before calling deserializeYml() the ESP.getMaxAllocHeap() returns around 77.000 and still deserializing results in a "Not enough memory" error.

I've tried enabling debug or verbose level logging, but that doesn't give me any more logging then the "Not enough memory" error message

image

Why does it take more than 77kB to parse a 10.000 byte stream? I would think that te purpose of a stream is to not have all the data in memory at once

If I comment out the "currentcontoller" map at the bottom of the yaml file, the parsing does work ...

sensors:
  Main_Vref:
    id: 11
    nodes:
      main:
        type: ADCVoltage
        terminalId: 17
        pollingInterval: 10000
        r1: 36000
        r2: 10000

  OutsideTemp:
    id: 13
    nodes:
      iomodule:
        type: ADCThermistor
        terminalId: 44
        pollingInterval: 30000
        vRefNodeId: 11
      main:
        type: RemoteIntegerTimes10
        communicatorId: 4
        readOnly: true

  CabinetTemp:
    id: 25
    nodes:
      iomodule:
        type: ADCThermistor
        terminalId: 18
        pollingInterval: 30000
        vRefNodeId: 11
      main:
        type: RemoteIntegerTimes10
        communicatorId: 4
        readOnly: true

  LightsLargeBedroom:
    id: 33
    nodes:
      iomodule:
        type: PMW
        terminalId: 51
        storage: pwm
        key: large
      main:
        type: LightZoneRemoteSensor
        communicatorId: 4

  LightsBathroom:
    id: 30
    nodes:
      main:
        type: PWMLightZone
        terminalId: 24
        storage: pwm
        key: bathroom

  LightsSmallroom:
    id: 31
    nodes:
      main:
        type: PWMLightZone
        terminalId: 22
        storage: pwm
        key: smallroom

  LightsLivingroomLedstrip:
    id: 34
    nodes:
      iomodule:
        type: PMW
        terminalId: 49
        storage: pwm
        key: livingmood
      main:
        type: LightZoneRemoteSensor
        communicatorId: 4

  LightLivingroomTable:
    id: 35
    nodes:
      iomodule:
        type: PMW
        terminalId: 47
        storage: pwm
        key: living
      main:
        type: LightZoneRemoteSensor
        communicatorId: 4

  LightsOutside:
    id: 36
    nodes:
      iomodule:
        type: PMW
        terminalId: 45
        storage: pwm
        key: outside
      main:
        type: LightZoneRemoteSensor
        communicatorId: 4

  Outlets:
    id: 37
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 43
        stateNodeId: 107
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  OutletState:
    id: 107
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 10
        interruptEnabled: true
        pollingInterval: 10300

  Boiler:
    id: 41
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 42
        stateNodeId: 111
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  BoilerState:
    id: 111
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 11
        interruptEnabled: true
        pollingInterval: 10700

  HighpowerOutlet:
    id: 38
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 41
        stateNodeId: 108
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  HigpowerOutletState:
    id: 108
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 12
        interruptEnabled: true
        pollingInterval: 10500

  HeaterLiving:
    id: 39
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 40
        stateNodeId: 109
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  HeaterLivingState:
    id: 109
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 13
        interruptEnabled: true
        pollingInterval: 10600

  HeaterBathroom:
    id: 40
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 39
        stateNodeId: 110
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  HeaterBathroomState:
    id: 110
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 14
        interruptEnabled: true
        pollingInterval: 10400

  Stove:
    id: 42
    nodes:
      iomodule:
        type: MxPulseRelay
        terminalId: 38
        stateNodeId: 112
      main:
        type: ControlledRemoteBoolean
        currentControllerId: 180
        communicatorId: 4

  StoveState:
    id: 112
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 15
        interruptEnabled: true
        pollingInterval: 10800

  # Not present on testboard
  # Current:
  #   id: 43
  #   nodes:
  #     iomodule:
  #       type: ADCCurrent
  #       terminalId: 1
  #       pinMode: INPUT
  #       pollingInterval: 500
  #     main:
  #       type: RemoteIntegerTimes10
  #       readOnly: true
  #       communicatorId: 4

  # Not present on testboard
  # KwhCounter:
  #   id: 45
  #   nodes:
  #     iomodule:
  #       type: MxInterruptCount
  #       terminalId: 26
  #       pollingInterval: 10200
  #       divider: 100
  #       interruptMode: FALLING
  #       storage: intcount
  #       key: kwh
  #     main:
  #       type: StoredSettings<InterruptCounterType>
  #       storage: haaksnano
  #       key: powerusage

  WaterflowCounter:
    id: 50
    nodes:
      iomodule:
        type: InterruptCount
        terminalId: 3
        divider: 1
        debounceMs: 1000
        pollingInterval: 10900
        storage: intcount
        key: water
      main:
        type: StoredSettingsInterruptCounterType
        storage: haaksnano
        key: waterusage

  FlushRelais:
    id: 83
    nodes:
      iomodule:
        type: MxDigital
        terminalId: 37
        pinMode: OUTPUT
      main:
        type: RemoteBoolean
        communicatorId: 4

  SunUpdown:
    id: 21
    nodes:
      main:
        type: SunUpDown
        lightzoneIds:
          - 36
        pollingInterval: 60000

currentcontroller:
  id: 180
  main:
    maxCurrent: 25
    currentNodeId: 43
    currentNodeMode: PositiveLoad
    mainSwitchNode: 129
    mainSwitchReverseLogic: true
    pollingInterval: 10000
    nodes:
      - nodeId: 42
        prio: 100
      - nodeId: 37
        prio: 90
      - nodeId: 38
        prio: 80
      - nodeId: 182
        prio: 70
      - nodeId: 41
        prio: 60
      - nodeId: 181
        prio: 50

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.