Giter Site home page Giter Site logo

anymarkup's Introduction

anymarkup

Build Status Code Health Coverage

Parse or serialize any markup. Currently supports ini, json, json5, toml, xml and yaml. Report bugs and new functionality requests at https://github.com/bkabrda/anymarkup/issues.

Parsing:

>>> import anymarkup
>>> anymarkup.parse('foo: bar')
{'foo': 'bar'}
>>> anymarkup.parse_file('foo/bar.ini')
{'section': {'subsection': {'opt2': 'bar'}, 'opt1': 'foo'}}

$ cat foo/bar.ini
[section]
opt1=foo
[[subsection]]
opt2=bar

Serializing:

>>> import anymarkup
>>> anymarkup.serialize({'foo': 'bar'}, 'json')
b'{\n  "foo": "bar"\n}'
>>> anymarkup.serialize_file({'foo': 'bar'}, 'foo/bar.json')

$ cat foo/bar.json
{
  "foo": "bar"
}

anymarkup is licensed under BSD license. You can download official releases from https://pypi.python.org/pypi/anymarkup or install them via pip install anymarkup.

anymarkup works with Python 2.7 and >= 3.3.

Automatic Markup Language Recognition

When using anymarkup.parse(input), anymarkup will try to guess markup language of input. This usually works fine except:

  • ini vs toml: These two look almost the same and in fact have common subset (which, however, yields different parsing result). Because of this, anything with an ini-like look will be parsed with ini parser. If you want an input string to be parsed as toml, you have to explicitly specify that using format=toml (see below for examples).
  • json vs json5: json5 is superset of json, but not very widely used. Because of practicality of json usage, everything that looks like json is parsed as json. If you want input string to be parsed as json5, you have to explicitly specify that using format=json5.

When using anymarkup.parse_file(path), anymarkup will try to guess format based on file extension and then fallback to guessing as explained above. This means that if the file has .toml pr .json5 extension, you don't have to provide format=<format> explicitly.

Notes on Parsing Basic Types

When parsing, anymarkup recognizes basic types - NoneType, int, float and bool (and long on Python 2) and converts all values to these types. If you want to get everything as strings, just use force_types=False with parse or parse_file. Finally, you can also use force_types=None to get whatever the parsing backend returned:

>>> anymarkup.parse('a: 1')
{'a': 1}
>>> anymarkup.parse('a: 1', force_types=False)
{'a': '1'}
>>> anymarkup.parse('a: 1', force_types=None)
{'a': 1}

CLI

To install the CLI, run the following command:

pip install anymarkup

Example of conversion from JSON to XML:

anymarkup convert --from-format json --to-format xml <somefile.json

For full help on the CLI run the following commands:

anymarkup --help anymarkup convert --help

Backends

anymarkup uses:

Notes on OrderedDict

Parsing certain types of markup can yield Python's OrderedDict type - namely XML documents and YAML !!omap (see http://yaml.org/type/omap.html). anymarkup handles this without a problem, but note that if you serialize these as JSON or INI and then parse again, you'll lose the ordering information (meaning you'll get just dict back).

This is because JSON and INI parsers (to my knowledge) don't consider ordering key-value structures important and there's no direct means in these markup languages to express ordering key-value structures.

Notes on Dependencies

Read this section if you want anymarkup functionality only for subset of supported markup languages without the need to install all parsers.

Since version 0.5.0, anymarkup is just a wrapper library around anymarkup-core (https://github.com/bkabrda/anymarkup-core) and doesn't actually contain any code, except of imports from anymarkup-core.

anymarkup-core goal is to not explicitly depend on any of the parsers, so people can install it with only a specified subset of dependencies. For example, you can install anymarkup-core only with PyYAML, if you know you'll only be parsing YAML.

If you install anymarkup, you will always get a full set of dependencies and you will be able to parse any markup language that's supported.

The CLI requires click as indicated in the requirements.txt file.

Examples

Parsing examples:

ini = """
[a]
foo = bar"""

json = """
{"a": {
    "foo": "bar"
}}"""

xml = """<?xml version="1.0" encoding="UTF-8"?>
<a>
    <foo>bar</foo>
</a>"""

yaml = """
a:
  foo: bar
"""

# these will all yield the same value (except that xml parsing will yield OrderedDict)
anymarkup.parse(ini)
anymarkup.parse(json)
anymarkup.parse(xml)
anymarkup.parse(yaml)

# explicitly specify a type of format to expect and/or encoding (utf-8 is default)
anymarkup.parse('foo: bar', format='yaml', encoding='ascii')

# by default, anymarkup recognizes basic types (None, booleans, ints and floats)
#   if you want to get everything as strings, just use force_types=False

# will yield {'a': 1, 'b': True, 'c': None}
anymarkup.parse('a: 1\nb: True\nc: None')
# will yield {'a': '1', 'b': 'True', 'c': 'None'}
anymarkup.parse('a: 1\nb: True\nc: None', force_types=False)

# or parse a file
anymarkup.parse_file('foo.ini')

# if a file doesn't have a format extension, pass it explicitly
anymarkup.parse_file('foo', format='json')

# you can also pass encoding explicitly (utf-8 is default)
anymarkup.parse_file('bar', format='xml', encoding='ascii')

Serializing examples:

struct = {'a': ['b', 'c']}

for fmt in ['ini', 'json', 'xml', 'yaml']:
    # any of the above formats can be used for serializing
    anymarkup.serialize(struct, fmt)

# explicitly specify encoding (utf-8 is default)
anymarkup.serialize(struct, 'json', encoding='utf-8')

# or serialize directly to a file
anymarkup.serialize_file(struct, 'foo/bar.ini')

# if a file doesn't have a format extension, pass it explicitly
anymarkup.serialize_file(struct, 'foo/bar', format='json')

# you can also pass encoding explicitly (utf-8 is default)
anymarkup.serialize_file(struct, 'foo/bar', format='json', encoding='ascii')

anymarkup's People

Contributors

bkabrda avatar nightwatchcyber avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

anymarkup's Issues

Able to detect what format?

Is there a function / command to be able to actually detect what format the file is in?

Ex.

print anymarkup.format(content)
"json"

EDIT: Just found _guess_fmt_from_bytes ๐Ÿ’ƒ nvm!

PyPI package doesn't include test/__init__.py

When I download the tarball on PyPI, the tests won't run:

test/test_parse.py:13: in <module>
    class TestParse(object):
test/test_parse.py:34: in TestParse
    (example_ini, example_as_dict),
E   NameError: name 'example_ini' is not defined

RFE: HCL Support

Hashicorp developed HCL as an alternative to JSON and YAML and allows its use with some of their tools. As some of our tools use anymarkup and interact with some Hashicorp tools, I would like anymarkup to support HCL.

When force_types=None used, strings with numeric values are changed to ints

IIUC this code should provide basically the same input regarding value types.

import anymarkup

data = anymarkup.parse_file("file.json", force_types=None)
anymarkup.serialize_file(data, "output.json")

But if you feed this file into it:
http://pastebin.centos.org/27686/
you'll get this:
http://pastebin.centos.org/27691/

The change can be seen in env object:

"env":[
                { "name": "SERVER_ID", "value": "1" }
              ],

which changes to

"env": [
                {
                  "name": "SERVER_ID",
                  "value": 1
                }
              ],

Which then fails to load by openshift/kubernetes parser as it expects value to be a string. I tried to do the same with plain json.load/json.dump and it preserves types correctly.

Consider tomlkit to support TOML markup

Description

Hi there, first of all thanks much for anymarkup. It's a great module and I'm using it for quite some time in the KIWI
appliance builder project. Recently there was the request to also support the TOML markup for describing an image description. I added a pull request to support this using anymarkup and its toml backend. You can see the open PR here:

As you can see it would add a new python module dependency to the toml module because that's how anymarkup
implements support for it. It all works nicely but a review comment was made stating that support for TOML has been
split into tomllib in the Python 3.11+ standard library for reading, and tomlib-w for writing.

From that perspective I'd like to ask if you would consider to support the new tomllib implementation as
alternative backend to support TOML with python 3.11+ ?

For us from the KIWI side this would be great as we don't need to add a dependency to the old toml module
and benefit from features of the new tomllib e.g style and comment preserving

Thanks much

Long key names result in odd line wrapping behavior when serializing to yaml

This might be part of the yaml spec, I am not sure. When I have a long key name it can end up wrapping like so:

- ? Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod 
    tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, 
    quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
  : 

Is there any way to keep that as a single line with standard formatting?

Also, is it possible to select the alternate list format?

Currently, A list will serialize into the format:

attributes:
  - itemA
  - itemB
  - itemC

I would like to be able to have it output in the format:

attributes: [itemA, itemB, itemC]

if possible.

Another weird case:
If my key in python is "3'b000 : 0", in the yaml serial it is '3''b000 : 0':The single quoting seems to be a bit odd to me. I am not sure if this is valid or not.

Idea: Provide a command line tool

First of all, i don't need this, I'm just proposing this on the "nice to have" level.
I think it could help anymarkup to gain more glory.

The ultimate /usr/bin/anymarkup

Reads from stdin or file, outputs to stdout or file. Has --from and --to optional format switches, guesses the format otherwise based on file extension and/or provided content. (Actually when writing to stdout, --to needs to be mandatory).

All the logic is already there, all this needs is a nice entry point and click.

Imagine the glory of this tool, you could do:

curl http://fedora.portingdb.xyz/stats.json | anymarkup --to yaml | anymarkup --to json

I might provide pull request if you approve the idea.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.