tamasfe/taplo

A TOML toolkit written in Rust

Website

Taplo

This is the repository for Taplo, a TOML v1.0.0 toolkit, more details on the website.

• Taplo
  • Status
  • Contributing

Status

The project is very young, so bugs and incomplete features are expected, so any help is welcome!

The correctness of the TOML parsing and decoding is not yet entirely guaranteed (as there is no official 1.0.0 compliance test suite yet).

Contributing

All kinds of contributions are welcome. Make sure to read the CONTRIBUTING.md first!

taplo's Issues

Cursor detection for symbols

If a top level table (or table in array) has only a single entry, the range of its value is not correctly set.

Documentation

Proper documentation must be written before releasing 1.0.0.
This issue is just simply a reminder and an entry in the checklist of the milestone.

Multi-line literal strings + CRLF line endings = False positive 'Invalid character in string'

On my Windows machine the Even Better TOML extension for VS Code reports a bunch of 'Invalid character in string' problems with the multi-line literal string (three single quotes) in this example pyproject.toml file taken from https://black.readthedocs.io/en/stable/pyproject_toml.html.

[tool.black]
line-length = 88
target-version = ['py37']
include = '\.pyi?$'
exclude = '''

(
  /(
      \.eggs         # exclude a few common directories in the
    | \.git          # root of the project
    | \.hg
    | \.mypy_cache
    | \.tox
    | \.venv
    | _build
    | buck-out
    | build
    | dist
  )/
  | foo.py           # also separately exclude a file named foo.py in
                     # the root of the project
)
'''

I suspect it has to do with line endings. When I convert from CRLF to LF the extension does not report any problems in this file.

`=` inside string breaks syntax coloring

Having a = inside any string (double or single quoted) breaks all syntax coloring after the string in a TOML file in VSCode.

Simplest reproduction:

1. Have any TOML file open in VSCode
2. Put test = "=" near top of file

I'm on newest VSCode on Windows 10 using Even Better TOML extension.

Incorrect indentation

Tables sharing the same keys are indented, but they shouldn't be.

For now indentation was disabled by default (can still be enabled) , as it is rarely seen in toml documents anyway.

Defining a key multiple times should be invalid as per spec.

This crate advertises as TOML v1.0.0-rc.1 compliant and the spec forbids defining a key multiple times however as of v1.0.0-alpha.4, i can define multiple keys:

# taplo = "1.0.0-alpha.4"

const SOURCE: &str =
"ip = '127.0.0.1'
ip = 'xxx'

[secret]
github = 'github-xxxxxxxxxxxxxxxxx'
travis = 'travis-xxxxxxxxxxxxxxxxx'
azure = 'azure-xxxxxxxxxxxxxxxxx'";

let rect = taplo::parser::parse(SOURCE);

assert_eq!(rect.errors.len(), 1);

Crates.io releases past 1.0.0-alpha.1 missing in Git history here?

Hey there! I'm currently using 1.0.0-alpha.6 to implement some CI tooling in a closed-source project, and I noticed that there seems to be no history in this repo containing releases marked in a Cargo.toml file beyond the 1.0.0-alpha.1 release. Is this just something still on somebody's local machine? It seems sort of unlikely, since options like allowed_blank_lines is now in master, yet not in a Crates.io release, and taplo/Cargo.toml is still set to 1.0.0-alpha.1.

Incorrect arrays are accepted

This is a regression that was caused by making the parser more tolerant to syntax errors, it ended up being a bit too tolerant.

TOML files are touch/modified even if no textual change was done

Basically this means that files are considered modified even when content wasn't changed which causes me to run rebuilds on every toml formatting.
maybe we can just not modify files if content isn't really changing.

Bare keys and quoted keys are not equivalent

About

Bare keys and quoted keys are equivalent on toml v1.0.0-rc.2 but taplo doesn't raise error

Reproduction

• create toml file and write

  # THIS WILL NOT WORK
  spelling = "favorite"
  "spelling" = "favourite"

• create multiple-key-2.toml in test-data/invalid and write

  # https://toml.io/en/v1.0.0-rc.2#keys
  # THIS WILL NOT WORK
  spelling = "favorite"
  "spelling" = "favourite"
  

  after that,run cargo run --bin test-gen --package test-gen -- -i ../../test-data -o ../../taplo/src/tests and run test at project root

Notes

I'm trying to make tests from TOML 1.0.0-rc.2 now. It may solve #3, but I don't know which ones in test-data folder are needed. (almost all are same as toml.io's, but I can't find some ones.) I create prototype of test data so please check and tell me I should proceed as it is or not if you can. (It refers to only toml.io and I've finished up to end of the keys section.)

Symbols Within Arrays

Provide symbols inside arrays in the LSP.

Support VS Code remote development with extensionKind

Hi,

Can you please give proper support for VS Code remote development by setting the proper value for extensionKind in the extension package.json file?

I believe the correct option for your "Even Better TOML" is:

{
  "extensionKind": ["workspace"]
}

But please read this documentation first and test on your side.

Kind rergards.

vscode Even Better TOML: Authors regex pattern for validation of pyproject.toml doesn't match email addresses surrounded by <>

When using Even Better TOML with vscode, I noticed that my author byline wasn't matching.

I had written: authors = ["Travis Bradshaw <[email protected]>"], which created an error that reads "the value doesn't match the given pattern".

I found the regex in the source to perform a cursory check, but after pasting the regex into a couple online validators it didn't seem to parse for them. (I think maybe this is a flavor of regex that the online validators didn't support? The discrepancies appeared to be very minor, but the regex for matching this field is a bit of a whopper and so I didn't investigate further.)

Through just trial and error, I found that "Travis Bradshaw [email protected]" does work, but "Travis Bradshaw [email protected]" does not.

Checking the spec for pyproject.toml, I think that this is in error: https://www.python.org/dev/peps/pep-0621/#authors-maintainers

Thank you for your time,

-- Travis

Add hover for values

Hover docs are only shown for keys right now.

Can't build webpack-server in `vscode`

I've tried to build taplo-ide in my environment for transplant to vim, but I got error.

I'm beginner in wasm and rust, So I can't find what cause this bug and evaluate my way to build is correct or not. I suspect this issue is related to this error but I'm not sure. Sorry if I'm wrong.

Error message

Debugger listening on ws://127.0.0.1:46543/63284125-1b94-4376-bde6-7c6f630fa7a9
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.

> [email protected] webpack-server /(project_root)/taplo-ide/vscode
> webpack --config webpack.config.server.js --mode production

Debugger listening on ws://127.0.0.1:45305/c9a15a5e-777e-41b3-8929-039fc99e2277
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Debugger listening on ws://127.0.0.1:35467/98246438-8f84-49c7-ae87-db298bc34fc3
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Waiting for the debugger to disconnect...
Hash: 7426d92239a271a038f0
Version: webpack 4.43.0
Time: 2249ms
Built at: 2020-09-14 17:43:27
 3 assets
Entrypoint main = server.js server.js.map
[0] ../pkg/taplo_ide_bg.js 9.99 KiB {0} [built]
[1] ../pkg/taplo_ide_bg.wasm 3.01 MiB {0} [built]
[2] ./src/server.ts 599 bytes {0} [built]
[3] ../pkg/taplo_ide.js 79 bytes {0} [built]
[4] (webpack)/buildin/harmony-module.js 573 bytes {0} [built]
[5] external "fs" 42 bytes {0} [built]

ERROR in ../pkg/taplo_ide_bg.wasm
WebAssembly module is included in initial chunk.
This is not allowed, because WebAssembly download and compilation must happen asynchronous.
Add an async splitpoint (i. e. import()) somewhere between your entrypoint and the WebAssembly module:
* ./src/server.ts --> ../pkg/taplo_ide.js --> ../pkg/taplo_ide_bg.wasm
* ... --> ../pkg/taplo_ide.js --> ../pkg/taplo_ide_bg.js --> ../pkg/taplo_ide_bg.wasm
* ... --> ../pkg/taplo_ide.js --> ../pkg/taplo_ide_bg.wasm --> ../pkg/taplo_ide_bg.js --> ../pkg/taplo_ide_bg.wasm
* ... --> ../pkg/taplo_ide_bg.js --> ../pkg/taplo_ide_bg.wasm --> ../pkg/taplo_ide_bg.js --> ../pkg/taplo_ide_bg.wasm
Debugger listening on ws://127.0.0.1:45305/c9a15a5e-777e-41b3-8929-039fc99e2277
For help, see: https://nodejs.org/en/docs/inspector
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] webpack-server: `webpack --config webpack.config.server.js --mode production`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the [email protected] webpack-server script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /$HOME/.npm/_logs/2020-09-14T08_43_27_211Z-debug.log

Reproduction

#(on project root directory)
cargo build
cd cargo-ide
cargo build
wasm-pack build
cd vscode
npm install -D
npm webpack

versions

cargo 1.45.1
rustup 1.22.1
wasm-pack 0.9.1
npm 6.14.6
node 12.18.3

Properly handle dotted keys and arrays of tables

Right now everything is based on the DOM of the TOML documents, which makes sense for validation, but less so for completion.

The problem is that tables created with dotted keys and arrays of tables are merged into one, so additional spans are lost, as visible from the keys below:

Completion of dotted keys also don't work if there is already an entry sharing the same dotted key parent tables.

A solution would be to either include the additional information in the DOM, or just use the syntax for completion, or probably both.

Invalid syntax causes vscode extension to freeze entirely

I've been converting my YAML config files into TOML, so I naturally stumbled across this while renaming .yml files to .toml. Once I rename any file the VSCode extension stops working entirely. If I switch to a different TOML file it also stops for that file. Restarting VSCode appears to solve it, but this is a rather annoying bug.

Serialization

Taplo should support some sort of TOML encoding, as it shouldn't be too hard to implement.

Autocompletion erases the end of inline tables

Indexed path completion is not handled properly

Disable `alignEntries` by default

I believe that the evenBetterToml.formatter.alignEntries setting should be disabled by default. The reason:

Everyone uses version control systems (e.g. git) these days. Most people don't align keys/values in their Cargo.toml. This means that contributors have to disable the setting anyway, so their PRs don't reformat the entire Cargo.toml file.

On the other hand, if you are the maintainer of a crate and decide that keys/values should be aligned, contributors either have to use "Even Better TOML", or they have to manually align it. Furthermore, if they use cargo add, they have to reformat the file afterwards to make sure that the added value is properly aligned.

There is another problem: If you add very_long_dependency_name = "*", all keys must be moved to the right. Even though it modifies only whitespace, the risk of merge conflicts is increased.

That's why I think that aligning entries should be discouraged by making the setting opt-in.

---

Note that rustfmt has a similar setting, struct_field_align_threshold, which is set to 0 by default, so struct fields are not aligned by default in Rust.

Add enum links

Enum links in schemas are currently ignored.

Alter settings for a specific part of a document

E.g. disable formatting for a few lines, or just disable sorting for a few entries, and so on.

This could be done with comments like # taplo:ignore but this either gets very limiting or adds too much noise in the comments.

I would much prefer a way to do this with the help of #19.

Unexpected error on publish in Cargo.toml

With the below content in a Cargo.toml file, i get an error displayed on the true value of publish.
The error message is, or the messages are:

the value must be valid for at least one of the 2 schemas - Even Better TOML (Publish)
-----
invalid type: expected array, not boolean - Even Better TOML
-----
invalid enum value, expected to be one of {false} - Even Better TOML (Disable)

I guess this has something to do with the Cargo.toml scheme?

You can reproduce this with the following content.

[package]
name = "async-graphql-rocket"
version = "1.17.10"
authors = [""]
edition = "2018"
description = "async-graphql for Rocket.rs"
publish = true  # this is where the error occurs
license = "MIT/Apache-2.0"
documentation = "https://docs.rs/async-graphql/"
homepage = "https://github.com/async-graphql/async-graphql"
repository = "https://github.com/async-graphql/async-graphql"
keywords = ["futures", "async", "graphql", "rocket"]
categories = ["network-programming", "asynchronous"]

Cargo.toml dependency keys shouldn't be links

This should be removed during schema generation.

Request: Opt-out of collapsing multiple lines of whitespace between sections

I'm really enjoying working with this extension, but when I save a modified TOML file I find that multiple lines (two in my case) of whitespace between sections are collapsed into one.

I looked, but didn't find an existing setting to opt out of this behavior. Is there an option for this that I've overlooked?

Thanks in advance and also for creating the Even Better TOML VSCode extension!

Comment formatting is broken when comments have = in them

For example:

foo = { bar = "" }# what = "even"

renders as

Don't highlight arrays and objects differently

This is how my [dependencies] section looks like:

I don't think that the different colors are useful, I find them distracting. It would be better to use the same color for all keys. If someone wants different colors, they can customize them using the semantic token types.

Remove recursions

I thought I could get away with using recursive functions to build trees while parsing.

Well, I was wrong, as one of the first TOML tests contained 1000 nested arrays and inline tables which caused a stack overflow.

While 1000s of nested data in TOML shouldn't appear out in the wild, this is definitely something I want to do about for 1.0.0.

The code parts related to this issue are marked with FIXME(recursion) in the comments.

Configuration file

Formatter and linter options should be configurable from an .rc file (taplo.toml?).

[BUG] Wrong Poetry schema

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
isort = "*"
flake8 = "*"

generates such an error:

the required property "python" is missing Even Better TOML

which is not true - python property is only needed in dependencies section.

Allow non-local schemas

Allow external (http, https) schemas for associations.

Do not format comment in array

std = [
	# crates
	"codec/std",
	"serde",
	"serde_json", 
	# darwinia
	"array-bytes/std",
	"darwinia-support/std", 
	# substrate
	"frame-support/std",
	"frame-system/std",
	"sp-io/std",
	"sp-runtime/std",
	"sp-std/std",
]

will be

std = [
	# crates
	"codec/std",
	"serde",
	"serde_json", # darwinia
	"array-bytes/std",
	"darwinia-support/std", # substrate
	"frame-support/std",
	"frame-system/std",
	"sp-io/std",
	"sp-runtime/std",
	"sp-std/std",
]

Reported errors in VSCode aren't removed when files are deleted.

When a TOML file with syntax errors is deleted, the errors aren't removed and persist in the folder icon and problems tab. I'll attach screenshots below.

Incorrect warning about duplicate keys in array-of-table syntax (vscode extension)

the TOML spec says the following is valid:

[[thing]]
    [thing.test]
    test = 'other data'


[[thing]]
    [thing.test]
    test = 'data'

both tomljson (from the go-toml lib) and python's toml lib: python3 -c "import sys, toml, json; print(json.dumps(toml.loads(sys.stdin.read()), indent=2))" both parse this correctly to the following json

{
  "thing": [
    {
      "some_data": "data!",
      "table": {
        "test": "other data"
      }
    },
    {
      "some_data": "data!",
      "table": {
        "test": "data"
      }
    }
  ]
}

However the VSCode extention indicates that thing.table and test under it are duplicate keys with the above example:

Add an option to control array space

Add an option to control the space between [ ] and data.

# array_filling_space = 0
array_compact = [0, [1], [[2]]]
# array_filling_space = 1
array_free        = [ 0, [ 1 ], [ [ 2 ] ] ]

Escapes in keys are ignored

"\"quotes\"" = "also \"quotes\""

This is turned into the following JSON:

{
  "\\\"quotes\\\"": "also \"quotes\""
}

Duplicate fields in completions

Don't show the same fields for oneOf schemas in completions.

Support 1.0.0-rc.3

See what changed in rc2, and update accordingly if needed.

Question: standalone LSP

I'm planning to create standalne lsp with taplo, taplo-ide and lsp-server crate from rust-analyzer.
But If taplo will soon include standalone lsp feature or @tamasfe will create new crate, I think it's better to contribute to it instead of mine. Please tell me if you have loadmap about standalone lsp.

relation?: #39 #33

standalone formatter

can this be packaged as a binary crate and be used as a toml formatter?
getting a glob path and than formatting all toml files based on that path?

Proper Test Suite

The current tests are not up to date with TOML 1.0.0-rc.1, and are not entirely correct.

The library will need proper TOML test examples with example JSON outputs.

Unstable cargo features not supported by the vscode extension

Cargo nightly has unstable features, and they don't seem to be supported by the vscode extension

For example, in my case, after enabling the profile strip option with

cargo-features = ["strip"]

The following code block still emits an error unknown property Even Better TOML (Profile) pointing at strip

[profile.release]
strip = "symbols"

Here's a screenshot of the error

Section names "inherit" links of their children

When hovering over a section name, it displays the links of its children:

Extension has absurdly high resource usage

I'm connecting to a VPS using the VSCode remote extension, and this extension has to be installed on the VPS to work. But when I do that, my VPS is maxed at 100% CPU and memory usage (cpu never goes above 5% normally, and memory is never higher than 20%). Uninstalling the extension instantly fixes the resource usage and installing it again makes the usage quickly return, so I'm fairly certain this extension is the cause. This is definitely a low-end VPS, but this still seems absurd.

Running cli with --help exists with exit code 1

You have a run function that is called and if it returns false, you exit with 1.
but help invocation which is valid, returns false as well.
so doing taplo --help or taplo format --help always exists with 1.

Add an option to turn off entry alignment

It is often undesirable to align on items as shown on the following image, it would be nice to be able to format with one whitespace around the equals sign.

Tables plus dotted keys generates incorrect error (vscode extension)

I am running version 0.8.7 as reported in vscode and listed at time of writing as the up to date version on the vscode marketplace.
I am running VSCode 1.50.1 on Windows 10 x64.

If I mix both fully qualified table syntax and table plus dotted key syntax, the "Even Better TOML" extension appears to incorrectly give me a duplicate table definition error if the first subtable matches a top level table name.
While there is no explicit sentence that I can find in the TOML spec, either rc1, or (current) rc3, stating it's not an error, the spec does appear to specify (and the code appears to handle this in isolation) that dotted keys under a table definition append under that table, not the global scope, and thus I would expect the intended result would not have it error about a conflict with a global table. Especially since using a dotted table definition works as expected.
TOML https://toml.io/en/v1.0.0-rc.3#table has an explicit example of dotted keys inside a table, and the example is the same in https://toml.io/en/v1.0.0-rc.1#table.

Hopefully, these examples make it clear what the issue is.

Invalid per the extension:

[foo]
bar = 1

[baz]
foo.bar = 2 # Not Ok per the extension, conflicts with table "foo"

[bar.foo] # Ok
baz = 3

Expected JSON Tree:

{
  "foo": {
    "bar": 1
  },
  "baz": {
    "foo": {
      "bar": 2
    }
  },
  "bar": {
    "foo": {
      "baz": 3
    }
  }
}

Do note that if I copy just the middle TOML snippet:

[baz]
foo.bar = 2

as JSON, the contents of my clipboard are the expected:

{
  "baz": {
    "foo": {
      "bar": 2
    }
  }
}

JSON Schema

I'm planning to implement JSON schema validation, but there doesn't seem to be an actively maintained validation library that is up to date with the latest drafts.

I've started working on one, but it's rather tedious, and will take some time.

Table and array table conflict is allowed

This is currently incorrectly accepted: