dlang-community/d-scanner

Swiss-army knife for D source code

D-Scanner

D-Scanner is a tool for analyzing D source code

Building and installing

First make sure that you have all the source code. Run git submodule update --init --recursive after cloning the project.

To build D-Scanner, run make (or the build.bat file on Windows). The build time can be rather long with the -inline flag on front-end versions older than 2.066, so you may wish to remove it from the build script. The makefile has "ldc" and "gdc" targets if you'd prefer to compile with one of these compilers instead of DMD. To install, simply place the generated binary (in the "bin" folder) somewhere on your $PATH.

Testing

Testing does not work with DUB. Under linux or OSX run the tests with make test. Under Windows run the tests with build.bat test.

Installing with DUB

> dub fetch dscanner && dub run dscanner

Installing with Docker

With Docker no installation is required:

docker run --rm -v $(pwd):/src dlangcommunity/dscanner

Usage

The following examples assume that we are analyzing a simple file called helloworld.d

import std.stdio;
void main(string[] args)
{
	writeln("Hello World");
}

Linting

Use

dscanner lint source/

to view a human readable list of issues.

Diagnostic types can be enabled/disabled using a configuration file, check out the --config argument / dscanner.ini file for more info. Tip: some IDEs that integrate D-Scanner may have helpers to configure the diagnostics or help generate the dscanner.ini file.

Auto-Fixing issues

Use

dscanner fix source/

to interactively fix all fixable issues within the source directory. Call with --applySingle to automatically apply fixes that don't have multiple automatic solutions.

Tooling integration

Many D editors already ship with D-Scanner.

For a CLI / tool parsable output use either

dscanner -S source/
# or
dscanner --report source/

The --report switch includes all information, plus cheap to compute autofixes that are already resolved ahead of time, as well as the names for the autofixes that need to be resolved using the --resolveMessage switch like described below.

You can also specify custom formats using -f / --errorFormat, where there are also built-in formats for GitHub Actions:

# for GitHub actions: (automatically adds annotations to files in PRs)
dscanner -S -f github source/
# custom format:
dscanner -S -f '{filepath}({line}:{column})[{type}]: {message}' source/

To resolve automatic issue fixes for a given location use

# collecting automatic issue fixes
# --resolveMessage <line>:<column> <filename>
dscanner --resolveMessage 11:3 file.d
# --resolveMessage b<byteIndex> <filename>
dscanner --resolveMessage b512 file.d
# <filename> may be omitted to read from stdin

outputs JSON:

// list of available auto-fixes at the given location
[
	{
		"name": "Make function const",
		// byte range `[start, end)` what code to replace
		// this is sorted by range[0]
		"replacements": [
			// replace: range[0] < range[1], newText != ""
			{"range": [10, 14], "newText": "const "},
			// insert: range[0] == range[1], newText != ""
			{"range": [20, 20], "newText": "auto"},
			// remove: range[0] < range[1], newText == ""
			{"range": [30, 40], "newText": ""},
		]
	}
]

Algorithm to apply replacements:

foreach_reverse (r; replacements)
	codeBytes = codeBytes[0 .. r.range[0]] ~ r.newText ~ codeBytes[r.range[1] .. $];

Replacements are non-overlapping, sorted by range[0] in ascending order. When combining multiple different replacements, you first need to sort them by range[0] to apply using the algorithm above.

Other features

Token Count

The "--tokenCount" or "-t" option prints the number of tokens in the given file

$ dscanner --tokenCount helloworld.d
20

Import Listing

The "--imports" or "-i" option prints a listing of modules imported by the given source file.

$ dscanner --imports helloworld.d
std.stdio

Passing "-I" arguments (import locations) will cause D-Scanner to also attempt to resolve the locations of the imported modules.

$ dscanner --imports helloworld.d -I ~/.dvm/compilers/dmd-2.071.1-b2/src/phobos/ -I ~/.dvm/compilers/dmd-2.071.1-b2/src/druntime/src/
/home/brian/.dvm/compilers/dmd-2.071.1-b2/src/phobos/std/stdio.d

Remember to pass map the import locations when you use Docker:

docker run --rm -v $(pwd):/src -v /usr/include/dlang/dmd:/d dlangcommunity/dscanner --imports helloworld.d -I/d
/d/std/stdio.d

The "--recursiveImports" option is similar to "--imports", except that it lists imports of imports (and so on) recursively. The recursive import option requires import paths to be specified in order to work correctly.

Limitations:

• The import listing feature DOES NOT IGNORE imports that may be unused to to version or static if.
• The import listing DOES NOT INCLUDE imports introduced by mixins.

Syntax Check

The "--syntaxCheck" or "-s" option prints a listing of any errors or warnings found while lexing or parsing the given source file. It does not do any semantic analysis and it does not compile the code. The format of the errors or warnings can be configured with the "--errorFormat" or "-f" option.

Style Check

The "--styleCheck" or "-S" option runs some basic static analysis checks against the given source files, the sources contained in the given folders, or the sources contained in the current working directory (when nothing is supplied). The format of the errors or warnings can be configured with the "--errorFormat" or "-f" option.

Skip style checks in the tests

Static checks in the unit tests can produce irrelevant warnings. For example, it's legit to declare a variable that's not used if the goal is to verify that a templatized function can be instantiated by inference of the type of this variable. To avoid these cases, it's possible to pass the "--skipTests" option.

Configuration

By default all checks are enabled. Individual checks can be enabled or disabled by using a configuration file. Such a file can be placed, for example, is the root directory of your project. Running dscanner --defaultConfig will generate a default configuration file and print the file's location. You can also specify the path to a configuration file by using the "--config" option if you want to override the default or the local settings.

For each check, three values are possible:

• "disabled": the check is not performed.
• "enabled": the check is performed.
• "skip-unittest": the check is performed but not in the unit tests.

Any other value deactivates a check.

Note that the "--skipTests" option is the equivalent of changing each "enabled" check by a "skip-unittest" check.

Implemented checks

• Old alias syntax (i.e "alias a b;" should be replaced with "alias b = a;").
• Implicit concatenation of string literals.
• Complex number literals (e.g. "1.23i").
• Empty declarations (i.e. random ";" characters).
• enum array literals in struct/class bodies.
• Avoid Pokémon exception handling.
• opCmp or opEquals, or toHash not declared "const".
• Format numbers for readability.
• delete keyword is deprecated.
• "fish operators" (floating point operators) are deprecated.
• Left side of a foreach or foreach_reverse range expression is larger than the right.
• Left side of a slice expression is larger than the right.
• Variable, struct, class, union, module, package, and interface names that do not comply with Phobos style guidelines.
• Struct constructors that have a single parameter that has a default argument.
• Assign expressions where the left side of the '=' operator is the same as the right.
• 'if' statements where the 'else' block is the same as the 'if' block.
• ||, &&, and == expressions where the left and right sides of the operator are identical.
• && and || expressions where the order of operations is confusing.
• Unused variables.
• Unused parameters (check is skipped if function is marked "override").
• Duplicate attributes.
• Declaring opEquals without toHash.
• Undocumented public declarations.
• Subtraction from .length properties. (These may be unsigned and could lead to integer underflow)
• Class, struct, and union member variables whose names conflict with built-in type properties.
• Confusing asm syntax.
• Placement of const, immutable, or inout before a function return type instead of after the parameters.
• Functions in interface declarations redundantly marked 'abstract'.
• Declaring a variable with the same name as a label.
• Variables that could have been declared const or immutable (experimental)
• Redundant parenthesis.
• Unused labels.
• Lines longer than max_line_length characters.
• Incorrect infinite range definitions.
• Some assertions that check conditions that will always be true.
• Auto functions without return statement. The compiler doesn't see an omission and it infers 'void' as return type.
• final attribute is used but in this context it's a noop.
• Check for properly documented public functions ("Returns" and "Params" sections). Initially implemented to lint Phobos. By default disabled.
• Check for explicitly annotated unittests (@system or @safe). Initially implemented to lint Phobos. By default disabled.
• Check for that imports are sorted. Initially implemented to lint Phobos. By default disabled.
• Virtual calls inside classes constructors.
• Useless initializers.
• Allman brace style
• Redundant visibility attributes
• Public declarations without a documented unittest. By default disabled.
• Asserts without an explanatory message. By default disabled.
• Indentation of if constraints
• Check that @trusted is not applied to a whole scope. Trusting a whole scope can be a problem when new declarations are added and if they are not verified manually to be trustable.
• Redundant storage class attributes
• Cyclomatic complexity threshold per function and unittest (starts at 1, increased by 1 at each if, switch case, loop, &&, ||, ?:, throw, catch, return, break, continue, goto and function literal)

Wishlist

See this list of open issues for the wishlist.

Reports

The "--report" option writes a JSON report on the static analysis checks document above to standard output. This file is usually used by the D plugin for SonarQube located here.

Using option "--reportFormat sonarQubeGenericIssueData" a report in a sonar-scanner supported Generic Issue Data format can be created.

$ dscanner --reportFormat sonarQubeGenericIssueData . > sonar-generic-issue-data.json

Reference the report filename in sonar-project.properties using key "sonar.externalIssuesReportPaths"

sonar.externalIssuesReportPaths=sonar-generic-issue-data.json

Find Declaration

Ack, grep, and The Silver Searcher are useful for finding usages of symbols, but their signal to noise ratio is not very good when searching for a symbol's declaration. The "--declaration" or "-d" options allow you to search for a symbols declaration. For example:

$ dscanner -d TokenStructure
./libdparse/src/std/lexer.d(248:8)

Line of Code Count

The "--sloc" or "-l" option prints the number of lines of code in the file. Instead of simply printing the number of line breaks, this counts the number of semicolon, while, if, do, else, switch, for, foreach, foreach_reverse, default, and case tokens in the file.

$ ./dscanner --sloc helloworld.d
2

Syntax Highlighting

The "--highlight" option prints the given source file as syntax-highlighted HTML to the standard output. The CSS styling uses the Solarized color scheme by default, but can be customised using the "--theme" option.

The following themes are available:

• solarized

• solarized-dark

• gruvbox

• gruvbox-dark

  No example. It would take up too much space

CTAGS Output

The "--ctags" or "-c" option generates CTAGS information and writes it to the standard output. Directory arguments are scanned recursively for .d and .di files.

$ dscanner --ctags helloworld.d
!_TAG_FILE_FORMAT	2
!_TAG_FILE_SORTED	1
!_TAG_FILE_AUTHOR	Brian Schott
!_TAG_PROGRAM_URL	https://github.com/Hackerpilot/Dscanner/
main	helloworld.d	3;"	f	arity:1

CTAGS output uses the following tag kinds:

• g -- enum declarataion
• e -- enum member
• v -- variable declaration
• i -- interface declaration
• c -- class declaration
• s -- struct declaration
• f -- function declaration
• u -- union declaration
• T -- template declaration
• a -- alias declarataion

More information on the CTAGS format can be found here.

Etags Output

The --etags, -e, and --etagsAll options are similar to --ctags except that an Emacs-compatible tags file is generated. The --etagsAll option generates tags for private and package declarations in addition to what --etags and -e generate.

Outline

The "--outline" option parses the given D source file and writes an simple outline of the file's declarations to stdout.

Configuration

If a dscanner.ini file is locate in the working directory or any of it's parents, it overrides any other configuration files.

As final location, D-Scanner uses the configuration file given in $HOME/.config/dscanner/dscanner.ini. Run --defaultConfig to regenerate it.

The --config option allows one to use a custom configuration file path.

AST Dump

The "--ast" or "--xml" options will dump the complete abstract syntax tree of the given source file to standard output in XML format.

$ dscanner --ast helloworld.d

<module>
<declaration>
<importDeclaration>
<singleImport>
<identifierChain>
<identifier>std</identifier>
<identifier>stdio</identifier>
</identifierChain>
</singleImport>
</importDeclaration>
</declaration>
<declaration>
<functionDeclaration line="3">
<name>main</name>
<type pretty="void">
<type2>
void
</type2>
</type>
<parameters>
<parameter>
<name>args</name>
<type pretty="string[]">
<type2>
<symbol>
<identifierOrTemplateChain>
<identifierOrTemplateInstance>
<identifier>string</identifier>
</identifierOrTemplateInstance>
</identifierOrTemplateChain>
</symbol>
</type2>
<typeSuffix type="[]"/>
</type>
<identifier>args</identifier>
</parameter>
</parameters>
<functionBody>
<blockStatement>
<declarationsAndStatements>
<declarationOrStatement>
<statement>
<statementNoCaseNoDefault>
<expressionStatement>
<expression>
<assignExpression>
<functionCallExpression>
<unaryExpression>
<primaryExpression>
<identifierOrTemplateInstance>
<identifier>writeln</identifier>
</identifierOrTemplateInstance>
</primaryExpression>
</unaryExpression>
<arguments>
<argumentList>
<assignExpression>
<primaryExpression>
<stringLiteral>Hello World</stringLiteral>
</primaryExpression>
</assignExpression>
</argumentList>
</arguments>
</functionCallExpression>
</assignExpression>
</expression>
</expressionStatement>
</statementNoCaseNoDefault>
</statement>
</declarationOrStatement>
</declarationsAndStatements>
</blockStatement>
</functionBody>
</functionDeclaration>
</declaration>
</module>

For more readable output, pipe the command through xmllint using its formatting switch.

$ dscanner --ast helloworld.d | xmllint --format -

Selecting modules for a specific check

It is possible to create a new section analysis.config.ModuleFilters in the dscanner.ini. In this optional section a comma-separated list of inclusion and exclusion selectors can be specified for every check on which selective filtering should be applied. These given selectors match on the module name and partial matches (std. or .foo.) are possible. Moreover, every selectors must begin with either + (inclusion) or - (exclusion). Exclusion selectors take precedence over all inclusion operators. Of course, for every check a different selector set can given:

[analysis.config.ModuleFilters]
final_attribute_check = "+std.foo,+std.bar"
useless_initializer = "-std."

A few examples:

• +std.: Includes all modules matching std.
• +std.bitmanip,+std.json: Applies the check only for these two modules
• -std.bitmanip,-std.json: Applies the check for all modules, but these two
• +.bar: Includes all modules matching .bar (e.g. foo.bar, a.b.c.barros)
• -etc.: Excludes all modules from .etc
• +std,-std.internal: Includes entire std, except for the internal modules

d-scanner's Issues

Lexer does not properly handle token strings

This can be observed by running echo "q{test}" | dscanner --highlight

Build error in autocomplete.d

autocomplete.d(237): Error: function langutils.combineTokens (ref const(Token[]) tokens) is not callable using argument types (const(Token)[])
/usr/include/d/std/range.d(611): Error: static assert "Cannot put a char[] into a Appender!(char[])"
/usr/include/d/std/array.d(1493): instantiated from here: put!(Appender!(char[]), char[])
autocomplete.d(341): instantiated from here: join!(MapResult!(unaryFun, const(Token)[]), string)
make: *** [all] Error 1

Properly handle unnamed enums

Currently Dscanner doesn't parse enum members in situation like
enum { ... }
i.e. when no name is given. Dscanner supposes there is always a name (if I understand correctly)

alias do not appear in JSON output

I tried Dscanner with the --json flag and alias declarations

alias int MyInt;

are not recognized. They do not appear in the JSON file, at least.

More weird D syntax that the parser rejects

From the Libevent D bindings in Deimos:

mixin("*msg.base." ~ member ~ "_assign")(msg, value);

I'm guessing this is a function call?

parser rejects valid: class B : public A {}

class A{}
class B : public A {}

Parser does not support user-defined attributes

Introduced to the language in 2.061

Is there a way for easy communication?

I've sent you an email (found on dlang.org), but probably there is some better way to contact you?
Also, what is your time zone? When would it be (potentially) convenient to ask questions, if I have any?

parser rejects valid: #!/usr/bin/rdmd

#!/usr/bin/rdmd

Parser does not support declaring multiple variables in a single statement

int a, b;
produces this json output

"variables" : [
  {
    "name" : "a",
    "line" : 1,
    "protection" : "public",
    "attributes" : [
    ],
    "type" : "int"
  }
],

Recursive option for ctags generation

Add a --recursive option to the ctags generation option so that a tags file can be generated for an entire directory at once. Switch should be --recursive or -R

ctags doesn't work with phobos

dscanner --ctags dmd_root/src/phobos/std/algorithm.d > out_ctags.txt
std.format.FormatException@dmd_root/src/phobos/std/string.d(2536): Orphan format arguments: args[4..5]

parser rejects valid: align(16) uint a = 0;

align(16) uint a = 0;
(originally from phobos:
align(16) private immutable uint[4] bswap_shufb_ctl =
[
0x0001_0203, 0x0405_0607, 0x0809_0a0b, 0x0c0d_0e0f
];
)

Funny characters in types.d

In static this(), where type properties are defined, in every instance of string "double".

Where is option of ctags output format for emacs?

ctags for etags format option is -e.
How in Dscanner?

Automated tests are needed

It would be great to have automated smoke tests to check that most important functionality is not broken after changes.

If not possible, at least manual instructions what to check and how.

Minor discrepancies with specification when lexing integers.

Implementation accepts some invalid inputs. I'm going to fix this in a way that corresponds to specification and will ask for review.

parser rejects valid: template A(alias T: B){}

class B{}
template A(alias T: B){}

libevent doesn't work

Hi. Trying to run --ctags on some dub packages:

~/code/d/Dscanner/dscanner /home/simendsjo/.dub/packages/libevent-master --recursive --ctags
[1]    1316 segmentation fault (core dumped)  ~/code/d/Dscanner/dscanner /home/simendsjo/.dub/packages/libevent-master

Lexer does not properly handle special token sequences

http://dlang.org/lex.html#Special%20Token%20Sequence

"structs" appears twice in JSON output

I just tried the --json flag and it appears to output twice the "structs":{} pair.

Switch statement and With statement

DMD 2.063 accepts the following:

enum Foo { Bar, Baz }

void main(string[] args)
{
    Foo foo;
    final switch (foo) with (Foo)
    {
        case Bar: break;
        case Baz: break;
    }
} 

Dscanner expects a '{' after the switch test and reports the following errors when using ---syntaxCheck:

test_switch.d(7:21)[error]: Expected { instead of with
test_switch.d(0:1)[error]: Expected } instead of EOF

End of file is not handled in a way described in D specification.

I will handle this myself if you don't mind (that's pretty straightforward).

Dscanner --json doesn't print variables

dscanner doesn't output variables info with --json option.

it would be useful if it can return info about a,b inside function main.

int main()
{
int a;
string b;
return 0;
}

compile error : isAlpha

stdx/d/lexer.d(1663): Error: std.uni.isAlpha at phobos/std/uni.d(7063) conflicts with std.ascii.isAlpha at phobos/std/ascii.d(92)

I get this error after upgrading to git master.
should it be std.uni.isAlpha or std.ascii.isAlpha?

Assertion when no file is specified on --sloc

Dscanner crashes with command dscanner --sloc, when no args are specified. Go make it read from stdin! ;)

parser rejects valid: auto timeout=0.msecs;

void test(){
  import core.time:msecs;
  auto timeout=0.msecs;  
}

Output etags for using with Emacs?

This is on behalf of my brother who uses Emacs, would it be possible to also output etags so that Emacs can use them?

Feature Request: Support parsing files from stdin

Integrating completion with an editor will require saving the file (either directly or to a temporary location) before running dscanner. If you support reading the file from stdin as an alternative to a file, you can avoid this issue.

parser rejects valid: 'version(none): '

void fun(){}

version(none):
void fun2(){}

parser rejects valid: @(A) void fun(){}

struct A{}
@(A) void fun(){}

parser rejects valid: foreach(i;0..100) auto t=fun();

int fun(){return 0;}
void test(){
foreach(i;0..100)
auto t=fun();
}

parser rejects valid: 'ubyte a[1]; '

ubyte a[1];

=> error: C-style variable declarations are not supported.

parser rejects valid: 'mixin(`1`).writeln;

import std.stdio;
void test(){
  mixin(`1`).writeln;
}

Will imaginary (and complex) literals be lexed?

They are part of specification, but I saw somewhere that it is planned to deprecate them.

parser rejects valid: enum a=A!3.length;

template A(int stop) {
alias int[3] A;
}
void test(){
enum a=A!3.length;
}

More AST classes should have location information

Location information makes the AST more useful for applications such as static code analysis. Right now it's possible to use the --ast option and xmllint to do things like search for if statements without braces, but you can't say where in the source file it happened.

parser rejects valid: '0.reduce!max([1])'

import std.algorithm;
void test(){
  0.reduce!max([1]);
}

parser.messageFunction: should accept an extra argument to distinguish bw errror vs warning

errors are not recoverable, warnings are, so the API should distinguish them to make it easy for client code using the parser

Linker error

I'm on windows and get these errors/warning if I run the build.bat:

C:\Users\Besitzer\Documents\Github\Dscanner>dmd main.d stats.d imports.d highlig
hter.d ctags.d astprinter.d outliner.d stdx/d/lexer.d stdx/d/parser.d stdx/d/ent
ities.d stdx/d/ast.d -wi -ofdscanner
stdx\d\parser.d(390): Warning: statement is not reachable
stdx\d\parser.d(405): Warning: statement is not reachable
stdx\d\parser.d(419): Warning: statement is not reachable
stdx\d\parser.d(433): Warning: statement is not reachable
stdx\d\parser.d(452): Warning: statement is not reachable
stdx\d\parser.d(466): Warning: statement is not reachable
stdx\d\parser.d(480): Warning: statement is not reachable
stdx\d\parser.d(494): Warning: statement is not reachable
stdx\d\parser.d(508): Warning: statement is not reachable
stdx\d\parser.d(526): Warning: statement is not reachable
stdx\d\parser.d(540): Warning: statement is not reachable
stdx\d\parser.d(554): Warning: statement is not reachable
stdx\d\parser.d(591): Warning: statement is not reachable
stdx\d\parser.d(611): Warning: statement is not reachable
stdx\d\parser.d(625): Warning: statement is not reachable
stdx\d\parser.d(3680): Warning: statement is not reachable
stdx\d\parser.d(5655): Warning: statement is not reachable
OPTLINK (R) for Win32 Release 8.00.13
Copyright (C) Digital Mars 1989-2010 All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
dscanner.obj(dscanner)
Error 42: Symbol Undefined _D9formatter7__arrayZ
dscanner.obj(dscanner)
Error 42: Symbol Undefined _D9formatter12__ModuleInfoZ
--- errorlevel 2

parsing templates

I have started using Dscanner to extract info about phobos modules. Currently not everything seems to be parsed. For example, in std.functional, the only functions I get back from Dscanner are adjoin and toDelegate.

Is this a known parsing issue (something to do with templates maybe) or must I be doing something wrong to not see all the function definitions?

Thanks and great tool!

parser rejects valid: alias extern(C) int function(int) foo;

alias extern(C) int function(int) foo;

parser rejects valid: writeln(0,)

import std.stdio;
void main(){
  writeln(0,);
}

Assertion failure in parser.d, line 32

This file triggers an assertion failure, here is a link, sorry if this is somewhat nonstandard:
https://dl.dropbox.com/u/5629472/abstractmenu.d
The D language provides few information about exception, so I currently don't know how to properly find it.

Range violation in function declaration parsing

void main(string[] args) causes a range violation.

Feature request: provide mechanism to know whether errors have been found

Some editors can display errors as you type. E.g., Visual Studio. Other tools might also benefit from having diagnostic information available.

Example design would be to add a parameter to tokenize():
Token[] tokenize(S)(S inputString, void delegate(string) logMessage, IterationStyle iterationStyle = IterationStyle.CODE_ONLY) if (isSomeString!S)

logMessage() would by default write to stderr.

Of course, details may vary, the concept could be generalized or implemented totally differently. For example, you might want to add command line arguments to specify logFileName or to pass formatString. Error code could be returned from main to indicate that error log is not empty.

On at least Windows, --highlight emits twice as many newlines as it should

And it looks really silly!

Hex strings are handled incorrectly: x"AB" should be treated like "\xAB"

I've put comment in source file to implement correct behavior later (whoever will do that)

please make parser/tokens public field; it's useful for clients

parser rejects valid: void foo(alias fun=a=>a) () {}

void foo(alias fun=a=>a) () {}