Hello Can ctags skip the new module import statements in C++ 20 plea

Oh, thank you, this is great ! If I don't use <code class="notransla

C++: support modules,about universal-ctags/ctags

Comments (42)

masatake commented on June 15, 2024 1

Eventually, we have to use C language. However, using regex for prototyping is an excellent idea.

I'll extend your code for discussion.

from ctags.

masatake commented on June 15, 2024 1

Can you please add the module: field to C++ for this purpose ? Is it possible for users to add it using the command line, for use with regexp tags ?

Could you show me a pair of (input.cpp and expected.tags) ?

  --_fielddef-<LANG>=<name>,<description>
       Define new field for <LANG>.
       Make NEWLANG parser pretend OLDLANG parser in lang: field.

This may be what you want. However, I'm not sure this is enough for your purpose.

So is there a way to add references to C++ ?

Yes. As far as I can remember I have not documented. I assumed none would enter this area.

Through this issue, I would like to improve the document, the hacking guide.

However, I cannot do all of them at once. So I will give you a big hint: --help-full option.
With the option, you can find a hidden option for power users.

Before using, roles must be defined with _roledef.

[yamato@dev64]/tmp/c++% cat input-import.cpp 
cat input-import.cpp 
import mylib;
import <iostream>;
import "libctags";
[yamato@dev64]/tmp/c++% cat ./cxx-modules.ctags 
cat ./cxx-modules.ctags 
# Derrived from https://github.com/universal-ctags/ctags/issues/3932
--kinddef-C++=M,module,modules
--_roledef-C++.M=imported,imported
--_roledef-C++.M=system,imported with <...>
--_roledef-C++.M=local,imported with "..."

# TODO: update the hacking guide.
# {_extra=reference} is not needed. _extra is for the parser specific extras.
-Dimport=
--regex-C++=/^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+([a-zA-Z_][a-zA-Z_0-9.:]*)[[:space:]]*;[[:space:]]*$/\2/M/{_role=imported}{exclusive}
--regex-C++=/^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+<([^>]+)>[[:space:]]*;[[:space:]]*$/\2/M/{_role=imported}{_role=system}{exclusive}
--regex-C++=/^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+"([^"]+)"[[:space:]]*;[[:space:]]*$/\2/M/{_role=imported}{_role=local}{exclusive}
--extras=+{reference}
--fields=+{roles}
[yamato@dev64]/tmp/c++% ctags --options=NONE --options=cxx-modules.ctags --sort=no -o - input-import.cpp 
ctags --options=NONE --options=cxx-modules.ctags --sort=no -o - input-import.cpp 
ctags: Notice: No options will be read from files or environment
mylib	input-import.cpp	/^import mylib;$/;"	M	roles:imported
iostream	input-import.cpp	/^import <iostream>;$/;"	M	roles:imported,system
libctags	input-import.cpp	/^import "libctags";$/;"	M	roles:imported,local
[yamato@dev64]/tmp/c++%

from ctags.

masatake commented on June 15, 2024 1

{_extra=reference} may not work at all. I cannot remember the details now.
If {_role=...} is given, the tag becomes a reference tag.
Reference tags are printed only when --extras=+{reference} is given.

from ctags.

masatake commented on June 15, 2024 1

Thank you for taking the time to track it down. Now I partially understand what happens. Please, give me some time.

from ctags.

terminatorul commented on June 15, 2024

Here's a quick way to add C++ module definitions (and module partitions) using regular expression arguments.

To replace the wrong definition tags generated by default from import statements, I also had to use:

    "--Dimport="

The regexps can help especially for module partitions and for module names containg a dot. The expresions try to match lines of the form

[export] module Module.Name [: Module.Partition] ;
module : private
[export] import Module.Name ;
[export] import : Module.Partition ;

Note that in C++ these staments can legally be split across multiple lines, but the bellow regexps are single-line only. Also there can be spaces around the dots in Module . Name (and Module . Partition) which are not supported by the regexps.

D:\Local\universal-ctags\ctags.exe
    	"--kinddef-C++=M,module,modules"
    	"--kinddef-C++=P,partition,module partitions"
    	"--regex-C++=/^^[[:space:]]*module([[:space:]]*:[[:space:]]*private)?[[:space:]]*;[[:space:]]*$//M/{placeholder}"
    	"--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?module[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.:]*)[[:space:]]*;[[:space:]]*$/\2/M/"
    	"--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?module[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*:[[:space:]]*([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*;[[:space:]]*$/\2:\3/P/"
    	"--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?module[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*:[[:space:]]*([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*;[[:space:]]*$/:\3/P/"
    	"--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*;[[:space:]]*$/\2/M/{placeholder}"
    	"--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?import[[:space:]]*:[[:space:]]*([a-zA-Z_][a-zA-Z_0-9\.]*)[[:space:]]*;[[:space:]]*$//P/{placeholder}"
    	"--kinds-C++=+{module}{partition}"

Beware tools like the Vim editor, also need hierarchical module names, were the statement module Module.Name ; generates a tag for module Name that is a "member" of module Module. This is mostly because C++ identifiers do not include the . character. So most tag searches in Vim will look for Name instead of Module.Name by default. Users can still write a Vim command by hand to explicitly search for Module.Name, like :tselect Module.Name, but that is more typing then the default shortcuts of Ctrl+] or g]

Can you please add the module: field to C++ for this purpose ? Is it possible for users to add it using the command line, for use with regexp tags ?

I also tried regular expressions that match C++ module references (import statements), but ctags tells me there is no reference extra for C++, and there is no option --roles-C++.h ... :(

D:\Local\universal-ctags\ctags.exe
      --recurse --languages=+C,C++
      --kinds-C=+px
      --kinds-C++=+px
      --fields=+lzkKErSt
      "--extras=+{qualified}{inputFile}{reference}"
      --totals
      --kinddef-C++=M,module,modules
      "--kinddef-C++=P,partition,module partitions"
      "--roles.C++.h=+{imported}"
      "--roles.C++.M=+{imported}"
      "--regex-C++=/^^[[:space:]]*module([[:space:]]*:[[:space:]]*private)?[[:space:]]*;[[:space:]]*$//M/{placeholder}"
      "--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?module[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.:]*)[[:space:]]*;[[:space:]]*$/\2/M/"
      "--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+([a-zA-Z_][a-zA-Z_0-9\.:]*)[[:space:]]*;[[:space:]]*$/\2/M/{_extra=reference}{_role=imported}"
      "--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+\<([^^>])\>[[:space:]]*;[[:space:]]*$/\2/M/{_extra=reference}{_role=imported}{_role=system}"
      "--regex-C++=/^^[[:space:]]*(export[[:space:]]+)?import[[:space:]]+""([^^""])""[[:space:]]*;[[:space:]]*$/\2/M/{_extra=reference}{_role=imported}{_role=local}"
      "--kinds-C++=+{module}"
        *

Can u-ctags allow the user to add the reference extra to C++, from the command line, if user chooses so ?

from ctags.

terminatorul commented on June 15, 2024

So is there a way to add references to C++ ?

So I can properly tag module imports, and header file imports and includes ?

from ctags.

terminatorul commented on June 15, 2024

Oh, thank you, this is great !

If I don't use {_extra=reference}, is there a way for the refernce tags (for module imports) to only be generated if users select references for their output, like with --extras=+{reference} ? Or is such behavior the default ?

from ctags.

masatake commented on June 15, 2024

The feature of adding a parser specific field may interest you.
Version 1:

[yamato@dev64]/tmp/c++% cat ./input-import+exported.cpp 
cat ./input-import+exported.cpp 
import mylib;
import <iostream>;
import "libctags";

export import xmylib;
export import <xiostream>;
export import "xlibctags";
[yamato@dev64]/tmp/c++% cat ./cxx-modules+exported.ctags 
cat ./cxx-modules+exported.ctags 
# Derrived from https://github.com/universal-ctags/ctags/issues/3932
--kinddef-C++=M,module,modules
--_roledef-C++.M=imported,imported
--_roledef-C++.M=system,imported with <...>
--_roledef-C++.M=local,imported with "..."

--_fielddef-C++=exported,whether the language objects is exported or not
# The field must be enabled explicitly.
--fields-C++=+{exported}

# TODO: update the hacking guide.
# {_extra=reference} is not needed. _extra is for the parser specific extras.
-Dimport=
--regex-C++=/^[[:space:]]*import[[:space:]]+([a-zA-Z_][a-zA-Z_0-9.:]*)[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{exclusive}
--regex-C++=/^[[:space:]]*import[[:space:]]+<([^>]+)>[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{_role=system}{exclusive}
--regex-C++=/^[[:space:]]*import[[:space:]]+"([^"]+)"[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{_role=local}{exclusive}
--regex-C++=/^[[:space:]]*export[[:space:]]+?import[[:space:]]+([a-zA-Z_][a-zA-Z_0-9.:]*)[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{exclusive}{_field=exported:1}
--regex-C++=/^[[:space:]]*export[[:space:]]+?import[[:space:]]+<([^>]+)>[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{_role=system}{exclusive}{_field=exported:1}
--regex-C++=/^[[:space:]]*export[[:space:]]+?import[[:space:]]+"([^"]+)"[[:space:]]*;[[:space:]]*$/\1/M/{_role=imported}{_role=local}{exclusive}{_field=exported:1}
--extras=+{reference}
--fields=+{roles}
[yamato@dev64]/tmp/c++% ctags --options=NONE --options=cxx-modules+exported.ctags  --sort=no -o - input-import+exported.cpp 
ctags --options=NONE --options=cxx-modules+exported.ctags  --sort=no -o - input-import+exported.cpp 
ctags: Notice: No options will be read from files or environment
mylib	input-import+exported.cpp	/^import mylib;$/;"	M	roles:imported
iostream	input-import+exported.cpp	/^import <iostream>;$/;"	M	roles:imported,system
libctags	input-import+exported.cpp	/^import "libctags";$/;"	M	roles:imported,local
xmylib	input-import+exported.cpp	/^export import xmylib;$/;"	M	roles:imported	exported:1
xiostream	input-import+exported.cpp	/^export import <xiostream>;$/;"	M	roles:imported,system	exported:1
xlibctags	input-import+exported.cpp	/^export import "xlibctags";$/;"	M	roles:imported,local	exported:1

See the end of lines for each tag; exported:1 is appended.

from ctags.

masatake commented on June 15, 2024

A tuple of input.cpp, a ctags cmdline, and expected tags output help me know what you want and what you did. After refining the .ctags, I'll try to convert it to the real parser written in C.

from ctags.

terminatorul commented on June 15, 2024

I am not familiar enough with the tags file format, and I do not have the regexps for the complete syntax.

But what I think is needed to make Vim behave normally, when using the default shortcut Ctrl+] (go to tag), plus other commands and shortcuts, is to parse statements like this:

C++:

module My.App . ConfigModule;      // main module header
// ...
import : Config.Data. IniSettings [[cats::meow(-1,8, "de"[1])]] ; // fall back to IniSettings

this should generate the bellow tags:

IniSettings kind: partition scope:partition:"My.App.ConfigModule:Config.Data" roles:imported
Config.Data.IniSettings kind:partition scope:module:My.App.ConfigModule roles:imported
:Config.Data.IniSettings kind:partition scope:module:My.App.ConfigModule roles:imported
My.App.ConfigModule:Config.Data.InitSettings kind:partition roles:imported

C++:

module My.App.ConfigModule;      // main module header
//...
export import : Config.Data. IniSettings ;

this should generate the bellow tags:

IniSettings kind: partition scope:partition:"My.App.ConfigModule:Config.Data" roles:imported,exported
Config.Data.IniSettings kind:partition scope:module:My.App.ConfigModule roles:imported,exported
:Config.Data.IniSettings kind:partition scope:module:My.App.ConfigModule roles:imported,exported
My.App.ConfigModule:Config.Data.InitSettings kind:partition roles:imported,exported

C++:

module My.App . ConfigModule: Config.Data. IniSettings ;

this should generate the bellow tags:

IniSettings kind:partition scope:partition:"My.App.ConfigModule:Config.Data" roles:def
Config.Data.IniSettings kind:partition scope:module:"My.App.ConfigModule" roles:def
:Config.Data.IniSettings kind:partition scope:module:"My.App.ConfigModule" roles:def
My.App.ConfigModule:Config.Data.InitSettings kind:partition roles:def

C++:

export module My.App . ConfigModule: ConfigData. IniSettings ;

this should generate the bellow tags:

IniSettings kind:partition scope:partition:"My.App.ConfigModule:ConfigData" roles:def
ConfigData.IniSettings kind:partition scope:module:My.App.ConfigModule roles:def
:ConfigData.IniSettings kind:partition scope:module:My.App.ConfigModule roles:def
My.App.ConfigModule:ConfigData.InitSettings kind:partition roles:def

C++:

export import My.App . ConfigModule  [[cats::meow(-1,8, "de"[1])]] ; // dedicated Config module

this should generate the bellow tags:

ConfigModule kind:module scope:module:My.App roles:imported,exported
My.App.ConfigModule kind:module roles:imported,exported

C++:

import My.App . ConfigModule  [[cats::meow(-1,8, "de"[1])]] ; // dedicated Config module

this should generate the bellow tags:

ConfigModule kind:module scope:module:My.App roles:imported
My.App.ConfigModule kind:module roles:imported

C++:

export module My.App . ConfigModule  [[cats::meow(-1,8, "de"[1])]] ; // dedicated Config module

this should generate the bellow tags:

ConfigModule kind:module scope:module:My.App roles:def,exported
My.App.ConfigModule kind:module roles:def,exported

C++:

module My.App . ConfigModule  [[cats::meow(-1,8, "de"[1])]] ; // dedicated Config module

this should generate the bellow tags:

ConfigModule kind:module scope:module:My.App roles:def
My.App.ConfigModule kind:module roles:def

C++:

module;
// ...
module My.App . Config;   // main module header

the first line module; is a placeholder (C++ defines it as a Global module fragment)

maybe generate some special tag like GLOBAL_MODULE, if user opts in to it, with kind:module roles:def

C++:

module My.App.ConfigModule;        // main module header
// ...
module: private;

maybe generate some special tag, like:
- PRIVATE kind:partition scope:module:My.App.ConfigModule roles:def
- :PRIVATE kind:partition scope:module:My.App.ConfigModule roles:def
- My.App.ConfigModule:PRIVATE kind:partition roles:def.

C++:

import "Local\App\ConfigModule.hh"   [[cats::meow(-1,8, "de"[1])]] [[deprecated]];

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:local,imported
Local\App\ConfigModule.hh kind:header roles:local,imported
Local/App/ConfigModule.hh kind:header roles:local,imported

C++:

export import "Local\App\ConfigModule.hh"   [[cats::meow(-1,8, "de"[1])]] [[deprecated]];

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:local,imported,exported
Local\App\ConfigModule.hh kind:header roles:local,imported,exported
Local/App/ConfigModule.hh kind:header roles:local,imported,exported

C++:

import <Local\App\ConfigModule.hh>;

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:system,imported
Local\App\ConfigModule.hh kind:header roles:system,imported
Local/App/ConfigModule.hh kind: header roles:system,imported

C++:

export import <Local\App\ConfigModule.hh>;

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:system,imported,exported
Local\App\ConfigModule.hh kind:header roles:system,imported,exported
Local/App/ConfigModule.hh kind:header roles:system,imported,exported

C++:

#include "Local\App\ConfigModule.hh"       // plain old included header

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:local,included
Local\App\ConfigModule.hh kind:header roles:local,included
Local/App/ConfigModule.hh kind:header roles:local,included

C++:

#include <Local\App\ConfigModule.hh>       // plain old included header

this should generate the bellow tags:

ConfigModule.hh kind:header scope:directory:Local/App roles:system,included
Local\App\ConfigModule.hh kind:header roles:system,included
Local/App/ConfigModule.hh kind:header roles:system,included

Note how qualified names usually require the --extras={qualified} option on the command line. But for C++ modules this should be the default. The reason is hierarchical modules (of the form My.App.ConfigModule) are a convention and are not formally defined in C++ language. The language only says module names and partition names can include the dot . character when needed.

The same could be said for header files. I should be able to search freely for either stat.h or sys/stat.h, and still find the include lines. Also if source file reads #include <sys\stat.h> (works on Windows, non-standard way), it would be nice to still find the line as sys/stat.h. Especially if the original include line uses a mix of slash and backslash characters. Collapsing multiple slash characters (sys//\\\stat.h =>sys/stat.h) is also a nice to have.

from ctags.

terminatorul commented on June 15, 2024

Hope that helps.

I am not sure scope field (or section, as the field is called in C++) is relevant for references, but I have included that above anyway, just to be sure. You will know best if scope should be excluded for these, which I guess it could be.

Note that module: and partion: act as scopes only for modules and partitions. For functions and variables, they are not scopes like namespaces and classes are. That is, the function and variable definitions still belong to their namespace for example, but also have some internal "label" (or field in ctags) attached that shows what module they are defined in. The C++ language says the function declarations are "attached" to a module, to suggest a different (weaker) relationship then the usual aggregation.

A new module: field and a new partition: field could be used at the same time with the actual scope (the namespace or class) for functions and variables. Put simply, it's like variables can have two scopes now: the namespace or class, and the module and partition (maybe ctags already supports multiple independent scopes like this, I don't actually know) However, this disscussion applies to functions and variables, not the modules themselves, so this is a different topic.

Notice how sometimes the main module header line:

[export] module My.App.Config;

is needed to parse some later module statements, of the form:

[export] import :Config.Data.IniSettings;
module : private ;

I can only hope ctags will end up just great at parsing these new statements !

from ctags.

terminatorul commented on June 15, 2024

Not sure how ctags is sorting multiple tags of the same name, but the exported definition of a module (with roles:def,exported) should have priority over the list of internal definitions (with roles:def).

For module partition references, the exported reference (with roles:imported,exported) should have priority over internal import references (with roles:imported).

from ctags.

terminatorul commented on June 15, 2024

I am trying to build an options file with all the regular expressions, and I have been working so hard for the regexps...

But now this command seems to trigger some bug where only pseudotags are output, and not the real tags:

D:\Local\universal-ctags\ctags.exe
     --languages=+C,C++
     --kinds-C=+px
     --kinds-C++=+px
     --fields=+lzkKErSt
    "--extras=+{qualified}{inputFile}{reference}{pseudo}"
     --totals
     -D_M_AMD64 -D_WINDOWS -D_MBCS -D_WIN64 -D_WIN32 -D_MSC_VER=1933 -D_MSC_FULL_VER=193331630
     --options=C:\Users\Adrian\vimfiles\ProjectConfig\plugin\ProjectConfig_CTagsCxxModules.ctags 
     -f -
    
 C:\Users\Adrian\packages\edk2\NvStrapsReBar\CxxModuleCTags.cc

If i remove pseudotags (or use console output, which does the same), I get the real tags. But I still want the pseudotags... :(

Can you help me please ? @masatake

I have attached the options file and the input C++ file.

Is this a known issue ? Am I doing something wrong ?
ProjectConfig_CTagsCxxModules.zip
CxxModuleCTags.zip

from ctags.

terminatorul commented on June 15, 2024

The issue shows up with the simple command: ctags.exe --extras=+{pseudo} -f - CxxModuleCTags.cc

from ctags.

masatake commented on June 15, 2024

[yamato@dev64]/tmp% ctags --version                                
Universal Ctags 6.0.0, Copyright (C) 2015-2022 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jul 19 2023, 00:00:00
  URL: https://ctags.io/
  Output version: 0.0
  Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +xpath, +json, +interactive, +sandbox, +yaml, +packcc, +optscript
[yamato@dev64]/tmp% ctags --extras=+{pseudo} -f - CxxModuleCTags.cc
!_TAG_EXTRA_DESCRIPTION	anonymous	/Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION	fileScope	/Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION	pseudo	/Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION	subparser	/Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION	epoch	/the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION	file	/File-restricted scoping/
!_TAG_FIELD_DESCRIPTION	input	/input file/
!_TAG_FIELD_DESCRIPTION	name	/tag name/
!_TAG_FIELD_DESCRIPTION	pattern	/pattern/
!_TAG_FIELD_DESCRIPTION	typeref	/Type and name of a variable or typedef/
!_TAG_FIELD_DESCRIPTION!C++	name	/aliased names/
!_TAG_FILE_FORMAT	2	/extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED	1	/0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++	c,class	/classes/
!_TAG_KIND_DESCRIPTION!C++	d,macro	/macro definitions/
!_TAG_KIND_DESCRIPTION!C++	e,enumerator	/enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++	f,function	/function definitions/
!_TAG_KIND_DESCRIPTION!C++	g,enum	/enumeration names/
!_TAG_KIND_DESCRIPTION!C++	h,header	/included header files/
!_TAG_KIND_DESCRIPTION!C++	m,member	/class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++	n,namespace	/namespaces/
!_TAG_KIND_DESCRIPTION!C++	s,struct	/structure names/
!_TAG_KIND_DESCRIPTION!C++	t,typedef	/typedefs/
!_TAG_KIND_DESCRIPTION!C++	u,union	/union names/
!_TAG_KIND_DESCRIPTION!C++	v,variable	/variable definitions/
!_TAG_OUTPUT_EXCMD	mixed	/number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP	slash	/slash or backslash/
!_TAG_OUTPUT_MODE	u-ctags	/u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION	0.0	/current.age/
!_TAG_PARSER_VERSION!C++	0.0	/current.age/
!_TAG_PATTERN_LENGTH_LIMIT	96	/0 for no limit/
!_TAG_PROC_CWD	/tmp/	//
!_TAG_PROGRAM_AUTHOR	Universal Ctags Team	//
!_TAG_PROGRAM_NAME	Universal Ctags	/Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL	https://ctags.io/	/official site/
!_TAG_PROGRAM_VERSION	6.0.0	//
!_TAG_ROLE_DESCRIPTION!C++!header	local	/local header/
!_TAG_ROLE_DESCRIPTION!C++!header	system	/system header/
!_TAG_ROLE_DESCRIPTION!C++!macro	undef	/undefined/
RemoteSystem	CxxModuleCTags.cc	/^module RemoteSystem:	    CTagsRemote [[cats::meow(-1.28, "dE"[1])]];$/;"	v	typeref:typename:module
SystemCTagsConfig	CxxModuleCTags.cc	/^import SystemCTagsConfig;$/;"	v	typeref:typename:import
[yamato@dev64]/tmp% ctags -f - CxxModuleCTags.cc 
RemoteSystem	CxxModuleCTags.cc	/^module RemoteSystem:	    CTagsRemote [[cats::meow(-1.28, "dE"[1])]];$/;"	v	typeref:typename:module
SystemCTagsConfig	CxxModuleCTags.cc	/^import SystemCTagsConfig;$/;"	v	typeref:typename:import

(I also tried with --options=NONE and ctags worked fine as well.)

I wonder which version of ctags you use.

from ctags.

terminatorul commented on June 15, 2024

6.1.0

C:\Users\Adrian\Projects\.vim\ProjectConfig>ctags --version
Universal Ctags 6.1.0(v6.1.0), Copyright (C) 2015-2023 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Dec 29 2023, 08:24:13
  URL: https://ctags.io/
  Output version: 0.0
  Optional compiled features: +win32, +wildcards, +regex, +gnulib_regex, +internal-sort, +unix-path-separator, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames, +packcc, +optscript, +pcre2

from ctags.

masatake commented on June 15, 2024

I have no idea where the bug comes from.

Could you try the following minimum test?

$ cat input.c
int x;
$ ctags --version
$ ctags --verbose --options=NONE --extras='{pseudo}' -f - input.c
...

I would like to see the whole output.

from ctags.

terminatorul commented on June 15, 2024

Now it works:
https://pastebin.com/17BaWB7w

I also tried with --options=NONE with my input file, still not working:

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>D:\Local\universal-ctags\ctags.exe --options=NONE --extras={pseudo} -f - CxxModuleCTags.cc
ctags.exe: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>C:\msys64\usr\bin\head.exe CxxModuleCTags.cc


module;

#include <Local/App/Config.hh>
#include "Remote/App/RemoteConfig.hh"

import <Local/App/Config.hh>;
import "Remote/App/RemoteConfig.hh";


C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

terminatorul commented on June 15, 2024

The verbose output for my input file is here:
https://pastebin.com/9Qcj5Mjt

I can see two symbols being logged, but never output:

OPENING CxxModuleCTags.cc as C++ language file [new]
Initialize parser: C++
Initialize parser: QtMoc
Initialize parser: LdScript
Initialize parser: CPreProcessor
symtbl[:=] *root*<-RemoteSystem/000001fd85d379d0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001fd85d37b80 (line: 22)
symtbl[:=] *root*<-RemoteSystem/000001fd85d379d0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001fd85d37b80 (line: 22)
sorting tag file
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/
 

    C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

masatake commented on June 15, 2024

Though

!_TAG_KIND_DESCRIPTION!C++	v,variable	/variable definitions/

should be printed, I cannot find it in the output.

Add --kinds-C++=+v to the command line like:

$ ctags --options=NONE --verbose  --extras=+'{pseudo}' -f - --kinds-C++=+v CxxModuleCTags.cc

I also want to see ctags --options=NONE --list-kinds-full=C++. On my environemtn I got:

#LETTER NAME       ENABLED REFONLY NROLES MASTER DESCRIPTION
A       alias      no      no      0      NONE   namespace aliases
D       macroparam no      no      0      C      parameters inside macro definitions
L       label      no      no      0      C      goto labels
N       name       no      no      0      NONE   names imported via using scope::symbol
U       using      no      no      0      NONE   using namespace statements
Z       tparam     no      no      0      NONE   template parameters
c       class      yes     no      0      NONE   classes
d       macro      yes     no      2      C      macro definitions
e       enumerator yes     no      0      C      enumerators (values inside an enumeration)
f       function   yes     no      0      C      function definitions
g       enum       yes     no      0      C      enumeration names
h       header     yes     yes     2      C      included header files
l       local      no      no      0      C      local variables
m       member     yes     no      0      C      class, struct, and union members
n       namespace  yes     no      0      NONE   namespaces
p       prototype  no      no      0      C      function prototypes
s       struct     yes     no      0      C      structure names
t       typedef    yes     no      0      C      typedefs
u       union      yes     no      0      C      union names
v       variable   yes     no      0      C      variable definitions
x       externvar  no      no      0      C      external and forward variable declarations
z       parameter  no      no      0      C      function parameters inside function or prototype definitions

from ctags.

terminatorul commented on June 15, 2024

--list-kinds appears fine:

C:\Users\Adrian>CD Packages\edk2\NvStrapsReBar

C:\Users\Adrian\Packages\edk2\NvStrapsReBar> ctags --options=NONE --list-kinds-full=C++
ctags: Notice: No options will be read from files or environment
#LETTER NAME       ENABLED REFONLY NROLES MASTER DESCRIPTION
A       alias      no      no      0      NONE   namespace aliases
D       macroparam no      no      0      C      parameters inside macro definitions
L       label      no      no      0      C      goto labels
N       name       no      no      0      NONE   names imported via using scope::symbol
U       using      no      no      0      NONE   using namespace statements
Z       tparam     no      no      0      NONE   template parameters
c       class      yes     no      0      NONE   classes
d       macro      yes     no      2      C      macro definitions
e       enumerator yes     no      0      C      enumerators (values inside an enumeration)
f       function   yes     no      0      C      function definitions
g       enum       yes     no      0      C      enumeration names
h       header     yes     yes     2      C      included header files
l       local      no      no      0      C      local variables
m       member     yes     no      0      C      class, struct, and union members
n       namespace  yes     no      0      NONE   namespaces
p       prototype  no      no      0      C      function prototypes
s       struct     yes     no      0      C      structure names
t       typedef    yes     no      0      C      typedefs
u       union      yes     no      0      C      union names
v       variable   yes     no      0      C      variable definitions
x       externvar  no      no      0      C      external and forward variable declarations
z       parameter  no      no      0      C      function parameters inside function or prototype definitions

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

Here is the output from:
ctags --options=NONE --verbose --extras=+'{pseudo}' -f - --kinds-C++=+v CxxModuleCTags.cc

https://pastebin.com/KLS93sj1

symtbl[:=] *root*<-RemoteSystem/0000011ed9177aa0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/0000011ed9177c50 (line: 22)
symtbl[:=] *root*<-RemoteSystem/0000011ed9177aa0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/0000011ed9177c50 (line: 22)
sorting tag file
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/
 
C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

masatake commented on June 15, 2024

Mysterious. I am trying to understand why v/variable is not listed in the output.
How is the output if you store the output in a tag file?

$ ctags --options=NONE --verbose  --extras=+'{pseudo}' --kinds-C++=+v CxxModuleCTags.cc
$

I tried https://github.com/universal-ctags/ctags-win32/releases/download/2023-12-29%2Fv6.1.0-4-ge4cb2a0/ctags-2023-12-29_v6.1.0-4-ge4cb2a0-x64.zip . I don't know well Windows at all. I started the Powershell console and ran the ctags executable in the archive.

The issue about the pseudo tags was not reproduced. However, I saw an interesting behavior.

Though the input is very similar, x in input.cc was not extracted.

I created input.cc with "PS C:...> echo 'int x;' > input.cc".
On the other hand, I created input-2.cc with notepad.exe.

Emacs on Fedora says "utf-16le-with-signature-dos" encoding is used in input.cc.

from ctags.

masatake commented on June 15, 2024

@terminatorul, @k-takata, @leleliu008

I need your help.

Could you show the output of the following command lines?

> echo '' > input.cpp                                                     
> ctags --options=NONE --extras=+'{pseudo}' -f - ./input.cpp
> ctags --version

I expect output includes following lines:

!_TAG_KIND_DESCRIPTION!C++	c,class	/classes/
!_TAG_KIND_DESCRIPTION!C++	d,macro	/macro definitions/
!_TAG_KIND_DESCRIPTION!C++	e,enumerator	/enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++	f,function	/function definitions/
!_TAG_KIND_DESCRIPTION!C++	g,enum	/enumeration names/
!_TAG_KIND_DESCRIPTION!C++	h,header	/included header files/
!_TAG_KIND_DESCRIPTION!C++	m,member	/class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++	n,namespace	/namespaces/
!_TAG_KIND_DESCRIPTION!C++	s,struct	/structure names/
!_TAG_KIND_DESCRIPTION!C++	t,typedef	/typedefs/
!_TAG_KIND_DESCRIPTION!C++	u,union	/union names/
!_TAG_KIND_DESCRIPTION!C++	v,variable	/variable definitions/

On my Windows:

from ctags.

masatake commented on June 15, 2024

@terminatorul How did you get your ctags executable?
Downloading the binary somewhere or building by yourself?

from ctags.

terminatorul commented on June 15, 2024

I downloaded my binary from the github project linked on the main README.md page, it looked official and also had the experimental downloads for the new clang-based parser as well. It must have been the recommended ctags-win32 project (linked from your README.md page) , where I looked for the latest stable release.

Here's the exact download URL from my browser download history:
https://objects.githubusercontent.com/github-production-release-asset-2e65be/94896628/a37a11c3-ae24-4176-b747-8ce17ef449da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240112T232245Z&X-Amz-Expires=300&X-Amz-Signature=5f46150592b848071cd4f46c3df9008df9c0235db02bef72fb846c3bc3b8a29d&X-Amz-SignedHeaders=host&actor_id=378924&key_id=0&repo_id=94896628&response-content-disposition=attachment%3B%20filename%3Dctags-v6.1.0-x64.zip&response-content-type=application%2Foctet-stream

In your test using PowerShell, it looks like PowerShell created a Unicode (UTF-16) file input.cc with the PS built-in echo command. This may be the reason the int x variable was not tagged. And notepad must have created an ASCII file as usual.

So can you just use cmd.exe instead of PowerShell ?

Here is my output to a real tags file, the same issue take place:
https://pastebin.com/qGg4PVng

Reading command line arguments
Get file language for CxxModuleCTags.cc
        pattern: CxxModuleCTags.cc
                #candidates: 1
                        0: C++ (extension: "cc")
                #candidates after sorting and filtering: 1
                        0: C++ (extension: "cc")
OPENING CxxModuleCTags.cc as C++ language file [new]
Initialize parser: CPreProcessor
symtbl[:=] *root*<-RemoteSystem/000001ce60337a30 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001ce60337be0 (line: 22)
symtbl[:=] *root*<-RemoteSystem/000001ce60337a30 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001ce60337be0 (line: 22)
sorting tag file
 
C:\Users\Adrian\Packages\edk2\NvStrapsReBar>Type tags
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/
 
C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

And here is my output from an empty .cpp file, like you requested. I have adjusted the commands as need for Windows, because on Windows the command echo '' would actually output two single-quote ' characters (and the same would be true for double-quote " characters).

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>echo.> input.cpp

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>Type input.cpp


C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp
ctags: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FIELD_DESCRIPTION!C++     name    /aliased names/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      c,class /classes/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++      f,function      /function definitions/
!_TAG_KIND_DESCRIPTION!C++      g,enum  /enumeration names/
!_TAG_KIND_DESCRIPTION!C++      h,header        /included header files/
!_TAG_KIND_DESCRIPTION!C++      m,member        /class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++      n,namespace     /namespaces/
!_TAG_KIND_DESCRIPTION!C++      s,struct        /structure names/
!_TAG_KIND_DESCRIPTION!C++      t,typedef       /typedefs/
!_TAG_KIND_DESCRIPTION!C++      u,union /union names/
!_TAG_KIND_DESCRIPTION!C++      v,variable      /variable definitions/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PARSER_VERSION!C++        1.1     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/
!_TAG_ROLE_DESCRIPTION!C++!header       local   /local header/
!_TAG_ROLE_DESCRIPTION!C++!header       system  /system header/
!_TAG_ROLE_DESCRIPTION!C++!macro        undef   /undefined/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --version
Universal Ctags 6.1.0(v6.1.0), Copyright (C) 2015-2023 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Dec 29 2023, 08:24:13
  URL: https://ctags.io/
  Output version: 0.0
  Optional compiled features: +win32, +wildcards, +regex, +gnulib_regex, +internal-sort, +unix-path-separator, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames, +packcc, +optscript, +pcre2

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

terminatorul commented on June 15, 2024

I unpacked and installed your binary (linked above) as ctags-6.1.0-4.

The issue is still present for me with your binary as well:
https://pastebin.com/91jGH2AF

Reading command line arguments
Get file language for CxxModuleCTags.cc
        pattern: CxxModuleCTags.cc
                #candidates: 1
                        0: C++ (extension: "cc")
                #candidates after sorting and filtering: 1
                        0: C++ (extension: "cc")
OPENING CxxModuleCTags.cc as C++ language file [new]
Initialize parser: CPreProcessor
symtbl[:=] *root*<-RemoteSystem/000001abca0f7aa0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001abca0f7c50 (line: 22)
symtbl[:=] *root*<-RemoteSystem/000001abca0f7aa0 (line: 19)
symtbl[:=] *root*<-SystemCTagsConfig/000001abca0f7c50 (line: 22)
sorting tag file
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /e4cb2a0/
 
C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

masatake commented on June 15, 2024

@terminatorul Thank you.

ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp
...
!_TAG_KIND_DESCRIPTION!C++ c,class /classes/
!_TAG_KIND_DESCRIPTION!C++ d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++ e,enumerator /enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++ f,function /function definitions/
!_TAG_KIND_DESCRIPTION!C++ g,enum /enumeration names/
!_TAG_KIND_DESCRIPTION!C++ h,header /included header files/
!_TAG_KIND_DESCRIPTION!C++ m,member /class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++ n,namespace /namespaces/
!_TAG_KIND_DESCRIPTION!C++ s,struct /structure names/
!_TAG_KIND_DESCRIPTION!C++ t,typedef /typedefs/
!_TAG_KIND_DESCRIPTION!C++ u,union /union names/
!_TAG_KIND_DESCRIPTION!C++ v,variable /variable definitions/
...

Nice. The pseudo tags are printed expectedly.

If you replace input.cpp with CxxModuleCTags.cc, these _TAG_KIND_DESCRIPTION!C++ aren't printed fully.
Do I understand the situation?

If yes, the way to debug becomes simple.
Making input.cpp similar to CxxModuleCTags.cc incrementally, you will see the number of _TAG_KIND_DESCRIPTION!C++ reduces suddenly.

from ctags.

terminatorul commented on June 15, 2024

The problem appears at line 25:

module: private;

The line is a valid statement in C++20 using modules. It is used to mark the start of the module private declarations and definitions (the module internal implementation).

But I remember you said everything still works fine on your version (6.0.0) ?

from ctags.

terminatorul commented on June 15, 2024

Do you think ctags will get confused if I use the option -Dprivate= ?
What would happen if ctags sees an extraneous : character at the start of a member declaration... (the previously private member) ?

But the -D option doesn't appear to do anything...:

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --options=NONE  --extras=+{pseudo} -f - --kinds-C++=+v -D private= -D module= -D export= -D import= CxxModuleCTags.cc
ctags: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

masatake commented on June 15, 2024

Do you mean !_TAG_KIND_DESCRIPTION!C++ v,variable /variable definitions/ is not found in the output of:

$ echo 'module: private;' > input.cpp
$ ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp

from ctags.

terminatorul commented on June 15, 2024

Yes, that is correct:

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>echo module: private; > input.cpp

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>Type input.cpp
module: private;

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp
ctags: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

I also tried with v6.0.0 binaries (for Windows), and the same problem is present. So for me no version has worked so far ...

from ctags.

masatake commented on June 15, 2024

Nice.

I would like you to find minimum input:

module: private;

module: private

module: privat

module: priva

...

Removing characters in the middle is also interesting:

module: privat;
module: p;
module:;
...

from ctags.

terminatorul commented on June 15, 2024

Minimum input is just the keyword private

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>echo private> input.cpp

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>Type input.cpp
private

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp
ctags: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

masatake commented on June 15, 2024

How about 'privat', 'public', 'publi', 'protected', 'protecte', and 'using' ?

from ctags.

terminatorul commented on June 15, 2024

private, protected and public trigger the issue.

using and other keywords are fine:

auto
break
catch
class
concept
constexpr
consteval
constinit
continue
default
delete
enum
explicit
export
extern
false
friend
goto
inline
mutable
namespace
noexcept
nullptr
operator
reflexpr
register
requires
return
static
template
this
throw
try
typedef
typename
virtual
volatile

from ctags.

terminatorul commented on June 15, 2024

Partial keywords like "privat" work fine:

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>echo privat> input.cpp

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>ctags --options=NONE --extras=+{pseudo} -f - ./input.cpp
ctags: Notice: No options will be read from files or environment
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FIELD_DESCRIPTION!C++     name    /aliased names/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C++      c,class /classes/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++      f,function      /function definitions/
!_TAG_KIND_DESCRIPTION!C++      g,enum  /enumeration names/
!_TAG_KIND_DESCRIPTION!C++      h,header        /included header files/
!_TAG_KIND_DESCRIPTION!C++      m,member        /class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++      n,namespace     /namespaces/
!_TAG_KIND_DESCRIPTION!C++      s,struct        /structure names/
!_TAG_KIND_DESCRIPTION!C++      t,typedef       /typedefs/
!_TAG_KIND_DESCRIPTION!C++      u,union /union names/
!_TAG_KIND_DESCRIPTION!C++      v,variable      /variable definitions/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_OUTPUT_VERSION    0.0     /current.age/
!_TAG_PARSER_VERSION!C++        1.1     /current.age/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  C:/Users/Adrian/Packages/edk2/NvStrapsReBar/    //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   6.1.0   /v6.1.0/
!_TAG_ROLE_DESCRIPTION!C++!header       local   /local header/
!_TAG_ROLE_DESCRIPTION!C++!header       system  /system header/
!_TAG_ROLE_DESCRIPTION!C++!macro        undef   /undefined/

C:\Users\Adrian\Packages\edk2\NvStrapsReBar>

from ctags.

leleliu008 commented on June 15, 2024

@masatake

How can I help you?

Is this a windows-specific issue?

from ctags.

masatake commented on June 15, 2024

@leleliu008 Yes, I need your help. Thank you.

I would like to see the output of ctags running on Windows.

steps:

prepare input.cpp; echo private > input.cpp
run ctags with: ctags --options=NONE --extras=+{pseudo} -f - input.cpp

I would like to see the output of the command line.

I wonder whether the following strings are included or not:

!_TAG_KIND_DESCRIPTION!C++      c,class /classes/
!_TAG_KIND_DESCRIPTION!C++      d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C++      e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C++      f,function      /function definitions/
!_TAG_KIND_DESCRIPTION!C++      g,enum  /enumeration names/
!_TAG_KIND_DESCRIPTION!C++      h,header        /included header files/
!_TAG_KIND_DESCRIPTION!C++      m,member        /class, struct, and union members/
!_TAG_KIND_DESCRIPTION!C++      n,namespace     /namespaces/
!_TAG_KIND_DESCRIPTION!C++      s,struct        /structure names/
!_TAG_KIND_DESCRIPTION!C++      t,typedef       /typedefs/
!_TAG_KIND_DESCRIPTION!C++      u,union /union names/
!_TAG_KIND_DESCRIPTION!C++      v,variable      /variable definitions/

@terminatorul reported they were omitted.
However, on my Windows VM, it was rarely turned on; I was the strings.

They should be printed.

I can utilize our CI/CD environments for debugging platform-specific bugs. However, the debugging technique may waste my time. Before jumping into the hello, I would like to gather the information.

from ctags.

leleliu008 commented on June 15, 2024

@masatake It seems that the internal sort has problem.

I tried on Ubuntu-23.10 and MSYS2/MINGW64.

I tried our master HEAD, if --disable-external-sort configure option is given, !_TAG_KIND_DESCRIPTION!C++ lines would not be printed out. otherwise, it would be printed out.

ctags-win32 use --disable-external-sort option.

https://github.com/universal-ctags/ctags-win32/blob/master/.github/workflows/build-ctags.yaml#L163C21-L163C44

from ctags.

masatake commented on June 15, 2024

@leleliu008 Thank you very much. Reproduced on my Fedora.
I will look into this.

See https://github.com/universal-ctags/ctags/pull/3936/files#diff-2cd63180d7ce2a22a03aaf13aa69580ce91b1842802267c162d75e925c3b70f3

from ctags.

masatake commented on June 15, 2024

@terminatorul based on your comment (#3932 (comment)), I extended the C++ parser supports modules.
See #3941.

from ctags.

masatake commented on June 15, 2024

#3941 was merged.

from ctags.

C++: support modules about ctags HOT 42 CLOSED

Comments (42)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent