Auto generate tags throws a unicode error for certain Windows paths

[jsfaint]

Auto generate tags throws a unicode error for certain Windows paths

[jsfaint]

ctags don't support unicode
It looks universal-ctags has limited unicode support, you can have a try.
https://github.com/universal-ctags/ctags

[jsfaint]

Oh, sorry. I misunderstand your issue.

Maybe this issue was imported in the rewrite of ctags_prune().
I am still in the holiday of national day, I'll look into this issue later.

[jsfaint]

I looked into this issue.

My universal-ctags version is download from https://github.com/universal-ctags/ctags-win32/releases/download/2018-10-08%2F208e2c0b/ctags-2018-10-08_208e2c0b-x64.zip

$ ctags --version
Universal Ctags 0.0.0(208e2c0b), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Oct  8 2018, 00:08:42
  URL: https://ctags.io/
  Optional compiled features: +win32, +wildcards, +regex, +internal-sort, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames

If the source code path is pure English, ctags will generate a tags file with charset 'us-ascii' which will be treat as 'utf-8' in python

$ ctags -R d:\ethtool
$ file tags --mime-encoding
tags: us-ascii

And if the path with CJK inside, the tags file will be use the OS native charset

$ ctags -R d:\测试ethtool
$ file --mime-encoding tags
tags: iso-8859-1

But in python, the data read from file will be treat as utf-8 by default, that's why you got this issue.
I'm not familiar with python, it looks the data need some convert before comparison.

The workaround you provide only mute the error message, but the ctags_prune() is still not work.
Maybe you should not use CJK in path or disable g:gen_tags#ctags_prune before we got a solution.

[embedded-kyle]

I have been using universal ctags, though my build is a bit older.

Universal Ctags 0.0.0(f4be7435), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Sep 27 2018, 00:07:08
  URL: https://ctags.io/
  Optional compiled features: +win32, +wildcards, +regex, +internal-sort, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames

All paths are in English and the tags file is in ascii.

> file .\prj_tags --mime-encoding
> .\prj_tags: us-ascii

As I mentioned I think it has to do with the path containing "C:\Users" as mentioned in here. The error does not occur if the path does not contain \U. Moving the project outside of C:\Users to a different folder or drive makes the error go away.

If I print out the paths you can see that the l:file variable isn't having it's path fixed.

  echom l:tagfile
  echom l:file
  "Prune tags content for saved file
  if g:gen_tags#ctags_prune
    call s:ctags_prune(l:tagfile, l:file)
  endif

C:/Users/kyle/Workspace/Trunk&Std/clean_trunk/.svn/tags_dir/prj_tags
C:\Users\kyle\Workspace\Trunk&Std\clean_trunk\MyApplication\Application.c

I thought fixing the escape sequence in the prune function in prune.py would have fixed it as well but the error remains after trying this:

f = open(tagfile.encode('unicode-escape').decode('ascii'), 'w')

How can you tell that ctags_prune() is not working? When I have my workaround in place, the prj_tags file is still written and contains properly escaped paths like this:

MK22FA12.h C:\\Users\\kyle\\Workspace\\Trunk&Std\\clean_trunk\\Executables\\source\\Application.c /^#include "MK22FA12.h"/;" h

Let me know if there is anything you would like me to try out.

[jsfaint]

with ctags_prune enabled, tag for current buffer will be deleted on save.
And append to the end of the tag file after update procedure done.
But with the workaround, I can't see that behavior.

[jsfaint]

https://github.com/jsfaint/gen_tags.vim/tree/fix_unicode
I rewrite ctags_prune() with pure vimscript which resolve this issue.

I do some profile on different platform, the performance is accpetable compare to the python version.

@embedded-kyle Can you have a try?

[embedded-kyle]

Updated and switched branches. The error is gone and the tags file was updated with the correct timestamp.

I still don't fully understand the structure of the tag file so I can't confirm that it updated correctly. I was editing Application.c which contains the primary task that my RTOS runs. The last two lines in the tag file are links for the "µVision build Log" but that only seems to be because the file is organised alphabetically?

Let me know if there is something specific that you would like me to try or to look for to confirm that this fix is working properly. I'm happy to help. Thanks for the update.

[jsfaint]

Yes, the tag is sorted by default
We can disable the sort behavior by add -u for ctags.

I already test on different platforms (Windows, Linux and macOS)
I think I can merge that branch now.

Thanks for your help @embedded-kyle 👍

[jsfaint]

I think we can close this issue now.
@embedded-kyle thanks again 😄