gvansickle/ucg

UniversalCodeGrep (ucg) is an extremely fast grep-like tool specialized for searching large bodies of source code.

ucg's Issues

Coverity CID 53715: Not checking fstat() return code on known-good file descriptor

I suppose fstat() could fail here, leading to problems:

30File::File(const std::string &filename)
31{
32 // open() the file. We have to do this regardless of whether we'll subsequently mmap() or read().
33 m_file_descriptor = open(filename.c_str(), O_RDONLY);
34
\1. Condition this->m_file_descriptor == -1, taking false branch

35 if(m_file_descriptor == -1)
36 {
37 // Couldn't open the file, throw exception.
38 throw std::system_error(errno, std::generic_category());
39 }
40
41 // Check the file size.
42 struct stat st;
CID 53715 (#1 of 1): Unchecked return value from library (CHECKED_RETURN)2. check_return: Calling fstat(this->m_file_descriptor, &st) without checking return value. This library function may fail and return an error code. [Note: The source code implementation of the function has been overridden by a builtin model.]
43 fstat(m_file_descriptor, &st);
44 m_file_size = st.st_size;
45 // If filesize is 0, skip.
46 if(m_file_size == 0)

Support all ack filetypes

Currently, we only support the C++ file type.

Add performance comparison to grep

We're on par with grep speed-wise. Add comparison to test suite.

Fill out the README

It's just stubbed in at the moment.

Remove dependency on Boost

It's only used for the message queues, which would be simple enough to implement directly in C++11. Eliminates an extra download for builders. Maybe make Boost optional, for comparison of queue implementations.

Coverity CID 53718: File descriptor leak in the m_start_paths logic of Globber::Run()

Not much of a leak (O(n) in number of paths on command line), but still:

54void Globber::Run()
55{
56 char * dirs[m_start_paths.size()+1];
57
58 int i = 0;
\1. Iterating over another element of this->m_start_paths
59 for(const std::string& path : m_start_paths)
60 {
61 dirs[i] = const_cast<char*>(path.c_str());
62
63 // Check if this start path exists and is a file or directory.
64 DIR *d = opendir(dirs[i]);
\2. open_fn: Returning handle opened by open.
\3. var_assign: Assigning: f = handle returned from open(dirs[i], 0).
65 int f = open(dirs[i], O_RDONLY);
66 if(d != NULL)
67 {
68 closedir(d);
69 }
70 else if(f != -1)
71 {
72 close(f);
73 }
74 else
75 {
76 m_bad_path = dirs[i];
77 return;
78 }
79
80 ++i;
CID 53718 (#1 of 1): Resource leak (RESOURCE_LEAK)4. leaked_handle: Handle variable f going out of scope leaks the handle.
81 }
82 dirs[m_start_paths.size()] = 0;

Switch from mmap to read (was: Linux threads all on one core)

Clean up for release 0.1.0.

Add config file support.

Add --match PATTERN option.

Explicit specification of PATTERN as an option arg. Should be pretty easy.

Files with ~100k matches result in long periods of no output

The way the MatchList/OutputTask mechanism and/or the line-finding mechanism currently work can result in long periods without any output in certain cases. E.g. if you do this:

ucg 'endif' boost_dir > endifs.txt

endifs.txt ends up being about 4MB/46000lines, every line containing an endif. If you then do a:

ucg 'endif'

and your only files are files such as endifs.txt, there can be long periods (minutes) where there is no console output, and ucg appears hung.

configure: Check for libpcre > 8.2

Currently an old PCRE will still be detected and will try to compile, with unsatisfactory results.

Get building on MinGW-64

Currently there are issues with at least the argp lib.

Get a .deb binary distribution solution

ucg builds on Ubuntu and Debian, but it's a minor hassle. Come up with a .deb binary install solution.

Line component parsing is incorrect in FileScanner::Run()

The positions in the lines aren't being calculated properly.

TypeManager::notype() too simplistic, doesn't match ack behavior.

When "--noTYPE" is given to ack, all extensions for that type are no longer matched, even if they appear in another type. Because notype() only removes the entry for the type from the active type map, ucg doesn't do this. The ack behavior is more correct. This behavior should probably extend to all file filter types.

Fix configure.ac BUG-REPORT-ADDR and website link

Update autoconf-archive m4 macros

New autoconf-archive version 2015.09.25, update.

Configure: Detect bad std::regex lib (e.g. gcc 4.8.x)

Some std::regex libs, in particular the one shipping with gcc 4.8.x, are essentially just stubs and do not function correctly, even though the compiler ostensibly supports C++11. Detect this situation at configure (or possibly build) time and output an appropriate error message.

Support directory argument

Currently unimplemented.

Improve logging

Currently we're just using std::clog and std::cerr. Come up with a better way, which can be at least somewhat controlled by command-line params.

ucg fails if pattern contains capturing subpatterns

Add at least a minimal man page

rpmlint complains, and we really should have something.

Add version to --help

Currently there's no version info printed with "--help". "--version" does contain the info.

Add file types that ack detects as text

ack 2.x scans every file to see if it's text or not. At least as a temporary measure, add as many of these as practical as types; we can come up with a scan-for-binary algorithm later.

Extra newline at the beginning of tty output.

Special case this one, maybe moving the newline generation to OutputTask.

Add optional pcre support

C++ implementations as of this writing (gcc 5.2, clang 3.7) are buggy (gcc) causing SIGSEGVs due to use of recursion, or non-existent (clang on Linux). Add configure-time support for libpcre and see if that's any better.

Get configure.ac order sorted

It's not in the order recommended by the Autoconf docs.

Add --help-types option.

Add --noenv

ax_cxx_compile_stdcxx_11.m4 puts flag in CFLAGS, should be AM_CFLAGS

Fix this, also change default of "-O0" to "-O2" or 3.

Add --TYPE support.

Add Autotest

• Compare performance to ack, ag
• Determine performance vs. number of threads.

Test for issue #17 is broken.

The test in sanity.at for issue #17 fails, when it should either pass or be an expected failure.

Clean up warnings

make distcheck is failing

"ERROR: files left in build directory after distclean:
./tests/perf_test_results.txt"

Add --no-recurse.

Separate from --recurse issue because this one requires additional logic to implement.

Test: Capture versions of other programs

Right now we only capture the version of ucg. Capture the ack and ag versions as well.

Add --recurse.

Gracefully handle regex compile errors

Currently FileScanner::FileScanner() throws an exception when regex compilation or study fails, and there's nothing to catch it. On Cygwin 64 (at least), this results in e.g.:
"
terminate called without an active exception
[2] 3448 abort (core dumped) ./ucg --noenv '[]' ~/src/boost_1_58_0
"
We should at least not dump core in this situation.

Add file type specification support.

Add autoversioning support to build infrastructure

Determine and append git repo hash that's being built, or use .tarversion. Determine if repo is modified and add that info to the version report.

Implement directory include/exclude

Currently the command line params do nothing.

Coverity CID 53716,53717: ArgParse::GetProjectRCFilename(): if homedirname.empty() == false, open() could still fail, the home_fd would be invalid in subsequent uses.

Could happen, should be fixed:

451std::string ArgParse::GetProjectRCFilename() const
452{
453 // Walk up the directory hierarchy from the cwd until we:
454 // 1. Get to the user's $HOME dir, in which case we don't return an rc filename even if it exists.
455 // 2. Find an rc file, which we'll then return the name of.
456 // 3. Can't go up the hierarchy any more (i.e. we hit root).
457 /// @todo We might want to reconsider if we want to start at cwd or rather at whatever
458 /// paths may have been specified on the command line. cwd is what Ack is documented
459 /// to do, and is easier.
460
461 std::string retval;
462
463 // Get a file descriptor to the user's home dir, if there is one.
464 auto homedirname = GetUserHomeDir();
465 int home_fd = -1;
\1. Condition !homedirname.empty(), taking true branch
466 if(!homedirname.empty())
467 {
\2. negative_return_fn: Function open(homedirname.c_str(), 65536) returns a negative number.
\3. var_assign: Assigning: signed variable home_fd = open.
468 home_fd = open(homedirname.c_str(), O_RDONLY | O_DIRECTORY);
469 }
470
471 // Get the current working directory's absolute pathname.
472 /// @note GRVS - get_current_dir_name() under Cygwin will currently return a DOS path if this is started
473 /// under the Eclipse gdb. This mostly doesn't cause problems, except for terminating the loop
474 /// (see below).
475 char _original_cwd = get_current_dir_name();
476
477 //std::clog << "INFO: cwd = "" << original_cwd << """ << std::endl;
478
479 auto current_cwd = original_cwd;
\4. Condition current_cwd != NULL, taking true branch
\5. Condition current_cwd[0] != '.', taking true branch
\13. Condition current_cwd != NULL, taking true branch
\14. Condition current_cwd[0] != '.', taking true branch
480 while((current_cwd != nullptr) && (current_cwd[0] != '.'))
481 {
482 // See if this is the user's $HOME dir.
483 auto cwd_fd = open(current_cwd, O_RDONLY | O_DIRECTORY);
CID 53716: Improper use of negative value (NEGATIVE_RETURNS) [select issue]
\6. Condition is_same_file(cwd_fd, home_fd), taking false branch
CID 53717 (#3-2 of 3): Improper use of negative value (NEGATIVE_RETURNS)15. negative_returns: home_fd is passed to a parameter that cannot be negative. [show details]
484 if(is_same_file(cwd_fd, home_fd))
485 {
486 // We've hit the user's home directory without finding a config file.
487 close(cwd_fd);
488 break;
489 }
490 close(cwd_fd);
491
492 // Try to open the config file.
493 auto test_rc_filename = std::string(current_cwd);
\7. Condition *std::__cxx11::basic_string<char, std::char_traits, std::allocator >::reverse_iterator(test_rc_filename.rbegin()) != '/', taking true branch
494 if(_test_rc_filename.rbegin() != '/')
495 {
496 test_rc_filename += "/";
497 }
498 test_rc_filename += ".ucgrc";
499 //std::clog << "INFO: checking for rc file "" << test_rc_filename << """ << std::endl;
500 auto rc_file = open(test_rc_filename.c_str(), O_RDONLY);
\8. Condition rc_file != -1, taking false branch
501 if(rc_file != -1)
502 {
503 // Found it. Return its name.
504 //std::clog << "INFO: found rc file "" << test_rc_filename << """ << std::endl;
505 retval = test_rc_filename;
506 close(rc_file);
507 break;
508 }
509
510 /// @note GRVS - get_current_dir_name() under Cygwin will currently return a DOS path if this is started
511 /// under the Eclipse gdb. This mostly doesn't cause problems, except for terminating the loop.
512 /// The clause below after the || handles this.
\9. Condition strlen(current_cwd) == 1, taking false branch
\10. Condition strlen(current_cwd) <= 4, taking true branch
\11. Condition current_cwd[1] == ':', taking false branch
513 if((strlen(current_cwd) == 1) || (strlen(current_cwd) <= 4 && current_cwd[1] == ':'))
514 {
515 // We've hit the root and didn't find a config file.
516 break;
517 }
518
519 // Go up one directory.
520 current_cwd = dirname(current_cwd);
\12. Jumping back to the beginning of the loop
521 }
522
523 // Free the cwd string.
524 free(original_cwd);
525
526 // Close the homedir we opened above.
CID 53716: Argument cannot be negative (NEGATIVE_RETURNS) [select issue]
CID 53717 (#1 of 3): Argument cannot be negative (NEGATIVE_RETURNS) [select issue]
527 close(home_fd);
528
529 return retval;
530}

No files found on Ubuntu/gcc5.2.1

ucg compiled with gcc (Ubuntu 5.2.1-22ubuntu2) 5.2.1 20151010 does not find any files in the ~/src/boost* test.

No error val return with non-existent search dir

If you give ucg a search dir that doesn't exist (or otherwise can't be opened), it does not return a non-zero error code. Also no error message is printed.

Add --literal option.

Add --word-regexp functionality

Currently not implemented.

Implement --type= option.

"ERROR:" message when user has no .ucgrc in home directory