farsightsec/nmsg

network message encapsulation library

nmsg's Issues

get rid of the protobuf version warnings

Explicitly specify the protocol version?

GEN nmsg/nmsg.pb-c.c
[libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
specified for the proto file: nmsg/nmsg.proto. Please use 'syntax =
"proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
to proto2 syntax.)
...
GEN nmsg/base/dnstap.pb-c.c
[libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
specified for the proto file: nmsg/base/dnstap.proto. Please use 'syntax =
"proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
to proto2 syntax.)

libnmsg can't unbuffer json output objects

nmsg_output_set_buffered() is ignorant of nmsg_output_type_json and therefore does not allow the option of unbuffering json output objects (ostensibly via nmsgtool's --unbuffered option).

dnsqr: optimize IPv6 capture?

It may be possible to optimize the BPF that we generate for IPv6 traffic so that it only matches port 53 traffic, similar to what we do for IPv4, given a new enough libpcap. (The BPF program must execute in the kernel, though.)

nmsgtool -o says -V and -T are required

presentation default is default and no -V and -T are required.
-o option in manual says -V and -T are required, but -o works without them.

Include examples/ with installation?

New SIE customers typically get a tutorial session on how to read NMSG data, and I typically start by using nmsgtool presentation output to demonstrate. When they are ready to do script or C programming, I refer to the examples that come with the source and compare it to nmsgtool functionality. I typically download the latest source file (nmsg-$version.tar.gz) and extract the examples/ subdirectory. During installation it may be a good idea to have the examples/ subdirectory copied to a system documentation area like /usr/local/share/doc/nmsg or /usr/share/doc/nmsg so that they're available to reference without extra work.

It's possible that we don't need to install the examples/ if we know that github or some other online place will be available (for example: https://github.com/farsightsec/nmsg/tree/master/examples). Thoughts?

The same request would would apply to python-nmsg script examples and maybe Perl Net::Nmsg script examples, but I thought I'd try asking for nmsg first. If there's consensus that it's a good idea for nmsg, I'll reach out for the others.

Thanks,
Eric Ziegast

Manpage for nmsgtool missing new Json features.

the flags for JSON operation appear in the command help, but are absent from the manpage.

nmsgtool manpage requests

This ticket is for adding some missing documentation to the nmsgtool manual:

-B --byterate

-X --readxchan

--readif (optional for -i)

-D --daemon

-P --pidfile

-U --username

-v --version

--unbuffered

Maybe while there alphabetize the options?

Also document the environment variables:

NMSG_BPF environment variable sets c->bpfstr (same as -b or --bpf)

NMSG_KICKER sets c->kicker (same as -k or --kicker)

These both override corresponding command line options.

NMSG_MSGMOD_DIR overrides the default directory for finding message modules. Later this could be in a nmsg.3 manual.

And either put into this manual the base environment variable settings. (I am fine with doing a new man page for this but since these are builtin, same manual is fine with me for now. Or it could be moved to a new nmsg.3 manual):

DNSQR_CAPTURE_QR

DNSQR_CAPTURE_RD

DNSQR_ZERO_RESOLVER_ADDRESS

DNSQR_STATE_TABLE_MAX

DNSQR_QUERY_TIMEOUT

DNSQR_AUTH_ADDRS

DNSQR_RES_ADDRS

DNSQR_FILTER_QNAMES_INCLUDE

DNSQR_FILTER_QNAMES_EXCLUDE

dnsqr: deprecate ICMP capture

Debian packaging issue? library path in nmsg-msg9-module-base 0.15.0-1+fsi9

It looks like a change was made to the .so plugin install path between nmsg-msg9-module-base versions 0.14.0-1+fsi9 (/usr/lib/nmsg/) and 0.15.0-1+fsi9 (/usr/lib/x86_64-linux-gnu/nmsg/):

version 0.14.0-1+fsi9

# dpkg --list | grep nmsg-msg9-module-base
ii  nmsg-msg9-module-base             0.14.0-1+fsi9                               amd64        base message module plugin for libnmsg

# dpkg --listfiles nmsg-msg9-module-base
/.
/usr
/usr/lib
/usr/lib/nmsg
/usr/lib/nmsg/nmsg_msg9_base.so
/usr/share
/usr/share/doc
/usr/share/doc/nmsg-msg9-module-base
/usr/share/doc/nmsg-msg9-module-base/changelog.Debian.gz
/usr/share/doc/nmsg-msg9-module-base/changelog.gz
/usr/share/doc/nmsg-msg9-module-base/copyright

version 0.15.0-1+fsi9

# dpkg --list | grep nmsg-msg9-module-base
ii  nmsg-msg9-module-base             0.15.0-1+fsi9                               amd64        base message module plugin for libnmsg

# dpkg --listfiles nmsg-msg9-module-base
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/nmsg
/usr/lib/x86_64-linux-gnu/nmsg/nmsg_msg9_base.so
/usr/share
/usr/share/doc
/usr/share/doc/nmsg-msg9-module-base
/usr/share/doc/nmsg-msg9-module-base/changelog.Debian.gz
/usr/share/doc/nmsg-msg9-module-base/changelog.gz
/usr/share/doc/nmsg-msg9-module-base/copyright

It appears to me that nmsgtool 0.15.0-1+fsi9 only loads plugins from /usr/lib/nmsg/, and is not finding the base module installed in /usr/lib/x86_64-linux-gnu/nmsg:

# dpkg --list | grep nmsgtool
ii  nmsgtool                          0.15.0-1+fsi9                               amd64        network message encapsulation tool

# dpkg --search /usr/bin/nmsgtool
nmsgtool: /usr/bin/nmsgtool

# /usr/bin/nmsgtool --version
_nmsg_msgmodset_init: unable to opendir /usr/lib/nmsg: No such file or directory
nmsgtool: unable to initialize libnmsg

# ln -s /usr/lib/x86_64-linux-gnu/nmsg /usr/lib/nmsg

# /usr/bin/nmsgtool --version
nmsgtool: version 0.15.0

-k/--kicker should work with -t as it does with -c in JSON output mode.

I find myself in a scenario where I'd like to capture 24 1-hour files for a DITL. The Kicker mode works fine if I'm willing to limit my file-size by (-c) number of records. How about allowing me to trigger kicking on (-t) the passage of time?

The JSON code appears to not properly handle the -t option.

configure check for yajl needs to check >= 2.1

If you try to compile nmsg against yajl 2.0.x (tested with 2.0.4) you will get a linking error with an undefined reference to yajl_gen_reset

The relevant configure.ac entry that needs to be updated is

AC_ARG_WITH([yajl], AS_HELP_STRING([--without-yajl], [Disable yajl support]))
if test "x$with_yajl" != "xno"; then
    PKG_CHECK_MODULES([yajl], [yajl >= 2])
    AC_DEFINE([HAVE_YAJL], [1], [Define to 1 if yajl support is enabled.])
    use_yajl="true"
else
    use_yajl="false"
fi

transparent_json.c: < 0 check on unsigned value

Compiler warning:

nmsg/msgmod/transparent_json.c:171:19: warning: comparison of unsigned
      expression < 0 is always false [-Wtautological-compare]
                        if (enum_value < 0 || enum_value >= enum_descr->...

type conflict when building w/o yajl

Hi,

Building nmsg 0.11.1 from source on Mac El Capitan configured w/o yajl, I seem to be hitting a type conflict:

CC nmsg/input.lo
nmsg/input.c:168:1: error: conflicting types for 'nmsg_input_open_json'
nmsg_input_open_json(int fd, nmsg_msgmod_t msgmod) {
^
./nmsg/input.h:173:1: note: previous declaration is here
nmsg_input_open_json(int fd);
^
1 error generated.
make[1]: *** [nmsg/input.lo] Error 1
make: *** [all] Error 2

FYI.

Thanks and regards,

Joe

Missing Mailing list .. it seems to have sneaked off into the ether! :)

README.md says:

"Questions about libnmsg, nmsgtool, pynmsg, the development of libnmsg
client programs or language bindings, or the NMSG binary protocol should be
directed to the nmsg-dev mailing list:

https://lists.farsightsecurity.com/mailman/listinfo/nmsg-dev"

but:
$ host !$
host lists.farsightsecurity.com
Host lists.farsightsecurity.com not found: 3(NXDOMAIN)

where did it go? (asking because I have a question about python nmsg and debian buster support of same)

docbook autoconf check uses a different stylesheet than the one used to build the manpage

the autoconf macro AX_CHECK_DOCBOOK_XSLT_MIN checks that the non-namespaced version of the DocBook stylesheets can be used:

http://docbook.sourceforge.net/release/xsl/current/...

while the Makefile actually uses the namespaced version:

http://docbook.sourceforge.net/release/xsl-ns/current/...

on debian systems the non-namespaced stylesheets are in the docbook-xsl package while the namespaced stylesheets are in the separate docbook-xsl-ns package, so it is possible for the configure check to succeed while the actual xsltproc invocation to build the manpage fails. the autoconf check needs to be updated in order to check for the namespaced stylesheets.

Buster - it's the new Wheezy / Stretch (or something like that)

We're well into the lifecycle of Debian Buster (10.6 even!) could the nmsg stack get debian packages built and published for buster please?

thanks!

README updates for msgmod/fltmod plugins

Update the README on how to build and install msgmod/fltmod plugins, etc., per Henry's comments on #41 here: #41 (comment).

nmsgtool incomplete json output?

Hi,

I have nmsg file with vendor base and type dnsqr.
When I use 'pres' output (nmsgtool -r file.nmsg -o -) I have full output with all sections (ANSWER, AUTHORITY, etc).

But when I use 'json' output format (nmsgtool -r file.nmsg -J -) I didnt see any answer, authority section data.

Can I get same data in 'json' output as in 'pres' somehow? It will be very convinient for later processing.

release tarballs should install manpage even without docbook

release tarballs currently do not install the nmsgtool(1) manpage if the docbook toolchain (and related dependencies) aren't installed even though they ship with a compiled doc/docbook/nmsgtool.1 file.

possibly commit doc/docbook/nmsgtool.1 (even though it is a generated file) to the repository, so that dist_man_MANS = doc/docbook/nmsgtool.1 can be made into an unconditional rule (and not dependent on BUILD_MAN).

or possibly come up with a workaround in configure.ac / Makefile.am that installs the nmsgtool.1 file if it's present in the source tree.

nmsg_input: export statistics from seqsrc tracking

there is no public interface exporting the results from seqsrc tracking. possibly we could have the nmsg_input implementation keep aggregate counters and export them via some new functions such as:

/**
* For UDP datagram socket nmsg_input_t objects, retrieve the total
* number of NMSG containers that have been received.
*
* \param[in] input UDP socket based NMSG input object.
* \param[out] count Total number of NMSG containers received by the
* nmsg_input_t object during its lifetime.
*
* \return #nmsg_res_success
* \return #nmsg_res_failure
*/
nmsg_res
nmsg_input_get_count_container_received(nmsg_input_t input,
                                        uint64_t *count);

/**
* For UDP datagram socket nmsg_input_t objects, retrieve the total
* number of NMSG containers that been dropped. Sequence number
* tracking must have been previously enabled by a call to
* #nmsg_input_set_verify_seqsrc().
*
* \param[in] input UDP socket based NMSG input object.
* \param[out] count Number of NMSG containers determined to have
* been dropped by the nmsg_input_t object since sequence number
* tracking was enabled.
*
* \return #nmsg_res_success
* \return #nmsg_res_failure
*/
nmsg_res
nmsg_input_get_count_container_dropped(nmsg_input_t input,
                                       uint64_t *count);

stale link to libxs

README.md in nmsg points to http://www.crossroads.io/ which appears to be down. After a lot of searching and bad links, I found a libxs v1.2.0 source blob at
http://ftp.de.debian.org/debian/pool/main/libx/libxs/libxs_1.2.0.orig.tar.gz

Sha256 Checksum: 525dc999cc6524779bc4eef510e423077b7f7ea491cb3ad6d8056ecaf99ff515

Need to update README.md to use it.

Maybe also update README.me to explain why it might be ok to configure --without-ns

Possible null pointer dereference

Internal jenkins' run with cppcheck reported:

State File Line Severity Type Inconclusive Message

unchanged nmsg/input_frag.c 226 error nullPointer false
Possible null pointer dereference: nmsg - otherwise it is redundant to
check if nmsg is null at line 227

unchanged nmsg/input_nmsg.c 62 error nullPointer false
Possible null pointer dereference: msg - otherwise it is redundant to
check if msg is null at line 63

nmsg documentation

As new channels have started appearing it's once again becoming obvious that the lack of nmsg documentation is slowing down adoption. While nmsgtool does produce a splash page of options, it lacks examples showing the correct combination of commands/syntax, the required support libraries, where to obtain those libraries and how to read the nmsg binary files with C and python.

I don't consider "source code" as documentation. :)

In summary, documentation regarding nmsg and nmsgtool would be greatly appreciated.

Andy

Replacement for libxs

So it seems that crossroads.io and libxs are pretty much dead. http://www.crossroads.io/ is down again and it's been a year since code was pushed to https://github.com/crossroads-io/libxs. Perhaps we should switch to another, more active, library for the functionality that libxs provides.

Compile warnings on OS X

The newest release of nmsg has two complaints when building on OS X.

Struct timespec warnings

In several places the following warning shows up:

...
In file included from ./nmsg/nmsg.h:93:
./nmsg/input.h:262:15: warning: declaration of 'struct timespec' will not be
      visible outside of this function [-Wvisibility]
                     struct timespec *ts, nmsg_message_t **msg, size_t *n_msg);
                            ^
1 warning generated.
...

Comparison always true

This doesn't complain on Linux, just OS X:

...
nmsg/res.c:43:10: warning: comparison of constant 16 with expression of type
      'enum nmsg_res' is always true
      [-Wtautological-constant-out-of-range-compare]
            val <= sizeof(res_strings) / sizeof(char *) &&
            ~~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
...

Both fixes are pushed to my fork and a PR has been issued: mschiffm@8c51879

warning: cast from pointer to integer of different size

On NetBSD I saw:
fltmod/nmsg_flt_sample.c: In function 'sample_thread_init':
fltmod/nmsg_flt_sample.c:223:65: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
uint32_t seed = (unsigned) tv.tv_sec + (unsigned) tv.tv_usec + (unsigned) pthread_self();

I didn't see on LInux, both x86_64 architecture. Both with same sizes of pthread_t and unsigned and int and uintptr_t. (My NetBSD gcc is 5.5.0 and debian gcc is 6.3.0.)

My workaround is:

--- a/fltmod/nmsg_flt_sample.c
+++ b/fltmod/nmsg_flt_sample.c
@@ -220,7 +220,7 @@ sample_thread_init(void *mod_data, void **thr_data)
        /* Initialize state->xsubi, seed for this thread's random generator. */
        struct timeval tv = {0};
        gettimeofday(&tv, NULL);
-       uint32_t seed = (unsigned) tv.tv_sec + (unsigned) tv.tv_usec + (unsigned) pthread_self();
+       uint32_t seed = (unsigned) tv.tv_sec + (unsigned) tv.tv_usec + (uintptr_t) pthread_self();
        memcpy(state->xsubi, &seed, sizeof(seed));

check return value of fchown()

src/process_args.c:45:10: warning: ignoring return value of ‘fchown’, declared with attribute warn_unused_result [-Wunused-result]

libnmsg needs a way to extract payload size from nmsg_message_t

Something like: size_t nmsg_message_get_payload_size(nmsg_message_t msg)

need way to get maximum val_idx for a field

This is the only current way to get all instances of a field:

for (val_idx = 0; ; ++val_idx)  {
    res = nmsg_message_get_field_by_idx(...val_idx...)
    if (res != nmsg_res_success)
         break;

Robert said "There ought to be a better way (maybe some new function) ..."
I added "Another tolerable way might be a new error value for val_idx>max, but that would be less convenient and might be slower than a function that simply returns the maximum.

Errors displaying empty string fields.

NMSG data with zero-length string fields presented the following issues on display:

• In JSON format, nmsgtool experienced a segmentation fault.
• In presentation format, the field value was displayed as "(null)" rather than the empty string.

Issues reproduced in nmsg 0.13.0