csirtgadgets/cif-v1

DEPRECATED USE v3!

cif-v1's Issues

cif-smrt: -P flag misplacement fails

08:33 < akreffett> I found something wonky with cif_crontool/cif_smrt: depending on where you put a -P on the command line post 
                   processors will either be enabled or not
08:34 < _wes_> :-/
08:34 < _wes_> RC3?
08:34 < akreffett> For instance this does not seem to enable post processors: /cif_smrt -C /opt/cif/cif.conf -r 
                   /opt/cif/etc/misc.cfg -f sshbl.org -d -T medium -A root -N 1 -P
08:35 < akreffett> While this does: /cif_smrt -P -C /opt/cif/cif.conf -r /opt/cif/etc/misc.cfg -f sshbl.org -d -T medium -A root 
                   -N 1
08:35 < _wes_> k
08:35 < _wes_> i've had issues with the underlying library in the past that does this
08:35 < akreffett> Let me check to see if I have all the relevant changesets in
08:35 < _wes_> i think i need to dig into their module.

revisit build instructions

https://groups.google.com/d/topic/ci-framework/gSaG4eS0sCE/discussion
https://groups.google.com/forum/?hl=en&fromgroups=#!topic/ci-framework/gSaG4eS0sCE

cif-smrt: pull file bug

1. File.pm applies the feed_limit, but the other Pull modules do not. ParseCsv and ParseDelim apply the feed_limit again. That causes data to be lost when using Csv/Delim files if the feed_limit index doesn't start with 0. E.g. if feed_limit is 1,5 then @lines comes out of File.pm with 5 members. If it runs through ParseCsv, it gets limited to members 1..5 again, but there is no index 5 anymore. With only 5 members, the highest index is 4. The first member gets dropped, and a null is pushed onto the end. The patch checks the number of elements in the array in ParseCsv and ParseDelim and doesn't apply the limit if the number of members is already correct.

Note: there's a potential error there if the size specified by the feed_limit matches the size of the input exactly, and the Csv/Delim didn't come from File.pm. Like if you want feed_limit = 1,10 to always remove the first line, and the data from an HTTP pull has exactly 10 lines already, the first line won’t be removed.

cif_feed: make 'whitelisting' configurable

malwarepatrol feeds

/opt/cif/bin/cif_smrt -C /home/cif/.cif -r /opt/cif/etc/mpatrol.cfg -f md5 -d -T medium -P at /opt/cif/bin/cif_crontool line 205.
[DEBUG][2013-04-24T21:03:15Z]: fail closed: 0
[DEBUG][2013-04-24T21:03:15Z]: postprocessing enabled…
[DEBUG][2013-04-24T21:03:15Z]: setting up zmq interfaces…
[DEBUG][2013-04-24T21:03:15Z]: sending ctrl warm-up msg…
[DEBUG][2013-04-24T21:03:15Z]: starting sender thread…
[DEBUG][2013-04-24T21:03:16Z]: creating 32 worker threads…
[DEBUG][2013-04-24T21:03:18Z]: done…
[DEBUG][2013-04-24T21:03:18Z]: running preprocessor routine…
[DEBUG][2013-04-24T21:03:18Z]: parsing…
[DEBUG][2013-04-24T21:03:18Z]: pulling feed: http://www.malware.com.br/cgi/submit?action=list_hashes
[cif-smrt] failure: malware.com.br
ERROR: failed to get feed: http://www.malware.com.br/cgi/submit?action=list_hashes
504 Gateway Time-out

check with the authors, see if they need help.

cif-smrt: net-dns miss-call of ->address()

I saw this today while running cif_crontool -p daily -d -P
Not sure it's an issue, but putting it out there.

[DEBUG][2013-07-18T12:02:42Z]: starting with 548 recs...

***
***  WARNING!!!  The program has attempted to call the method
***  "address" for the following RR object:
***
***  0/24.204.21.72.origin.asn.cymru.com.   14399   IN  TXT "16509 | 72.21.192.0/19 | US | arin | 2004-12-30"
***
***  This object does not have a method "address".  THIS IS A BUG
***  IN THE CALLING SOFTWARE, which has incorrectly assumed that
***  the object would be of a particular type.  The calling
***  software should check the type of each RR object before
***  calling any of its methods.
***
***  Net::DNS has returned undef to the caller.
*** 

lwpx-paranoid agent disables proxies

collectiveintel/cif-smrt/pull/55

cif-smrt: cvs parser chokes on header date

On a related note, I used the 1,9999 feed limit because CIF chokes on a header row when one of the columns should be a date:

Can't call method "epoch" on an undefined value at /opt/cif/bin/../lib/CIF/Smrt.pm line 411.

The first row of the Shadowserver CSV is a header:

"timestamp","ip","asn","geo","url","type" …

So it tried to normalize_timestamp() the string “timestamp” and then call epoch() on the result. Maybe it makes sense to have a header row parameter in the config file that makes CIF skip the first line?

documentation: security onion integration

osx .app

https://groups.google.com/forum/#!topic/ci-framework/fSwQSJlpYOk

documentation: whitelist

https://code.google.com/p/collective-intelligence-framework/wiki/Whitelist_v1

cif-smrt: cvs parser bug

1. The regex to match a CSV file doesn't match if the first column has white space in it. E.g. ..

          "2013-08-02 00:00:00","68.102.208.67",22773,"US"
   

... doesn't get matched by /^#?\s?"\S+","\S+"/

iodef-format: iptables/snort output should -j ACCEPT for whitelist

https://groups.google.com/forum/#!topic/ci-framework/DxBisGjBLqw

ipset output plugin

Are there any plans / thoughts about enabling cif to output ipsets instead of raw iptables rules? It seems like ipsets would work better since you can build up a set and then swap the new set with an existing set to update your rules without having to flush it. You could also use the same set for both output and input, referencing it in the appropriate iptables chains.

apikeys not creating

I receive the following error message when attempting to generate the initial apikey to be used by the client.

"Can't insert new CIF::APIKey: DBD::Pg::st execute failed: ERROR: relation "apikeys" does not exist"

I am attempting a new server install and cannot seem to get past this point.
Distro: Ubuntu 12.04

Thank you in advance,

libcif: multiple query parsing

should auto-clean up any spaces that show up in the query:

https://groups.google.com/forum/#!topic/ci-framework/nKiHiBrcJu4

RC4 - cif_apikeys doesn't recognize "--key" option

• lists keys as expected
  • cif_apikeys -k <key>
• gives an error
  • cif_apikeys --key <key>
  • error message:

CIF::APIKey can't SELECT uuid
FROM   apikeys
WHERE  uuid = ?
: DBD::Pg::st execute failed: ERROR:  invalid input syntax for uuid: "ey" [for Statement "SELECT uuid
FROM   apikeys
WHERE  uuid = ?
" with ParamValues: 1='ey'] at /usr/share/perl5/DBIx/ContextualFetch.pm line 52.
 at /usr/share/perl5/Class/DBI/Search/Basic.pm line 169

cif_apikeys remove-all flag

CentOS docfix

re: the comments section of the doc

https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_CentOS6_v1

cif-smrt: postprocessor flag in .cfg

Also, is it possible to more granularly control how that process works? For instance, if I only one one url feed list to be resolved, but not others, is there a cfg directive or something that I can use to do so?

ability to turn on / off postprocessors per feed in the config.

https://groups.google.com/d/msg/ci-framework/q-NYUO5A8p4/3yCwJij-rNoJ

RC4 - cif_apikeys description option not working

A few different issues

• cif_apikeys -a -u [email protected] -g group -G group -D group
  • doesn't add description
• cif_apikeys -a -u [email protected] -g group -G group --desc group
  • doesn't add description either
• cif_apikeys -a -u [email protected] -D group -g group -G group
  • when put in this position in the string, the description isn't added, and the specified group isn't used (the default "everyone" group is used)
• cif_apikeys -D group -a -u [email protected] -g group -G group
  • when put in this position, no user is created, and it just lists all the keys

cif_smrt ENV proxy settings

https://groups.google.com/forum/#!topic/ci-framework/GUlm675bHZM

-m flag (back in time)

[/opt/cif/bin/cif]

If the -m flag had the ability to specify how many minutes/hours back in time you wanted to go, one could use that in combination with a cronjob to display the new data in x minutes.

For example

cif -q 192.168.1.0/24 -m -min 60

This would give you all the results for 192.168.1.0/24 in the previous 60 minutes. Combine this with with an hourly cron and you get the ability to show everything new in the last 60 minutes.

cif_vacuum is spelled wrong.

dumbass.

iodef-pb: SiLK output plugin

http://infosecmatters.blogspot.co.uk/2013/05/using-cif-with-silk.html
http://tools.netsa.cert.org/silk/docs.html#analysis-set-bag-pmap

clean up cmd option flags

https://groups.google.com/d/msg/ci-framework/ovTSb3ts2gU/unR1YA2nY9oJ

https://github.com/collectiveintel/cif-smrt/commit/45ba16ddf4d25e6caf48baf0e5a360a458acb90c

access control: cif_apikeys -Enable flag

yea, technically is possible (we have the same issue), i'm just working out code "to make it easier"..

something like:

$ cif_apikeys -u [email protected] -E 'domain/botnet,AS701,malware/md5,AS123'

https://groups.google.com/d/msg/ci-framework/LKZTkJ5qero/LXzt1WiIXm0J

Issues with Client configuration

Can't locate Iodef/Pb/Simple.pm in @inc (@inc contains: /opt/cif/bin/../../libcif/lib /opt/cif/bin/../local/lib /opt/cif/bin/../lib /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at /opt/cif/bin/../lib/CIF/Client.pm line 13.
BEGIN failed--compilation aborted at /opt/cif/bin/../lib/CIF/Client.pm line 13.
Compilation failed in require at /opt/cif/bin/cif line 33.
BEGIN failed--compilation aborted at /opt/cif/bin/cif line 33.

cif-smrt SIGINT code

cif-smrt, cif-feeds, etc

v0 migration tools

documentation: cif-protocol

iodef-format: snort output incorrect for whitelist

Also, the Snort plugin perl module throws an uninitialized value error on line 45 when running the whitelist query:
Use of uninitialized value in lc at /usr/local/share/perl/5.14.2/Iodef/Pb/Format/Snort.pm line 45.

Here's a diff of my fix:

$ diff /usr/local/share/perl/5.14.2/Iodef/Pb/Format/Snort.pm /usr/local/share/perl/5.14.2/Iodef/Pb/Format/Snort.pm.orig
45,46c45
<       my $severity = ($_->{'severity'}) ? lc($_->{'severity'}) : 'none';
<         for($severity){

---
>         for(lc($_->{'severity'})){

ref: https://groups.google.com/forum/#!topic/ci-framework/DxBisGjBLqw

update RHEL bind doc

https://code.google.com/p/collective-intelligence-framework/wiki/BindSetup_v1

see comments.

Iptables output plugin error for ip addresses with '0' as an octet

Issue 1: If an IP address in a list contains a '0', the Iptables plugin omits that octet, resulting in a malformed IP address.

Fix to Iodef/Pb/Format/Iptables.pm:

45c45
<         next unless(/^0{1,2}/);

---
>         next unless(/^0{1,2}[1-9]{1,2}/);

Output diff between the original Iptables.pm and the fixed Iptables.pm:

< iptables -A CIF_IN -s 222.36..48 -j DROP
< iptables -A CIF_OUT -d 222.36..48 -j DROP

---
> iptables -A CIF_IN -s 222.36.0.48 -j DROP
> iptables -A CIF_OUT -d 222.36.0.48 -j DROP

< iptables -A CIF_IN -s 177..161.133 -j DROP
< iptables -A CIF_OUT -d 177..161.133 -j DROP

---
> iptables -A CIF_IN -s 177.0.161.133 -j DROP
> iptables -A CIF_OUT -d 177.0.161.133 -j DROP

< iptables -A CIF_IN -s 68.144..18 -j DROP
< iptables -A CIF_OUT -d 68.144..18 -j DROP

---
> iptables -A CIF_IN -s 68.144.0.18 -j DROP
> iptables -A CIF_OUT -d 68.144.0.18 -j DROP

cif_smrt: allow for processing of multiple feed files in a directory

https://groups.google.com/forum/?fromgroups=#!topic/ci-framework/ppM7Aza_3c8

tool to remove things like searches

need a way to purge data-sets from the database based on description or assessment, etc...

feed retention policies

limit_days (7)
feed_retention (7)

https://github.com/collectiveintel/libcif-dbi/blob/v1/lib/CIF/Feed.pm#L85

installer fixes

installer gets confused with certain updates...

https://code.google.com/p/collective-intelligence-framework/wiki/Upgrade_v1?ts=1376308997&updated=Upgrade_v1#v1-RC2

query testing

• guid testing (make sure group A can't see group B)
• data-type testing (make sure email addresses don't show up in the domains feed, and vs versa)
• build this into the ci-framework testing kit
• other mix-match apikey type testings??

dns proxy option for resolver code

Use case: I have a DNS resolver specifically for use by CIF and would like to point only CIF at this resolver without modifying the local system resolvers.  Think HTTP proxy for DNS.

I will hack this in to the Fqdn.pm module locally but a general solution would be the best fix and probably of use to others as well.

https://groups.google.com/forum/?hl=en&fromgroups=#!topic/ci-framework/dvwGdgCfFT8

document "filter by source uuid" flag in client

doc: group mappings

https://groups.google.com/d/msg/ci-framework/Ya7RqUF6OQ0/jW2BEoD7DR4J

RC4 - cif_apikeys doesn't recognize "-d" option

• displays the help menu
  • cif_apikeys -d -k <key>
• deletes key as expected
  • cif_apikeys --delete -k <key>

cif-smrt: execute something else and get result into cif-smrt

Executable pull method. Execute a program and insert whatever it outputs. This is a possible alternative method for the Shadowserver bot feeds. It would also support anything that needs to be fetched via complex API calls rather than simple HTTP pulls. E.g. it could support data pulled from Arbor or Titan or Symantec’s reputation feeds.

csv output -- turn off the magic bits

cif-smrt: pull file bug

1. Using a feed_limit value greater than the number of rows in the data generates a bunch of "uninitialized variable" warnings in CIF/Smrt/Plugin/Pull/File.pm. I.e. ...
   @lines = @lines[$start..$end];

... pads @lines with null values if $end is higher than $#lines.

FF plugin, ubuntu cpu spike / hang

https://groups.google.com/forum/#!topic/ci-framework/cN1D5OkZOTw

API Issue (Formatting)

Was the "fmt" option deprecated from v0 to v1? I'd like to get a table return instead of JSON as I am using the API to query from ESM.

apikeys not creating

I receive the following error message when attempting to generate the initial apikey to be used by the client.

"Can't insert new CIF::APIKey: DBD::Pg::st execute failed: ERROR: relation "apikeys" does not exist"

I am attempting a new server install and cannot seem to get past this point.
Distro: Ubuntu 12.04

Thank you in advance,

cif-smrt mutex lock checking

https://groups.google.com/forum/#!topic/ci-framework/qY44C7ieeqQ