greynoise-intelligence/pygreynoise

Python3 library and command line for GreyNoise

pygreynoise's Issues

Output as: XML

Jinja2 exception: no filter named 'max'

Python version:

$ python3 --version
Python 3.5.2

OS: Ubuntu 14.04

$ greynoise
Traceback (most recent call last):
  File "/home/andrew/.local/bin/greynoise", line 9, in <module>
    load_entry_point('greynoise==0.2.0', 'console_scripts', 'greynoise')()
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/subcommand.py", line 32, in wrapper
    output = formatter(result, obj["verbose"]).strip("\n")
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/formatter.py", line 44, in wrapper
    output = function(*args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/formatter.py", line 105, in gnql_query_formatter
    return template.render(results=results, verbose=verbose)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/templates/macros.txt.j2", line 29, in template
    {%- set left_width = elements | map(attribute=field_name) | map('length') | max %}
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 986, in render
    return concat(self.root_render_func(self.new_context(vars)))
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/templates/gnql_query.txt.j2", line 1, in top-level template code
    {% import "macros.txt.j2" as macros %}
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/templates/macros.txt.j2", line 29, in template
    {%- set left_width = elements | map(attribute=field_name) | map('length') | max %}
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 515, in _generate
    return generate(source, self, name, filename, defer_init=defer_init)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 62, in generate
    generator.visit(node)
  File "/usr/lib/python3/dist-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 816, in visit_Template
    self.blockvisit(node.body, frame)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 492, in blockvisit
    self.visit(node, frame)
  File "/usr/lib/python3/dist-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 1181, in visit_Macro
    macro_frame = self.macro_body(node, frame)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 730, in macro_body
    self.blockvisit(node.body, frame)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 492, in blockvisit
    self.visit(node, frame)
  File "/usr/lib/python3/dist-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 1400, in visit_Assign
    self.visit(node.node, frame)
  File "/usr/lib/python3/dist-packages/jinja2/visitor.py", line 38, in visit
    return f(node, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 1565, in visit_Filter
    self.fail('no filter named %r' % node.name, node.lineno)
  File "/usr/lib/python3/dist-packages/jinja2/compiler.py", line 427, in fail
    raise TemplateAssertionError(msg, lineno, self.name, self.filename)
jinja2.exceptions.TemplateAssertionError: no filter named 'max'

Add time series

Use pandas : https://chrisalbon.com/python/pandas_time_series_basics.html

Installation error, missing Click

error: The 'click' distribution was not found and is required by greynoise, click-default-group

It's resolved by manually running pip3 install Click==7.0 then re-running python3 setup.py install --user

OS: Ubuntu 16.04
Python version: Python 3.5.2

Bulk data download options

No results when using wildcards with JSON

greynoise -o json "raw_data.web.paths:*phpunit*" produces no results
greynoise "raw_data.web.paths:*phpunit*" produces results

JSON format from raw to newline delimited JSON for easy parsing

Long running commands crash with an HTTP timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/greynoise", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 101, in wrapper
    return function(api_client, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 29, in wrapper
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 59, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 131, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/subcommand.py", line 96, in query
    results = [api_client.query(query=query) for query in queries]
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/subcommand.py", line 96, in <listcomp>
    results = [api_client.query(query=query) for query in queries]
  File "/usr/local/lib/python3.7/site-packages/greynoise/api.py", line 140, in query
    response = self._request(self.EP_GNQL, params={"query": query})
  File "/usr/local/lib/python3.7/site-packages/greynoise/api.py", line 98, in _request
    url, headers=headers, timeout=self.timeout, params=params, json=json
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='enterprise.api.greynoise.io', port=443): Read timed out. (read timeout=7)

I propose a long timeout (like one minute)

Recreate: $ greynoise query -o json "raw_data.web.paths:*wp-config*" | jq -r '.[].data[].raw_data.web.paths[]' | grep "wp-config" | sort | uniq -c | sort -nr

Tab autocomplete for facets

metadata, classification, raw_data, etc

Optimize greynoise analyze subcommand by doing a bulk quick check prior to attempting to look up context

Sweet colors

Receiving Jinja2 error

[andrew] ~/GreyNoise/Projects/pygreynoise $ git branch
* master
  preserve-cli-functionality
[andrew] ~/GreyNoise/Projects/pygreynoise $ git checkout
Your branch is up to date with 'origin/master'.
[andrew] ~/GreyNoise/Projects/pygreynoise $ git pull
Already up to date.
[andrew] ~/GreyNoise/Projects/pygreynoise $ python3 setup.py install
running install
running bdist_egg
running egg_info
writing src/greynoise.egg-info/PKG-INFO
writing dependency_links to src/greynoise.egg-info/dependency_links.txt
writing entry points to src/greynoise.egg-info/entry_points.txt
writing requirements to src/greynoise.egg-info/requires.txt
writing top-level names to src/greynoise.egg-info/top_level.txt
reading manifest file 'src/greynoise.egg-info/SOURCES.txt'
writing manifest file 'src/greynoise.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.14-x86_64/egg
running install_lib
running build_py
creating build/bdist.macosx-10.14-x86_64/egg
creating build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/gncli.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/gnutils.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/util.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/client.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/__init__.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
creating build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
copying build/lib/greynoise/cli/formatter.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
copying build/lib/greynoise/cli/__init__.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
copying build/lib/greynoise/cli/parser.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
creating build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/ip_context.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/gnql_stats.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/gnql.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/actors.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/macros.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/ip_multi_quick_check.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/templates/ip_quick_check.txt.j2 -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/templates
copying build/lib/greynoise/cli/subcommand.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
copying build/lib/greynoise/cli/parameter.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise/cli
copying build/lib/greynoise/api.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/cli.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
copying build/lib/greynoise/exceptions.py -> build/bdist.macosx-10.14-x86_64/egg/greynoise
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/gncli.py to gncli.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/gnutils.py to gnutils.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/util.py to util.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/client.py to client.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/formatter.py to formatter.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/parser.py to parser.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/subcommand.py to subcommand.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli/parameter.py to parameter.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/api.py to api.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/cli.py to cli.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/greynoise/exceptions.py to exceptions.cpython-37.pyc
creating build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/PKG-INFO -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/SOURCES.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/dependency_links.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/entry_points.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/not-zip-safe -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/requires.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying src/greynoise.egg-info/top_level.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
creating 'dist/greynoise-0.1.5-py3.7.egg' and adding 'build/bdist.macosx-10.14-x86_64/egg' to it
removing 'build/bdist.macosx-10.14-x86_64/egg' (and everything under it)
Processing greynoise-0.1.5-py3.7.egg
removing '/usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg' (and everything under it)
creating /usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg
Extracting greynoise-0.1.5-py3.7.egg to /usr/local/lib/python3.7/site-packages
greynoise 0.1.5 is already the active version in easy-install.pth
Installing greynoise script to /usr/local/bin

Installed /usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg
Processing dependencies for greynoise==0.1.5
Searching for requests==2.21.0
Best match: requests 2.21.0
Adding requests 2.21.0 to easy-install.pth file

Using /usr/local/lib/python3.7/site-packages
Searching for dicttoxml==1.7.4
Best match: dicttoxml 1.7.4
Processing dicttoxml-1.7.4-py3.7.egg
dicttoxml 1.7.4 is already the active version in easy-install.pth

Using /usr/local/lib/python3.7/site-packages/dicttoxml-1.7.4-py3.7.egg
Searching for click-default-group==1.2.1
Best match: click-default-group 1.2.1
Processing click_default_group-1.2.1-py3.7.egg
click-default-group 1.2.1 is already the active version in easy-install.pth

Using /usr/local/lib/python3.7/site-packages/click_default_group-1.2.1-py3.7.egg
Searching for Click==7.0
Best match: Click 7.0
Adding Click 7.0 to easy-install.pth file

Using /usr/local/lib/python3.7/site-packages
Searching for chardet==3.0.4
Best match: chardet 3.0.4
Adding chardet 3.0.4 to easy-install.pth file
Installing chardetect script to /usr/local/bin

Using /usr/local/lib/python3.7/site-packages
Searching for certifi==2019.3.9
Best match: certifi 2019.3.9
Adding certifi 2019.3.9 to easy-install.pth file

Using /usr/local/lib/python3.7/site-packages
Searching for urllib3==1.24.1
Best match: urllib3 1.24.1
Adding urllib3 1.24.1 to easy-install.pth file

Using /usr/local/lib/python3.7/site-packages
Searching for idna==2.8
Best match: idna 2.8
Adding idna 2.8 to easy-install.pth file

Using /usr/local/lib/python3.7/site-packages
Finished processing dependencies for greynoise==0.1.5
[andrew] ~/GreyNoise/Projects/pygreynoise $ greynoise
Traceback (most recent call last):
  File "/usr/local/bin/greynoise", line 11, in <module>
    load_entry_point('greynoise==0.1.5', 'console_scripts', 'greynoise')()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
    return self.resolve()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg/greynoise/cli/__init__.py", line 10, in <module>
    from greynoise.cli.subcommand import actors, gnql, ip, setup
  File "/usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg/greynoise/cli/subcommand.py", line 8, in <module>
    from greynoise.cli.formatter import FORMATTERS
  File "/usr/local/lib/python3.7/site-packages/greynoise-0.1.5-py3.7.egg/greynoise/cli/formatter.py", line 10, in <module>
    from jinja2 import Environment, PackageLoader
ModuleNotFoundError: No module named 'jinja2'

Support CVE in ip, query, stats, and analyze

Agnostic filtering from STDIN

Identify and analyze / filter IP addresses from data coming into STDIN

Asciinema screencast examples

For installation, usage, etc

Auto-scroll abstraction option for large queries

Automatically cycle through pages with scroll token

IP command should not break when new fields are added to the JSON results in the REST API

Support spoofable in ip, query, stats, and analyze

Examples for everything

Update docs to include CLI documentation + examples

Implement tests

Feature: push on pypi

https://pypi.python.org/pypi

Increase result size

From @andrew-morris :
There's a hidden feature that allows you to include an API key POST param to increase the amount of results you get from 500 to 5,000. I'll add a config file.

Refactor CLI arguments

• Query string should always be the final argument(s)
• Query string should not need to be encapsulated in quotes, CLI should be smart enough to combine all final query facets together with a join(" ")

`-k` option for API key not working, needs swapped

pygreynoise/src/greynoise/cli/subcommand.py

Line 39 in e818d95

@click.option("-k", "--api-key", required=True, help="Key to include in API requests")

Click documentation seems to suggest the long argument should always be first so "--api-key", "-k" instead.

Add support for (non-enterprise) api.greynoise.io

I've had a poke with this library, and although it would be really useful, it looks like the enterprise.api.greynoise.io endpoints are different from those on your public api. (Not that I noticed this until after I'd changed the GreyNoise object's BASE_URL to point to https://api.greynoise.io/v1...)

I'm not sure whether the enterprise API is planned to be a completely distinct product, if so then this incompatibility is understandable, but otherwise a flag to initialise the object in 'free' mode which would point it to the correct endpoints would be very handy.

In the mean time, I can just write the requests by hand which isn't going to be an issue.

greynoise setup not working

andrew@rpi:~/.local/bin $ ./greynoise setup -k

Error: API key not found.

To fix this problem, please use any of the following methods (in order of precedence):
- Pass it using the -k/--api-key option.
- Set it in the GREYNOISE_API_KEY environment variable.
- Run 'greynoise setup' to save it to the configuration file.

andrew@rpi:~/.local/bin $ ./greynoise setup -k XXXXXXXX

Error: API key not found.

To fix this problem, please use any of the following methods (in order of precedence):
- Pass it using the -k/--api-key option.
- Set it in the GREYNOISE_API_KEY environment variable.
- Run 'greynoise setup' to save it to the configuration file.

andrew@rpi:~/.local/bin $ ./greynoise setup --api-key XXXXXXX

Error: API key not found.

To fix this problem, please use any of the following methods (in order of precedence):
- Pass it using the -k/--api-key option.
- Set it in the GREYNOISE_API_KEY environment variable.
- Run 'greynoise setup' to save it to the configuration file.

andrew@rpi:~/.local/bin $ ./greynoise setup

Error: API key not found.

To fix this problem, please use any of the following methods (in order of precedence):
- Pass it using the -k/--api-key option.
- Set it in the GREYNOISE_API_KEY environment variable.
- Run 'greynoise setup' to save it to the configuration file.

andrew@rpi:~/.local/bin $ ./greynoise setup

Error: API key not found.

To fix this problem, please use any of the following methods (in order of precedence):
- Pass it using the -k/--api-key option.
- Set it in the GREYNOISE_API_KEY environment variable.
- Run 'greynoise setup' to save it to the configuration file.

Output as: CSV

Feature: read/write from file

Read IPs from a file
Write results to a file

Update for v1.1 API

From @andrew-morris GreyNoise-Intelligence/api.greynoise.io#9 (comment) :

• Going to separate out the categories of tags
  • Actors
  • Activity
  • Hosting
  • Worms
  • Tools
  • Search Engines
• Allow queries based on
  • CIDR (maybe)
  • Organization
  • ASN
  • rDNS
  • Datacenter
• Combination tag queries
  • e.g. TELNET_WORM_HIGH + RESIDENTIAL
  • or NTP_SCANNER_HIGH - SHODAN (maybe)
• Stats query
  • How many entries of a given tag were generated today? This week? This month?
• Metadata query
  • What does this analytic actually mean?
  • What are some reference pages for this analytic

Prettify JSON in documentation

Modify user agent in alpha, enterprise, and cli usage for aggregate stats

Install error with `--user` on OS X

$ python3 setup.py install --user
running install
error: can't combine user with prefix, exec_prefix/home, or install_(plat)base

TypeError when running quick on stdin

[andrew] ~ $ greynoise query -f json "last_seen:today classification:benign" | jq '.[].data[].ip' -r | head
146.88.240.16
146.88.240.32
40.77.167.109
137.226.113.10
198.108.66.192
213.180.203.34
146.88.240.41
146.88.240.40
104.237.144.22
66.249.79.136

[andrew] ~ $ greynoise query -f json "last_seen:today classification:benign" | jq '.[].data[].ip' -r | tail
185.173.35.21
78.136.44.8
50.57.61.23
139.162.99.58
196.52.43.126
139.162.110.42
92.118.160.9
198.108.67.39
216.218.206.79
172.105.207.40

[andrew] ~ $ greynoise query -f json "last_seen:today classification:benign" | jq '.[].data[].ip' -r | wc -l
    1235

[andrew] ~ $ greynoise query -f json "last_seen:today classification:benign" | jq '.[].data[].ip' -r | greynoise quick -i -
Traceback (most recent call last):
  File "/usr/local/bin/greynoise", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 98, in wrapper
    return function(api_client, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 28, in wrapper
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/decorator.py", line 56, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/greynoise/cli/subcommand.py", line 202, in quick
    results.extend(api_client.quick(ip_addresses=ip_addresses))
  File "/usr/local/lib/python3.7/site-packages/greynoise/api.py", line 148, in quick
    ip_address = api_result["ip"]
TypeError: string indices must be integers

Tab autocomplete for tags

Gracefully handle bad API key

Example of using a bad/expired API key

$ greynoise 8.0.0.0/8
Traceback (most recent call last):
  File "/home/andrew/.local/bin/greynoise", line 9, in <module>
    load_entry_point('greynoise==0.2.0', 'console_scripts', 'greynoise')()
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/subcommand.py", line 25, in wrapper
    result = function(obj, *args, **kwargs)
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/cli/subcommand.py", line 125, in query
    results.append(api_client.run_query(query=query))
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/api.py", line 177, in run_query
    response = self._request(self.EP_GNQL, params={"query": query})
  File "/home/andrew/.local/lib/python3.5/site-packages/greynoise-0.2.0-py3.5.egg/greynoise/api.py", line 94, in _request
    raise RequestFailure(response.status_code, response.content)
greynoise.exceptions.RequestFailure: (401, b'{"error":"forbidden","status":"error"}')

GNQL API endpoint

Need an API abstraction endpoint for GNQL, that supports scrolling, etc etc.

Key error when single non-routable IP is passed to quick() function

While using GreyNoise SDK, a KeyError Exception is raised when any internal IP address is provided.

For example,
greynoise quick 10.0.11.34 returns the following traceback:

  File "C:\Python27\Scripts\greynoise-script.py", line 11, in <module>
    load_entry_point('greynoise==0.3.0', 'console_scripts', 'greynoise')()
  File "c:\python27\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\python27\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\python27\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\python27\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python27\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\python27\lib\site-packages\greynoise\cli\decorator.py", line 105, in wrapper
    return function(api_client, *args, **kwargs)
  File "c:\python27\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\python27\lib\site-packages\greynoise\cli\decorator.py", line 30, in wrapper
    result = function(*args, **kwargs)
  File "c:\python27\lib\site-packages\greynoise\cli\decorator.py", line 60, in wrapper
    return function(*args, **kwargs)
  File "c:\python27\lib\site-packages\greynoise\cli\decorator.py", line 164, in wrapper
    return function(*args, **kwargs)
  File "c:\python27\lib\site-packages\greynoise\cli\subcommand.py", line 95, in quick
    results.extend(api_client.quick(ip_addresses=ip_addresses))
  File "c:\python27\lib\site-packages\greynoise\api.py", line 218, in quick
    ip_address = api_result["ip"]
KeyError: 'ip'

APIs for getting noise status returns different result in case of internal IPs. For instance:

1. In case of noise/quick/{ip_address} endpoint, the response for IP = 10.0.11.34 will be:

{
    "error": "non-routable ip"
}

2. In case of noise/multi/quick endpoint, the response for IP = 10.0.11.34 is:

{
    "ip": "10.0.11.34",
    "noise": false,
    "code": "0x07"
}

In this case, we can solve the KeyError Exception in these two ways:

1. To make the response of both the APIs consistent. But this might have an impact on the existing integrations using the above APIs.
2. Handle the Exception in the code:
   Replace this in api.py line 240 and 260

api_results.append(self._request(endpoint))

with

response = self._request(endpoint)
if "non-routable ip" in response.values():
    response["ip"] = ip_address
    response["noise"] = False
    response["code"] = "0x07"
api_results.append(response)

Quick start for API

version subcommand to include git commit or installation method

IP extraction via regex helper

Update library documentation for readthedocs

Feature: Improve output format

The real magic will be lots of different output methods

• CSV
• JSON
• Greppable
• Pprint

Include API server address as an item in the config file

instead of hardcoding research.api.greynoise.io

greynoise ip context bug when a commonly spoofed ip

greynoise ip context 8.8.8.8
Returns:
Traceback (most recent call last):
File "/usr/local/bin/greynoise", line 11, in
load_entry_point('greynoise==0.2.0', 'console_scripts', 'greynoise')()
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/greynoise-0.2.0-py3.7.egg/greynoise/cli/subcommand.py", line 25, in wrapper
result = function(obj, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/greynoise-0.2.0-py3.7.egg/greynoise/cli/subcommand.py", line 67, in context
results.append(api_client.get_context(ip_address=ip_address))
File "/usr/local/lib/python3.7/site-packages/greynoise-0.2.0-py3.7.egg/greynoise/api.py", line 160, in get_context
response = self._request(endpoint)
File "/usr/local/lib/python3.7/site-packages/greynoise-0.2.0-py3.7.egg/greynoise/api.py", line 98, in _request
raise RequestFailure(response.status_code, body)
greynoise.exceptions.RequestFailure: (200, {'error': 'commonly spoofed ip'})

Output as: JSON

Implement legacy alpha API support in separate endpoints

Jinja template error on stats query via CLI

$ greynoise stats "Google LLC"
Traceback (most recent call last):
  File "/usr/local/bin/greynoise", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/greynoise/cli/decorator.py", line 105, in wrapper
    return function(api_client, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/greynoise/cli/decorator.py", line 39, in wrapper
    output = formatter(result, params.get("verbose", False)).strip("\n")
  File "/usr/local/lib/python2.7/site-packages/greynoise/cli/formatter.py", line 45, in wrapper
    output = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/greynoise/cli/formatter.py", line 115, in gnql_stats_formatter
    return template.render(results=results, verbose=verbose, max_width=max_width)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/greynoise/cli/templates/macros.txt.j2", line 34, in template
    {%- set right_width = elements_slice | map(attribute='count') | map('string') | map('length') | max %}
jinja2.exceptions.TemplateAssertionError: no filter named 'max'

version:

greynoise 0.3.0
  Python 2.7.16
  Darwin-18.7.0-x86_64-i386-64bit```

Smart single IP query

If you receive a query for simple one IP, hit the /v2/noise/context/:ip endpoint, instead of GNQL. Easy to do with a regex.

Allow facets to be specified by user

Inspiration:

$ shodan search --fields ip,org --limit 10 apache