cert-polska/mwdblib

Client library for the mwdb service by CERT Polska.

mwdblib's Issues

CLI: mwdb fetch confirmation is printed to stdout

Confirmation printed to stdout interferes with mwdb fetch <id> - which prints output to stdout.

We need to rework that part a bit

CLI: Implement caching of session token for password-authenticated sessions

Currently, every CLI command performs /auth/login request if password was stored using mwdb login command.

It would be nice to securely store session token for multiple password-authenticated CLI invocations.

Add website to the repository sidebar

Might be a nice addition to have the docs website in the sidebar of this reposiotry.

CLI: Use click.Choice for formatting options

not a problem of this PR, but not a fan of manually listing (and checking) many options. Maybe in the future consider click.Choice (found this by googling: pallets/click#605)

Originally posted by @msm-code in #43 (comment)

`self.flush` removes "type" key from data

mwdblib/mwdblib/object.py

Lines 347 to 352 in 9e2e55b

	def flush(self) -> None:
	"""
	Flushes local object state in case of pending updates.
	All object-specific properties will be lazy-loaded using API
	"""
	self.data = {"id": self.data["id"]}

while type is expected to be loaded eagerly

mwdblib/mwdblib/object.py

Lines 92 to 96 in 9e2e55b

	def object_type(self) -> str:
	"""
	Object type ('file', 'static_config' or 'text_blob')
	"""
	return cast(str, self.data["type"])

mwdblib CLI list is shrinked with latest beautifultable

/usr/local/lib/python3.8/dist-packages/beautifultable/utils.py:136: FutureWarning: 'max_width' has been deprecated in 'v1.0.0' and will be removed in 'v1.2.0'. Use 'maxwidth' instead.
  warnings.warn(message, FutureWarning)

API authentication is broken when using the cli

Storing the API key with mwdb login command silently fails:

[user@shell]$ mwdb login -a "<valid api key>"
[user@shell]$ mwdb search "name:file.exe"
Error: Not authenticated. Use `mwdb login` first to set credentials.
Aborted!

First half of the issue lies in how store_credentials method of APIClientOptions class validates its arguments. The method will silently fail if self.username is not set. This is likely due to keyring's set_password method requiring some username to store any secret, even an API key.

The second half is in the login command handler (login_command in mwdb/cli/login.py). This function incorrectly calls set_api_key as a member of the mwdb object rather than its member api (which is an APIClient instance). However even if it correctly invokes set_api_key, the API key won't be stored due to the aforementioned issue in store_credentials. Thus login_credentials has to ensure that the username is set prior to storing the API key.

[2.9.0] Add support for chunk count param in listing methods

Add support for CERT-Polska/mwdb-core#755 to improve listing time (useful for statistics evaluation)

Allow to store credentials in plaintext, without keyring

In some keyring implementations, it's really difficult to keep keyring opened for a longer time and not being asked for keyring password periodically.

We should provide an optional way to store credentials directly in ~/.mwdb file e.g. via additional mwdb login flag

listen_for_objects: Prevent iterating over whole database when pivot object has been deleted

Additionally check upload time if current object was not uploaded before the last object.

Interrupt fetching next objects and show warning in that case.

[cli] Store API url provided in `mwdb login` command

Current behavior

Currently we don't store API url so it must be provided to all of the commands. It's not very convenient for self-hosted Malwarecage instances.

$ mwdb --api-url http://127.0.0.1/api/ login
Username: admin
Password: 

results in ~/.mwdb contents:

[mwdb]
username = admin

Expected behavior

$ mwdb login http://127.0.0.1/api/
Username: admin
Password: 

$ cat ~/.mwdb
[mwdb]
username = admin
api_url = http://127.0.0.1/api/

Additional TODO: Change in documentation needed.

'json' parameter handling is no longer required in APIClient.request

https://github.com/CERT-Polska/mwdblib/blob/master/src/api.py#L130-L133

Current API uses multipart/form-data only for file upload. Additional options are passed via options field instead of json (https://github.com/CERT-Polska/mwdblib/blob/master/src/core.py#L579)

Support ~/.mwdb in Python as well

Authenticating using mwdb = MWDB() from python should honor the ~/.mwdb config file, like it does with the CLI.
Code reference:
https://github.com/CERT-Polska/mwdblib/blob/master/src/cli/authenticator.py#L15

use of deprecated functions

mwdb get $REDACTED_HASH
Please enter password for encrypted keyring:
/home/bonus/.local/lib/python3.9/site-packages/beautifultable/utils.py:125: FutureWarning: 'BeautifulTable.append_row' has been deprecated in 'v1.0.0' and will be removed in 'v1.2.0'. Use 'BTRowCollection.append' instead.
  warnings.warn(message, FutureWarning)
/home/bonus/.local/lib/python3.9/site-packages/beautifultable/utils.py:125: FutureWarning: 'BeautifulTable.get_string' has been deprecated in 'v1.0.0' and will be removed in 'v1.2.0'. Use 'str' instead.
  warnings.warn(message, FutureWarning)
 File name:    Mozi.m.elf
 File size:    11.6 kB
 File type:    ELF 32-bit MSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, no section header

search_files - AttributeError: 'str' object has no attribute 'get'

When searching using mwdblib against my local instance of mwdb, mwdblib throws this error.
When using the search params 'tag:mirai' on the public cert.pl instance, it works just fine.

from mwdblib import MWDB
from configparser import ConfigParser

config = ConfigParser()
config.read('mwdb.ini')

mwdb = MWDB(api=config.get('mwdb', 'url'), api_key=config.get('mwdb', 'api'))

results = mwdb.search_files('tag:plugx')

for r in results:
    print(r.sha256)
    
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-56-f6c20a1a115a> in <module>
----> 1 for r in results:
      2     print(r.sha256)

~/venv/lib/python3.8/site-packages/mwdblib/core.py in _recent(self, object_type, query)
    103                     params["query"] = query
    104                 # 'object', 'file', 'config' or 'blob'?
--> 105                 result = self.api.get(object_type.URL_TYPE, params=params)
    106                 key = object_type.URL_TYPE + "s"
    107                 if key not in result or len(result[key]) == 0:

AttributeError: 'str' object has no attribute 'get'

Thoughts? Python version issue maybe...?

Feature request: implement remove_comment function

Currently it's not possible to remove comments from samples, even though it's possible to add them with add_comment().
There are functions to remove tags and attributes, and it would be helpful if it's also possible to remove comments.

Comments already have a unique ID associated with them which could be used as the parameter to a new remove_comment() function.

I have not checked if mwdb-core needs support for this or if a DELETE request is already handled.

Support for new attributes features from mwdb-core

Feature Category

• Correctness
• Performance

Describe the problem

New attribute api for integration with objects has been introduced in mwdb-core in mwdblib there is legacy version of it.
Furthermore JSON attributes was introduced so we need to expand attributes in mwdblib.

Describe the solution you'd like

• Adaptation of the current mwdblib api functions to the new api attributes in mwdb-core
• Main naming changed from metakey to atrributes
• Expand attributes for JSON support
• It would be nice to safe backwards compatibility for the old metakeys.

Implementation steps

1. Analyze mwdb-core api references from mwdblib related to metakeys and attributes
2. Adapting mwdblib functions to the new mwdb-core api, changing references to new endpoints.
3. Change of naming from metakeys to attributes
4. Check the code for backward compatibility to metakeys
5. Introduction of JSON attributes

Implement support for Karton analysis API

Allow user to specify search query in "listen_for_objects"

Being able to query specific objects could be pretty useful.
Current workaround is probably to get all available objects and then do the filtering which is a lot less convenient and efficient.

Usage example:

for config in mwdb.listen_for_configs(query="family:danabot"):
     print(config.cfg)

Delete sample functionality

Removing stored samples from mwdb Web UI is already possible in few clicks, yet the functionality is not available though any API endpoint. It would be a valuable addition, considering the automation / scripting functionality offered by mwdblib.

On the one hand, it could be integrated as an additional method within mwdblib.file.MWDBFile, for single delete operation, or in addition, introduce a "bulk" delete function consuming results from, e.g MWDB.recent_files(), etc.

Document -oshort

There is an undocumented -o short option:

mwdblib

$ mwdb search "meta.karton2:12117120-fdc0-4f4d-9798-738b17450e51" -oshort
b5f0aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa60

But the docs are silent about it:

  -o, --output TEXT    Format attributes separated by commas. Supported
                       values: nocolor, nopager, nohuman

Change docs to the Read-The-Docs theme

It is nicer and easier to navigate :)

More useful access to attributes

One of use-cases found during some code review:

def remove_attribute(sample: mwdblib.MWDBFile, key: str, value: Any = None) -> None:
    for attribute in sample.attributes[key]:
        if value is None or attribute.value == value:
            sample.remove_attribute(attribute.id)

Code above is wrong but shows expected usage of new API.

get_attributes_detailed exposes identifiers, but attributes are not grouped by key. In addition, it would be nice to have them in named tuples (accessible by getattr, .value) instead of regular dicts with ["value"] syntax to access the value

get_terminal_size

Getting this error when running a query using the mwdb cli tool:

  File "/home/name/venv_mwdb/lib/python3.9/site-packages/mwdblib/cli/formatters/tabular.py", line 34, in format_table
    term_width, term_height = click.get_terminal_size()
AttributeError: module 'click' has no attribute 'get_terminal_size'

Should this be os.get_terminal_size() instead of click.get_terminal_size()? https://github.com/CERT-Polska/mwdblib/blob/master/mwdblib/cli/formatters/tabular.py#L20

Requirements not covering all dependencies for proper working

I encountered that requirements.txt is not covering all dependencies for mwdblib to work.
Missing dependencies:

• click
• keyring
• beautifultable
• humanize
• click_default_group

Error encountered in attached file hardtimes.txt

`api_key` doesn't work on Python 3.5

mwdb = Malwarecage(api_key=MWDB_TOKEN)
Traceback (most recent call last):
  File "/home/[redacted]/lib/python3.5/site-packages/mwdblib/api.py", line 79, in set_api_key
    self.logged_user = json.loads(base64.b64decode(self.api_key.split(".")[1] + "=="))["login"]
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/[redacted]/lib/python3.5/site-packages/mwdblib/core.py", line 51, in __init__
    self.api = api or MalwarecageAPI(**api_options)
  File "/home/[redacted]/lib/python3.5/site-packages/mwdblib/api.py", line 64, in __init__
    self.set_api_key(api_key)
  File "/home/[redacted]/lib/python3.5/site-packages/mwdblib/api.py", line 81, in set_api_key
    raise InvalidCredentialsError("Invalid API key format. Verify whether actual token is provided "
mwdblib.exc.InvalidCredentialsError: Invalid API key format. Verify whether actual token is provided instead of its UUID.

Probably related with https://docs.python.org/3/library/json.html#json.loads

Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

Docs: missing package description

https://pypi.org/project/mwdblib/

The author of this package has not provided a project description

So we need to fill it in!

CLI: JSON formatter

Provide -o json option which prints data into JSON format, which can be used for further processing (e.g. piped to jq)

Docs: Methods decorated with APIClient.requires are missing in documentation

It seems functools.wraps is missing

In [16]: file.remove_attribute
Out[16]: functools.partial(<bound method APIClient.requires.<locals>.VersionDependentMethod.__call__ of <mwdblib.api.api.APIClient.requires.<locals>.VersionDependentMethod object at 0x7f4c1dc3e4a0>>, <mwdblib.file.MWDBFile object at 0x7f4c1ecaec20>)

In [17]: file.remove_attribute.__doc__
Out[17]: 'partial(func, *args, **keywords) - new function with partial application\n    of the given arguments and keywords.\n'