calidog/axeman

Axeman is a utility to retrieve certificates from Certificate Transparency Lists (CTLs)

axeman's Issues

Process getting killed.

Hey.

After running axeman within a few seconds my process is getting killed.

Axeman is only pulling from one log

Hi I have set up Axeman on a Google Compute Engine machine running Ubuntu 16.04 LTS server. When I run the program:
axeman -c 16
The program downloads 32GB of data from https://ct.googleapis.com/logs/argon2019/ct/v1 and haults.
If I try to download a specific log:
Axeman -c 16 -u ‘https://ct.googleapis.com/logs/argon2019/ct/v1‘
The program outputs
[INFO:root] - the current time - Starting...
Although no files are downloaded and the console allows input as if the program didn’t begin properly.

Converting as_der to OpenSSL.crypto.X509 object

Hi Mate

Love your work.

You have the conversion below. How would you reverse this to get a OpenSSL.crypto.X509 object?

"as_der": base64.b64encode(crypto.dump_certificate(crypto.FILETYPE_ASN1, certificate)).decode('utf-8')

Andrei

Cut new pypi release?

The URL fix in 1.15 is needed for this project to function, pypi is on 1.14 https://pypi.org/project/axeman/

Can you cut the new release?

UnicodeEncodeError: 'ascii' codec can't encode character '\u010d'

UnicodeEncodeError: 'ascii' codec can't encode character '\u010d' in position 22113: ordinal not in range(128)

Seen while downloading .csv's from ct.googleapis.com/logs/argon2022

File "/usr/local/lib/python3.5/dist-packages/axeman/core.py", line 227, in process_worker
f.write("".join(lines))

System: Debian 9
Python: 3.5

Crashes with "KeyError: 'entries'"

axeman pulled down a few TBs of CSVs from various CTLs and then crashed:

[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72408700-72408766.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72407530-72407596.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410455-72410521.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72410195-72410261.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72412535-72412601.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72414160-72414226.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415460-72415526.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72415850-72415916.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72417410-72417476.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419880-72419946.csv written!
[450] Parsing...
[450] Finished, writing CSV...
[450] CSV /tmp/certificates/ct2.digicert-ct.com_log/72419360-72419426.csv written!
[INFO:root] 2019-10-17 06:47:18,526 - Completed DigiCert Log Server 2, stored at /tmp/ct2.digicert-ct.com_log.csv!
[INFO:root] 2019-10-17 06:47:18,526 - Finished downloading and processing ct2.digicert-ct.com/log
[INFO:root] 2019-10-17 06:47:18,526 - Downloading certificates for DigiCert Yeti2019 Log
[INFO:root] 2019-10-17 06:47:30,526 - Starting processing coro and process pool
[1] Making dir...
[INFO:root] 2019-10-17 06:47:31,057 - Getting things to process...
[INFO:root] 2019-10-17 06:47:31,058 - Queue Status: Processing Queue Size:0 Downloaded blocks:50/1130984 (0.0044%)
Traceback (most recent call last):
  File "/usr/local/bin/axeman", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 295, in main
    loop.run_until_complete(retrieve_certificates(loop, concurrency_count=args.concurrency_count))
  File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 112, in retrieve_certificates
    await download_tasks
  File "/usr/local/lib/python3.7/site-packages/axeman/core.py", line 53, in download_worker
    for index, entry in zip(range(start, end + 1), entry_list['entries']):
KeyError: 'entries'

-z (ctl_offset) messes up output unless value is exactly devidable by block_size

The use of the -z (ctl_offset) option will result in missing entries, or double entries if its not devidable by the block_size.

Having a non dividable amount will cause Axeman to send requests lager then block_size to the CT (resulting in less results then expected and other weird behaviour)

How to reproduce:

(Using argon2023 which has a blocksize of 32)
Good looking output (128 / 32 = 4):
axeman -z 128 -u ct.googleapis.com/logs/argon2023

Bad looking output (126 / 32 = ~3.94):
axeman -z 126 -u ct.googleapis.com/logs/argon2023

In this example the bad looking output will be 32 block big files, with only 2 entries (and the other 30 not written)

-s (start_offset) doesn't work

-s (start_offset) is not used in the code, and thus does not work.

(Is this option replaced by -z (ctl_offset)?)

Wrong with client session

Traceback (most recent call last):
File "/home/dcs/.local/bin/axeman", line 11, in
sys.exit(main())
File "/home/dcs/.local/lib/python3.5/site-packages/axeman/core.py", line 280, in main
loop.run_until_complete(get_certs_and_print())
File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/dcs/.local/lib/python3.5/site-packages/axeman/core.py", line 239, in get_certs_and_print
with aiohttp.ClientSession(conn_timeout=5) as session:
File "/home/dcs/.local/lib/python3.5/site-packages/aiohttp/client.py", line 745, in enter
raise TypeError("Use async with instead")
TypeError: Use async with instead
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7ff03a30fcf8>

Axeman also includes non-website certs (Code signing) and does not escape them.

Hi,

While running a custom wrapper around axeman that inserts records into a custom database format, I've found out that every now and then a special certificate comes along, example being this certificate (id 17395 on ct.googleapis.com/logs/argon2022) being a Code Signing certificate given to "SOOSAN INT CO., Ltd"

This gives multiple issues, not only does axeman not properly escape the "," character in this name but it also ends up in the all_domains field. Although the last might be wanted behaviour, the non escaped "," character isn't.