emalderson/thephish

ThePhish: an automated phishing email analysis tool

thephish's Issues

[Question] ThePhish Several Questions

I have some questions about ThePhish:

• Will ThePhish have their modules updated?
• Will ThePhish be made compatible with Docker on Windows?
• Can you pretty please make a video tutorial on how to install and configure ThePhish for test environment and do a test-email?

Thanks in advance.

Startup script and systemd service [Ubuntu]

Hello, thank you for this project!

I would like to add startup script and Systemd service:

ThePhish_service.sh

#!/bin/bash
#
#

case "$1" in
  start)
    cd /opt/ThePhish/app/ && /opt/ThePhish/app/venv/bin/python3 thephish_app.py
    ps aux | grep thephish_app.py | grep -v grep | awk -F' ' '{print $2}'
    echo "[  OK  ] ThePhish was successfuly started as PID $PID."
    ;;
  stop)
    PID=$(ps aux | grep thephish_app.py | grep -v grep | awk -F' ' '{print $2}')
    kill -9 $PID
    echo "[  OK  ] ThePhish was successfully stopped."
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}" >&2
    exit 1
    ;;
esac

Create file /lib/systemd/system/thephish.service

[Unit]
Description=ThePhish
Documentation= https://github.com/emalderson/ThePhish#install-it-using-docker-and-docker-compose
Wants=network-online.target davmail.service
After=network.target network-online.target davmail.service

[Service]
Type=simple
RemainAfterExit=yes
User=root
Group=root
TimeoutStartSec=0
Restart=on-failure
RestartSec=30s
#ExecStartPre=
ExecStart=/opt/ThePhish/ThePhish_service.sh start
ExecStop=/opt/ThePhish/ThePhish_service.sh stop


[Install]
WantedBy=multi-user.target

[Question] running docker-compose up causes entire kali linux system to crash

Question

I am using a Kali Linux virtual machine, but when i execute docker-compose up, the whole virtual machine crashes. I have ensured to download docker and docker compose, and ensured that they both have the correct versions installed

How do I prevent the whole Kali Linux virtual machine from crashing?

Custom email notification

I think that it could be useful to add the possibility to customize the emails that thephish send to the users that forwarded the email. One way could be to add in the json file configuration.json or in a new json file something like that

 "email" : {
                 "notification" : "Notification text. [{0}] to include the subject of the email",
                 "safe" : "Safe email text. [{0}] to include the subject of the email",
                 "malicius" : "Malicius email text. [{0}] to include the subject of the email"
        }

The suspicius case is not included because the email notification isn't automatic.
A possible change to the code could be:

task_notification = thehive4py.models.CaseTask(
                                        id=task_id,
                                        description = "mailto:" + mail_to + "\n" + config['notification'].format(case.json()['title'][11:]),
                                        status = 'InProgress'
                                        )

Where config['notification'] is equal to the value of the object Notification.

[BUG] - Docker compose - java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager

When starting the containers for the The Phish using docker composer, an error line appears more than one time:

thehive | [error] o.t.s.u.Retry [|] An error occurs
thehive | java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
thehive | at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)

...

thehive | Caused by: java.lang.reflect.InvocationTargetException: null
thehive | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

...

thehive | Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.21.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.21.0.2:9042] Cannot connect))
thehive | at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268)

...

Here is the YAML file:

version: "3.8"
services:

cassandra:
image: cassandra:3.11
container_name: cassandra
restart: unless-stopped
hostname: cassandra
environment:
- MAX_HEAP_SIZE=1G
- HEAP_NEWSIZE=1G
- CASSANDRA_CLUSTER_NAME=thp
- CASSANDRA_LISTEN_ADDRESS=127.0.0.1
volumes:
- ./vol/cassandra-data:/var/lib/cassandra/data

thehive:
image: thehiveproject/thehive4:4.1.9-1
container_name: thehive
restart: unless-stopped
depends_on:
- cassandra
ports:
- '0.0.0.0:9005:9005'
volumes:
- ./thehive/application.conf:/etc/thehive/application.conf
- ./vol/data:/opt/data
- ./vol/index:/opt/index
command: '--no-config --no-config-secret --cortex-keys SGTlcSntWXJb/rzrcZSGlb/cidaTUl4U'

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1
container_name: elasticsearch
restart: unless-stopped
ports:
- '0.0.0.0:9200:9200'
environment:
- http.host=0.0.0.0
- discovery.type=single-node
- cluster.name=hive
- script.allowed_types= inline
- thread_pool.search.queue_size=100000
- thread_pool.write.queue_size=10000
- gateway.recover_after_nodes=1
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- 'ES_JAVA_OPTS=-Xms256m -Xmx256m'
ulimits:
nofile:
soft: 65536
hard: 65536
volumes:
- ./vol/elasticsearch_data:/usr/share/elasticsearch/data
- ./vol/elasticsearch_logs:/usr/share/elasticsearch/logs

cortex:
image: thehiveproject/cortex:3.1.1-1
container_name: cortex
restart: unless-stopped
volumes:
- ./cortex/application.conf:/etc/cortex/application.conf
- /var/run/docker.sock:/var/run/docker.sock
- /tmp:/tmp
environment:
- http_proxy=${http_proxy}
- https_proxy=${https_proxy}
depends_on:
- elasticsearch
ports:
- '0.0.0.0:9001:9001'

redis:
image: redis:6.2.5
container_name: redis
restart: unless-stopped

mysql:
image: mysql:8.0.26
container_name: mysql
restart: unless-stopped
command: --default-authentication-plugin=mysql_native_password
environment:
- "MYSQL_USER=misp"
- "MYSQL_PASSWORD=example"
- "MYSQL_ROOT_PASSWORD=password"
- "MYSQL_DATABASE=misp"
volumes:
- ./vol/mysql:/var/lib/mysql

misp:
image: coolacid/misp-docker:core-v2.4.148a
container_name: misp
restart: unless-stopped
depends_on:
- redis
- mysql
ports:
- '0.0.0.0:80:80'
- '0.0.0.0:443:443'
environment:
- "MYSQL_HOST=mysql"
- "HOSTNAME=https://localhost"
- "REDIS_FQDN=redis"
- "INIT=true"
- "CRON_USER_ID=1"
- "DISIPV6=true"

thephish:
image: emalderson/thephish:latest
container_name: thephish
restart: unless-stopped
depends_on:
- thehive
- cortex
- misp
ports:
- '0.0.0.0:8080:8080'
volumes:
- ./thephish_conf_files/analyzers_level_conf.json:/root/thephish/analyzers_level_conf.json
- ./thephish_conf_files/configuration.json:/root/thephish/configuration.json
- ./thephish_conf_files/whitelist.json:/root/thephish/whitelist.json

I don't know if it's related, but when I try to access TheHive, the browser does not find the app on port 9000 or gives a "Connection Reset" on port 9005.

Thanks for any help you can provide.

AttributeError: 'NoneType' object has no attribute 'span'

i login in success，but recieving mails has error，the error show this：

could u tell me how i fix it？i will thanks very much！

I receive attachment .eml but happen error after click list emails

Question

After click on button "list emails" appear ERROR, this message attachment is type .eml

python3 thephish_app.py
[INFO][list_emails]: Connected to xxxxxx@[email protected]:993/inbox
[INFO][list_emails]: 1 unread messages to process
[INFO][list_emails]: Message from: b'aleatory_spam@local' with subject: Phishing 3
**[ERROR][list_emails]: Error while trying to retrieve the emails: Traceback (most recent call last):
File "/opt/ThePhish/app/list_emails.py", line 250, in main
emails_info = retrieve_emails(connection)
File "/opt/ThePhish/app/list_emails.py", line 112, in retrieve_emails
decode = email.header.decode_header(internal_msg['Subject'])
File "/usr/lib/python3.7/email/header.py", line 80, in decode_header
if not ecre.search(header):
TypeError: expected string or bytes-like object**

Any help with this?

[BUG] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_6/

Describe the bug

A clear and concise description of what the bug is.

Work environment

Question	Answer
OS version (server)	Ubuntu Server
OS version (client)	20.04
Python version
Type of email address used	Gmail, Hotmail, ...
Mail client type & version	Thunderbird, Gmail, ...
Browser type & version	Firefox, Chrome, Edge, Opera, ...
Virtualized Env.	True / False
Dedicated RAM	8 GB
vCPU	4
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True / False
Docker Version	If applicable
Docker Compose version	If applicable

To Reproduce

Steps to reproduce the behavior:

Run docker-compose up

and get:

I already tried the steps with:

Thanks for help!

Do you have a working ISO Image for the Phish ?

Work environment

Question	Answer
OS version (server)	Ubuntu, Kali linux
OS version (client)	XP, Seven, 10, Ubuntu, ...
Python version
Type of email address used	Gmail,
Mail client type & version	Gmail, ...
Browser type & version	Firefox, Chrome, Edge, Opera, ...
Virtualized Env.	True /

                     |

| vCPU | 4 / 8 / 16 / 32 |
| ThePhish version | |
| TheHive version | |
| Cortex version | |
| MISP version | |
| Installed using Docker and Docker Compose | True / False |
| Docker Version | If applicable |
| Docker Compose version | If applicable |

Question
I have tried so many times until we succeeded set the cortex, hive, misp, thePhis alive but it seems a gmail not working anymore these, cannot run thePhish when try to connect gmail
Do you have a working iso image or OVA, OVF I can analyze with ?
where can I download ? please help

[BUG]no response when clicking "list emails"

Describe the bug

A clear and concise description of what the bug is.

Work environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	10
Python version	3
Type of email address used	company email, ...
Mail client type & version	O365 ...
Browser type & version	Firefox
Virtualized Env.	True / False
Dedicated RAM	XX GB
vCPU	4
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True
Docker Version	If applicable
Docker Compose version	If applicable

To Reproduce

Steps to reproduce the behavior:

1. Go to 'http://localhost:8080/'
2. Click on 'list emails'
3. But no response

Expected behavior

list emails.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Possible solutions

seems some code is missing?

[ERROR]: Error while trying to notify the start of analysis

Work environment

Question	Answer
OS version (server)	centos-release-7-9.2009.1.el7.centos.x86_64
OS version (client)	-
Python version	Python 2.7.5
Type of email address used	Gmail
Mail client type & version	Gmail
Browser type & version	Firefox, Chrome
Virtualized Env.	True /
Dedicated RAM	6 GB
vCPU	8
ThePhish version	ThePhish 1
TheHive version	5
Cortex version	3.1.7-1
MISP version	MISP - 2.4.162
Installed using Docker and Docker Compose	False
Docker Version	-
Docker Compose version	-

I ask you to help with solving the problem, during the launch of the analysis of the message, after creating the case in thehive, the error indicated in the screenshot appears. When manually running the responder mailer from thehive, the email is sent to the addressee successfully

[BUG] Error parsing and getting mails

Describe the bug

When i push over the "List Mails" i get an error. It appears to be a issue with the list_emails.py. on line 184. I have tried with plain/text mails and html mails

Work environment

Question	Answer
OS version (server)	Ubuntu Desktop 22.04
OS version (client)	Ubuntu, ...
Python version	3.10.4
Type of email address used	office 365
Browser type & version	Chrome
Virtualized Env.	True
Dedicated RAM	8 GB
vCPU	4
ThePhish version
TheHive version	4.1.9-1
Cortex version	3.1.1-1
MISP version	2.4.148
Installed using Docker and Docker Compose	True
Docker Version	20.10.12
Docker Compose version	1.29.2

Screenshots

Log

thephish | AttributeError: 'NoneType' object has no attribute 'contents'
thephish |
thehive | [info] o.t.s.AccessLogFilter [00000004|] 172.19.0.1 GET /api/status took 3ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000005|] 172.19.0.1 GET /api/status took 2ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000006|] 192.168.46.213 GET /api/status took 2ms and returned 200 752 bytes
thephish | [INFO][list_emails]: Connected to myemail@[email protected]:993/inbox
thephish | [INFO][list_emails]: 3 unread messages to process
thephish | [INFO][list_emails]: Message from: b' [email protected]' with subject: hola
thephish | [INFO][list_emails]: Message from: b' [email protected]' with subject: prueba 4
thephish | [ERROR]_[list_emails]: Error while trying to retrieve the emails: Traceback (most recent call last):
thephish | File "/root/thephish/list_emails.py", line 250, in main
thephish | emails_info = retrieve_emails(connection)
thephish | File "/root/thephish/list_emails.py", line 184, in retrieve_emails
thephish | body = soup.body.div.p.span.contents[0]
thephish | AttributeError: 'NoneType' object has no attribute 'contents'
thephish |
thehive | [info] o.t.s.AccessLogFilter [00000007|] 172.19.0.1 GET /api/status took 2ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000008|] 172.19.0.1 GET /api/status took 1ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000009|] 192.168.46.213 GET /api/status took 1ms and returned 200 752 bytes

Ciao non riesco nell'installazione [Question]

Work environment

Question	Answer
OS version (server)	Ubuntu server 22.03 macchina virtuale con 16 gb ram 2 cpu e 250 di disco

Nell'installazione ricevo questo errore
potresti aiutarmi?

Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
elasticsearch | output:
elasticsearch | [0.000s][error][logging] Error opening log file 'logs/gc.log': Permission denied
elasticsearch | [0.000s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
elasticsearch | error:
elasticsearch | Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
elasticsearch | Error: Could not create the Java Virtual Machine.
elasticsearch | Error: A fatal exception has occurred. Program will exit.
elasticsearch | at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
elasticsearch | at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
elasticsearch exited with code 1

Question

Describe the question/requirement as clearly as possible.

ModuleNotFoundError: No module named 'flask'

Describe the bug

Installed ThePhish from scratch. Have TheHive, Cortex, MISP, and mail all setup correctly. Followed the instructions to the letter. When running the command to start thephish, I receive an error regarding flask module not being found. I've tried uninstalling flask and installing for the whole system using python3 -m pip install flask but this doesn't seem too work.

Work environment

Question	Answer
OS version (server)	Ubuntu Server 22.04
OS version (client)	n/a
Python version	3.10
Type of email address used	Gmail
Mail client type & version	Gmail
Browser type & version	Firefox (really n/a)
Virtualized Env.	True
Dedicated RAM	16 GB
vCPU	8
ThePhish version	Unknown / latest version
TheHive version	5.2.11-1 (I'm aware of the issues with Hive4py, was hoping to work on this a bit
Cortex version	3.1.8-1
MISP version	2.4.185
Installed using Docker and Docker Compose	False
Docker Version	n/a
Docker Compose version	n/a

To Reproduce

Steps to reproduce the behavior:

1. Install everything from scratch per instructions, including running in a python3 virtual environment.
2. Run command sudo python3 thephish_app.py
3. Receive the following error.
   Traceback (most recent call last): File "/home/tordormore/ThePhish/app/thephish_app.py", line 1, in <module> import flask ModuleNotFoundError: No module named 'flask'

Expected behavior

I would expect to see the application start and then be able to get to the main website for thephish.

Screenshots

ERROR: Version in "./docker-compose.yml" is unsupported

Hi, i just cloned and try to start from compose file, but I am getting:

root@123:/opt/ThePhish/docker# docker-compose up
WARNING: The http_proxy variable is not set. Defaulting to a blank string.
WARNING: The https_proxy variable is not set. Defaulting to a blank string.
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.

With compose:
docker-compose version 1.25.0, build unknown

Automatic running analyze after new email was arrived.

Question

Is there any way to run automatic analyze after new email was arrived?

Thank you.

Majo

[BUG]Documentation

you need to have a Case Template created in TheHive named 'ThePhish' or you get an error. When my only Case Template was named reported-email I kept getting errors and it would not kick off Analysis.

[BUG]

Describe the bug

when i am trying to connect outlook to retrieve the emails i got this error. seems like parsing issue

Work environment

Question	Answer
OS version (host)	windows, , , ...
OS version (client)	Ubuntu 20
Python version
Type of email address used	outlook
Mail client type & version	outlook
Browser type & version	Chrome
Virtualized Env.	docker
Dedicated RAM
vCPU
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True
Docker Version	Version: 20.10.21
Docker Compose version	Docker Compose version v2.12.2

To Reproduce

Steps to reproduce the behavior:

1. Go to Thephish url
2. Click on retrieve emails
3. See error in console and screen

Expected behavior

retrieving of email.

Screenshots

[Question] How can we update the container version?

Question

Hello, first of all thank you for the amazing tools, I'm using docker installation method and had fun for last few days, wanted to know if there is a way to update the docker like theHive/Cortex/MISP's version without losing all the configuration?

Thank you again!

How to change the email message that thephish sends.

Work environment

Question	Answer
OS version (server)	Debian, Ubuntu, CentOS, RedHat, ...
OS version (client)	XP, Seven, 10, Ubuntu, ...
Python version
Type of email address used	Gmail, Hotmail, ...
Mail client type & version	Thunderbird, Gmail, ...
Browser type & version	Firefox, Chrome, Edge, Opera, ...
Virtualized Env.	True / False
Dedicated RAM	XX GB
vCPU	4 / 8 / 16 / 32
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True / False
Docker Version	If applicable
Docker Compose version	If applicable

Question

Describe the question/requirement as clearly as possible.

[ERROR]: Error during the analysis task - Why does this happen?

The error happens during the analysis, ThePhish GUI displays the error message, and the analysis stops.

Looking into the logs, we can see this message:

[error] o.t.t.c.c.s.CortexActor [|] CortexActor received job output OutputJob(7Rsx1IwB-UJszOBmP3Bj,f8f874decbb877a41361ee9c94feb746,TorProject_1_0,TorProject_1_0,Thu Jan 04 11:17:26 UTC 2024,Some(Thu Jan 04 11:17:26 UTC 2024),Some(Thu Jan 04 11:18:12 UTC 2024),Success,Some(XXX.XXX.XXX.XXX),None,CIB1,ip,Some(OutputReport(Vector(OutputMinireport(info,TorProject,Node,false)),Some({}),true,Vector(),Vector(),None,None)),analyzer) but did not have it in state List()

XXX.XXX.XXX.XXX is a valid IPv4 address. I just changed it for privacy issues.

Before that error message, there is another one:

thephish | [INFO][run_analysis]: Started analyzer TorProject_1_0 for ip XXX.XXX.XXX.XXX
thephish | [ERROR][run_analysis]: Error during the analysis task: Traceback (most recent call last):
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
thephish | httplib_response = self._make_request(
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
thephish | six.raise_from(e, None)
thephish | File "", line 3, in raise_from
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
thephish | httplib_response = conn.getresponse()
thephish | File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
thephish | response.begin()
thephish | File "/usr/lib/python3.8/http/client.py", line 307, in begin
thephish | version, status, reason = self._read_status()
thephish | File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
thephish | raise RemoteDisconnected("Remote end closed connection without"
thephish | http.client.RemoteDisconnected: Remote end closed connection without response

Again, XXX.XXX.XXX.XXX is a valid IPv4 address.

I'll disable this analyzer and see if it works smoothly.

Best regards and a happy new year!!

[Bug] Attachments (txt) are not parsed

Work environment

Question	Answer
OS version (server)	Ubuntu 20.04.03
OS version (client)	Ubuntu 20.04.03
Python version	Python 3.8.10
Type of email address used	Gmail
Mail client type & version	Thunderbird
Browser type & version	Firefox
Virtualized Env.	True
Dedicated RAM	4 GB
vCPU	2
ThePhish version	recent
TheHive version	4.1.11-1
Cortex version	3.1.1-1
MISP version	not used
Installed using Docker and Docker Compose	False
Docker Version	If applicable
Docker Compose version	If applicable

Question

Hi,
first of all thank you for developing this tool.

A) - Attachments are not parsed

My Problem is that Attachments of an E-Mail are not parsed.
This is what i do:

1. Want to analyze this Mail with invoice.txt attached
   

2. Forward mail via Thunderbird to ThePish Gmail-Account
   

3. Start ThePhish Analysis

4. invoice.txt is not listed in observable and not analyzed
   

5. If i download the mail in TheHive (ziped) invoice.txt is still attached to the mail so it seems that the mail is received correctly by ThePhish but invoice.txt is not recolonized.

.txt is not whitelisted.

Am i missing something here? Can you help me out?

B) - html-tags for hyperlinks gets recolonized as part of the URL

Moreover some html-tags for hyperlinks gets recolonized as part of the URL as u can see here:

C) - mini report from the URL-Haus Analyzer is not shown in observables

Independent from the ThePhish i recognized another little issue, maybe u have a solution for that or you can give me hint how to debug this:
The mini report from the URL-Haus Analyzer is not shown in observables

But it has hits

Do you have any idea how to fix that?

Thank you!

Claudio

[ERROR]: Error while trying to create the case

Hi,

After click open "Analyze" button, some messages shows up and suddenly it appears a message saying: [ERROR]: Error while trying to create the case.

How can I debug that?

Any idea?

[Question] authentication/login page for ThePhish

hi,
is it possible to authenticate a user who wants to access thephish page? Did not see anything relevant in documentation or among questions/issues.

regards.

[Question]

Work environment

Question	Answer
OS version (server)	Debian, Ubuntu
OS version (client)	any
Python version	3.9.2
Type of email address used	any
Mail client type & version	any
Browser type & version	any
Virtualized Env.	False
Dedicated RAM	16 GB
vCPU	4
ThePhish version	603eca6
TheHive version	4.1.11-1
Cortex version	3.1.1-1
MISP version	2.4.150
Installed using Docker and Docker Compose	False
Docker Version	n/a
Docker Compose version	n/a

Question

This may not be specifically an issue for ThePhish but hopefully you can shed some light on how to go about solving it appropriately.

In Cortex, we want to use HaveIBeenPwned (HIBP). When we submit a message from ThePhish to be analyzed, the EmlParser grabs all the email addresses in a message, including the To. Then Cortex dutifully sends each of these addresses to HIBP. If any of them comes back positive, ThePhish flags the message as malicious. The problem, of course, is if the recipient's email is in this database, the message gets flagged as malicious!

What is the correct way to get some finer grain control over what gets sent to HIBP to make it work say on other addresses in the mail than the envelope recipient or to header?

Currently ThePhish closes the message as malicious if any one of the results comes back malicious. It seems like we need some finer control like if-this-and-that sort of control. I don't know if that belongs in ThePhish or what.

Issue regarding configuration

Work environment

Question	Answer
OS version (server)	Ubuntu,
OS version (client)	Ubuntu 22
Python version
Type of email address used	Gmail
Mail client type & version	Gmail
Browser type & version	Firefox
Virtualized Env.	True
Dedicated RAM	8 GB
vCPU	8
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True

Question

When accessing ThePhish web interface, I am trying to click the TheHive link from left menu but it is not opening( I am getting the message "Trouble finding the site"). below is my configuration file:

{
"imap" : {
"host" : "imap.gmail.com",
"port" : "993",
"user" : "@gmail.com",
"password" : "**",
"folder" : "inbox"
},
"thehive" : {
"url" : "http://thehive:9000",
"apikey" : "gLFo6Dz+g805UxbEwvWdimW"
},
"cortex" : {
"url" : "http://cortex:9001",
"apikey" : "ZuNzKNKRVa8YE6zf2l7YJsUwL",
"id" : "local"
},
"misp" : {
"id" : "MISP THP"
},
"case" : {
"tlp" : "2",
"pap" : "2",
"tags" : ["email", "ThePhish"]
}
}
~

[Question]

does ThePhish work with TheHive version 5?

[BUG] TheHive, Cortex and MISP instances are not reachable on thephish

Describe the bug
Hi i have a bug i installed ThePhish with the github and see that :
" The URLs at which TheHive, Cortex and MISP instances are reachable should also be replaced in the file templates/index.html so that the buttons on the web interface will be able to reach them. To do that, replace the last three href of this portion of code: "

So i did that for also be redirected on the instances by thephish interface but nothing change when i inspect the page. The href are not modify while i restart with docker compose.

I also changed Urls in thehive/application.conf , thephish_conf_files/configuration.json with my server IP address but nothing change.

I have an other issue when i go in /ThePhish/docker/vol/mysql$ and try to connect to the database with this command : sudo mysql -u misp -p misp.
i got this answer sudo: mysql : commande introuvable and i do not know how to resolved it as mysql is in docker.

Work environment
| OS version (server) | Debian |
| OS version (client) | Debian11 Bullseye|
| Python version | |
| Type of email address used | Outlook|
| Mail client type & version | Outlook|
| Browser type & version | Chrome |
| Virtualized Env. | False |
| ThePhish version | |
| TheHive version | |
| Cortex version | |
| MISP version | |
| Installed using Docker and Docker Compose | True |
| Docker Version | Docker version 24.0.4|
| Docker Compose version | Docker Compose version v2.19.1|

[BUG] List emails not working

Describe the bug

ThePhish mentioned no e-mails to read, but mentioned that there are unread messages to process

Work environment

Question	Answer
OS version (server)	Kali Linux
OS version (client)	Kali Linux
Python version
Type of email address used	Gmail
Mail client type & version	Gmail
Browser type & version	Edge
Virtualized Env.	True
Dedicated RAM	6.5 GB
vCPU	4 / 8 / 16 / 32
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True
Docker Version	If applicable
Docker Compose version	If applicable

To Reproduce

Expected behavior

ThePhish should list the email

Screenshots

Additional context

I think I have enabled everything correct through the Docker method, but ThePhish is still unable to list emails.
I also ensured to forward as attachment to "[email protected]", but still unable to list emails

Possible solutions

Keep this section if you have suggestions on how to solve the problem, otherwise delete it.

[Question] External API Call

Work environment

Question	Answer
OS version (server)	RHEL7, Docker
OS version (client)	NA
Python version	NA
Type of email address used	NA
Mail client type & version	NA
Browser type & version	Firefox
Virtualized Env.	True
Dedicated RAM	NA
vCPU	NA
ThePhish version	Latest
TheHive version	NA
Cortex version	NA
MISP version	NA
Installed using Docker and Docker Compose	True
Docker Version	20.10.3
Docker Compose version	1.29.0

Question

I started to POC ThePhish. Without configuring anything, I noticed in DevTools that ThePhish does external call to "fonts.googleapis.com" and "cdnjs.cloudflare.com".

ThePhish is installed in an environnement without internet access by default.

My question is : is it mandatory to open some external web address in proxies ?

Thanks in advance.

[BUG] Error while trying to connect to IMAP server

Describe the bug

After setting it up all up and despite using a virtual environment I still am unable to get the gmail authentication to work. I have tried two different app passwords with gmail resulting in the same issue.

Work environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	20.04.5 LTS
Python version	3.8.10
Type of email address used	Gmail
Mail client type & version	Gmail
Browser type & version	Chrome
Virtualized Env.	True
Dedicated RAM	4 GB
vCPU	6
ThePhish version	Where do I find this info? latest git clone
TheHive version	5.0.18-1
Cortex version	cortex0 - 3.1.7-1
MISP version	MISP - 2.4.164
Installed using Docker and Docker Compose	False

To Reproduce

Steps to reproduce the behavior:

1. Go to the Phish URL'
2. Click on List emails'
3. An error is produced

Expected behavior

Expect a valid connection to gmail

Screenshots

Dependencies met

App password configured

Config file

Error

Additional context

My Hive, Cortex and MISP is hosted on another container within my proxmox server.

The only thing I cant seem to get working is the "Add the run_responder()". Is it still relevant for my case? Do I run the command on my ThePhish container or my other container holding thehive?

[Problème de réception d'email au niveau de la BAL ThePhish]

Work environment

Question	Answer
OS version (server)	RedHat 8
Python version	3.6
Type of email address used	Exchange
Mail client type & version	Outlook
Browser type & version	Chrome, Firefox
Virtualized Env.	False
Dedicated RAM	XX GB
vCPU	8
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True / False
Docker Version	If applicable
Docker Compose version	If applicable

*I actually have questions about the current version
I installed from scratch but during tests, the thephish interface cannot retrieve emails from the dedicated exchange BAL for thephish
Furthermore, I am also unable to see the thephish logs
I think several people have already reported this error on git but I can't find a solution to this, even on git

Describe the question/requirement as clearly as possible.

[Question]

Work environment

Question	Answer
Python version	3.8
ThePhish version	Latest
TheHive version	4.1.9-1
Cortex version	3.1.1-1

Question

Hello, I had other issue after phase notification mail sent, appear a error about [run_analysis] with something Cortex Job Id

[INFO][run_analysis]: Notification mail sent
[ERROR][run_analysis]: Error during the analysis task: Traceback (most recent call last):
File "/opt/ThePhish/app/run_analysis.py", line 576, in main
observables_info, reports_observables = analyze_observables(case, task_ids['Analysis'], wsl)
File "/opt/ThePhish/app/run_analysis.py", line 240, in analyze_observables
job['job_id'] = analyzer_job.json()['cortexJobId']
KeyError: 'cortexJobId'

Have some idea about this?

[BUG] listing Emails

Hello again,
i set up ThePhish from scratch and integrated theHive, MISP and Cortex correctly. When trying to fetch Emails though i get the following traceback. I tested 2 Email-Adresses, using the data i already used sucessfully on a docker build.

[ERROR]_[list_emails]: Error while trying to connect to IMAP server: Traceback (most recent call last):
File "/etc/ThePhish/app/list_emails.py", line 243, in main
connection = connect_to_IMAP_server()
File "/etc/ThePhish/app/list_emails.py", line 21, in connect_to_IMAP_server
connection.login(config['imapUser'], config['imapPassword'])
File "/usr/lib/python3.8/imaplib.py", line 601, in login
typ, dat = self._simple_command('LOGIN', user, self._quote(password))
File "/usr/lib/python3.8/imaplib.py", line 1205, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python3.8/imaplib.py", line 1030, in _command_complete
raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: LOGIN command error: BAD [b'Not enough arguments provided 8mb428246745wrb']

Work environment
| -------------------------- | ------------------------------------- |
| OS version (server) | Ubuntu 20.04.4 |
| OS version (client) | Ubuntu 20.04.4 |
| Python version | 3.8.10 |
| Type of email address used | Gmail, 365 |
| Browser type & version | Firefox, Chromium |
| Virtualized Env. | True |
| Dedicated RAM | 24 GB |
| vCPU | 8 |
| ThePhish version | latest |
| TheHive version | 4.1.22-1 |
| Cortex version | 3.1.6-1 |
| MISP version | 2.4.159 |
| Installed using Docker and Docker Compose | False |

To Reproduce

start ThePhish with:
cd /etc/ThePhish/app/
python3 thephish_app.py

navigate to localhost:8080 and click "List Emails"

Expected behavior

ThePhish connecting to inbox and getting Emails.

Screenshots

Solution
I tried to search for that problem but couldn't find something helpfull. Would be great if you have an idea on how to resolve this issue.

Thanks in advance

There are no e-mails to read

Hi, I installed the system on Windows configured and it seems to work without errors. The thing I can't understand is why when I launch it it tells me emails are not present when in relation to the logs I see that it downloads them. can you help me?

[BUG] - Crowdsec_Analyzer_1_0: Job AobGi4wB0KV4xh-jmuSh has be updated (JsDefined("Failure"))

When running Crowdsec Analyzer against an IP address, Cortex always fails:

cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=1, status=Pulling from cortexneurons/crowdsec_analyzer, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=null, status=Digest: sha256:d99a82be3ef69a823d14db0ed34bed1151a028f968f9b4a87aef6e27984b268c, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=null, status=Status: Image is up to date for cortexneurons/crowdsec_analyzer:1, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] o.t.c.s.DockerJobRunnerSrv - Execute container 41c6783a56caec06399fd0167b3fc15a15a211739141b68502f3f005505d6d67
cortex | timeout: 30 minutes
cortex | image : docker.io/cortexneurons/crowdsec_analyzer:1
cortex | volume : /tmp/cortex-jobs/cortex-job-AobGi4wB0KV4xh-jmuSh-239385392839758287:/job
cortex | [info] c.s.d.c.DefaultDockerClient - Starting container with Id: 41c6783a56caec06399fd0167b3fc15a15a211739141b68502f3f005505d6d67
cortex | [info] o.t.c.s.AuditActor - Job AobGi4wB0KV4xh-jmuSh has be updated (JsDefined("Failure"))
cortex | [info] o.t.c.s.JobSrv - Job AobGi4wB0KV4xh-jmuSh has finished with status Failure

Looking into the message on Cortex web interface, a Python exception is shown:

Traceback (most recent call last):
File "/worker/Crowdsec/crowdsec_analyzer.py", line 4, in
from crowdsec_api import Crowdsec
File "/worker/Crowdsec/crowdsec_api.py", line 2, in
from requests.compat import urljoin
ModuleNotFoundError: No module named 'requests'

I tried to install "requests" on the cortex container, but I then realized that the neurons run on their own containers....

Thanks for any help

[Feature request] phishing test campaign whitelist & notification

Hello and thank you for this great work.

It would be very interesting to add a phishing test campaign whitelist feature. If a domain is in this list, a third type of indicator could be generated to classify the mail as a "test campaign". And possibly notify the user that he has not fallen into the trap.

It's one of the little things missing for production use.

Thank you.
Alban

[Question]

Work environment

Question	Answer
OS version (server)	Debian
OS version (client)	Debian
Python version
Type of email address used	outlook
Mail client type & version	outlook
Browser type & version	Firefox
Virtualized Env.	True
Dedicated RAM	16 GB
vCPU	8
ThePhish version
TheHive version
Cortex version
MISP version
Installed using Docker and Docker Compose	True / False
Docker Version	If applicable
Docker Compose version	If applicable

Question
Error in cases
[ERROR]_[run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 106, in do_post
thephish | response.raise_for_status()
thephish | File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
thephish | raise HTTPError(http_error_msg, response=self)
thephish | requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://cortex:9001/api/responder/_search?range=0-1
thephish |
thephish | The above exception was the direct cause of the following exception:
thephish |
thephish | Traceback (most recent call last):
thephish | File "/root/thephish/run_analysis.py", line 568, in main
thephish | notify_start_of_analysis(case, task_ids['Notification'], mail_to, wsl)
thephish | File "/root/thephish/run_analysis.py", line 55, in notify_start_of_analysis
thephish | mailer_responder = api_cortex.responders.get_by_name('Mailer_1_0')
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/controllers/responders.py", line 22, in get_by_name
thephish | return self._wrap(self._find_one_by(Eq('name', name)), Responder)
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/controllers/abstract.py", line 29, in _find_one_by
thephish | collection = self._api.do_post(url, {'query': query or {}}, params).json()
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 109, in do_post
thephish | self.__recover(ex)
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 46, in __recover
thephish | raise AuthenticationError("Authentication error") from exception
thephish | cortex4py.exceptions.AuthenticationError: Authentication error
thephish |
cassandra | INFO [Service Thread] 2022-08-02 15:15:47,309 GCInspector.java:285 - ConcurrentMarkSweep GC in 430ms. CMS Old Gen: 49304 -> 65512; Par Eden Space: 858980352 -> 51411816; Par Survivor Space: 43551056 -> 0
thehive | [info] o.t.t.s.IntegrityCheckActor [|] End of CaseTemplate global check: Map(orphans -> 0, duration -> 24)

[Question] Handling URL rewrite

My environment passes all email through Cisco Secure Email Security, which will rewrite URLs with a neutral or unknown reputation to redirect them to the Cisco Web Security Proxy for click-time evaluation of their safety.

For those emails which slip through and are reported as suspicious/malicious, I would like to use ThePhish as my analysis and logging platform.

Does ThePhish have a capability to decode these URL rewrites so that the true URL is analyzed ?

Cortex integration in TheHive

| OS version (server) | Debian 11 bullseye |
| Python version | 3.9.2 |
| Type of email address used | Gmail |
| Browser type & version | Firefox |
| Virtualized Env. | True |
| Dedicated RAM | 8 GB |
| vCPU | 2 |
| ThePhish version | - |
| TheHive version | 4.1.9-1 |
| Cortex version | 3.1.1-1 |
| MISP version | 2.4.148 |
| Installed using Docker and Docker Compose | True |
| Docker Version | 20.10.16 |
| Docker Compose version | 2.5.0 |

Hi Emalderson,
i really like your Platform but im having problems setting it up. I used your Guide for installation with docker compose and keep getting "AUTH_ERROR" on theHive. I did the integration of the API-keys in theHive application-conf aswell as thephish_conf_files config. In Hive though i get this message:

Are there any other steps i can take to assure a connection between Hive and Cortex? Fetching Emails on ThePhish doesn't work (An error occured). I tried doing a stacktrace but im not used to Linux and couldn't quiet figure it out. Also i don't know how to view thePhish version.

CortexJobID

Hi!!

First of all, thank you with all my heart for the development you launched, because I was doing something similar, but you already had everything practically done.

Second, I would like to see if you can help me with an error, since I do not know if I am misconfiguring the system.

When I send the email to be analyzed, it creates the case in thehive and sends me a notification email. M error lies when calling parsers ... it throws me the following error:

How could I solve this error?

Regards from Chile.

TheHive5 analysis in alert

With TheHive 5 it is today possile to run analyzers in alerts, Is this possible to get a version that create an alert with all artifacts analyzed and if we got a Safe or a Malicious verdict, a case creation. But not when we got a Suspicious verdict. Like that an analyst can take the alert later.

Amazing tool thank for your work !

Regards

.msg functionality

Hello,

I love the tool but I need to run it using .msg extension instead of eml and it's seems like not possible currently.

Any plan on extending it to msg ?

[Question]

Work environment

Question	Answer
Type of email address used	Exchange 2019 (Mail Corporate)
Mail client type & version	Outlook

Question

Have some other options besides imap? I'd like using my company but we have exchange 2019 (account corporate).

"imap" : {
"host" : "imap.gmail.com",
"port" : "993",
"user" : "",
"password" : "",
"folder" : "inbox"
},

[Question] How to troubleshoot "Error while trying to notify the start of analysis"

Work environment

Question	Answer
OS version (server)	Debian 11..
Python version	3.9
Type of email address used	Gmail
Mail client type & version	gmail
Browser type & version	Edge
Virtualized Env.	True
Dedicated RAM	16 GB
vCPU	4 / 8
ThePhish version
TheHive version	4.1.11-1
Cortex version	3.1.1-1
MISP version	2.4
Installed using Docker and Docker Compose	False
Docker Version	If applicable
Docker Compose version	If applicable

Question

Describe the question/requirement as clearly as possible.
Can't figure out how to get past this error when emailing a notification status. The analysis part runs, but then get stuck here.

[INFO][case_from_email]: Added observable file Payment Confirmation Invoice no#333339.eml to case 18
[ERROR][run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
File "/home/xxxxx/ThePhish/app/run_analysis.py", line 573, in main
KeyError: 'Notification'

[ERROR]_[run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
File "/home/xxxxx/ThePhish/app/run_analysis.py", line 573, in main
KeyError: 'Notification'

[Question]CaseTemplate not found

I came across this error, can you tell what template should I set for thehive?
[ERROR]: Cannot create template: 404 ({"type":"NotFoundError","message":"CaseTemplate not found"})

[Question]

I'm trying to install ThePhish with the docker configuration on an openSUSE 15.4 server (without a graphical interface), but I can't connect to TheHive. Does anyone know why?

[Feature request] Add Modern Auth for O365 email checking/sending

With Microsoft dropping support for Legacy Authentication in October 2022, it would be great to have the ability to use modern auth to the app for checking for new emails to analyze as well as sending emails to the end users.

[Error] Imap connection

Imap Connection

Hi, i have an imap connection error.
Why this happens?
I'll be grateful if you can help me.