emalderson / thephish Goto Github PK
View Code? Open in Web Editor NEWThePhish: an automated phishing email analysis tool
License: GNU Affero General Public License v3.0
ThePhish: an automated phishing email analysis tool
License: GNU Affero General Public License v3.0
does ThePhish work with TheHive version 5?
I think that it could be useful to add the possibility to customize the emails that thephish send to the users that forwarded the email. One way could be to add in the json file configuration.json or in a new json file something like that
"email" : {
"notification" : "Notification text. [{0}] to include the subject of the email",
"safe" : "Safe email text. [{0}] to include the subject of the email",
"malicius" : "Malicius email text. [{0}] to include the subject of the email"
}
The suspicius case is not included because the email notification isn't automatic.
A possible change to the code could be:
task_notification = thehive4py.models.CaseTask(
id=task_id,
description = "mailto:" + mail_to + "\n" + config['notification'].format(case.json()['title'][11:]),
status = 'InProgress'
)
Where config['notification'] is equal to the value of the object Notification.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu 20.04.03 |
OS version (client) | Ubuntu 20.04.03 |
Python version | Python 3.8.10 |
Type of email address used | Gmail |
Mail client type & version | Thunderbird |
Browser type & version | Firefox |
Virtualized Env. | True |
Dedicated RAM | 4 GB |
vCPU | 2 |
ThePhish version | recent |
TheHive version | 4.1.11-1 |
Cortex version | 3.1.1-1 |
MISP version | not used |
Installed using Docker and Docker Compose | False |
Docker Version | If applicable |
Docker Compose version | If applicable |
Question
Hi,
first of all thank you for developing this tool.
My Problem is that Attachments of an E-Mail are not parsed.
This is what i do:
Start ThePhish Analysis
If i download the mail in TheHive (ziped) invoice.txt is still attached to the mail so it seems that the mail is received correctly by ThePhish but invoice.txt is not recolonized.
Am i missing something here? Can you help me out?
Moreover some html-tags for hyperlinks gets recolonized as part of the URL as u can see here:
Independent from the ThePhish i recognized another little issue, maybe u have a solution for that or you can give me hint how to debug this:
The mini report from the URL-Haus Analyzer is not shown in observables
But it has hits
Do you have any idea how to fix that?
Thank you!
Claudio
Describe the bug
A clear and concise description of what the bug is.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu Server |
OS version (client) | 20.04 |
Python version | |
Type of email address used | Gmail, Hotmail, ... |
Mail client type & version | Thunderbird, Gmail, ... |
Browser type & version | Firefox, Chrome, Edge, Opera, ... |
Virtualized Env. | True / False |
Dedicated RAM | 8 GB |
vCPU | 4 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True / False |
Docker Version | If applicable |
Docker Compose version | If applicable |
To Reproduce
Steps to reproduce the behavior:
Run docker-compose up
and get:
I already tried the steps with:
Thanks for help!
Work environment
Question | Answer |
---|---|
OS version (server) | Debian, Ubuntu, CentOS, RedHat, ... |
OS version (client) | XP, Seven, 10, Ubuntu, ... |
Python version | |
Type of email address used | Gmail, Hotmail, ... |
Mail client type & version | Thunderbird, Gmail, ... |
Browser type & version | Firefox, Chrome, Edge, Opera, ... |
Virtualized Env. | True / False |
Dedicated RAM | XX GB |
vCPU | 4 / 8 / 16 / 32 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True / False |
Docker Version | If applicable |
Docker Compose version | If applicable |
Question
Describe the question/requirement as clearly as possible.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu, |
OS version (client) | Ubuntu 22 |
Python version | |
Type of email address used | Gmail |
Mail client type & version | Gmail |
Browser type & version | Firefox |
Virtualized Env. | True |
Dedicated RAM | 8 GB |
vCPU | 8 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True |
Question
When accessing ThePhish web interface, I am trying to click the TheHive link from left menu but it is not opening( I am getting the message "Trouble finding the site"). below is my configuration file:
{
"imap" : {
"host" : "imap.gmail.com",
"port" : "993",
"user" : "@gmail.com",
"password" : "**",
"folder" : "inbox"
},
"thehive" : {
"url" : "http://thehive:9000",
"apikey" : "gLFo6Dz+g805UxbEwvWdimW"
},
"cortex" : {
"url" : "http://cortex:9001",
"apikey" : "ZuNzKNKRVa8YE6zf2l7YJsUwL",
"id" : "local"
},
"misp" : {
"id" : "MISP THP"
},
"case" : {
"tlp" : "2",
"pap" : "2",
"tags" : ["email", "ThePhish"]
}
}
~
Work environment
Question | Answer |
---|---|
OS version (server) | Debian, Ubuntu |
OS version (client) | any |
Python version | 3.9.2 |
Type of email address used | any |
Mail client type & version | any |
Browser type & version | any |
Virtualized Env. | False |
Dedicated RAM | 16 GB |
vCPU | 4 |
ThePhish version | 603eca6 |
TheHive version | 4.1.11-1 |
Cortex version | 3.1.1-1 |
MISP version | 2.4.150 |
Installed using Docker and Docker Compose | False |
Docker Version | n/a |
Docker Compose version | n/a |
Question
This may not be specifically an issue for ThePhish but hopefully you can shed some light on how to go about solving it appropriately.
In Cortex, we want to use HaveIBeenPwned (HIBP). When we submit a message from ThePhish to be analyzed, the EmlParser grabs all the email addresses in a message, including the To. Then Cortex dutifully sends each of these addresses to HIBP. If any of them comes back positive, ThePhish flags the message as malicious. The problem, of course, is if the recipient's email is in this database, the message gets flagged as malicious!
What is the correct way to get some finer grain control over what gets sent to HIBP to make it work say on other addresses in the mail than the envelope recipient or to header?
Currently ThePhish closes the message as malicious if any one of the results comes back malicious. It seems like we need some finer control like if-this-and-that sort of control. I don't know if that belongs in ThePhish or what.
Hi, i just cloned and try to start from compose file, but I am getting:
root@123:/opt/ThePhish/docker# docker-compose up
WARNING: The http_proxy variable is not set. Defaulting to a blank string.
WARNING: The https_proxy variable is not set. Defaulting to a blank string.
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
With compose:
docker-compose version 1.25.0, build unknown
Question
Hello, first of all thank you for the amazing tools, I'm using docker installation method and had fun for last few days, wanted to know if there is a way to update the docker like theHive/Cortex/MISP's version without losing all the configuration?
Thank you again!
Describe the bug
Installed ThePhish from scratch. Have TheHive, Cortex, MISP, and mail all setup correctly. Followed the instructions to the letter. When running the command to start thephish, I receive an error regarding flask module not being found. I've tried uninstalling flask and installing for the whole system using python3 -m pip install flask
but this doesn't seem too work.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu Server 22.04 |
OS version (client) | n/a |
Python version | 3.10 |
Type of email address used | Gmail |
Mail client type & version | Gmail |
Browser type & version | Firefox (really n/a) |
Virtualized Env. | True |
Dedicated RAM | 16 GB |
vCPU | 8 |
ThePhish version | Unknown / latest version |
TheHive version | 5.2.11-1 (I'm aware of the issues with Hive4py, was hoping to work on this a bit |
Cortex version | 3.1.8-1 |
MISP version | 2.4.185 |
Installed using Docker and Docker Compose | False |
Docker Version | n/a |
Docker Compose version | n/a |
To Reproduce
Steps to reproduce the behavior:
sudo python3 thephish_app.py
Traceback (most recent call last): File "/home/tordormore/ThePhish/app/thephish_app.py", line 1, in <module> import flask ModuleNotFoundError: No module named 'flask'
Expected behavior
I would expect to see the application start and then be able to get to the main website for thephish.
Describe the bug
A clear and concise description of what the bug is.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu |
OS version (client) | 10 |
Python version | 3 |
Type of email address used | company email, ... |
Mail client type & version | O365 ... |
Browser type & version | Firefox |
Virtualized Env. | True / False |
Dedicated RAM | XX GB |
vCPU | 4 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True |
Docker Version | If applicable |
Docker Compose version | If applicable |
To Reproduce
Steps to reproduce the behavior:
Expected behavior
list emails.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Possible solutions
seems some code is missing?
Describe the bug
Hi i have a bug i installed ThePhish with the github and see that :
" The URLs at which TheHive, Cortex and MISP instances are reachable should also be replaced in the file templates/index.html so that the buttons on the web interface will be able to reach them. To do that, replace the last three href of this portion of code: "
So i did that for also be redirected on the instances by thephish interface but nothing change when i inspect the page. The href are not modify while i restart with docker compose.
I also changed Urls in thehive/application.conf , thephish_conf_files/configuration.json with my server IP address but nothing change.
I have an other issue when i go in /ThePhish/docker/vol/mysql$ and try to connect to the database with this command : sudo mysql -u misp -p misp.
i got this answer sudo: mysql : commande introuvable and i do not know how to resolved it as mysql is in docker.
Work environment
| OS version (server) | Debian |
| OS version (client) | Debian11 Bullseye|
| Python version | |
| Type of email address used | Outlook|
| Mail client type & version | Outlook|
| Browser type & version | Chrome |
| Virtualized Env. | False |
| ThePhish version | |
| TheHive version | |
| Cortex version | |
| MISP version | |
| Installed using Docker and Docker Compose | True |
| Docker Version | Docker version 24.0.4|
| Docker Compose version | Docker Compose version v2.19.1|
Work environment
Question | Answer |
---|---|
Type of email address used | Exchange 2019 (Mail Corporate) |
Mail client type & version | Outlook |
Question
Have some other options besides imap? I'd like using my company but we have exchange 2019 (account corporate).
"imap" : {
"host" : "imap.gmail.com",
"port" : "993",
"user" : "",
"password" : "",
"folder" : "inbox"
},
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu, Kali linux |
OS version (client) | XP, Seven, 10, Ubuntu, ... |
Python version | |
Type of email address used | Gmail, |
Mail client type & version | Gmail, ... |
Browser type & version | Firefox, Chrome, Edge, Opera, ... |
Virtualized Env. | True / |
|
| vCPU | 4 / 8 / 16 / 32 |
| ThePhish version | |
| TheHive version | |
| Cortex version | |
| MISP version | |
| Installed using Docker and Docker Compose | True / False |
| Docker Version | If applicable |
| Docker Compose version | If applicable |
Question
I have tried so many times until we succeeded set the cortex, hive, misp, thePhis alive but it seems a gmail not working anymore these, cannot run thePhish when try to connect gmail
Do you have a working iso image or OVA, OVF I can analyze with ?
where can I download ? please help
Hi,
After click open "Analyze" button, some messages shows up and suddenly it appears a message saying: [ERROR]: Error while trying to create the case.
How can I debug that?
Any idea?
Hello and thank you for this great work.
It would be very interesting to add a phishing test campaign whitelist feature. If a domain is in this list, a third type of indicator could be generated to classify the mail as a "test campaign". And possibly notify the user that he has not fallen into the trap.
It's one of the little things missing for production use.
Thank you.
Alban
The error happens during the analysis, ThePhish GUI displays the error message, and the analysis stops.
Looking into the logs, we can see this message:
[error] o.t.t.c.c.s.CortexActor [|] CortexActor received job output OutputJob(7Rsx1IwB-UJszOBmP3Bj,f8f874decbb877a41361ee9c94feb746,TorProject_1_0,TorProject_1_0,Thu Jan 04 11:17:26 UTC 2024,Some(Thu Jan 04 11:17:26 UTC 2024),Some(Thu Jan 04 11:18:12 UTC 2024),Success,Some(XXX.XXX.XXX.XXX),None,CIB1,ip,Some(OutputReport(Vector(OutputMinireport(info,TorProject,Node,false)),Some({}),true,Vector(),Vector(),None,None)),analyzer) but did not have it in state List()
XXX.XXX.XXX.XXX is a valid IPv4 address. I just changed it for privacy issues.
Before that error message, there is another one:
thephish | [INFO][run_analysis]: Started analyzer TorProject_1_0 for ip XXX.XXX.XXX.XXX
thephish | [ERROR][run_analysis]: Error during the analysis task: Traceback (most recent call last):
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
thephish | httplib_response = self._make_request(
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
thephish | six.raise_from(e, None)
thephish | File "", line 3, in raise_from
thephish | File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
thephish | httplib_response = conn.getresponse()
thephish | File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
thephish | response.begin()
thephish | File "/usr/lib/python3.8/http/client.py", line 307, in begin
thephish | version, status, reason = self._read_status()
thephish | File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
thephish | raise RemoteDisconnected("Remote end closed connection without"
thephish | http.client.RemoteDisconnected: Remote end closed connection without response
Again, XXX.XXX.XXX.XXX is a valid IPv4 address.
I'll disable this analyzer and see if it works smoothly.
Best regards and a happy new year!!
My environment passes all email through Cisco Secure Email Security, which will rewrite URLs with a neutral or unknown reputation to redirect them to the Cisco Web Security Proxy for click-time evaluation of their safety.
For those emails which slip through and are reported as suspicious/malicious, I would like to use ThePhish as my analysis and logging platform.
Does ThePhish have a capability to decode these URL rewrites so that the true URL is analyzed ?
With TheHive 5 it is today possile to run analyzers in alerts, Is this possible to get a version that create an alert with all artifacts analyzed and if we got a Safe or a Malicious verdict, a case creation. But not when we got a Suspicious verdict. Like that an analyst can take the alert later.
Amazing tool thank for your work !
Regards
Hello,
I love the tool but I need to run it using .msg extension instead of eml and it's seems like not possible currently.
Any plan on extending it to msg ?
Work environment
Question | Answer |
---|---|
OS version (server) | RedHat 8 |
Python version | 3.6 |
Type of email address used | Exchange |
Mail client type & version | Outlook |
Browser type & version | Chrome, Firefox |
Virtualized Env. | False |
Dedicated RAM | XX GB |
vCPU | 8 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True / False |
Docker Version | If applicable |
Docker Compose version | If applicable |
*I actually have questions about the current version
I installed from scratch but during tests, the thephish interface cannot retrieve emails from the dedicated exchange BAL for thephish
Furthermore, I am also unable to see the thephish logs
I think several people have already reported this error on git but I can't find a solution to this, even on git
Describe the question/requirement as clearly as possible.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu server 22.03 macchina virtuale con 16 gb ram 2 cpu e 250 di disco |
Nell'installazione ricevo questo errore
potresti aiutarmi?
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
elasticsearch | output:
elasticsearch | [0.000s][error][logging] Error opening log file 'logs/gc.log': Permission denied
elasticsearch | [0.000s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
elasticsearch | error:
elasticsearch | Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
elasticsearch | Error: Could not create the Java Virtual Machine.
elasticsearch | Error: A fatal exception has occurred. Program will exit.
elasticsearch | at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
elasticsearch | at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
elasticsearch | at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
elasticsearch exited with code 1
Question
Describe the question/requirement as clearly as possible.
Describe the bug
When i push over the "List Mails" i get an error. It appears to be a issue with the list_emails.py. on line 184. I have tried with plain/text mails and html mails
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu Desktop 22.04 |
OS version (client) | Ubuntu, ... |
Python version | 3.10.4 |
Type of email address used | office 365 |
Browser type & version | Chrome |
Virtualized Env. | True |
Dedicated RAM | 8 GB |
vCPU | 4 |
ThePhish version | |
TheHive version | 4.1.9-1 |
Cortex version | 3.1.1-1 |
MISP version | 2.4.148 |
Installed using Docker and Docker Compose | True |
Docker Version | 20.10.12 |
Docker Compose version | 1.29.2 |
Log
thephish | AttributeError: 'NoneType' object has no attribute 'contents'
thephish |
thehive | [info] o.t.s.AccessLogFilter [00000004|] 172.19.0.1 GET /api/status took 3ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000005|] 172.19.0.1 GET /api/status took 2ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000006|] 192.168.46.213 GET /api/status took 2ms and returned 200 752 bytes
thephish | [INFO][list_emails]: Connected to myemail@[email protected]:993/inbox
thephish | [INFO][list_emails]: 3 unread messages to process
thephish | [INFO][list_emails]: Message from: b' [email protected]' with subject: hola
thephish | [INFO][list_emails]: Message from: b' [email protected]' with subject: prueba 4
thephish | [ERROR]_[list_emails]: Error while trying to retrieve the emails: Traceback (most recent call last):
thephish | File "/root/thephish/list_emails.py", line 250, in main
thephish | emails_info = retrieve_emails(connection)
thephish | File "/root/thephish/list_emails.py", line 184, in retrieve_emails
thephish | body = soup.body.div.p.span.contents[0]
thephish | AttributeError: 'NoneType' object has no attribute 'contents'
thephish |
thehive | [info] o.t.s.AccessLogFilter [00000007|] 172.19.0.1 GET /api/status took 2ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000008|] 172.19.0.1 GET /api/status took 1ms and returned 200 752 bytes
thehive | [info] o.t.s.AccessLogFilter [00000009|] 192.168.46.213 GET /api/status took 1ms and returned 200 752 bytes
With Microsoft dropping support for Legacy Authentication in October 2022, it would be great to have the ability to use modern auth to the app for checking for new emails to analyze as well as sending emails to the end users.
When running Crowdsec Analyzer against an IP address, Cortex always fails:
cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=1, status=Pulling from cortexneurons/crowdsec_analyzer, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=null, status=Digest: sha256:d99a82be3ef69a823d14db0ed34bed1151a028f968f9b4a87aef6e27984b268c, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] c.s.d.c.LoggingPullHandler - pull docker.io/cortexneurons/crowdsec_analyzer:1: ProgressMessage{id=null, status=Status: Image is up to date for cortexneurons/crowdsec_analyzer:1, stream=null, error=null, progress=null, progressDetail=null}
cortex | [info] o.t.c.s.DockerJobRunnerSrv - Execute container 41c6783a56caec06399fd0167b3fc15a15a211739141b68502f3f005505d6d67
cortex | timeout: 30 minutes
cortex | image : docker.io/cortexneurons/crowdsec_analyzer:1
cortex | volume : /tmp/cortex-jobs/cortex-job-AobGi4wB0KV4xh-jmuSh-239385392839758287:/job
cortex | [info] c.s.d.c.DefaultDockerClient - Starting container with Id: 41c6783a56caec06399fd0167b3fc15a15a211739141b68502f3f005505d6d67
cortex | [info] o.t.c.s.AuditActor - Job AobGi4wB0KV4xh-jmuSh has be updated (JsDefined("Failure"))
cortex | [info] o.t.c.s.JobSrv - Job AobGi4wB0KV4xh-jmuSh has finished with status Failure
Looking into the message on Cortex web interface, a Python exception is shown:
Traceback (most recent call last):
File "/worker/Crowdsec/crowdsec_analyzer.py", line 4, in
from crowdsec_api import Crowdsec
File "/worker/Crowdsec/crowdsec_api.py", line 2, in
from requests.compat import urljoin
ModuleNotFoundError: No module named 'requests'
I tried to install "requests" on the cortex container, but I then realized that the neurons run on their own containers....
Thanks for any help
When starting the containers for the The Phish using docker composer, an error line appears more than one time:
thehive | [error] o.t.s.u.Retry [|] An error occurs
thehive | java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
thehive | at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
...
thehive | Caused by: java.lang.reflect.InvocationTargetException: null
thehive | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
...
thehive | Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.21.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.21.0.2:9042] Cannot connect))
thehive | at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268)
...
Here is the YAML file:
version: "3.8"
services:
cassandra:
image: cassandra:3.11
container_name: cassandra
restart: unless-stopped
hostname: cassandra
environment:
- MAX_HEAP_SIZE=1G
- HEAP_NEWSIZE=1G
- CASSANDRA_CLUSTER_NAME=thp
- CASSANDRA_LISTEN_ADDRESS=127.0.0.1
volumes:
- ./vol/cassandra-data:/var/lib/cassandra/data
thehive:
image: thehiveproject/thehive4:4.1.9-1
container_name: thehive
restart: unless-stopped
depends_on:
- cassandra
ports:
- '0.0.0.0:9005:9005'
volumes:
- ./thehive/application.conf:/etc/thehive/application.conf
- ./vol/data:/opt/data
- ./vol/index:/opt/index
command: '--no-config --no-config-secret --cortex-keys SGTlcSntWXJb/rzrcZSGlb/cidaTUl4U'
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1
container_name: elasticsearch
restart: unless-stopped
ports:
- '0.0.0.0:9200:9200'
environment:
- http.host=0.0.0.0
- discovery.type=single-node
- cluster.name=hive
- script.allowed_types= inline
- thread_pool.search.queue_size=100000
- thread_pool.write.queue_size=10000
- gateway.recover_after_nodes=1
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- 'ES_JAVA_OPTS=-Xms256m -Xmx256m'
ulimits:
nofile:
soft: 65536
hard: 65536
volumes:
- ./vol/elasticsearch_data:/usr/share/elasticsearch/data
- ./vol/elasticsearch_logs:/usr/share/elasticsearch/logs
cortex:
image: thehiveproject/cortex:3.1.1-1
container_name: cortex
restart: unless-stopped
volumes:
- ./cortex/application.conf:/etc/cortex/application.conf
- /var/run/docker.sock:/var/run/docker.sock
- /tmp:/tmp
environment:
- http_proxy=${http_proxy}
- https_proxy=${https_proxy}
depends_on:
- elasticsearch
ports:
- '0.0.0.0:9001:9001'
redis:
image: redis:6.2.5
container_name: redis
restart: unless-stopped
mysql:
image: mysql:8.0.26
container_name: mysql
restart: unless-stopped
command: --default-authentication-plugin=mysql_native_password
environment:
- "MYSQL_USER=misp"
- "MYSQL_PASSWORD=example"
- "MYSQL_ROOT_PASSWORD=password"
- "MYSQL_DATABASE=misp"
volumes:
- ./vol/mysql:/var/lib/mysql
misp:
image: coolacid/misp-docker:core-v2.4.148a
container_name: misp
restart: unless-stopped
depends_on:
- redis
- mysql
ports:
- '0.0.0.0:80:80'
- '0.0.0.0:443:443'
environment:
- "MYSQL_HOST=mysql"
- "HOSTNAME=https://localhost"
- "REDIS_FQDN=redis"
- "INIT=true"
- "CRON_USER_ID=1"
- "DISIPV6=true"
thephish:
image: emalderson/thephish:latest
container_name: thephish
restart: unless-stopped
depends_on:
- thehive
- cortex
- misp
ports:
- '0.0.0.0:8080:8080'
volumes:
- ./thephish_conf_files/analyzers_level_conf.json:/root/thephish/analyzers_level_conf.json
- ./thephish_conf_files/configuration.json:/root/thephish/configuration.json
- ./thephish_conf_files/whitelist.json:/root/thephish/whitelist.json
I don't know if it's related, but when I try to access TheHive, the browser does not find the app on port 9000 or gives a "Connection Reset" on port 9005.
Thanks for any help you can provide.
Question
I am using a Kali Linux virtual machine, but when i execute docker-compose up, the whole virtual machine crashes. I have ensured to download docker and docker compose, and ensured that they both have the correct versions installed
How do I prevent the whole Kali Linux virtual machine from crashing?
| OS version (server) | Debian 11 bullseye |
| Python version | 3.9.2 |
| Type of email address used | Gmail |
| Browser type & version | Firefox |
| Virtualized Env. | True |
| Dedicated RAM | 8 GB |
| vCPU | 2 |
| ThePhish version | - |
| TheHive version | 4.1.9-1 |
| Cortex version | 3.1.1-1 |
| MISP version | 2.4.148 |
| Installed using Docker and Docker Compose | True |
| Docker Version | 20.10.16 |
| Docker Compose version | 2.5.0 |
Hi Emalderson,
i really like your Platform but im having problems setting it up. I used your Guide for installation with docker compose and keep getting "AUTH_ERROR" on theHive. I did the integration of the API-keys in theHive application-conf aswell as thephish_conf_files config. In Hive though i get this message:
Are there any other steps i can take to assure a connection between Hive and Cortex? Fetching Emails on ThePhish doesn't work (An error occured). I tried doing a stacktrace but im not used to Linux and couldn't quiet figure it out. Also i don't know how to view thePhish version.
you need to have a Case Template created in TheHive named 'ThePhish' or you get an error. When my only Case Template was named reported-email I kept getting errors and it would not kick off Analysis.
Hello, thank you for this project!
I would like to add startup script and Systemd service:
ThePhish_service.sh
#!/bin/bash
#
#
case "$1" in
start)
cd /opt/ThePhish/app/ && /opt/ThePhish/app/venv/bin/python3 thephish_app.py
ps aux | grep thephish_app.py | grep -v grep | awk -F' ' '{print $2}'
echo "[ OK ] ThePhish was successfuly started as PID $PID."
;;
stop)
PID=$(ps aux | grep thephish_app.py | grep -v grep | awk -F' ' '{print $2}')
kill -9 $PID
echo "[ OK ] ThePhish was successfully stopped."
;;
restart)
stop
start
;;
*)
echo "Usage: $0 {start|stop|restart}" >&2
exit 1
;;
esac
Create file /lib/systemd/system/thephish.service
[Unit]
Description=ThePhish
Documentation= https://github.com/emalderson/ThePhish#install-it-using-docker-and-docker-compose
Wants=network-online.target davmail.service
After=network.target network-online.target davmail.service
[Service]
Type=simple
RemainAfterExit=yes
User=root
Group=root
TimeoutStartSec=0
Restart=on-failure
RestartSec=30s
#ExecStartPre=
ExecStart=/opt/ThePhish/ThePhish_service.sh start
ExecStop=/opt/ThePhish/ThePhish_service.sh stop
[Install]
WantedBy=multi-user.target
Work environment
Question | Answer |
---|---|
Python version | 3.8 |
ThePhish version | Latest |
TheHive version | 4.1.9-1 |
Cortex version | 3.1.1-1 |
Question
Hello, I had other issue after phase notification mail sent, appear a error about [run_analysis] with something Cortex Job Id
[INFO][run_analysis]: Notification mail sent
[ERROR][run_analysis]: Error during the analysis task: Traceback (most recent call last):
File "/opt/ThePhish/app/run_analysis.py", line 576, in main
observables_info, reports_observables = analyze_observables(case, task_ids['Analysis'], wsl)
File "/opt/ThePhish/app/run_analysis.py", line 240, in analyze_observables
job['job_id'] = analyzer_job.json()['cortexJobId']
KeyError: 'cortexJobId'
Have some idea about this?
Work environment
Question | Answer |
---|---|
OS version (server) | RHEL7, Docker |
OS version (client) | NA |
Python version | NA |
Type of email address used | NA |
Mail client type & version | NA |
Browser type & version | Firefox |
Virtualized Env. | True |
Dedicated RAM | NA |
vCPU | NA |
ThePhish version | Latest |
TheHive version | NA |
Cortex version | NA |
MISP version | NA |
Installed using Docker and Docker Compose | True |
Docker Version | 20.10.3 |
Docker Compose version | 1.29.0 |
Question
I started to POC ThePhish. Without configuring anything, I noticed in DevTools that ThePhish does external call to "fonts.googleapis.com" and "cdnjs.cloudflare.com".
ThePhish is installed in an environnement without internet access by default.
My question is : is it mandatory to open some external web address in proxies ?
Thanks in advance.
Hi!!
First of all, thank you with all my heart for the development you launched, because I was doing something similar, but you already had everything practically done.
Second, I would like to see if you can help me with an error, since I do not know if I am misconfiguring the system.
When I send the email to be analyzed, it creates the case in thehive and sends me a notification email. M error lies when calling parsers ... it throws me the following error:
How could I solve this error?
Regards from Chile.
Work environment
Question | Answer |
---|---|
OS version (server) | Debian 11.. |
Python version | 3.9 |
Type of email address used | Gmail |
Mail client type & version | gmail |
Browser type & version | Edge |
Virtualized Env. | True |
Dedicated RAM | 16 GB |
vCPU | 4 / 8 |
ThePhish version | |
TheHive version | 4.1.11-1 |
Cortex version | 3.1.1-1 |
MISP version | 2.4 |
Installed using Docker and Docker Compose | False |
Docker Version | If applicable |
Docker Compose version | If applicable |
Question
Describe the question/requirement as clearly as possible.
Can't figure out how to get past this error when emailing a notification status. The analysis part runs, but then get stuck here.
[INFO][case_from_email]: Added observable file Payment Confirmation Invoice no#333339.eml to case 18
[ERROR][run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
File "/home/xxxxx/ThePhish/app/run_analysis.py", line 573, in main
KeyError: 'Notification'
[ERROR]_[run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
File "/home/xxxxx/ThePhish/app/run_analysis.py", line 573, in main
KeyError: 'Notification'
Question
After click on button "list emails" appear ERROR, this message attachment is type .eml
python3 thephish_app.py
[INFO][list_emails]: Connected to xxxxxx@[email protected]:993/inbox
[INFO][list_emails]: 1 unread messages to process
[INFO][list_emails]: Message from: b'aleatory_spam@local' with subject: Phishing 3
**[ERROR][list_emails]: Error while trying to retrieve the emails: Traceback (most recent call last):
File "/opt/ThePhish/app/list_emails.py", line 250, in main
emails_info = retrieve_emails(connection)
File "/opt/ThePhish/app/list_emails.py", line 112, in retrieve_emails
decode = email.header.decode_header(internal_msg['Subject'])
File "/usr/lib/python3.7/email/header.py", line 80, in decode_header
if not ecre.search(header):
TypeError: expected string or bytes-like object**
Any help with this?
Work environment
Question | Answer |
---|---|
OS version (server) | Debian |
OS version (client) | Debian |
Python version | |
Type of email address used | outlook |
Mail client type & version | outlook |
Browser type & version | Firefox |
Virtualized Env. | True |
Dedicated RAM | 16 GB |
vCPU | 8 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True / False |
Docker Version | If applicable |
Docker Compose version | If applicable |
Question
Error in cases
[ERROR]_[run_analysis]: Error while trying to notify the start of analysis: Traceback (most recent call last):
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 106, in do_post
thephish | response.raise_for_status()
thephish | File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
thephish | raise HTTPError(http_error_msg, response=self)
thephish | requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://cortex:9001/api/responder/_search?range=0-1
thephish |
thephish | The above exception was the direct cause of the following exception:
thephish |
thephish | Traceback (most recent call last):
thephish | File "/root/thephish/run_analysis.py", line 568, in main
thephish | notify_start_of_analysis(case, task_ids['Notification'], mail_to, wsl)
thephish | File "/root/thephish/run_analysis.py", line 55, in notify_start_of_analysis
thephish | mailer_responder = api_cortex.responders.get_by_name('Mailer_1_0')
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/controllers/responders.py", line 22, in get_by_name
thephish | return self._wrap(self._find_one_by(Eq('name', name)), Responder)
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/controllers/abstract.py", line 29, in _find_one_by
thephish | collection = self._api.do_post(url, {'query': query or {}}, params).json()
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 109, in do_post
thephish | self.__recover(ex)
thephish | File "/usr/local/lib/python3.8/dist-packages/cortex4py/api.py", line 46, in __recover
thephish | raise AuthenticationError("Authentication error") from exception
thephish | cortex4py.exceptions.AuthenticationError: Authentication error
thephish |
cassandra | INFO [Service Thread] 2022-08-02 15:15:47,309 GCInspector.java:285 - ConcurrentMarkSweep GC in 430ms. CMS Old Gen: 49304 -> 65512; Par Eden Space: 858980352 -> 51411816; Par Survivor Space: 43551056 -> 0
thehive | [info] o.t.t.s.IntegrityCheckActor [|] End of CaseTemplate global check: Map(orphans -> 0, duration -> 24)
Describe the bug
ThePhish mentioned no e-mails to read, but mentioned that there are unread messages to process
Work environment
Question | Answer |
---|---|
OS version (server) | Kali Linux |
OS version (client) | Kali Linux |
Python version | |
Type of email address used | Gmail |
Mail client type & version | Gmail |
Browser type & version | Edge |
Virtualized Env. | True |
Dedicated RAM | 6.5 GB |
vCPU | 4 / 8 / 16 / 32 |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True |
Docker Version | If applicable |
Docker Compose version | If applicable |
To Reproduce
Expected behavior
ThePhish should list the email
Screenshots
Additional context
I think I have enabled everything correct through the Docker method, but ThePhish is still unable to list emails.
I also ensured to forward as attachment to "[email protected]", but still unable to list emails
Possible solutions
Keep this section if you have suggestions on how to solve the problem, otherwise delete it.
Describe the bug
when i am trying to connect outlook to retrieve the emails i got this error. seems like parsing issue
Work environment
Question | Answer |
---|---|
OS version (host) | windows, , , ... |
OS version (client) | Ubuntu 20 |
Python version | |
Type of email address used | outlook |
Mail client type & version | outlook |
Browser type & version | Chrome |
Virtualized Env. | docker |
Dedicated RAM | |
vCPU | |
ThePhish version | |
TheHive version | |
Cortex version | |
MISP version | |
Installed using Docker and Docker Compose | True |
Docker Version | Version: 20.10.21 |
Docker Compose version | Docker Compose version v2.12.2 |
To Reproduce
Steps to reproduce the behavior:
Expected behavior
retrieving of email.
Screenshots
I have some questions about ThePhish:
Thanks in advance.
I came across this error, can you tell what template should I set for thehive?
[ERROR]: Cannot create template: 404 ({"type":"NotFoundError","message":"CaseTemplate not found"})
I'm trying to install ThePhish with the docker configuration on an openSUSE 15.4 server (without a graphical interface), but I can't connect to TheHive. Does anyone know why?
Hi, I installed the system on Windows configured and it seems to work without errors. The thing I can't understand is why when I launch it it tells me emails are not present when in relation to the logs I see that it downloads them. can you help me?
Question
Is there any way to run automatic analyze after new email was arrived?
Thank you.
Majo
Work environment
Question | Answer |
---|---|
OS version (server) | centos-release-7-9.2009.1.el7.centos.x86_64 |
OS version (client) | - |
Python version | Python 2.7.5 |
Type of email address used | Gmail |
Mail client type & version | Gmail |
Browser type & version | Firefox, Chrome |
Virtualized Env. | True / |
Dedicated RAM | 6 GB |
vCPU | 8 |
ThePhish version | ThePhish 1 |
TheHive version | 5 |
Cortex version | 3.1.7-1 |
MISP version | MISP - 2.4.162 |
Installed using Docker and Docker Compose | False |
Docker Version | - |
Docker Compose version | - |
I ask you to help with solving the problem, during the launch of the analysis of the message, after creating the case in thehive, the error indicated in the screenshot appears. When manually running the responder mailer from thehive, the email is sent to the addressee successfully
Describe the bug
After setting it up all up and despite using a virtual environment I still am unable to get the gmail authentication to work. I have tried two different app passwords with gmail resulting in the same issue.
Work environment
Question | Answer |
---|---|
OS version (server) | Ubuntu |
OS version (client) | 20.04.5 LTS |
Python version | 3.8.10 |
Type of email address used | Gmail |
Mail client type & version | Gmail |
Browser type & version | Chrome |
Virtualized Env. | True |
Dedicated RAM | 4 GB |
vCPU | 6 |
ThePhish version | Where do I find this info? latest git clone |
TheHive version | 5.0.18-1 |
Cortex version | cortex0 - 3.1.7-1 |
MISP version | MISP - 2.4.164 |
Installed using Docker and Docker Compose | False |
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Expect a valid connection to gmail
Screenshots
Dependencies met
App password configured
Config file
Error
Additional context
My Hive, Cortex and MISP is hosted on another container within my proxmox server.
The only thing I cant seem to get working is the "Add the run_responder()". Is it still relevant for my case? Do I run the command on my ThePhish container or my other container holding thehive?
Hello again,
i set up ThePhish from scratch and integrated theHive, MISP and Cortex correctly. When trying to fetch Emails though i get the following traceback. I tested 2 Email-Adresses, using the data i already used sucessfully on a docker build.
[ERROR]_[list_emails]: Error while trying to connect to IMAP server: Traceback (most recent call last):
File "/etc/ThePhish/app/list_emails.py", line 243, in main
connection = connect_to_IMAP_server()
File "/etc/ThePhish/app/list_emails.py", line 21, in connect_to_IMAP_server
connection.login(config['imapUser'], config['imapPassword'])
File "/usr/lib/python3.8/imaplib.py", line 601, in login
typ, dat = self._simple_command('LOGIN', user, self._quote(password))
File "/usr/lib/python3.8/imaplib.py", line 1205, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python3.8/imaplib.py", line 1030, in _command_complete
raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.IMAP4.error: LOGIN command error: BAD [b'Not enough arguments provided 8mb428246745wrb']
Work environment
| -------------------------- | ------------------------------------- |
| OS version (server) | Ubuntu 20.04.4 |
| OS version (client) | Ubuntu 20.04.4 |
| Python version | 3.8.10 |
| Type of email address used | Gmail, 365 |
| Browser type & version | Firefox, Chromium |
| Virtualized Env. | True |
| Dedicated RAM | 24 GB |
| vCPU | 8 |
| ThePhish version | latest |
| TheHive version | 4.1.22-1 |
| Cortex version | 3.1.6-1 |
| MISP version | 2.4.159 |
| Installed using Docker and Docker Compose | False |
To Reproduce
start ThePhish with:
cd /etc/ThePhish/app/
python3 thephish_app.py
navigate to localhost:8080 and click "List Emails"
Expected behavior
ThePhish connecting to inbox and getting Emails.
Solution
I tried to search for that problem but couldn't find something helpfull. Would be great if you have an idea on how to resolve this issue.
Thanks in advance
hi,
is it possible to authenticate a user who wants to access thephish page? Did not see anything relevant in documentation or among questions/issues.
regards.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.