freetakteam/freetakserver

Situational Awareness Server compatible with TAK clients

freetakserver's Issues

Save CoT in the DB

There should be an option to Record None, Record until Disconnect, Record last X time, or Record all.

0.8 dataport is hardcoded in two instances

At two points it uses the const instead of the variable.

https://github.com/Tapawingo/FreeTakServer/blob/d09895503dad4f23bb9e962d95c577455bc29cc6/FreeTAKServer/controllers/DataPackageServer.py#L147

https://github.com/Tapawingo/FreeTakServer/blob/d09895503dad4f23bb9e962d95c577455bc29cc6/FreeTAKServer/controllers/DataPackageServer.py#L198

fix geochat Chat

direct messages are transmitted to other users (check marti tag)

CreateLoggerController.py missing from install

Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-106-generic x86_64)

I followed the new install instructions on the home page.

I noticed that "CreateLoggerController.py" is missing from /usr/local/lib/python3.6/dist-packages/FreeTAKServer/controllers

Is this CreateLoggerController.py required or should it be included?

send back Server version

manage FTS version, send to connected clients

Cannot start - [Errno -3] Temporary failure in name resolution

Application fails to start do to [Errno -3] Temporary failure in name resolution. My DNS resolution (1.1.1.1) is functioning fine.

Running:
python3 /TAKlib/TAKfreeServer/server.py -p 8087

Output

python3 /TAKlib/TAKfreeServer/server.py -p 8087

Traceback (most recent call last):
File "/TAKlib/TAKfreeServer/server.py", line 350, in
ThreadedServer(host=const.IP, port=port).listen()
File "/TAKlib/TAKfreeServer/server.py", line 64, in init
self.sock.bind((self.host, self.port))
socket.gaierror: [Errno -3] Temporary failure in name resolution

Server disconnects after 20 seconds

\\n'

<Element 'event' at 0x7f29ce4ded68>

uid is ANDROID-XXXXXXXXXXXX-ping

is ping

client disconnected now setting as disconnected

^CTraceback (most recent call last):

File "TAKFreeServer.py", line 199, in

ThreadedServer('',port_num).listen()

File "TAKFreeServer.py", line 40, in listen

client, address = self.sock.accept()

File "/usr/lib64/python3.6/socket.py", line 205, in accept

fd, addr = self._accept()

KeyboardInterrupt

Validate incoming data before broadcast and general security issues

Re-write the current connection handler to properly validate data before broadcasting it to other clients.

Goals:

• Parse XML and validate it's integrity.
• Grab user UID from initial thumbprint and require for all received packages as either a direct UID match or a linked UID match
• Allow proper handling for TLS/SSL connections

PIP installation is not setting execution path

in some OS the execution path is not set by PIP

Typo 'recieved'

Support Mission

My understanding is that a mission is a set of COTS, packed together, describing a certain situation. the question is how is the mission created? In CIVTAK I found nothing, in WinTak I can create the envelope of it (name, description, keyworrds) , but I cannot not add COTS.

Clarify Python 3.6+ minimum requirement

The README file states that FTS is Python3 software, however due to extensive use of f-strings (introduced in 3.6) it is incompatible with any Python3 version pre 3.6.

Currently, attempting to run the software in Python 3.4 fails ugly:

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    import httpServer
  File "/home/takserv/FreeTakServer-master/TAKfreeServer/httpServer.py", line 31
    print(f"Creating directory at {const.LOGDIRECTORY}")
                                                      ^
SyntaxError: invalid syntax

I suggest this should be indicated in the README, as well as checked and reported elegantly at runtime.

device timeout

If there is more than one TAK device connected all the other devices connected after the first device will get a socket.timeout exception.
The device will eventually just reconnect and syncronize it's data so the server still does it's job.

failed to send data [WinError 10022] An invalid argument was supplied

when using send command in ExampleServer.py it fails with exception:

failed to send data [WinError 10022] An invalid argument was supplied

FTS: ROADMAP

• 0.7 Alpha

  • data packages

• 0.8 BETA stabilize release:

  • Code reorganization for non-functional requirements support

• 0.9 RC1 stabilize release

  • Services
  • Code reorganization for non-functional requirements support
  • re-locate messages in conf file
  • create console view application
  • improve Pip installation

• 1.0 first release (Aug 2020)

  • Initial/ Internal REST API (start / stop services, configure, show users)
  • new CLI based on the API

• 1.1 (Sept 2020)

  • Send welcome message to users when they connect
  • Save User tables in the DB
  • Save CoTs in the DB (deactivated by Default)
  • Abstraction layer for the Database
  • Ability to list all Data Packages in the FTS CLI
  • Ability to delete an existing Data Package
  • Ability to store COTs into the DB
  • Ability to list stored COT's in the CLI

• 1.2 (Oct 2020)

  • API service: REST services included are
    • generic COT
    • Alert/emergency
    • chat
    • WSGI: Eventlet

• 1.3 (Nov 2020)

  • ExCheck
  • Encryption | SSL encryption

• 1.4 (Jan 2020)

  • Web Plugin : use a web framework for web admin, Dynamically change IP / ports, start, stop FTS services from web UI
  • Web Plugin : view connected clients in real time (connected IP / users, welcome message, kick out user)
  • Web Plugin : CPU and RAM usage meter as a dashboard
  • Web Plugin : add FTS healthcheck
  • Web Plugin : web configurator to create a Mission/ data package Package zip file
  • Web Plugin : Playback history
  • Web Plugin : Administer API / users

• 1.5 (Feb 2021)

  • Federation Service: ProtoBuff Support
  • Federation Service: Ability to connect a FTS (TAKServer) with another server

• 1.6 (Mar 2021)

  • Packaging for Android : Porting FTS to Android
  • Web Ui now can connect to multiple backends

• 1.7 (Apr 2021)

  • Security patches
  • REST API enhancement: search COT around me
  • REST API enhancement: create routes
  • REST API enhancement: calculate distances
  • REST API enhancement: resolve addresses

• 1.8 (May 2021)

  • Notification system (Alarm, critical FTS errors, warning resource limits)
  • Optimisation of CPU usage
  • re-implement linux demon
  • REST API additional types
  • Web Map
  • TelegramTAK

• 1.9 (Jun 2021)

  • RTSP video server
  • kick-out user from server
  • deactivate certs
  • APi for managing of Drones
  • API for managing on Video streams
  • API for managing of KML

• 1.9.9

  • #217
  • Refactoring and Repackaging

• 2.0 (Mid 2022)

  • #230
  • #209
  • #208
  • #240
  • #239
  • #238
  • #231
  • #237
  • #226
  • #228
  • #227
  • #232
  • #218
  • #219
  • #220
  • #226
  • #222
  • #223
  • #225
  • #224
  • #231
  • #232
  • #230
  • #229
  • #228
  • #225
  • #223
  • #224
  • #219
  • #242
  • #244
  • #245
  • #246
  • #248
  • #251
  • #252
    - [ ] #254
  • #255
  • #256
  • #250
  • Deploy
  • Config
  • Administration
  • Sync
  • Event management
  • #259

• 2.1 (TBD)

  • #232
  • #249
  • #253

• 2.2 (TBD)

  • #221
  • #243

• 2.3 (TBD)

  • #218
  • #241

• 2.4 (TBD)

  • #258

• 2.5 (TBD)

  • #220
  • #210
  • #257

• 2.6

  • FreeTAKTeam/UI#37

• 2.7

• 2.8

  • #227

• 2.9

• Future TAK Server (3.0+) (2023)

  • Machine Learning based Situational Awareness
  • Federation Service: support different type / TLS versions
  • script to convert to FreeTAKServer DB to POSTGIS
  • deploy in Kubernetes.
  • API danger zones (the map you could show potential areas of movement by the adversary. There's multiple reasons. From an emergency perspective to move people out of harms way. If you know a target is 100 degrees nw, Basically place potential safe zones on the map)
  • request user creation
  • Plugin System
  • Meshtastic microservice

Allow proper handling for TLS/SSL connections

Produce proper HTTP replies

Introduction

TAK devices creates an asynchronous connection on port 8080 where they send various http requests.
Upon opening a connection on the main socket it will send 2 HTTP/1.1 GET requests, one for the server version and one for the server config. It may also later be used for sharing mission packages.

Goals:

• Re-write current HTTP handler to produce proper responses and provide the materials requested.
• Allow for storing and distribution of mission packages

ModuleNotFoundError: No module named 'ReceiveConnections'

verbose.txt

Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-106-generic x86_64)

I followed the new install instructions on the home page.

Tried to starting with both commands below and received a couple different errors for each with ModuleNotFoundError: No module named 'ReceiveConnections

I checked /usr/local/lib/python3.6/dist-packages/FreeTAKServer/controllers for ReceiveConnections.py and it is their.

#pip3 check FreeTakServer No broken requirements found.

#python3 -m FreeTAKServer.controllers.Orchestrator Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.6/dist-packages/FreeTAKServer/controllers/Orchestrator.py", line 11, in <module> from ReceiveConnections import ReceiveConnections ModuleNotFoundError: No module named 'ReceiveConnections'
and

#python3 -m FreeTAKServer.controllers.Orchestrator -CoTPort 8099 -CoTIP 192.168.2.65 -DataPort 9999 Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.6/dist-packages/FreeTAKServer/controllers/Orchestrator.py", line 11, in <module> from ReceiveConnections import ReceiveConnections ModuleNotFoundError: No module named 'ReceiveConnections'

The TAK users on TAKFreeServer does not stale out

TBD

implement variable logging

-none

• Default
• Debug
• Trace

Update readme.md and the wiki

As of version 0.3a all documentation in Readme.md and the wiki is outdated.

manage datapackages within FTS

create functions to create, read, update and delete data packages in FTS

Optimize CPU consumption

in 0.8 the CPU of an average server is 50% even whaen the server has zero connections

Bind Address used with assumption that it's external

The IP address variable is defined at https://github.com/Tapawingo/FreeTakServer/blob/master/TAKfreeServer/constants.py#L15 is overloaded.

This is used both as the network component of the bind address (https://github.com/Tapawingo/FreeTakServer/blob/master/TAKfreeServer/httpServer.py#L255, https://github.com/Tapawingo/FreeTakServer/blob/master/TAKfreeServer/server.py#L58), and as the assumed external facing address to advertise (https://github.com/Tapawingo/FreeTakServer/blob/master/TAKfreeServer/httpServer.py#L147, https://github.com/Tapawingo/FreeTakServer/blob/master/TAKfreeServer/constants.py#L42).

If the externally accessible hostname or ip address is simplified it would enable running the server behind a proxy or within a docker container. Additionally it makes a bind address of 0.0.0.0 more feasible as the default.

FTS stop receive/send after a period X

send to client server version

send the server version to the ATAK clients

Do propper exception handling

Create propper exception handling for TAK class

Implement Meshstatic

Error starting TAK Windows Service

Disclaimer: I'm new here so I apologize if I've missed something. Please let me know if you need more information. Thanks!

Summary

When I try to run the Windows service (beginProgramAsService.py) I'm never prompted for an IP/port and I get the following error:

PS C:\Users\rcaud\Downloads\TAKfreeServerV0.6\TAKfreeServer\TAKWinService> python .\beginProgramAsService.py install
Traceback (most recent call last):
  File ".\beginProgramAsService.py", line 17, in <module>
    ThreadedServer()
  File "C:\Users\rcaud\Downloads\TAKfreeServerV0.6\TAKfreeServer\server.py", line 56, in __init__
    self.sock.bind((self.host, self.port))
socket.gaierror: [Errno 11001] getaddrinfo failed
PS C:\Users\rcaud\Downloads\TAKfreeServerV0.6\TAKfreeServer\TAKWinService>

Environment Info

• Windows 10 Pro 1909
• Clean python install

PS C:\Users\rcaud\Downloads\TAKfreeServerV0.6\TAKfreeServer\TAKWinService> python --version
Python 3.8.2
PS C:\Users\rcaud\Downloads\TAKfreeServerV0.6\TAKfreeServer\TAKWinService> pip install pywin32
Requirement already satisfied: pywin32 in c:\users\rcaud\appdata\local\programs\python\python38-32\lib\site-packages (227)

• Download latest FTS releasehttps://github.com/Tapawingo/TAKlib/releases/download/V0.6/TAKfreeServerV0.6.zip

script to convert to FreeTAKServer DB to POSTGIS.

There should also be a script to convert to POSTGIS.

This would still be much faster than TAKServer, which uses POSTGIS natively, but provide near-full compatibility

TAK users on TAKFreeServer does not stale out.

Users do not disappear on TAKFreeServer even if they have been logged off or exited. This is on latest GIT version. And TAKFreeServer invoked with "-p port" command on Linux shell.

I can still see TAK users active (colored SPI) and last seen is still in seconds value even if they've been inactive for houre... Even if app is closed.

45 minute timeout

after approximately 45 minutes without any connected clients, the server will cease to transmit recieved messages to other connected clients though the server will continue to accept connections.

Implement a listener

The TAKListener exposes an API to receive calls that are translated into proper COTS
Evaluating Flask for Rest API
see architecture

ports remain open after closing server

after closing the server, trying to restart provoke an error because the socket/port is still open.

Workaround: changing the port / killing the socket

Update and clean up documentation

Introduction

The current documentation located at: ./wiki is too sparse and doesn't go in depth enough. It is also out of date with the current version of TAKlib and thus is missing the new functions.

Goals:

• Go more in depth on each of the function notations and provide more info on exception handling, data handling and expected data types
• Update doumentation to align with the current TAKlib version

Send data

It's done using the data package function? all the communication we are aware of, until now, works over XML . Do you have an example of a message containing data (e.g. pictures?), it's embedded in the XML?
I wonder how they are wrapped in a COT message. Until now all the communication that I have seen is within a XML structure. XML has the ability to handle Binaries, however we need an example of a log.
The ZIP files are dropped in the /atak/tools/datapackage folder.

server with no clients

if the all clients disconnect the server will stop functioning properly.

Bad security

Currently the library has extremly bad security, there is no authentication or validation of senders nor received packages. This is extremly bad as it would allow a malicious party to send malicous data to the server and have it broadcast to all connected clients.

TAK devices fails to synchronize over WAN-like networks

When trying to sync a local device with a remote device it fails. As of now it's unknown what's causing this.

Currently tested:

4G --> 4G fails
Local (behind firewall) --> 4G fails
Local (behind firewall) <-- 4G fails
Local (behind firewall) --> Local (behind firewall) succeeds

datapackage server not functioning

datapackages are not working as expected in 0.8.
Planning to revert to database table containing user information

Support Emergency Beacon / 911

Emergency Beacon / 911 on a server, and it can be disabled by the user and instantaneously reflects to other server users. But behaves differently on FreeTAKServer. Could be 'hanging' when processed through TAKFreeServer. SPI's also has a delay when moving it live. Maybe because of the same reason.

Can't run new TAKDaemon.py on CentOS 7 VPS.

-bash-4.2$ python3 TAKDaemon.py
Traceback (most recent call last):
File "TAKDaemon.py", line 21, in
const = constant.vars()
AttributeError: module 'constant' has no attribute 'vars'

TypeError: an integer is required (got type NoneType)

Currently getting the error 'TypeError: an integer is required (got type NoneType)' when running command:

python3 server.py

Full read out -

Traceback (most recent call last)
File "server.py", line 350, in < module >
ThreadedServer (host=const.ip, port=port)).listen()
File "server.py", line 64, in _ init _
self.sock.bind((self.host, self.port))
TypeError: an integer is required (got type NoneType)

Any suggestions on what I need to do to rectify this? Cheers

Route sharing

route sharing is currently not working

Not an ISSUS a Question

I'm a python developer where can I get the ATAK.apk to start testing?

FTS dies randomly (possible orphaned sockets)

after a non-determined amount of time the server becomes non-responsive.
Possibly the socket that is linked to the thread is not being closed properly

TAKLinuxService: a demon for linux not starting

Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-99-generic x86_64)

running:
root@#####:/TAKlib/TAKfreeServer/TAKLinuxService# python3 beginServerFromLinux.py -p 8087

output:
nohup: appending output to 'nohup.out'

No process ever starts. Am I missing something? I tried with and without port switch "-p" with and got the same results.

This works good though.. python3 /TAKlib/TAKfreeServer/server.py -p 8087

Sending routes via FTS does not show on receiving EUD.

When sending a route through FTS via "Send" it does not appear on receiving EUD (End User Device).

Version: FTS 0.6

No such file or directory: '/logs/FTS_debug.log'

After recent pull I now get No such file or directory: '/logs/FTS_debug.log'. The new folder needs to be added to the repo or the application fails to start.

Ruining:
python3 /TAKlib/TAKfreeServer/server.py -p 8087

Output:
Traceback (most recent call last):
File "/TAKlib/TAKfreeServer/server.py", line 47, in
logger.addHandler(newHandler(const.DEBUGLOG, logging.DEBUG, log_format))
File "/TAKlib/TAKfreeServer/server.py", line 37, in newHandler
backupCount=const.BACKUPCOUNT
File "/usr/lib/python3.6/logging/handlers.py", line 150, in init
BaseRotatingHandler.init(self, filename, mode, encoding, delay)
File "/usr/lib/python3.6/logging/handlers.py", line 57, in init
logging.FileHandler.init(self, filename, mode, encoding, delay)
File "/usr/lib/python3.6/logging/init.py", line 1032, in init
StreamHandler.init(self, self._open())
File "/usr/lib/python3.6/logging/init.py", line 1061, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/logs/FTS_debug.log'

FTS intermittent connection on non-root.

WARNING:2020-05-06 01:51:11,215:error in connection setup: unable to open database file

Permissions are appropriate to the user running. Also tried chmod 777 all files, same outcome.

However, running as root. Works as expected.

Some end-users (like myself) are uncomfortable running daemons as root.