junzis/flight-data-processor

Flight data clustering and flight phase identification

flight-data-processor's Issues

cannot encode object: 379, of type: <class 'numpy.int64'>

I get the following error when I run flightextract.py on my data:

bson.errors.InvalidDocument: cannot encode object: 379, of type: <class 'numpy.int64'>

My data looks just like your sample dataset, except larger. When I run flightextract.py on your data, I don't get any errors.

The full error log that I get when I run it on my data is:

[1] Querying database.
[2] 1133 number of valid ICAOs.
[3][0-50 of 1133] ICAOs beening processed.
  [a] fetching records
flightextract.py:59: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  ids = dfchunk['icao'].as_matrix()
flightextract.py:60: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  lats = dfchunk['lat'].as_matrix()
flightextract.py:61: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  lons = dfchunk['lon'].as_matrix()
flightextract.py:62: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  alts = dfchunk['alt'].as_matrix()
flightextract.py:63: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  spds = dfchunk['spd'].as_matrix()
flightextract.py:64: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  hdgs = dfchunk['hdg'].as_matrix()
flightextract.py:65: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  rocs = dfchunk['roc'].as_matrix()
flightextract.py:66: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  times = dfchunk['ts'].as_matrix()
  [b] data scaling
/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:475: DataConversionWarning: Data with input dtype int64 was converted to float64 by MinMaxScaler.
  warnings.warn(msg, DataConversionWarning)
  [c] creating machine learning dataset.
  [d] start clustering, and saving results to DB
    * processing  . Traceback (most recent call last):
  File "flightextract.py", line 140, in <module>
    ('roc',  c[:, 6].tolist()),
  File "/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 700, in insert_one
    session=session),
  File "/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 614, in _insert
    bypass_doc_val, session)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 602, in _insert_one
    acknowledged, _insert_command, session)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1280, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1233, in _retry_with_session
    return func(session, sock_info, retryable)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 597, in _insert_command
    retryable_write=retryable_write)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/pool.py", line 589, in command
    self._raise_connection_failure(error)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/pool.py", line 750, in _raise_connection_failure
    raise error
  File "/anaconda3/lib/python3.6/site-packages/pymongo/pool.py", line 584, in command
    user_fields=user_fields)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/network.py", line 121, in command
    codec_options, ctx=compression_ctx)
  File "/anaconda3/lib/python3.6/site-packages/pymongo/message.py", line 678, in _op_msg
    flags, command, identifier, docs, check_keys, opts)
bson.errors.InvalidDocument: cannot encode object: 288, of type: <class 'numpy.int64'>

I wonder whether this error might be related to the ts variable?

syntax error on print()

I reproduced the first usage as in the README.

$ python flightextract.py --csv data/sample_adsb_decoded.csv --db test_db --coll flights

but I got syntax errors on a couple of print statements as shown below.
I was able to have all run successfully (the flights are inserted in the DB) after commenting the culprit statements out.

spi@ESSP1240 MINGW64 /c/repos/flight-data-processor (master)
$ python flightextract.py --csv data/sample_adsb_decoded.csv --db test_db --coll flights
  File "flightextract.py", line 110
    print('    * processing ', end=' ')
                                  ^
SyntaxError: invalid syntax
(py27)
spi@ESSP1240 MINGW64 /c/repos/flight-data-processor (master)
$ python flightextract.py --csv data/sample_adsb_decoded.csv --db test_db --coll flights
  File "flightextract.py", line 113
    print('.', end=' ')
                  ^
SyntaxError: invalid syntax
(py27)
spi@ESSP1240 MINGW64 /c/repos/flight-data-processor (master)
$ python flightextract.py --csv data/sample_adsb_decoded.csv --db test_db --coll flights
C:\opt\Miniconda3\envs\py27\lib\site-packages\sklearn\utils\validation.py:475: DataConversionWarning: Data with input dtype int64 was converted to float64 by MinMaxScaler.
  warnings.warn(msg, DataConversionWarning)
[1] Querying database.
[2] 20 number of valid ICAOs.
[3][0-50 of 20] ICAOs beening processed.
  [a] fetching records
  [b] data scaling
  [c] creating machine learning dataset.
  [d] start clustering, and saving results to DB

()
[4] All completed
(py27)

I am working in a conda environment as follows:

$ conda info

     active environment : py27
    active env location : C:\opt\Miniconda3\envs\py27
            shell level : 1
       user config file : C:\Users\spi\.condarc
 populated config files : C:\Users\spi\.condarc
          conda version : 4.5.3
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : C:\opt\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\opt\Miniconda3\pkgs
                          C:\Users\spi\AppData\Local\conda\conda\pkgs
       envs directories : C:\opt\Miniconda3\envs
                          C:\Users\spi\AppData\Local\conda\conda\envs
                          C:\Users\spi\.conda\envs
               platform : win-64
             user-agent : conda/4.5.3 requests/2.18.4 CPython/3.6.5 Windows/7 Windows/6.1.7601
          administrator : False
             netrc file : None
           offline mode : False

and the following packages:

$ conda list
# packages in environment at C:\opt\Miniconda3\envs\py27:
#
# Name                    Version                   Build  Channel
_license                  1.1                      py27_1
alabaster                 0.7.9                    py27_0
anaconda                  4.3.0               np111py27_0
anaconda-client           1.6.0                    py27_0
anaconda-navigator        1.4.3                    py27_0
argcomplete               1.0.0                    py27_1
arm_pyart                 1.7.0               np111py27_0    conda-forge
astroid                   1.4.9                    py27_0
astropy                   1.3                 np111py27_0
babel                     2.3.4                    py27_0
backports                 1.0                      py27_0
backports_abc             0.5                      py27_0
basemap                   1.0.7               np111py27_0
beautifulsoup4            4.5.3                    py27_0
bitarray                  0.8.1                    py27_1
blaze                     0.10.1                   py27_0
bokeh                     0.12.4                   py27_0
boto                      2.45.0                   py27_0
bottleneck                1.2.0               np111py27_0
bzip2                     1.0.6                     vc9_3  [vc9]
cdecimal                  2.3                      py27_2
certifi                   2018.4.16                py27_0
cffi                      1.9.1                    py27_0
chardet                   2.3.0                    py27_0
chest                     0.2.3                    py27_0
click                     6.7                      py27_0
cloudpickle               0.2.2                    py27_0
clyent                    1.2.2                    py27_0
colorama                  0.3.7                    py27_0
comtypes                  1.1.2                    py27_0
configobj                 5.0.6                    py27_0
configparser              3.5.0                    py27_0
console_shortcut          0.1.1                    py27_1
contextlib2               0.5.4                    py27_0
cryptography              1.7.1                    py27_0
curl                      7.52.1                    vc9_0  [vc9]
cycler                    0.10.0                   py27_0
cython                    0.25.2                   py27_0
cytoolz                   0.8.2                    py27_0
dask                      0.13.0                   py27_0
datashape                 0.5.4                    py27_0
decorator                 4.0.11                   py27_0
dill                      0.2.5                    py27_0
docutils                  0.13.1                   py27_0
entrypoints               0.2.2                    py27_0
enum34                    1.1.6                    py27_0
et_xmlfile                1.0.1                    py27_0
fastcache                 1.0.2                    py27_1
flask                     0.12                     py27_0
flask-cors                3.0.2                    py27_0
freetype                  2.5.5                     vc9_2  [vc9]
funcsigs                  1.0.2                    py27_0
functools32               3.2.3.2                  py27_0
futures                   3.0.5                    py27_0
get_terminal_size         1.0.0                    py27_0
gevent                    1.2.1                    py27_0
greenlet                  0.4.11                   py27_0
grin                      1.2.1                    py27_3
h5py                      2.6.0               np111py27_2
hdf5                      1.8.15.1                  vc9_4  [vc9]
heapdict                  1.0.0                    py27_1
icc_rt                    2017.0.4             h97af966_0
icu                       57.1                      vc9_0  [vc9]
idna                      2.2                      py27_0
imagesize                 0.7.1                    py27_0
intel-openmp              2018.0.0                      8
ipaddress                 1.0.18                   py27_0
ipykernel                 4.5.2                    py27_0
ipython                   5.1.0                    py27_0
ipython_genutils          0.1.0                    py27_0
ipywidgets                5.2.2                    py27_1
isort                     4.2.5                    py27_0
itsdangerous              0.24                     py27_0
jdcal                     1.3                      py27_0
jedi                      0.9.0                    py27_1
jinja2                    2.9.4                    py27_0
jpeg                      9b                        vc9_0  [vc9]
jsonschema                2.5.1                    py27_0
jupyter                   1.0.0                    py27_3
jupyter_client            4.4.0                    py27_0
jupyter_console           5.0.0                    py27_0
jupyter_core              4.2.1                    py27_0
lazy-object-proxy         1.2.2                    py27_0
libnetcdf                 4.3.3.1                   vc9_4  [vc9]
libpng                    1.6.27                    vc9_0  [vc9]
libtiff                   4.0.6                     vc9_3  [vc9]
llvmlite                  0.15.0                   py27_0
locket                    0.2.0                    py27_1
lxml                      3.7.2                    py27_0
markupsafe                0.23                     py27_2
matplotlib                2.0.0               np111py27_0
menuinst                  1.4.4                    py27_0
mistune                   0.7.3                    py27_0
mkl                       2018.0.2                      1
mkl-service               1.1.2                    py27_3
mkl_fft                   1.0.1            py27hc997a72_0
mpmath                    0.19                     py27_1
multipledispatch          0.4.9                    py27_0
nbconvert                 4.2.0                    py27_0
nbformat                  4.2.0                    py27_0
netcdf4                   1.2.4               np111py27_0
networkx                  1.11                     py27_0
nltk                      3.2.2                    py27_0
nose                      1.3.7                    py27_1
notebook                  4.3.1                    py27_1
numba                     0.30.1              np111py27_0
numexpr                   2.6.1               np111py27_2
numpy                     1.14.3           py27h297e335_0
numpy-base                1.14.3           py27h0bb1d87_0
numpydoc                  0.6.0                    py27_0
odo                       0.5.0                    py27_1
openpyxl                  2.4.1                    py27_0
openssl                   1.0.2k                    vc9_0  [vc9]
pandas                    0.22.0           py27hc56fc5f_0
partd                     0.3.7                    py27_0
path.py                   10.0                     py27_0
pathlib2                  2.2.0                    py27_0
patsy                     0.4.1                    py27_0
pep8                      1.7.0                    py27_0
pickleshare               0.7.4                    py27_0
pillow                    4.0.0                    py27_0
pip                       10.0.1                   py27_0
ply                       3.9                      py27_0
prompt_toolkit            1.0.9                    py27_0
psutil                    5.0.1                    py27_0
py                        1.4.32                   py27_0
pyasn1                    0.1.9                    py27_0
pycosat                   0.6.1                    py27_1
pycparser                 2.17                     py27_0
pycrypto                  2.6.1                    py27_5
pycurl                    7.43.0                   py27_2
pyflakes                  1.5.0                    py27_0
pygments                  2.1.3                    py27_0
pylint                    1.6.4                    py27_1
pymongo                   3.4.0                    py27_0
pyopenssl                 16.2.0                   py27_0
pyparsing                 2.1.4                    py27_0
pyqt                      5.6.0                    py27_2
pytables                  3.2.2               np111py27_4
pytest                    3.0.5                    py27_0
python                    2.7.15               he216670_0
python-dateutil           2.7.2                    py27_0
pytz                      2018.4                   py27_0
pywin32                   220                      py27_2
pyyaml                    3.12                     py27_0
pyzmq                     16.0.2                   py27_0
qt                        5.6.2                     vc9_3  [vc9]
qtawesome                 0.4.3                    py27_0
qtconsole                 4.2.1                    py27_2
qtpy                      1.2.1                    py27_0
requests                  2.12.4                   py27_0
rope                      0.9.4                    py27_1
scandir                   1.4                      py27_0
scikit-image              0.12.3              np111py27_1
scikit-learn              0.19.1           py27he00e316_0
scipy                     1.1.0            py27ha6b90e7_0
seaborn                   0.7.1                    py27_0
setuptools                39.1.0                   py27_0
simplegeneric             0.8.1                    py27_1
singledispatch            3.4.0.3                  py27_0
sip                       4.18                     py27_0
six                       1.11.0           py27ha5e1701_1
snowballstemmer           1.2.1                    py27_0
sockjs-tornado            1.0.3                    py27_0
sphinx                    1.5.1                    py27_0
spyder                    3.1.2                    py27_0
sqlalchemy                1.1.5                    py27_0
ssl_match_hostname        3.4.0.2                  py27_1
statsmodels               0.6.1               np111py27_1
subprocess32              3.2.7                    py27_0
sympy                     1.0                      py27_0
tk                        8.5.18                    vc9_0  [vc9]
toolz                     0.8.2                    py27_0
tornado                   4.4.2                    py27_0
traitlets                 4.3.1                    py27_0
unicodecsv                0.14.1                   py27_0
vc                        9                    h7299396_1
vs2008_runtime            9.00.30729.5054               0
vs2015_runtime            14.0.25123                    3
wcwidth                   0.1.7                    py27_0
werkzeug                  0.11.15                  py27_0
wheel                     0.31.0                   py27_0
widgetsnbextension        1.2.6                    py27_0
win_unicode_console       0.5                      py27_0
wincertstore              0.2              py27hf04cefb_0
wrapt                     1.10.8                   py27_0
xlrd                      1.0.0                    py27_0
xlsxwriter                0.9.6                    py27_0
xlwings                   0.10.2                   py27_0
xlwt                      1.2.0                    py27_0
zlib                      1.2.8                     vc9_3  [vc9]

error when running python test_phases.py

It looks like test_phases.py uses a python 3.6 version of pickle.load() which accepts an encoding argument, while the 2.7 version does not.

Here is the error I get when running python test_phases.py as in the README:

$ python test_phases.py
Traceback (most recent call last):
  File "test_phases.py", line 12, in <module>
    dataset = pickle.load(open(datadir+'/data/test_segment.pkl', 'rb'), encoding='bytes')
TypeError: load() got an unexpected keyword argument 'encoding'
(py27)

I tried to remove encoding='bytes' but I get further issues (after I close the plots from flightphase.plot_logics()):

$ python test_phases.py
Traceback (most recent call last):
  File "test_phases.py", line 12, in <module>
    dataset = pickle.load(open(datadir+'/data/test_segment.pkl', 'rb'))
  File "C:\opt\Miniconda3\envs\py27\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\opt\Miniconda3\envs\py27\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\opt\Miniconda3\envs\py27\lib\pickle.py", line 1096, in load_global
    klass = self.find_class(module, name)
  File "C:\opt\Miniconda3\envs\py27\lib\pickle.py", line 1130, in find_class
    __import__(module)
ImportError: No module named datetime
(py27)

Error when running the test_phases.py example

Hello, I've just discovered this toolkit - which looks really useful...but I have run into a problem:
I'm trying to run the test_phases example file and am running into an error (see below). This also happens with my own data: It seems that the labels array returned by fuzzylabels is larger than the original data arrays.
Any ideas what could be happening here? I will investigate this myself but thought I'd create an issue in case you have a quick insight into what went wrong.

  File "/home/s/miniconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 4291, in _parse_scatter_color_args
    raise ValueError
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_phases.py", line 38, in <module>
    plt.scatter(times, alts, marker='.', c=colors, lw=0)
  File "/home/s/miniconda3/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2847, in scatter
    None else {}), **kwargs)
  File "/home/s/miniconda3/lib/python3.7/site-packages/matplotlib/__init__.py", line 1601, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/home/s/miniconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 4454, in scatter
    get_next_color_func=self._get_patches_for_fill.get_next_color)
  File "/home/s/miniconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 4298, in _parse_scatter_color_args
    .format(nc=n_elem, xs=xsize, ys=ysize)
ValueError: 'c' argument has 1048 elements, which is not acceptable for use with 'x' with size 934, 'y' with size 934.

This is with a python 3.7.3 conda environment with the following library versions:
numpy 1.17.0
scikit-fuzzy 0.4.1
matplotlib 3.1.1
scipy 1.3.0

ADS-B 数据中RoC 字段的含义是什么？

我下载到adsb数据并没有包含这个字段