Giter Site home page Giter Site logo

vaexio / vaex Goto Github PK

View Code? Open in Web Editor NEW
8.2K 146.0 588.0 133.58 MB

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second ๐Ÿš€

Home Page: https://vaex.io

License: MIT License

JavaScript 24.53% Python 60.90% Shell 0.26% C++ 10.27% Batchfile 0.03% CSS 0.11% HTML 3.11% PHP 0.57% Makefile 0.02% Vue 0.15% CMake 0.06%
dataframe python bigdata tabular-data visualization memory-mapped-file hdf5 machine-learning machinelearning data-science

vaex's Introduction

Supported Python Versions Documentation Slack

What is Vaex?

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It calculates statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid for more than a billion (10^9) samples/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

Installing

With pip:

$ pip install vaex

Or conda:

$ conda install -c conda-forge vaex

For more details, see the documentation

Key features

Instant opening of Huge data files (memory mapping)

HDF5 and Apache Arrow supported.

opening1a

opening1b

Read the documentation on how to efficiently convert your data from CSV files, Pandas DataFrames, or other sources.

Lazy streaming from S3 supported in combination with memory mapping.

opening1c

Expression system

Don't waste memory or time with feature engineering, we (lazily) transform your data when needed.

expression

Out-of-core DataFrame

Filtering and evaluating expressions will not waste memory by making copies; the data is kept untouched on disk, and will be streamed only when needed. Delay the time before you need a cluster.

occ-animated

Fast groupby / aggregations

Vaex implements parallelized, highly performant groupby operations, especially when using categories (>1 billion/second).

groupby

Fast and efficient join

Vaex doesn't copy/materialize the 'right' table when joining, saving gigabytes of memory. With subsecond joining on a billion rows, it's pretty fast!

join

More features

Contributing

See contributing page.

Slack

Join the discussion in our Slack channel!

Learn more about Vaex

vaex's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vaex's Issues

python 3 errors in imports and syntax

I was able to correct a couple issues and make simple plots.

Patch below

diff --git a/python/vaex/image.py b/python/vaex/image.py
index 70fe8a0..78f63fb 100644
--- a/python/vaex/image.py
+++ b/python/vaex/image.py
@@ -1,12 +1,15 @@
 __author__ = 'maartenbreddels'
 import PIL.Image
 import PIL.ImageDraw
-import StringIO
 import collections
 import numpy as np
 import matplotlib.colors
 import warnings
 from base64 import b64encode
+try:
+    from StringIO import StringIO
+except:
+    from io import StringIO

 def rgba_2_pil(rgba):
        with warnings.catch_warnings():
@@ -15,7 +18,7 @@ def rgba_2_pil(rgba):
        return im

 def pil_2_data(im, format="png"):
-       f = StringIO.StringIO()
+       f = StringIO()
        im.save(f, format)
        return f.getvalue()

@@ -37,7 +40,7 @@ cairo_modes["saturate"] = (lambda aA, aB: np.minimum(1, aA + aB),
 cairo_modes["add"] = (lambda aA, aB: np.minimum(1, aA + aB),
                                                   lambda aA, xA, aB, xB, aR: (aA*xA + aB*xB)/aR)

-modes = pdf_modes.keys() + cairo_modes.keys()
+modes = list(pdf_modes.keys()) + list(cairo_modes.keys())
 def background(shape, color="white", alpha=1, bit8=True):
        rgba = np.zeros(shape + (4,))
        rgba[:] = np.array(matplotlib.colors.colorConverter.to_rgba(color))

Live demo: Cross-Origin Request Blocked

I'm trying to zoom the interactive "Yellow taxi pickup locations in New York City" plot on the web site http://vaex.astro.rug.nl/, however in the browser console I observe the following messages:

13:31:14.782 do debounched ajax call
hello.js:51:17
13:31:14.783 debounched ajax call
hello.js:75:13
13:31:14.783
Object { widget_manager: {โ€ฆ}, state_change: Promise { "fulfilled" }, _buffered_state_diff: {โ€ฆ}, pending_msgs: 0, msg_buffer: null, state_lock: null, id: "08ced375127f4de7be13a63e2e4e65ee", isNew: constructor/this.isNew(), views: {}, comm_live: false, โ€ฆ }
hello.js:77:13
13:31:14.785
Object { expressions: [โ€ฆ], variables: [], f: "log1p", virtual_columns: [], size: [โ€ฆ] }
hello.js:79:13
13:31:14.787
Object { limits: "[[-73.95398471742892,-73.91576366890979],[40.719841585654585,40.751078493433006]]", expressions: "[\"pickup_longitude\",\"pickup_latitude\"]", variables: "[]", f: "\"log1p\"", virtual_columns: "[]", size: "[600,600]" }
hello.js:86:13
13:31:14.811
GETXHR
http://www.astro.rug.nl/~breddels/cgi-bin/proxy.cgi/jansky:9002/datasets/nyt2015/image_rgba_url
[HTTP/1.1 500 Unexpected Error 169ms]
13:31:14.983 ajax error...
hello.js:92:21
13:31:14.982 Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://www.astro.rug.nl/~breddels/cgi-bin/proxy.cgi/jansky:9002/datasets/nyt2015/image_rgba_url?limits=%5B%5B-73.95398471742892%2C-73.91576366890979%5D%2C%5B40.719841585654585%2C40.751078493433006%5D%5D&expressions=%5B%22pickup_longitude%22%2C%22pickup_latitude%22%5D&variables=%5B%5D&f=%22log1p%22&virtual_columns=%5B%5D&size=%5B600%2C600%5D. (Reason: CORS header โ€˜Access-Control-Allow-Originโ€™ missing).

It seems the new "zoomed" data is not coming to the client due to this error. Could it be fixed?

A side question - are there any instructions how to embed this kind of interactive plots to a Jupyter notebook?

Broken lightboxes on website

The lightboxes on the project website are broken, because they contain a tags with a href attribute that does not include image/ (e.g. href="nyc_taxi_full.png" rather than href="image/nyc_taxi_full.png").

ModuleNotFoundError: No module named 'vaex.ext.bqplot'

Just started using vaex, trying to 3d plot galaxy sim, 2D plots work. Installed bqplot seperately - still not recognised by vaex.

Full error is:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-1b6052421b12> in <module>()
----> 1 ds.plot3d(ds.x,ds.y,ds.z,f='log')

/usr/local/lib/python3.6/site-packages/vaex/dataset.py in plot3d(self, x, y, z, vx, vy, vz, vwhat, limits, grid, what, shape, selection, f, vcount_limits, smooth_pre, smooth_post, grid_limits, normalize, colormap, figure_key, fig, lighting, level, opacity, level_width, show, **kwargs)
   2464                show=True, **kwargs):
   2465         """Use at own risk, requires ipyvolume"""
-> 2466         import vaex.ext.ipyvolume
   2467         # vaex.ext.ipyvolume.
   2468         cls = vaex.ext.ipyvolume.PlotDefault

/usr/local/lib/python3.6/site-packages/vaex/ext/ipyvolume.py in <module>()
----> 1 from .bqplot import PlotBase
      2 import ipyvolume.pylab as p3
      3 import ipyvolume.examples
      4 import traitlets
      5 import vaex.dataset

ModuleNotFoundError: No module named 'vaex.ext.bqplot'

Using Python3, installed vaex through pip3. Using conda ruined a lot of stuff, so not risking it atm

Edit: vaex.ext is completely empty and for some reason I can't find it in the vaex repository

Feature Request: support for datetime data

It would be great if there could be support for numpy.datetime64 objects in a data series and axis scale as I regularly get time data quoted in this format and charted using matplotlib. The data is getting large, so I would ideally like to be able to scatter plot this using vaex.

Feature request: 2D line plot

Hi,

I have looked at the documentation/code and I don't think a 2D line plot is supported in Vaex. I would like to plot 100M rows of data from a hdf5 column to create an interactive plot in Jupyter. I would prefer bqplot to matplotlib if possible.

Vaex demo not working on chrome 63

Interactive demos on https://vaex.io/ are not working: zoom-in works, but data is not updated...
Browser: Google Chrome Version 63.0.3239.132 (Official Build) (64-bit), OpenSuse Linux
Error message:
Mixed Content: The page at 'https://vaex.io/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://www.astro.rug.nl/~breddels/cgi-bin/proxy.cgi/jansky:9002/datasets/gaia-dr1-lb-10percent/image_rgba_url?limits=%5B%5B31.003323621705984%2C18.258769770183576%5D%2C%5B-4.969398881812663%2C1.4028780439485458%5D%5D&expressions=%5B%22(l%2B180)%25360-180%22%2C%22b%22%5D&variables=%5B%5D&f=%22log1p%22&virtual_columns=%5B%5D&size=%5B512%2C512%5D'. This request has been blocked; the content must be served over HTTPS.

Non conda dependencies

I was looking at making a conda recipe for vaex. The following dependecies are not available on the main conda channels

aplus
attrdict
cachetools
jprops
kapteyn
progressbar2

BUG?: pip install fail because of missing css

Hi,

I tried to install vaex with pip install vaex and got the following error

   error: can't copy 'docs/build/html/_static/css': doesn't exist or not a regular file
    
    ----------------------------------------
Command "/home/jrking/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-JpRy3M/vaex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-46rAqR-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-JpRy3M/vaex/

I have using anaconda2.

any idea?

Support for substring search in selection

I'd like to be able to perform a selection by searching for a substring in a given field of my dataset.
Something like this:

t = vaex.open('simbad.fits')
t.select("object_type.find('galax')>=0")

For the record: in numpy, you can use this syntax for this use case:

foo = np.array(['star', 'galaxy', 'quasar', 'cluster of galaxies'])
np.where(np.char.find(foo,  'gal') > -1)

Python 3

Incompatible with python 3. Do you plan to make a version?

Installation error in sampy.py through pip

I was trying to install Vaex through pip (9.0.1). After installing all the requirements an error occurred in sampy installation.
Although its not a Vaex issue i couldn't find the sampy package to try to fix or raise an issue, so i attended here.

Collecting sampy>=1.2 (from vaex)
Downloading sampy-1.2.1.tar.gz (63kB)
100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 71kB 176kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-LgDLkG/sampy/setup.py", line 3, in
import sampy
File "sampy.py", line 914, in
class HTTPSConnection(httplib.HTTPConnection):
File "sampy.py", line 921, in HTTPSConnection
ssl_version=ssl.PROTOCOL_SSLv3, strict=None):
AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

I updated urllib3 (1.22), requests (2.6.0), pip (9.0.1) etc packages as mentioned in other sources.

Tutorial run through issues on Windows 10

Platform, Windows 10, Python 3.5
The same codes works correctly according to tutorial on CENTOS/LINUX

#1 'warning'

>>> import vaex
>>> ds = vaex.example()
lib\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

NOTE

>>> ds.get_column_names()
['E', 'FeH', 'L', 'Lz', 'random_index', 'vx', 'vy', 'vz', 'x', 'y', 'z']

>>> ds.max('x')
array(271.365997)

#2 'Is this due to an installation error'?

import vaex
ds = vaex.example()  # open the example dataset provided with vaex
ds[ds.x < 0]

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-bd4c28c4abd5> in <module>()
      1 import vaex
      2 ds = vaex.example()  # open the example dataset provided with vaex
----> 3 ds[ds.x < 0]

AttributeError: 'Hdf5MemoryMapped' object has no attribute 'x'

Broken export_hdf5

I have no idea what happened but it seems the export function is looking for groups that do not exist.

~/anaconda/lib/python3.6/site-packages/vaex/dataset.py in export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
   4993         """
   4994         import vaex.export
-> 4995         vaex.export.export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress=progress, virtual=virtual, sort=sort, ascending=ascending)
   4996 
   4997     def export_fits(self, path, column_names=None, shuffle=False, selection=False, progress=None, virtual=False, sort=None, ascending=True):

~/anaconda/lib/python3.6/site-packages/vaex/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    235     kwargs = locals()
    236     import vaex.hdf5.export
--> 237     vaex.hdf5.export.export_hdf5(**kwargs)
    238 
    239 

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    174 
    175         # after this the file is closed,, and reopen it using out class
--> 176         dataset_output = vaex.hdf5.dataset.Hdf5MemoryMapped(path, write=True)
    177 
    178 	column_names = vaex.export._export(dataset_input=dataset, dataset_output=dataset_output, path=path, random_index_column=random_index_name,

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in __init__(self, filename, write)
     61                 self._version = 1
     62                 try:
---> 63                         self._load()
     64                 finally:
     65                         self.h5file.close()

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in _load(self)
    145                 if "table" in self.h5file:
    146                         self._version = 2
--> 147                         self._load_columns(self.h5file["/table"])
    148                         self.h5table_root_name = "/table"
    149                 # TODO: shall we rename it vaex... ?

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in _load_columns(self, h5data, first)
    228                                         if unitname == "system.get('S.I.').base('mass')":
    229                                                 self.units[column_name] = astropy.units.Unit("kg")
--> 230                                 data = column if self._version == 1 else column['data']
    231                                 if hasattr(data, "dtype"):
    232                                         #print column, column.shape

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

~/anaconda/lib/python3.6/site-packages/h5py/_hl/group.py in __getitem__(self, name)
    165                 raise ValueError("Invalid HDF5 object reference")
    166         else:
--> 167             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    168 
    169         otype = h5i.get_type(oid)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.open()

KeyError: "Unable to open object (object 'data' doesn't exist)"

Dealing with nans

Hi,

First of all, thanks for this awesome package. May I ask, can you please add a way of dealing with nans (or missing values) in the dataset, especially if the columns are a numeric type of some sort. This could be dropping columns or rows that have nans, or maybe imputing a user defined values.

Thank you,
J.

Rotation in 3d?

Can add_virtual_columns_rotation also work for three dimensions, rotating over theta and phi? Or is it necessary to do this in steps?

Documentation: Vaex / Numpy equivalent commands

In the documentation, it would be v useful to have a list / table of Numpy /Pandas / Dask commands alongside the equivalent Vaex command.

Suggested commands could include:

  • numpy.histogram(a, array) | vx.count(....., binby(a,) )
  • numpy.2dhistogram | vx.count(....., binby(a,b) )
  • numpy.histogram(.... , weights = weightsArray )
  • numpy.diff
  • pandas.rolling_mean
  • numpy.cumsum
  • numpy.fft

(N.B, all the commands above are supported by Dask)

There could be many more commands listed. There may be no equivalent Vaex command but it would be useful if the documentation could state either:

  • Planned for release #.# (see github issue ###)
  • help wanted
  • no plans to support
  • no direct equivalent command but here is a hack.......

add an option to cache selection in memory

It seems that vaex is not caching selection in memory. I think it will be useful to have an option to cache a selection in memory. It will avoid to go trough all the data again when working on a subset.

One should manage memory limitation, by example by setting a memory buffer limit. One should be warned if the selection exceed buffer limit.

Syntax error with `pip install` on Python 3

$ pip3 install vaex
Collecting vaex
  Using cached vaex-0.3.10.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/4j/n8nxnsy12y92t475g318yptm0000gn/T/pip-build-in7iizfc/vaex/setup.py", line 18
        print "Executing: ", cmd
                          ^
    SyntaxError: Missing parentheses in call to 'print'

    ----------------------------------------

I don't get the error when installing from Git:

$ pip3 install git+https://github.com/maartenbreddels/vaex/

My system info:

$ pip3 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)

$ python3 --version
Python 3.6.2

Calculating limits with selection

Hi,

There seems to be a problem when calculating the limits of a column while having specified a selection.
For example:
dataset.limits('param1', value='99%', selection='sel')
fully ignores the selection.
However
dataset.limits('param1', value='minmax', selection='sel')
works as expected with the specified selection.

Thanks,
Jovan.

Acknowledgement

Is there an official acknowledgement text somewhere to give credit to Vaex and your work in publications?

Bug in Windows: os.system( wget ... )

Hi,

The fetch() command doesn't seem to work in Windows. For example:

dataset = nyctaxi_yellow_201x.fetch()

I get an error message because Vaex is looking for a wget utility. Perhaps this utility only exists by standard in Linux and Mac OS.

In dataset.py I suggest to replace:
os.system("wget -c -P %s %s" % (os.path.join(data_dir, self.subdir), url))

with something like:
import wget
wget.download(self.url_list[i], os.path.join(data_dir, self.subdir))

I cloned the project and attempted to implement it but I couldn't get it to work. I just get an error. HTTPError: HTTP Error 403: Forbidden. More work required....

healpix_plot & healpix_count expecting Gaia conventions

The current API is based on Gaia defaults which are not obvious outside Gaia:
healpix_expression='source_id/34359738368', ... nest=True, ...

These are not the defaults in healpy. Healpy by defaults does nest=False for instance.

Maybe providing a virtual column that does the healpix calculations from RA, DEC columns similarly to the coordinate conversions?

maybe adapting properly something like:

def add_columns_healpix_data(ds, coords=('ra', 'dec'), 
                          name='source_id', nest=True,
                          lonlat=True, NSIDE=4096):
    """ Vaex defaults to Gaia conventions for healpix handling
    
    namely it defaults to having Healpix cell numbers as
    'source_id / 34359738368'
    
    It also uses nested pixels (not rings)
    """
    import healpy
    # add healpix numbers for plotting
    # using Gaia conventions
    ra, dec = coords
    _ra = ds.evaluate(ra)[:]
    _dec = ds.evaluate(dec)[:]
    hpix = healpy.ang2pix(NSIDE, _ra, _dec, 
                          nest=nest, lonlat=lonlat)
    # convention from gaia
    ds.add_column(name, (34359738368 * hpix).astype(int))
    return ds

Bug: 'vaex.ext.bqplot' has no attribute 'get_class'

Hi,

The plot_bq doesn't seem to work. See code and error message below. There doesn't appear to be a get_class function in the repo. I have noticed that you mentioned in the code comments that the bqplot in Vaex is experimental so perhaps I should just be using matplotlib for now. However, the bqplot demo in the 'Billion stars in Jupyter' presentation you did is very impressive! I tried to replicate it but I can't.

import vaex as vx
dataset = vx.datasets.helmi_de_zeeuw.fetch() # get a cup of coffee while this downloads
dataset.plot_bq("x", "y", f="log1p", show=True)


AttributeError Traceback (most recent call last)
in ()
----> 1 dataset.plot_bq("x", "y", f="log1p", show=True)

c:\users\doug\documents\github\vaexdr\vaex\vaex\dataset.py in plot_bq(self, x, y, grid, shape, limits, what, figsize, f, figure_key, fig, axes, xlabel, ylabel, title, show, selection, colormap, grid_limits, normalize, grid_before, what_kwargs, type, scales, tool_select, bq_cleanup, **kwargs)
1991 **kwargs):
1992 import vaex.ext.bqplot
-> 1993 cls = vaex.ext.bqplot.get_class(type)
1994 plot2d = cls(dataset=self, x=x, y=y, grid=grid, shape=shape, limits=limits, what=what,
1995 f=f, figure_key=figure_key, fig=fig,

AttributeError: module 'vaex.ext.bqplot' has no attribute 'get_class'

not opening vaex file (h5vaex False) -- can't find a bug and no error message

Very strangely, I had install vaex on a clean python distribution and now opening a file does not seems to do anything without raising any issue.

vaex.open('./gums18_mw_S000322010.hdf5')
I expected to return a Hdf5MemoryMapped instance but returns None

The file works on a different machine (not going through updates)
list(h5py.File('gums18_mw_S000322010.hdf5', "r")['/table'].items())
Does list columns and expected content.

it seems it does not recognize the format for some unknown reasons, below I expect
h5vaex True which appends on the other machine.

for name, class_ in list(vaex.file.other.dataset_type_map.items()):
     print(name, class_.can_open('./gums18_mw_S000322010.hdf5'))
    
buist False
fits False
h5vaex False
amuse False
gadget-hdf5 False
soneira-peebles False
zeldovich False
ascii False
gadget-plain False
votable False
tap False

I do not know if this bug is from Vaex or a bunker library, but I do not have any reported info as to where it fails... any idea?

progressbar.widgets has no attribute FormatWidgetMixin

Hi,

trying to run a pip installed git:master version i got the following import error:

epinux@epinux:~$ vaex
Traceback (most recent call last):
  File "/usr/local/bin/vaex", line 11, in <module>
    load_entry_point('vaex==1.0.0b6', 'console_scripts', 'vaex')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/vaex/__init__.py", line 71, in <module>
    import vaex.dataset
  File "/usr/local/lib/python3.5/dist-packages/vaex/dataset.py", line 18, in <module>
    import vaex.utils
  File "/usr/local/lib/python3.5/dist-packages/vaex/utils.py", line 179, in <module>
    class CpuUsage(progressbar.widgets.FormatWidgetMixin, progressbar.widgets.TimeSensitiveWidgetBase):
AttributeError: module 'progressbar.widgets' has no attribute 'FormatWidgetMixin'

Equal aspect without square figure?

Is it possible to have equal aspect axes in the GUI, without having a square image?
In this case, an x axis ranging from 0-1 and an y axis ranging from 0-2 with equal aspect would result in a rectangular figure (e.g. with similar x/y ratio as in the figure without equal aspect).

expressions don't work

Hi, I tried adding a virtual column in Zeldovich sample data, and get an error message. The same thing happens if I try to create a variable. I have installed Vaex in a virtual-env.

An unexpected error occured, you may press ok and continue, but the program might be unstable.

username: 'johannes'
program: 'vaex'
version: '1.0.0-beta.4'
full name: 'vaex-1.0.0-beta.4'
arguments: ['/home/johannes/.local/share/workon/vaex/bin/vaex']
Qt version: '5.8.1'
architecture: ('64bit', '')
dist: ('fedora', '25', 'Twenty Five')
java_ver: ('', '', ('', '', ''), ('', '', ''))
libc_ver: ('glibc', '2.3.4')
linux_distribution: ('Fedora', '25', 'Twenty Five')
mac_ver: ('', ('', '', ''), '')
machine: 'x86_64'
node: 'ESLT0041'
platform: 'Linux-4.9.14-200.fc25.x86_64-x86_64-with-fedora-25-Twenty_Five'
processor: 'x86_64'
python_branch: ''
python_build: ('default', 'Mar 21 2017 17:21:33')
python_compiler: 'GCC 6.3.1 20161221 (Red Hat 6.3.1-1)'
python_implementation: 'CPython'
python_revision: ''
python_version: '3.5.3'
python_version_tuple: ('3', '5', '3')
release: '4.9.14-200.fc25.x86_64'
system: 'Linux'
uname: uname_result(system='Linux', node='ESLT0041', release='4.9.14-200.fc25.x86_64', version='#1 SMP Mon Mar 13 19:26:40 UTC 2017', machine='x86_64', processor='x86_64')
version: '#1 SMP Mon Mar 13 19:26:40 UTC 2017'
win32_ver: ('', '', '', '')

Traceback (most recent call last):
File "/home/johannes/.local/share/workon/vaex/lib/python3.5/site-packages/vaex/ui/columns.py", line 77, in setData
self.dataset.write_meta()
File "/home/johannes/.local/share/workon/vaex/lib/python3.5/site-packages/vaex/dataset.py", line 4382, in write_meta
path = os.path.join(self.get_private_dir(create=True), "meta.yaml")
File "/home/johannes/.local/share/workon/vaex/lib/python3.5/site-packages/vaex/dataset.py", line 4306, in get_private_dir
name = os.path.abspath(self.path).replace("/", "_")[:250] # should not be too long for most os'es
File "/home/johannes/.local/share/workon/vaex/lib64/python3.5/posixpath.py", line 358, in abspath
if not isabs(path):
File "/home/johannes/.local/share/workon/vaex/lib64/python3.5/posixpath.py", line 64, in isabs
return s.startswith(sep)
AttributeError: 'NoneType' object has no attribute 'startswith'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.