asuiu/pyxtension

Pure Python extensions library that includes Scala-like streams, Json with attribute access syntax, and other common use stuff

pyxtension's Issues

Test Slack2

error invalid syntax when startup

Hi there, got strange error,

i installed 2 version, local and prods server, but on prod server it gives me error like this :

Traceback (most recent call last):
File "versi1_telebot.py", line 1, in
from pyxtension.Json import Json
File "/usr/local/lib/python3.5/dist-packages/pyxtension/Json.py", line 14, in
from pyxtension.streams import *
File "/usr/local/lib/python3.5/dist-packages/pyxtension/streams.py", line 45
_IDENTITY_FUNC: Callable[[T], T] = lambda _: _
^
SyntaxError: invalid syntax

pip install version does not match

I installed pyxtension with pip, but it's weird the version shown doesn't match the real version.
I want to install version 1.11:

$ pip install pyxtension==1.11
Collecting pyxtension==1.11
Installing collected packages: pyxtension
Successfully installed pyxtension-1.0

then I use pip freeze to see the version:

$ pip freeze | grep pyxtension
pyxtension==1.0

It shows version 1.0.

Implement split_in_batches

Example:

def split_in_batches(self, itr: Iterable[T], batch_max_size: int = BATCH_SIZE) -> Generator[slist[T], None, None]:
        batch = slist()
        for el in itr:
            batch.append(el)
            if batch.size() == batch_max_size:
                yield batch
                batch = slist()
        if batch.size():
            yield batch

Implement logger function on `stream`

Implement try_map() method on streams

It should try the map function, and if it throws exception return an instance of custom "Fail" object that evaluates to False for filter() functions, and contain the exception data (stack trace, etc..)

Use pathos multiprocess for mpmap

Standard lib multiprocess is not able to handle lambda functions due to use of limited serializer (Pickle). Pathos multiprocess fork is able to serialize lambdas using Dill serializer.

Proposal is to make pyxtension use this improved multiprocessing library to handle cases like lambda functions and other advantages it brings

Installation with pip

Hello

Is there any plans to standardize installation process with pip?
P.S. Thank you for such a convenient library!

Implement __add__, __iadd__ custom methods on slist/sdict

Why:
slist()+slist() retuns a list() object

multiprocessing method mpmap() raises error even for the sample case provided

Hi Andrei ,

For the sample given:

corpus = [
        "MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.",
        "At Google, MapReduce was used to completely regenerate Google's index of the World Wide Web",
        "Conceptually similar approaches have been very well known since 1995 with the Message Passing Interface standard having reduce and scatter operations."]

    def reduceMaps(m1, m2):
        for k, v in m2.iteritems():
            m1[k] = m1.get(k, 0) + v
        return m1

    word_counts = stream(corpus). \
        mpmap(lambda line: stream(line.lower().split(' ')).countByValue()). \
        reduce(reduceMaps)

The processing results in following error:
Traceback (most recent call last):
File "C:\Users\ses\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 537, in _handle_tasks
put(task)
File "C:\Users\ses\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "C:\Users\ses\AppData\Local\Programs\Python\Python39\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function at 0x00000183A90173A0>: attribute lookup on main failed

Similar error occurs with other case like:

result = stream([v for v in entry_list.values()]). \
            mpmap(lambda entry: TwigBasic.split_multiple_yang_structures_in_one_line(entry) if TwigBasic._check_for_twig(entry) else [entry]). \
            flatMap(). \
            enumerate(). \
            map(lambda item: (str(item[0]), item[1])). \
            toMap()

Could you pls have a look on it!
Best regards,
Serge

Add UnicodeReader and UTF8Recoder to read unicode CSV files

csv module in Py2x standard library dails reading unicode csv files. Adding UnicodeReader and UTF8Recorder according to stdlib documentation solves the problem, but inconvenient to replicate in every project/module.

Fix openByExtension to properly open gzip files with 'rt' mode

Use gzip.open function

Create fastmap_mp() multiprocessed

explicit agree on license changing

for changing from GPL to MIT type of licenses, the explicit agree from all contributors, including @asuiu @marian-rusu , must be collected

Implement on sset() union/__and__/__or__/__xor__, etc...

When performing a sset() | sset() it returns a set. We need it return sset so it can be chained further.

Create fastfilter() stream method.

Use already implemented fastmap()

Test Slack