specklesystems/specklepy

Python SDK 🐍

Speckle | specklepy 🐍

The Python SDK

Speckle is the data infrastructure for the AEC industry.

About Speckle

What is Speckle? Check our

Features

• Object-based: say goodbye to files! Speckle is the first object based platform for the AEC industry
• Version control: Speckle is the Git & Hub for geometry and BIM data
• Collaboration: share your designs collaborate with others
• 3D Viewer: see your CAD and BIM models online, share and embed them anywhere
• Interoperability: get your CAD and BIM models into other software without exporting or importing
• Real time: get real time updates and notifications and changes
• GraphQL API: get what you need anywhere you want it
• Webhooks: the base for a automation and next-gen pipelines
• Built for developers: we are building Speckle with developers in mind and got tools for every stack
• Built for the AEC industry: Speckle connectors are plugins for the most common software used in the industry such as Revit, Rhino, Grasshopper, AutoCAD, Civil 3D, Excel, Unreal Engine, Unity, QGIS, Blender and more!

Try Speckle now!

Give Speckle a try in no time by:

• ⇒ creating an account at our public server
• ⇒ deploying an instance in 1 click

Resources

• for help, feature requests or just to hang with other speckle enthusiasts, check out our community forum!
• our tutorials portal is full of resources to get you started using Speckle
• reference on almost any end-user and developer functionality

Repo structure

Usage

Send and receive data from a Speckle Server with operations, interact with the Speckle API with the SpeckleClient, create and extend your own custom Speckle Objects with Base, and more!

Head to the 📚 specklepy docs for more information and usage examples.

Developing & Debugging

Installation

This project uses python-poetry for dependency management, make sure you follow the official docs to get poetry.

To bootstrap the project environment run $ poetry install. This will create a new virtual-env for the project and install both the package and dev dependencies.

If this is your first time using poetry and you're used to creating your venvs within the project directory, run poetry config virtualenvs.in-project true to configure poetry to do the same.

To execute any python script run $ poetry run python my_script.py

Alternatively you may roll your own virtual-env with either venv, virtualenv, pyenv-virtualenv etc. Poetry will play along an recognize if it is invoked from inside a virtual environment.

Style guide

All our repo wide styling linting and other rules are checked and enforced by pre-commit, which is included in the dev dependencies. It is recommended to set up pre-commit after installing the dependencies by running $ pre-commit install. Commiting code that doesn't adhere to the given rules, will fail the checks in our CI system.

Local Data Paths

It may be helpful to know where the local accounts and object cache dbs are stored. Depending on on your OS, you can find the dbs at:

• Windows: APPDATA or <USER>\AppData\Roaming\Speckle
• Linux: $XDG_DATA_HOME or by default ~/.local/share/Speckle
• Mac: ~/.config/Speckle

Contributing

Please make sure you read the contribution guidelines and code of conduct for an overview of the practices we try to follow.

Community

The Speckle Community hangs out on the forum, do join and introduce yourself & feel free to ask us questions!

Security

For any security vulnerabilities or concerns, please contact us directly at security[at]speckle.systems.

License

Unless otherwise described, the code in this repository is licensed under the Apache-2.0 License. Please note that some modules, extensions or code herein might be otherwise licensed. This is indicated either in the root of the containing folder under a different license file, or in the respective file's header. If you have any questions, don't hesitate to get in touch with us via email.

specklepy's Issues

Recursion exception when receiving object via transport

Executing this:

from speckle.api.client import SpeckleClient
from speckle.api.credentials import get_default_account, get_local_accounts
from speckle.api import operations
from speckle.transports.server import ServerTransport

def main():
    account = get_default_account()

    if not account:
        raise Exception("No accounts available!")

    client = SpeckleClient(host=account.serverInfo.url, use_ssl=True)
    client.authenticate(account.token)

    streams = client.stream.list()
    stream = client.stream.get(id=streams[0].id)

    commit = stream.branches.items[0].commits.items[0]

    transport = ServerTransport(client, stream.id)
    res = operations.receive(commit.referencedObject, transport)
    
main()

Expected vs. Actual Behavior

Expecting to receive res object. Instead it goes into a loop:

Traceback (most recent call last):
  File "C:\git\SpeckleBlender\SpeckleBlender-2.0-test01.blend\Text.001", line 23, in <module>
  File "C:\git\SpeckleBlender\SpeckleBlender-2.0-test01.blend\Text.001", line 21, in main
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\api\operations.py", line 69, in receive
    return serializer.read_json(obj_string=obj_string)
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 218, in read_json
    return self.recompose_base(obj=obj)
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 276, in recompose_base
    base.__setattr__(prop, self.handle_value(value))
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 297, in handle_value
    obj_list = [self.handle_value(o) for o in obj]
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 297, in <listcomp>
    obj_list = [self.handle_value(o) for o in obj]
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 311, in handle_value
    return self.recompose_base(obj=obj)
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\serialization\base_object_serializer.py", line 276, in recompose_base
    base.__setattr__(prop, self.handle_value(value))
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\objects\geometry.py", line 352, in __setattr__
    super().__setattr__(name, value)
  File "\\TAS-03\assets\3d modelling\Plug-ins\Blender\modules\speckle\objects\base.py", line 96, in __setattr__
    super().__setattr__(name, value)
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 296, in __setattr__
    value, error_ = known_field.validate(value, self.dict(exclude={name}), loc=name, cls=self.__class__)

...

  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 675, in _iter
    exclude_none=exclude_none,
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 582, in _get_value
    exclude_none=exclude_none,
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 343, in dict
    exclude_none=exclude_none,
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 333, in <dictcomp>
    return {
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 675, in _iter
    exclude_none=exclude_none,
  File "C:\Users\tomsv\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\modules\pydantic\main.py", line 574, in _get_value
    if isinstance(v, BaseModel):
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\abc.py", line 139, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded while calling a Python object
Error: Python script failed, check the message in the system console

Reproduction Steps & System Config (win, osx, web, etc.)

Example code above. Running on Windows 10.

Proposed Solution (if any)

Some small examples of querying available streams and receiving all objects from a stream would help to compare code.

Running:

res = client.object.get(stream_id=stream.id, object_id=commit.referencedObject)

works:

Base(id: a8f62919a00d274391d59abf44931457, speckle_type: Base, totalChildrenCount: 13)

However it is not entirely clear how to get a list of children objects from this result...

Improve error reporting from the Server Transport Batch Sender

v unclear error if something goes wrong w the send on server side

🧪 Use docker image to spin up server for tests

Due to server complications, tests using the server all fail. This change will make local testing more difficult, but make testing more robust in general.

Same as specklesystems/speckle-sharp#374

🛑 Cancellation tokens

Cancellation tokens should be added to long-running operations and checked often to exit the operation gracefully.

⚠ Catch those damn JSON decode errors before they happen

Expected vs. Actual Behavior

server response needs to be checked in operations, not just the client

eg: from the forum

⚠ Validation of Dynamic `Base` Attribute Names

implement attr name validation as per specklesystems/speckle-sharp#37

🖌 Refactor Out `AbstractTransport`

The AbstractTransport was initially imagined with some useful shared functionality for all transports, but as the Transport system evolved this no longer seemed necessary. At this point, the AbstractTransport should probably get canned unless there are any suggestions for functionality that would be helpful to future transport authors.

🌯 Stream Wrapper for ez scripting and adding streams from URL

Async specklepy API

ServerTransport uploading is inefficient for many small objects

The server transport currently makes a server request for every object that needs to be sent.
If we have many small objects, it can get very slow.

I think we should batch the objects before sending

Ci: update package version based on git tag before publishing

so I don't need to remember to update the pyproject.toml every time

Read Receipts: Core API Call

Same as specklesystems/speckle-sharp#782 but in python ;)

Also related to the checklist in specklesystems/speckle-sharp#783, as this blocks the Blender and QGIS read receipts implementation.

🏛 Registering Custom `SpeckleBase` Derived Classes

Currently, only the /speckle/objects/ dir is searched for Speckle classes. This was for ease of development and presuming we would have a large number of generated Speckle Objects from speckle-sharp by now. As the Objects generation has been temporarily parked, there is a more pressing need to allow users to register their own Speckle Objects to be picked up by the parser.

Implementation for this is up for discussion. @gjedlicska has proposed a solution involving metaclass magic, though this may be blocked by our current use of the pydantic base. Pydantic was chosen for the code generation niceties, but as this is not being utilised at the moment we are considering dropping it and finding a different solution for code generation when we get to that.

🧪 ❌ Some tests are no longer passing on CI or locally

Expected vs. Actual Behavior

Some tests are failing on CI and locally too. See CI log for more details: https://app.circleci.com/pipelines/github/specklesystems/speckle-py/127/workflows/adc4e3d8-6c81-48a1-b3b4-b391295367b6/jobs/440

Reproduction Steps & System Config (win, osx, web, etc.)

Run your tests and see them 💥 . Any OS will do, tested in Mac/Windows and CI is Linux.

Proposed Solution (if any)

I have absolutely no idea what's going on, I'll leave it to @izzylys' magic.

💾 Speckle data path on MacOS

The SQLite transport is pointing to the wrong location on MacOS. The default path for the Objects and Accounts db should be ~/.config/Speckle. Will be a quick fix!

Don't use blocking IO for send and receive

Expected vs. Actual Behavior

Server transport uses synchronous blocking requests.

Proposed Solution (if any)

Other parts of the app uses aiohttp, replace the server transport requests call with aiohttp.

📦 Add some basic objects classes to enable simple geometry sending

Code generation from .NET is on hold, but some of the basic classes from sharp/objects would be great to bring in manually to allow people to play around with creating and sending geometry to the viewer.

🌀 Concurrency for serialisation and transports

Transports should not be blocking in order for decomposition / serialisation to happen concurrently with saving objects using the transports. The serialiser working its way through a root object with loads of child objects should be able to chug along with the decomposition and serialisation and chuck finished objects into a queue for the transport to pick up and send as they become available. However, we don't want to complicate usage for the end user meaning this magic should be somehow contained. How do we want to do this?

🥒 Bug with brep receiving (curve encoding)

Expected vs. Actual Behavior

Something is going wrong in deserialisation - 30 is not a valid CurveTypeEncoding but it's received fine in other connectors

Reproduction Steps & System Config (win, osx, web, etc.)

receive this object:
https://latest.speckle.dev/streams/24c3741255/objects/cf0bc147f341381c8f7bce854ad1b063

not super high prio at the moment as I think @AntoineDao is the only person I know of who is using breps in py 😜 (blender just receives and converts their display mesh)

Including `specklepy.api.operations` breaks AWS Lambda

Hi guys,

We come across a really quirky issue with this library and how it is deployed. On a scale from 1-10 I would rate it as 2 when it comes to how important it is, but I thought I would let you guys know.

Expected vs. Actual Behavior

When including specklepy.api.operations the following stack trace is thrown:

  File "/var/lang/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/datadog_lambda/handler.py", line 30, in <module>
    handler_module = import_module(modified_mod_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/import_lambdas/exchange2gltf.py", line 6, in <module>
    from .transform.speckle import get_base_from_raw_transport, get_trimesh_scene_from_base
  File "/var/task/import_lambdas/transform/speckle.py", line 3, in <module>
    from specklepy.api import operations
  File "/var/task/specklepy/api/operations.py", line 4, in <module>
    from specklepy.transports.sqlite import SQLiteTransport
  File "/var/task/specklepy/transports/sqlite.py", line 14, in <module>
    class SQLiteTransport(AbstractTransport):
  File "/var/task/specklepy/transports/sqlite.py", line 22, in SQLiteTransport
    __queue: Queue = Queue()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

This only happens within an AWS lambda. Further information here.

Expected behaviour is that this should not crash unless using SQLite transport explicitly.

Reproduction Steps & System Config (win, osx, web, etc.)

Deploy and run the following file in an AWS lambda:

from specklepy.api.operations import receive

Proposed Solution (if any)

There are several ways to solve this. One way could be to try to not initialize the SQLIte libraries before the transport is created. Another way could be to do a best effort to load SQLIte, but if it crashes the error is ignored.

Dream solution I think is to add sqlite as an pypi extra.

Let me know what you think! :D

Please upgrade the ujson dependency, which has a CVE

There was recently posted a CVE for ujson >= 4.0.2, < 5.1.0 for an out-of-bounds write: GHSA-fh56-85cw-5pq6

This is fixed in ujson 5.1.0, but since specklepy depends on ujson ^4.3.0, that isn't used. Could you please upgrade ujson to version 5.x?

The only breaking change in ujson 5.x is that they dropped support for Python 3.6, which was EOL a month ago.

Key Error in speckle SDK docs

Expected vs. Actual Behavior

I think found an error in the Speckle Python SDK docs here: https://speckle.guide/dev/py-examples.html
on line 16 there seems to be an invalid key named "obj_id"

The actual key in the specklepy is the following:

specklepy/specklepy/api/resources/commit.py

Line 102 in af50afe

object_id: str,

Reproduction Steps & System Config (win, osx, web, etc.)

I am running the code on a windows machine, in a venv. If you were to do the same and copy paste the code and run it, you would get this error:

    commid_id = myClient.commit.create(
TypeError: create() got an unexpected keyword argument 'obj_id'

Proposed Solution (if any)

Change the docs to reflect the correct key name for the method sited above

Optional: Affected Projects

Anybody such as myself who is using your awesome python sdk 😄

Fail to active SpeckleBlender 2.0 - path process error

Expected vs. Actual Behavior

Expected to be able to successfully activate SpeckleBlender 2.0 in Blender Add-ons UI but I get an error.

Reproduction Steps & System Config (win, osx, web, etc.)

OS: Manjaro Linux
python: 3.9.5

1. Install specklepy from this repo in python site-packages
2. symlink bpy_speckle from speckle-blender repo in ~/.config/blender/2.92/scripts/addons/bpy_speckle/
3. Activate SpeckleBlender 2.0 in Blender Add-ons

Error message:

Traceback (most recent call last):
  File "/usr/lib/python3.9/pkgutil.py", line 416, in get_importer
    importer = sys.path_importer_cache[path_item]
KeyError: PosixPath('/usr/lib/python3.9/site-packages/specklepy/api/resources')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/blender/2.92/scripts/modules/addon_utils.py", line 351, in enable
    mod = __import__(module_name)
  File "/home/cyril/.config/blender/2.92/scripts/addons/bpy_speckle/__init__.py", line 56, in <module>
    from specklepy.api.client import SpeckleClient  # , SpeckleCache
  File "/usr/lib/python3.9/site-packages/specklepy/api/client.py", line 6, in <module>
    from specklepy.api import resources
  File "/usr/lib/python3.9/site-packages/specklepy/api/resources/__init__.py", line 8, in <module>
    for (_, name, _) in pkgutil.iter_modules([Path(__file__).parent]):
  File "/usr/lib/python3.9/pkgutil.py", line 130, in iter_modules
    for i in importers:
  File "/usr/lib/python3.9/pkgutil.py", line 420, in get_importer
    importer = path_hook(path_item)
  File "<frozen importlib._bootstrap_external>", line 1601, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 1476, in __init__
  File "<frozen importlib._bootstrap_external>", line 177, in _path_isabs
AttributeError: 'PosixPath' object has no attribute 'startswith'

Proposed Solution (if any)

Except there is a better way path apparently needs to be converted to a string.

Optional: Affected Projects

• speckle-py
• speckle-blender

Stream Wrapper Issues w Nested Branches

Expected vs. Actual Behavior

Oopsie, the stream wrapper expects a certain no. of segments after splitting by /! Need to amend this to handle nested branches.

🛰 Update to new telemetry

🔍 Mysterious decode errors when receiving revit test stream

The naughty streams:
https://speckle.xyz/streams/fb53c10a53/commits/e10a43c1ee
https://latest.speckle.dev/streams/c544db35f5/commits/30943d0e48
The error:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files (x86)\Python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\izzy lyseggen\.vscode\extensions\ms-python.python-2021.5.840043038\pythonFiles\lib\python\debugpy\__main__.py", line 45, in <module>
    cli.main()
  File "c:\Users\izzy lyseggen\.vscode\extensions\ms-python.python-2021.5.840043038\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 444, in main
    run()
  File "c:\Users\izzy lyseggen\.vscode\extensions\ms-python.python-2021.5.840043038\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))  
  File "C:\Program Files (x86)\Python38-32\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Program Files (x86)\Python38-32\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Program Files (x86)\Python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\izzy lyseggen\Documents\dev\next\pyspeckle\scratch.py", line 755, in <module>
    main()
  File "c:\Users\izzy lyseggen\Documents\dev\next\pyspeckle\scratch.py", line 714, in main
    receive_debugging()
  File "c:\Users\izzy lyseggen\Documents\dev\next\pyspeckle\scratch.py", line 697, in receive_debugging        
    base = operations.receive(commit.referencedObject, transport)
  File "c:\Users\izzy lyseggen\Documents\dev\next\pyspeckle\specklepy\api\operations.py", line 76, in receive  
    obj_string = remote_transport.copy_object_and_children(
  File "c:\Users\izzy lyseggen\Documents\dev\next\pyspeckle\specklepy\transports\server\server.py", line 76, in copy_object_and_children
    root_obj = json.loads(root_obj_serialized)
  File "C:\Program Files (x86)\Python38-32\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files (x86)\Python38-32\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files (x86)\Python38-32\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting 
value", s, err.value) from None      
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)  

Commit with branchName=None

Expected vs. Actual Behavior

When a commit is fetched using client.commit.get(), it gets assigned a proper value for the branchName attribute. However, when it is fetched using client.commit.list(), branchName is only set to the default value None.

Reproduction Steps & System Config

specklepy v2.4.2

1. Create commits in one or more branches on a stream.
2. Fetch the commits on the stream, e.g. commits = client.commit.list(stream_id=stream_id).
3. Print the branchName attribute, e.g. for commit in commits: print(commit.branchName)

Proposed Solution

The branchName attribute is missing in the gql query in client.commit.list().

I have added it and tested it locally, and am happy to create a PR.

📚 Exception when receiving after cancelling a receive first

If the parent object is in the local db, we assume the children are all there as well. After cancelling a receive, it seems we are now getting an exception. Not sure why it wasn't happening before?? The top level parent is not saved until the very end so this should be a safe assumption??? I'm confused and it's friday afternoon. More investigation needed.

🧪 Tests!

Already in progress - needs some more additions and tidying up

📢 Automated releases to PyPI

⚡ Path process error rears it's head again (on mac this time!)

Same issue as #96 - this time on mac using the same py 3.9.5

Not sure why #93 didn't fix it for this case. Will need to investigate further.

thanks @AlanRynne for reporting. poke him if needed to replicate / test fix hehe

Modern python project and dependency management

Expected vs. Actual Behavior

The project dependecies are now defined in the standard requirements.txt file. This is completely functional, but it lacks some niceties. Ie easy separate dev dependecy definitions, reproducible builds etc.

Proposed Solution (if any)

Since PEP517 and 518 the new project definition standard is going to be using a pyproject.toml file. There are a few tools that already implement a solution utilizing the new standard.

I've had great success with Poetry.
I propose moving the project dependency management to poetry.

Manually add an account on GNU/Linux

From docs I see that there is apparently no SpeckleManager for GNU/Linux systems. Although I see also that it is possible to add an account manually.

Can you please point me out where in the code it looks for this .json file to see where I need to put it or if code needs to be modified ?

🐘 Enforced Type Checking

Currently, most stuff is type hinted. This is nice and helpful, but in many situations it would be better to more strictly enforce this.

Enforced type checking with mypy would be a good addition to the project.

👻 Support sending / receiving `null` property values

align and coincide with the same change in sharp

specklesystems/speckle-sharp#938

🏬 Flesh Out SQLite Transport

The SQLite transport doesn't have all the functionality it needs! Finish implementing this so it can work as the default local transport for operations.

Error details not surfaced on server validation

Expected vs. Actual Behavior

Seen here: https://speckle.community/t/is-speckle-xyz-accessible-from-specklepy/2190

When a SpeckleClient is created with an unreachable / invalid speckle server details, the details of what the error was is not surfaced to the user (my bad 😛 ).

The exception thrown is a generic Couldn't get ServerInfo message, but it should contain the actual details

Reproduction Steps & System Config (win, osx, web, etc.)

client = SpeckleClient(host="invalid-server.example.com", use_ssl=True)

Proposed Solution (if any)

If an exception is returned, we should throw that instead of the generic message

Optional: Affected Projects

🧨 Dynamic prop chunking

Same as specklesystems/speckle-sharp#170 but for py

🕸 Handle host input to make sure it's in the correct format

At the moment, there are no checks on the client host input. There should be some handling to make sure it is in the expected format.

🐘 Deserialisation: fall back on `json` when `ujson` throws int `ValueError`

Ints that are too long fail to deserialise w ujson 🤡 (ultrajson/ultrajson#49)

Very annoying, but somewhat of an edgecase and I don't think it's worth reverting to json given the speed improvements ujson provided. I think the best way to go is to just try except this and fall back on json on ValueError

Closes specklesystems/speckle-blender#74

👩‍💻 Dev Enhancements: Code Style, Linting, Commit Format, etc

It has been brought up that some enforcement of code style, linting, and commit formatting would be a nice enhancement to this project. While this is a low priority at the moment, it is totally something I agree with eventually implementing.

I am reluctant to be very strict about this to avoid too much overhead and complexity for new contributors to the project, but some general guidelines would be fab and are open for discussion.

On code style, I am fiercely loyal to black as you can tell from the current codebase and you'll have to put up a very strong argument for me to waver on this. On the rest, it's pretty flexible.

Commit formatting follows commitizen somewhat loosely as do the rest of the specklesystems projects. Linting is up in the air - we are currently using pylint for no better reason than that I am used to it so this can totally be changed.

Base model deserializer fails for mandatory __init__ arguments

Expected vs. Actual Behavior

This is a new issue that arises after solving #50 via #58.

Given a model below that subclasses the speckle Base type, it can only be deserialized by the BaseObjectSerializer

from speckle.objects.base import Base
from speckle.api import operations

class FakeBase(Base):
    """Just a test class type."""
    foo: str

if __name__ == "__main__":
    serialized = operations.serialize(fake_model)
    # this fails to deserialize the mandatory arg `foo`
   # because BaseObjectSerializer.recompose_base() assumes to initialize a new object without arguments.
    deserialized = operations.deserialize(serialized)

    assert fake_model.get_id() == deserialized.get_id()

Proposed Solution (if any)

Not really sure about a short term fix, but when loading objects with json.loads() the object hook method could be used to directly parse into the type. I'm not 100% on all the recompose step implications yet.

Align StreamWrapper.ValidateWithAccount behaviour with speckle-sharp

Expected vs. Actual Behavior

Not a problem! We just added a check in the validation process to ensure the account and the url are from the same server before proceeding to fetch the stream to ensure account has access.

If the account was not from the same server, it made no sense to even try to fetch, so now we output a more appropriate exception.

Proposed Solution (if any)

Do the same as here -> specklesystems/speckle-sharp#899

MemoryTransport mutable default value

Expected vs. Actual Behavior

Expected: Two separate instances of a MemoryTransport contain separate data.
Actual: The dict mutable default value is leaking value between the two instances.

Describe the problem here.

In python the default values are evaluated in parsing time. This results that mutable default values (that are created as reference types in the background, not copy types) are only created once on a single memory address. If a function has a mutable default value, ie. empty list. It is created once if that list is mutated, this can result in unexpected behaviour.

Reproduction Steps & System Config (win, osx, web, etc.)

Just run this script

from pydantic.dataclasses import dataclass
from abc import ABC
from typing import Dict

@dataclass
class AbsDefaultValue(ABC):
    """Just an abstract base."""

class WrongDefaultValue(AbsDefaultValue):
    mutable_default: Dict[str,str] = {}

if __name__ == "__main__":
    
    a = WrongDefaultValue()
    b = WrongDefaultValue()

    a.mutable_default["foo"] = "bar"

    print(a.mutable_default)
    print(b.mutable_default)

Proposed Solution (if any)

Pydantic actually protects from this behaviour with its internal check. So the first fix is, to add @DataClass decorator to the MemoryTransport class. In fact each class that is intended to be a dataclass must be decorated as a dataclass.

By changing the code to the proper behaviour, raises a meaningful exception.

from pydantic.dataclasses import dataclass
from abc import ABC
from typing import Dict

@dataclass
class AbsDefaultValue(ABC):
    """Just an abstract base."""

@dataclass
class WrongDefaultValue(AbsDefaultValue):
    mutable_default: Dict[str,str] = {}

This also affects the ServerTransport implementation with possibly a much greater impact, since there, the workers are shared across Transport instances.

I guess currently multiple transport are not initiated, so this hasn't come up yet, but it will cause serious bugs in the long run.

The fix is fortunately quite easy. You have to explicitly pass in a default factory callable that will be called by pydantic in the post init hook.

import dataclasses
from pydantic.dataclasses import dataclass
from pydantic import Field
from abc import ABC
from typing import Dict, List

@dataclass
class AbsDefaultValue(ABC):
    """Just an abstract base."""

@dataclass
class WrongDefaultValue(AbsDefaultValue):
    mutable_default: Dict[str, str] = dataclasses.field(default_factory=dict)

if __name__ == "__main__":

    print(dict())
    
    a = WrongDefaultValue()
    b = WrongDefaultValue()

    a.mutable_default["foo"] = "bar"
    b.mutable_default["foo"] = "not so bar"

    print(a.mutable_default)
    print(b.mutable_default)

http://speckle.xyz is not a compatible Speckle Server

Expected vs. Actual Behavior

I am trying to simply login to the demo speckle server to try some object serializations, but get this error:

Traceback (most recent call last):
  File "/Users/max/Documents/modugen/speckle-py/example/using_speckle.py", line 7, in <module>
    client = SpeckleClient(host=account.serverInfo.url, use_ssl=False)
  File "/Users/max/Documents/modugen/speckle-py/specklepy/api/client.py", line 56, in __init__
    raise SpeckleException(f"{self.url} is not a compatible Speckle Server", ex)
specklepy.logging.exceptions.SpeckleException: SpeckleException: http://speckle.xyz is not a compatible Speckle Server

I've tried manually providing the Token I've created in the web app and also tried the Speckle Manager for Mac.
With the Manager approach, I can retrieve the Account from the SQLTransport but using this code still get the error:

from specklepy.api.client import SpeckleClient
from specklepy.api.credentials import get_default_account

account = get_default_account()
print(account)
client = SpeckleClient(host=account.serverInfo.url, use_ssl=False)
client.authenticate(account.token)

print(stream_list = client.stream.list())

Does the python client work with the beta server or do I have to run the server locally?

Optional: Affected Projects

Does this issue propagate to other dependencies or dependents? If so, list them here!

BrepLoopType and other Brep subclasses are strings instead of Enum types

In C#, currently all Brep subclasses like BrepLoop and BrepTrim have a type property that is an enum. These are serialised as integers in the C# SDK.

The current object model in Python deserialises them as str, which doesn't cause an issue when receiving those Breps, but it prevents them from being sent from Python and received back in other C# connectors.

Pinging @AntoineDao as he may be interested in commenting about this. Changing this would break any Brep implementations that assume the type to be str.

Here's the type in the BrepLoop in C#:
https://github.com/specklesystems/speckle-sharp/blob/e08e51db07d532e7553d0904a43c408760a910b2/Objects/Objects/Geometry/Brep.cs#L310

and the BrepLoopType implementation:
https://github.com/specklesystems/speckle-sharp/blob/e08e51db07d532e7553d0904a43c408760a910b2/Objects/Objects/Geometry/Brep.cs#L412-L420

Here's the type of the BrepLoop

specklepy/specklepy/objects/geometry.py

Line 510 in f494916

Type: str = None

It looks like BrepTrimType does exist as an enum but it's not being used in the class:

specklepy/specklepy/objects/geometry.py

Line 542 in f494916

TrimType: str = None

The current error we're getting is Cannot deserialise string to BrepLoopType.

To reproduce:

1. send a Brep from Rhino/GH.
2. receive that brep in python
3. send it back to speckle
4. try to receive it in Rhino/GH.

Add test coverage report

Running the test suite should generate coverage report locally and in CI

Pytest has a plugin that enables coverage reports. It can at the end export the report in multiple formats. That should be hooked up to CircleCI / Codecov.

I can get the former done, the infra part is out of reach for me.

🧪 Test refactoring

split up the tests a bit more as they got a bit messy!

📰 Publish to pypi

so we can reach more users!

also so cristi can use for testing

🤙 Client GQL Operations Parity w/ Core

The client implements most, but not all of the queries and mutations. Finish this up!