jupyter-xeus / xeus Goto Github PK
View Code? Open in Web Editor NEWImplementation of the Jupyter kernel protocol in C++
License: BSD 3-Clause "New" or "Revised" License
Implementation of the Jupyter kernel protocol in C++
License: BSD 3-Clause "New" or "Revised" License
cursor_pos (and other fields related to cursor positions) should be casted to size_t before calling completion methods.
xeusConfig.cmake
, we replaced find_dependency
with pkg_check_modules
for the ZeroMQ
dependency.if (WIN32)
find_dependency(ZeroMQ 4.2.3)
else()
find_package(PkgConfig)
pkg_check_modules(ZeroMQ libzmq>=4.2.3 REQUIRED)
endif()
This causes the packages that depend on xeus to require pkg-config at build time.
Would it be more appropriate to not include that section in the xeusConfig.cmake
?
The current conda package of xeus for linux and macOS is still at 0.3.0 (https://anaconda.org/conda-forge/xeus). Any chance of getting that updated to make installation easier?
xkernel_core
should have two m_parent_header
members now that the shell
and the control
can have dedicated threads.
As per discussion in-person with @gouarin.
Currently, the buffers
member is missing in xmessage
. It is a requirement for the implementation of the COMM protocol.
xeus
dynamically maps the signing scheme specified in the kernel messages to the corresponding function pointer in OpenSSL.
This is done in function asevp
, at https://github.com/QuantStack/xeus/blob/0.23.3/src/xauthentication.cpp#L106-L124.
This function internally makes use of a std::map
with std::string
keys and const EVP_MD*(*)()
(function pointers).
Unfortunately, depending of the build options of OpenSSL, some of these functions may or may not be defined, so we had to comment out some of the supported schemes.
We should investigate if this information can be discovered at runtime.
When I was trying to create a xeus
port (microsoft/vcpkg#5351) for vcpkg
, I encountered a link error caused by the overriding of CMAKE_CXX_FLAGS_DEBUG
.
Install vcpkg
on Win10, and install all packages needed by xeus
via vcpkg
:
vcpkg install cppzmq cryptopp
Clone xeus
and try to build it with vcpkg
's cmake toolchain:
cd xeus
mkdir build
cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=E:/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x86-windows -DCMAKE_GENERATOR_PLATFORM=Win32 -DXEUS_USE_SHARED_CRYPTOPP=OFF -DCMAKE_BUILD_TYPE=Debug
Open the VS solution generated in previous step and build the xeus
project (Win32 Debug):
1>cryptopp-static.lib(cryptlib.cpp.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in xauthentication.obj
1>cryptopp-static.lib(cryptlib.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MDd_DynamicDebug' doesn't match value 'MD_DynamicRelease' in xauthentication.obj
1>cryptopp-static.lib(misc.cpp.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in xauthentication.obj
1>cryptopp-static.lib(misc.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MDd_DynamicDebug' doesn't match value 'MD_DynamicRelease' in xauthentication.obj
......
If I change the Runtime Library
from Multi-threaded DLL (/MD)
to Multi-threaded Debug DLL (/MDd)
(that is not overriding the default CMAKE_CXX_FLAGS_DEBUG
) in the xeus
project Properties dialog, this link error will disapper.
History is independent from the language and should be part of Xeus instead of being delegated to interpreters.
At the moment xeus
defaults to 10 entries for the history when n
is not specified: https://github.com/QuantStack/xeus/blob/871cfd3993b63b725ef98a0a597ce534ac05e387/src/xhistory_manager.cpp#L40
To get the whole history at once, clients can pass a very high number, such as 10000
, assuming the history has less than 10000
entries.
But when that number is bigger than what can fit in an int
, it defaults to 0
. Maybe n
could in that case default to the size of the history?
When compiling xeus from source on Ubuntu 18.04, I get the following error:
Linking CXX shared library libxeus.so
/usr/bin/x86_64-linux-gnu-ld: cannot find -lnlohmann_json
collect2: error: ld returned 1 exit status
CMakeFiles/xeus.dir/build.make:435: recipe for target 'libxeus.so.1.0.1' failed
make[2]: *** [libxeus.so.1.0.1] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/xeusget.dir/all' failed
make[1]: *** [CMakeFiles/xeus.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
However, nlohmann_json is a header-only library. If I've install nlohmann_json from source, how do I get xeus to locate it?
Hi!
Is there any plan to give an option to build xeus
as a static library? The reason is explained here: Microsoft/vcpkg#5351 (comment)
Thanks! Good night. :-)
Running conda install -n xeus-cling-env xeus -c QuantStack -c conda-forge
gives me an error. Conda downloads everything and then spits out the following:
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ ERROR conda.core.link:_execute_post_link_actions(658): An error occurred while installing package 'QuantStack::gcc-7-7.2.0-2'.
LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>
Attempting to roll back.
failed
ERROR conda.core.link:_execute(568): An error occurred while installing package 'QuantStack::gcc-7-7.2.0-2'.
LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>
Attempting to roll back.
Rolling back transaction: done
LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>
Build optoins should start with XEUS
to avoid propagating when including xeus as a subproject (or external project)
One area IPython shines is analysis on static data. Computing, charting and visualisation for real time, streaming, changing data is minimal. This would be an area perhaps that can be improved on.
Also see: http://www.ankhor.com/, https://github.com/MicrosoftResearch/Dryad, https://github.com/MicrosoftResearch/Naiad
We currently have
target_link_libraries(${target_name} PUBLIC LibUUID::LibUUID)
Right now, the kernel
owns the interpreter
and the history manager
. If the interpreter
owns the history manager
and is responsible for using it, it brings more flexibility for xeus users, as they can easily create their own history manager
, plug it to the interpreter
, and use it the way they want.
The version of the Jupyter protocol implemented should probably be set in a macro (and the macro used for the kernel info reply).
Should we be doing the same in xeus?
I was wondering why xpublisher
and xheartbeat
are kept as pointers in xserver_zmq
. This change has been added during @jcfr's PR #68. Is this change useful in SlicerJupyter?
Also, we were wondering why you added non-virtual destructors for those classes on this commit: a569af0. Was your plan to create a custom publisher and a custom heartbeat?
bool zmq::detail::socket_base::send(zmq::message_t&, int)
is deprecated: from 4.3.1, use send
taking message_t
and send_flags
[-Wdeprecated-declarations]bool zmq::detail::socket_base::recv(zmq::message_t&, int)
is deprecated: from 4.3.1, use recv
taking a reference to message_t
and recv_flags
[-Wdeprecated-declarations]ipywidget's Output widget makes use of the private implementation details of ipykernel to redirect outputs, by accessing the private member _parent_header
.
https://github.com/jupyter-widgets/ipywidgets/blob/7.0.0/ipywidgets/widgets/widget_output.py#L50-L55
In xeus-python, we could monkey-patch that function, but our implementation would still require an API in xeus core to access m_parent_header
.
Cryptopp uses makefile. Should a link to https://github.com/noloader/cryptopp-cmake also be added to the readme?
https://github.com/QuantStack/xeus/blob/master/CMakeLists.txt#L58
cppcrypto
dropped cmake
support. There is crypto-cmake
project that breaks: noloader/cryptopp-cmake#45
If the project doesn't officially support cmake
, you probably shouldn't insist on expecting it.
In the FreeBSD port, I have this patch:
+FIND_PACKAGE(PkgConfig)
find_package(nlohmann_json 3.2.0 REQUIRED)
find_package(xtl 0.5 REQUIRED)
find_package(cppzmq 4.3.0 REQUIRED)
-find_package(cryptopp REQUIRED)
+pkg_check_modules(cryptopp libcryptopp REQUIRED)
that works fine with cryptocpp
project as it is distributed.
When I build xeus with Visual Studio 2019, it built failed and I got this error:
xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(28): error C2593: 'operator =' is ambiguous
xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(29): error C2593: 'operator =' is ambiguous
xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(38): error C2593: 'operator =' is ambiguous
This is source code in xkernel_configuration.cpp:
std::ifstream ifs(file_name);
nl::json doc;
ifs >> doc;
xconfiguration res;
res.m_transport = doc["transport"];
res.m_ip = doc["ip"];
res.m_control_port = std::to_string(doc["control_port"].get<int>());
res.m_shell_port = std::to_string(doc["shell_port"].get<int>());
res.m_stdin_port = std::to_string(doc["stdin_port"].get<int>());
res.m_iopub_port = std::to_string(doc["iopub_port"].get<int>());
res.m_hb_port = std::to_string(doc["hb_port"].get<int>());
res.m_signature_scheme = doc.value("signature_scheme", "");
if (res.m_signature_scheme != "")
{
res.m_key = doc["key"];
}
else
{
res.m_key = "";
}
The type of res.m_transport
is string. When converting nl::json doc to string, it failed. I try to modify doc["transport"]
to doc["transport"].get<std::string>()
, it build successfully. But I don't know this change is correct or not. Who can help me to confirm it?
Thanks.
The PUS/SUB sockets used by the main thread to stop the heartbeat and publisher threads should be replaced with pairs of synchronous sockets so we are sure that these threads have sent their remaining messages and are ready for shutdown.
This failure appeared after cryptopp
was upgraded to 7.0.0
.
/wrkdirs/usr/ports/devel/xeus/work/xeus-0.13.0/src/xauthentication.cpp:31:43: error: use of undeclared identifier 'byte'
using signature_type = std::array<byte, hmac_type::DIGESTSIZE>;
^
It builds fine with cryptopp-5.6.5
.
The current implementation of xserver implements the polling using a while loop and a call to zmq::poll
without any timeout.
Keeping the current server/server_impl layout, here are possible approaches:
xserver_impl
to be derived in application codeTo integrate with event-loop based application (e.g a Qt application), a possible approach is to let the application drives the polling with a short timeout (e.g 10ms)
To support this use case, the most straightforward would be to:
update xkernel
to support non-blocking implementation of kernel start()
function: this can be done by keeping track of kernel_id, session_id, server, core and interpreter as ivars
refactor xserver_impl::start_impl
introducing xserver_impl::poll_impl
and xserver::poll(long timeout)
publicly expose xkernel_core.hpp
, xpublisher.hpp
, xheartbeat.hpp
and xserver_impl.hpp
include/xeus/private/
xpublisher
and xheartbeat
using std::unique_ptr
in xserver_impl
is not possible because xserver_impl
has a virtual destructor and need to be a complete type. Same remark regarding xkernel_core
and xkernel
xpolling_strategy
objectThe xpolling_strategy
object would be passed to the xkernel
constructor.
This requires to package more recent versions of cryptopp on quantstack channel.
... instead of explicitly using xpub_message
with make_header("status", user_name, session_id)
.
While it is a bigger dependency, openssl is well packaged for all platforms.
When trying to build from source I found problems when running cmake .
CMake Error at CMakeLists.txt:52 (find_package):
By not providing "Findcryptopp.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "cryptopp",
but CMake did not find one.
Could not find a package configuration file provided by "cryptopp" with any
of the following names:
cryptoppConfig.cmake
cryptopp-config.cmake
Add the installation prefix of "cryptopp" to CMAKE_PREFIX_PATH or set
"cryptopp_DIR" to a directory containing one of the above files. If
"cryptopp" provides a separate development package or SDK, be sure it has
been installed.
master
and 0.14.11
versions of XeusIt makes sense, CyrptoPP 7 does not use CMake so I think it cannot find the cmake config or something.
I tried with an older version of CryptoPP (5.6.5), again, build from source, but this time I get a different message:
CMake Error at /opt/cryptopp-CRYPTOPP_5_6_5/cryptopp-config.cmake:1 (include):
include could not find load file:
/opt/cryptopp-CRYPTOPP_5_6_5/cryptopp-targets.cmake
Call Stack (most recent call first):
CMakeLists.txt:52 (find_package)
-- Checking for module 'libzmq>=4.2.3'
-- Found libzmq, version 4.2.5
-- Found LibUUID: /usr/lib/libuuid.so
CMake Error at CMakeLists.txt:156 (get_target_property):
get_target_property() called with non-existent target "cryptopp-static".
CMake Error at CMakeLists.txt:157 (get_target_property):
get_target_property() called with non-existent target "cryptopp-static".
We are currently lacking the interupt_[request/reply]
message handling on the control channel.
As mentionned by @spennihana, we should probably change the xserver
so that the shell socket is polled by a separate thread, and the control socket remains the only polled by the main thread.
The main issue with this design is that the kernel_shutdown
message, as per the documentation can be sent on either the control or shell channels.
Ideally we would need to be 100% sure that the kernel shutdown message would only be received on the control socket.
cc @minrk
The README.md says minimum version is 4.2.5, but the CMakeLists.txt says 4.2.3.
Move semantics is implemented in cppzmq when this is defined.
Building with OpenSSL 1.1.1c fails due to undefined variables:
/usr/bin/c++ -DGUID_LIBUUID -DXEUS_EXPORTS -Dxeus_EXPORTS -I/builddir/build/BUILD/xeus-0.23.1/include -isystem /usr/include/pgm-5.2 -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -Wunused-parameter -Wextra -Wreorder -march=native -std=gnu++14 -o CMakeFiles/xeus.dir/src/xauthentication.cpp.o -c /builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp: In function 'const EVP_MD* xeus::asevp(const string&)':
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp:111:27: error: 'EVP_mdc2' was not declared in this scope; did you mean 'EVP_md2'?
111 | {"hmac-mdc2", EVP_mdc2},
| ^~~~~~~~
| EVP_md2
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp:121:9: error: could not convert '{{"hmac-md5", EVP_md5}, {"hmac-sha1", EVP_sha1}, {"hmac-mdc2", <expression error>}, {"hmac-ripemd160", EVP_ripemd160}, {"hmac-blake2b512", EVP_blake2b512}, {"hmac-blake2s256", EVP_blake2s256}, {"hmac-sha224", EVP_sha224}, {"hmac-sha256", EVP_sha256}, {"hmac-sha384", EVP_sha384}, {"hmac-sha512", EVP_sha512}}' from '<brace-enclosed initializer list>' to 'const std::map<std::__cxx11::basic_string<char>, const evp_md_st* (*)()>'
121 | };
| ^
| |
| <brace-enclosed initializer list>
Fedora has some compat package for 1.0.x, but it's orphaned and soon to be deleted.
(base) C:\Users\tomtz>conda install xeus -c QuantStack
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
Current channels:
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Hello!
I'm sorry, I have never used Conda, since I just use pip to manage my software dependencies and environments. So, I'm afraid that installing Conda, when I already have several versions of Python, can cause problems. Is it possible to install Xeus w/o Conda?
For instance, I have already installed Cling and Cling Jupyter Kernel just via pip. But, unfortunately, it doesn't support cell magic :-/
My system is Ubuntu 16.04.
At the moment, we use xcomm::send_comm_message
to compose comm messages which adds the target name while it is not required for most comm messages besides comm_open
.
We should probably not encourage the use of the xeus::xjson
alias and move the examples and documentation to use nlohmann::json
directly.
(However, we probably need to keep the alias using
definition for backward compatibility).
The clear_output
message is not supported: https://jupyter-client.readthedocs.io/en/stable/messaging.html#clear-output
Building with #184 and -DBUILD_TESTS=ON
seems to be broken:
[ 89%] Linking CXX executable test_kernel
cd /builddir/build/BUILD/xeus-0.23.1/test && /usr/bin/cmake -E cmake_link_script CMakeFiles/test_kernel.dir/link.txt --verbose=1
/usr/bin/c++ -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -rdynamic CMakeFiles/test_kernel.dir/test_interpreter.cpp.o CMakeFiles/test_kernel.dir/main.cpp.o -o test_kernel -Wl,-rpath,/builddir/build/BUILD/xeus-0.23.1 ../libxeus.so.1.0.0 /usr/lib64/libzmq.so /usr/lib64/libcrypto.so /usr/lib64/libuuid.so -lpthread
make[2]: Leaving directory '/builddir/build/BUILD/xeus-0.23.1'
/usr/bin/ld: cannot open output file test_kernel: Is a directory
collect2: error: ld returned 1 exit status
It seems to be trying to create a test executable at an already existing directory.
It also looks like neither Travis nor AppVeyor CI enable this flag.
This is not particularly nice (especially with --sys-prefix
), and doesn't work on builders. At the very least, this is something to be done at install, not build (but I understand it's a bit difficult as it's for tests.)
Jupyter supports something like JUPYTER_PATH
which could be used to add a kernel spec without installing it system-wide.
Getting the following message from the logs while using xeus-python
:
ERROR: could not deserialize message
Signatures don't match
Which seems to be coming from: https://github.com/QuantStack/xeus/blob/b9b474577362d95e43c83556fa9d6c05b8c3c1fe/src/xkernel_core.cpp#L203-L204
Observing this when running a client side test suite that sends many messages on different channels (control
and shell
)
I suppose it's referring to the HMAC signature?
Building xeus master from sources does not work without GTEST
cmake -D CMAKE_INSTALL_PREFIX=$CONDA_PREFIX ..
outputs:
-- xeus version: v0.23.1
-- xeus binary version: v1.0.0
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- CMAKE_CXX_FLAGS:
-- CMAKE_CXX_FLAGS:
-- Forcing tests build type to Release
CMake Error at /home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR
GTEST_MAIN_LIBRARY)
Call Stack (most recent call first):
/home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindGTest.cmake:197 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
test/CMakeLists.txt:79 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/martin/github/martinRenou/xeus/build/CMakeFiles/CMakeOutput.log".
CMake Error at CMakeLists.txt:51 (find_package):
By not providing "FindZeroMQ.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "ZeroMQ", but
CMake did not find one.
Could not find a package configuration file provided by "ZeroMQ" (requested
version 4.2.3) with any of the following names:
ZeroMQConfig.cmake
zeromq-config.cmake
Add the installation prefix of "ZeroMQ" to CMAKE_PREFIX_PATH or set
"ZeroMQ_DIR" to a directory containing one of the above files. If "ZeroMQ"
provides a separate development package or SDK, be sure it has been
installed.
ZeroMQ's build doesn't normally install .cmake files.
ZeroMQ project officially recommends to use .pc files:
zeromq/libzmq#3212 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.