Giter Site home page Giter Site logo

jupyter-xeus / xeus Goto Github PK

View Code? Open in Web Editor NEW
880.0 28.0 84.0 9.5 MB

Implementation of the Jupyter kernel protocol in C++

License: BSD 3-Clause "New" or "Revised" License

C++ 86.06% CMake 13.94%
jupyter-kernels interpreter jupyter c-plus-plus

xeus's People

Contributors

antoineprv avatar bkmgit avatar btel avatar derthorsten avatar fingolfin avatar gouarin avatar hind-m avatar isabelparedes avatar jcfr avatar johanmabille avatar jtpio avatar lassoan avatar lilywangl avatar m-kus avatar marimeireles avatar martinrenou avatar myd7349 avatar pseudofiame avatar qulogic avatar romainfrancois avatar spennihana avatar sylvaincorlay avatar thibaultlacharme avatar wolfv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xeus's Issues

Change cursor_pos type

cursor_pos (and other fields related to cursor positions) should be casted to size_t before calling completion methods.

Build-time dependency on pkg config for downstream

  • zeromq cannot be found with cmake if not built with cmake, so we switched to pkg-config
  • in xeusConfig.cmake, we replaced find_dependency with pkg_check_modulesfor the ZeroMQ dependency.
if (WIN32)
    find_dependency(ZeroMQ 4.2.3)
else()
    find_package(PkgConfig)
    pkg_check_modules(ZeroMQ libzmq>=4.2.3 REQUIRED)
endif()

This causes the packages that depend on xeus to require pkg-config at build time.

Would it be more appropriate to not include that section in the xeusConfig.cmake?

Improve signing scheme mapping from std::string to OpenSSL function pointers

xeus dynamically maps the signing scheme specified in the kernel messages to the corresponding function pointer in OpenSSL.

This is done in function asevp, at https://github.com/QuantStack/xeus/blob/0.23.3/src/xauthentication.cpp#L106-L124.

This function internally makes use of a std::map with std::string keys and const EVP_MD*(*)() (function pointers).

Unfortunately, depending of the build options of OpenSSL, some of these functions may or may not be defined, so we had to comment out some of the supported schemes.

We should investigate if this information can be discovered at runtime.

Overriding CMAKE_CXX_FLAGS_DEBUG may cause link error in MSVC

When I was trying to create a xeus port (microsoft/vcpkg#5351) for vcpkg, I encountered a link error caused by the overriding of CMAKE_CXX_FLAGS_DEBUG.

How to reproduce?

  1. Install vcpkg on Win10, and install all packages needed by xeus via vcpkg:

    vcpkg install cppzmq cryptopp

  2. Clone xeus and try to build it with vcpkg's cmake toolchain:

    cd xeus
    mkdir build
    cd build
    cmake .. -DCMAKE_TOOLCHAIN_FILE=E:/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x86-windows -DCMAKE_GENERATOR_PLATFORM=Win32 -DXEUS_USE_SHARED_CRYPTOPP=OFF -DCMAKE_BUILD_TYPE=Debug
  3. Open the VS solution generated in previous step and build the xeus project (Win32 Debug):

     1>cryptopp-static.lib(cryptlib.cpp.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in xauthentication.obj
     1>cryptopp-static.lib(cryptlib.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MDd_DynamicDebug' doesn't match value 'MD_DynamicRelease' in xauthentication.obj
     1>cryptopp-static.lib(misc.cpp.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in xauthentication.obj
     1>cryptopp-static.lib(misc.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MDd_DynamicDebug' doesn't match value 'MD_DynamicRelease' in xauthentication.obj
    ......
    

If I change the Runtime Library from Multi-threaded DLL (/MD) to Multi-threaded Debug DLL (/MDd)(that is not overriding the default CMAKE_CXX_FLAGS_DEBUG) in the xeus project Properties dialog, this link error will disapper.

History implementation

History is independent from the language and should be part of Xeus instead of being delegated to interpreters.

Problems compiling from source due to nlohmann_json linking

When compiling xeus from source on Ubuntu 18.04, I get the following error:

Linking CXX shared library libxeus.so
/usr/bin/x86_64-linux-gnu-ld: cannot find -lnlohmann_json
collect2: error: ld returned 1 exit status
CMakeFiles/xeus.dir/build.make:435: recipe for target 'libxeus.so.1.0.1' failed
make[2]: *** [libxeus.so.1.0.1] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/xeusget.dir/all' failed
make[1]: *** [CMakeFiles/xeus.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

However, nlohmann_json is a header-only library. If I've install nlohmann_json from source, how do I get xeus to locate it?

Failed install with conda

Running conda install -n xeus-cling-env xeus -c QuantStack -c conda-forge gives me an error. Conda downloads everything and then spits out the following:

Preparing transaction: done
Verifying transaction: done
Executing transaction: \ ERROR conda.core.link:_execute_post_link_actions(658): An error occurred while installing package 'QuantStack::gcc-7-7.2.0-2'.
LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>

Attempting to roll back.

failed
ERROR conda.core.link:_execute(568): An error occurred while installing package 'QuantStack::gcc-7-7.2.0-2'.
LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>

Attempting to roll back.

Rolling back transaction: done

LinkError: post-link script failed for package QuantStack::gcc-7-7.2.0-2
running your command again with `-v` will provide additional information
location of failed script: /home/jupyter/.conda/envs/xeus-cling-env/bin/.gcc-7-post-link.sh
==> script messages <==
<None>

Rename build options

Build optoins should start with XEUS to avoid propagating when including xeus as a subproject (or external project)

Link broken on the docs

On this page. The example link links to a 404.

Should it maybe link to xeus-calc?
(I can fix it, just tell me where to link or if I should remove it =) )

xinterpreter should be responsible for using the history manager

Right now, the kernel owns the interpreter and the history manager. If the interpreter owns the history manager and is responsible for using it, it brings more flexibility for xeus users, as they can easily create their own history manager, plug it to the interpreter, and use it the way they want.

Protocol version macro

The version of the Jupyter protocol implemented should probably be set in a macro (and the macro used for the kernel info reply).

Question concerning #68

I was wondering why xpublisher and xheartbeat are kept as pointers in xserver_zmq. This change has been added during @jcfr's PR #68. Is this change useful in SlicerJupyter?

Also, we were wondering why you added non-virtual destructors for those classes on this commit: a569af0. Was your plan to create a custom publisher and a custom heartbeat?

cppzmq deprecated function calls

  • bool zmq::detail::socket_base::send(zmq::message_t&, int) is deprecated: from 4.3.1, use send taking message_t and send_flags [-Wdeprecated-declarations]
  • bool zmq::detail::socket_base::recv(zmq::message_t&, int) is deprecated: from 4.3.1, use recv taking a reference to message_t and recv_flags [-Wdeprecated-declarations]

Please consider using pkg-config for cryptopp

https://github.com/QuantStack/xeus/blob/master/CMakeLists.txt#L58

cppcrypto dropped cmake support. There is crypto-cmake project that breaks: noloader/cryptopp-cmake#45

If the project doesn't officially support cmake, you probably shouldn't insist on expecting it.

In the FreeBSD port, I have this patch:

+FIND_PACKAGE(PkgConfig)
 find_package(nlohmann_json 3.2.0 REQUIRED)
 find_package(xtl 0.5 REQUIRED)
 find_package(cppzmq 4.3.0 REQUIRED)
-find_package(cryptopp REQUIRED)
+pkg_check_modules(cryptopp libcryptopp REQUIRED)

that works fine with cryptocpp project as it is distributed.

Building xeus failed with Visual Studio 2019 due to implicit conversion failed

When I build xeus with Visual Studio 2019, it built failed and I got this error:

xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(28): error C2593: 'operator =' is ambiguous
xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(29): error C2593: 'operator =' is ambiguous
xeus\src\ee70dfa02b-e9875865ca\src\xkernel_configuration.cpp(38): error C2593: 'operator =' is ambiguous

This is source code in xkernel_configuration.cpp:

        std::ifstream ifs(file_name);

        nl::json doc;
        ifs >> doc;

        xconfiguration res;
        res.m_transport = doc["transport"];
        res.m_ip = doc["ip"];
        res.m_control_port = std::to_string(doc["control_port"].get<int>());
        res.m_shell_port = std::to_string(doc["shell_port"].get<int>());
        res.m_stdin_port = std::to_string(doc["stdin_port"].get<int>());
        res.m_iopub_port = std::to_string(doc["iopub_port"].get<int>());
        res.m_hb_port = std::to_string(doc["hb_port"].get<int>());
        res.m_signature_scheme = doc.value("signature_scheme", "");
        if (res.m_signature_scheme != "")
        {
            res.m_key = doc["key"];
        }
        else
        {
            res.m_key = "";
        }

The type of res.m_transport is string. When converting nl::json doc to string, it failed. I try to modify doc["transport"] to doc["transport"].get<std::string>(), it build successfully. But I don't know this change is correct or not. Who can help me to confirm it?

Thanks.

Determinist shutdown

The PUS/SUB sockets used by the main thread to stop the heartbeat and publisher threads should be replaced with pairs of synchronous sockets so we are sure that these threads have sent their remaining messages and are ready for shutdown.

Fails to build with cryptopp-7.0.0

This failure appeared after cryptopp was upgraded to 7.0.0.

/wrkdirs/usr/ports/devel/xeus/work/xeus-0.13.0/src/xauthentication.cpp:31:43: error: use of undeclared identifier 'byte'
        using signature_type = std::array<byte, hmac_type::DIGESTSIZE>;
                                          ^

It builds fine with cryptopp-5.6.5.

Support integration of xeus in event-loop

The current implementation of xserver implements the polling using a while loop and a call to zmq::poll without any timeout.

Keeping the current server/server_impl layout, here are possible approaches:

Approach 1: Allow xserver_impl to be derived in application code

To integrate with event-loop based application (e.g a Qt application), a possible approach is to let the application drives the polling with a short timeout (e.g 10ms)

To support this use case, the most straightforward would be to:

  • update xkernel to support non-blocking implementation of kernel start() function: this can be done by keeping track of kernel_id, session_id, server, core and interpreter as ivars

  • refactor xserver_impl::start_impl introducing xserver_impl::poll_impl and xserver::poll(long timeout)

  • publicly expose xkernel_core.hpp, xpublisher.hpp, xheartbeat.hpp and xserver_impl.hpp

    • they can be moved into the directory include/xeus/private/
    • Note: forward declaring xpublisher and xheartbeat using std::unique_ptr in xserver_impl is not possible because xserver_impl has a virtual destructor and need to be a complete type. Same remark regarding xkernel_core and xkernel

Approach 2: Introduce a xpolling_strategy object

The xpolling_strategy object would be passed to the xkernel constructor.

Cmake support of cryptopp>=6.0 and cryptopp-cmake

When trying to build from source I found problems when running cmake .

CMake Error at CMakeLists.txt:52 (find_package):
  By not providing "Findcryptopp.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "cryptopp",
  but CMake did not find one.

  Could not find a package configuration file provided by "cryptopp" with any
  of the following names:

    cryptoppConfig.cmake
    cryptopp-config.cmake

  Add the installation prefix of "cryptopp" to CMAKE_PREFIX_PATH or set
  "cryptopp_DIR" to a directory containing one of the above files.  If
  "cryptopp" provides a separate development package or SDK, be sure it has
  been installed.
  • I downloaded and compiled and installed (successfully) CryptoPP version 7.0.0
  • I tried with both master and 0.14.11 versions of Xeus

It makes sense, CyrptoPP 7 does not use CMake so I think it cannot find the cmake config or something.

I tried with an older version of CryptoPP (5.6.5), again, build from source, but this time I get a different message:

CMake Error at /opt/cryptopp-CRYPTOPP_5_6_5/cryptopp-config.cmake:1 (include):
  include could not find load file:

    /opt/cryptopp-CRYPTOPP_5_6_5/cryptopp-targets.cmake
Call Stack (most recent call first):
  CMakeLists.txt:52 (find_package)


-- Checking for module 'libzmq>=4.2.3'
--   Found libzmq, version 4.2.5
-- Found LibUUID: /usr/lib/libuuid.so
CMake Error at CMakeLists.txt:156 (get_target_property):
  get_target_property() called with non-existent target "cryptopp-static".


CMake Error at CMakeLists.txt:157 (get_target_property):
  get_target_property() called with non-existent target "cryptopp-static".

interrupt_[request/reply] implementation

We are currently lacking the interupt_[request/reply] message handling on the control channel.

As mentionned by @spennihana, we should probably change the xserver so that the shell socket is polled by a separate thread, and the control socket remains the only polled by the main thread.

The main issue with this design is that the kernel_shutdown message, as per the documentation can be sent on either the control or shell channels.

Ideally we would need to be 100% sure that the kernel shutdown message would only be received on the control socket.

cc @minrk

Support newer OpenSSL

Building with OpenSSL 1.1.1c fails due to undefined variables:

/usr/bin/c++  -DGUID_LIBUUID -DXEUS_EXPORTS -Dxeus_EXPORTS -I/builddir/build/BUILD/xeus-0.23.1/include -isystem /usr/include/pgm-5.2  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC   -Wunused-parameter -Wextra -Wreorder -march=native -std=gnu++14 -o CMakeFiles/xeus.dir/src/xauthentication.cpp.o -c /builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp: In function 'const EVP_MD* xeus::asevp(const string&)':
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp:111:27: error: 'EVP_mdc2' was not declared in this scope; did you mean 'EVP_md2'?
  111 |             {"hmac-mdc2", EVP_mdc2},
      |                           ^~~~~~~~
      |                           EVP_md2
/builddir/build/BUILD/xeus-0.23.1/src/xauthentication.cpp:121:9: error: could not convert '{{"hmac-md5", EVP_md5}, {"hmac-sha1", EVP_sha1}, {"hmac-mdc2", <expression error>}, {"hmac-ripemd160", EVP_ripemd160}, {"hmac-blake2b512", EVP_blake2b512}, {"hmac-blake2s256", EVP_blake2s256}, {"hmac-sha224", EVP_sha224}, {"hmac-sha256", EVP_sha256}, {"hmac-sha384", EVP_sha384}, {"hmac-sha512", EVP_sha512}}' from '<brace-enclosed initializer list>' to 'const std::map<std::__cxx11::basic_string<char>, const evp_md_st* (*)()>'
  121 |         };
      |         ^
      |         |
      |         <brace-enclosed initializer list>

Fedora has some compat package for 1.0.x, but it's orphaned and soon to be deleted.

conda install fails

(base) C:\Users\tomtz>conda install xeus -c QuantStack
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  • xeus
  • cling-patches
  • xeus
  • cryptopp
  • *[track_features=cling]

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

Installation without Conda

Hello!
I'm sorry, I have never used Conda, since I just use pip to manage my software dependencies and environments. So, I'm afraid that installing Conda, when I already have several versions of Python, can cause problems. Is it possible to install Xeus w/o Conda?

For instance, I have already installed Cling and Cling Jupyter Kernel just via pip. But, unfortunately, it doesn't support cell magic :-/

My system is Ubuntu 16.04.

Comm messages and target name

At the moment, we use xcomm::send_comm_message to compose comm messages which adds the target name while it is not required for most comm messages besides comm_open.

xeus::xjson -> nlohmann::json

We should probably not encourage the use of the xeus::xjson alias and move the examples and documentation to use nlohmann::json directly.

(However, we probably need to keep the alias using definition for backward compatibility).

cc @JohanMabille @martinRenou

-DBUILD_TESTS=ON does not work for in-source builds

Building with #184 and -DBUILD_TESTS=ON seems to be broken:

[ 89%] Linking CXX executable test_kernel
cd /builddir/build/BUILD/xeus-0.23.1/test && /usr/bin/cmake -E cmake_link_script CMakeFiles/test_kernel.dir/link.txt --verbose=1
/usr/bin/c++  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG  -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -rdynamic CMakeFiles/test_kernel.dir/test_interpreter.cpp.o CMakeFiles/test_kernel.dir/main.cpp.o  -o test_kernel -Wl,-rpath,/builddir/build/BUILD/xeus-0.23.1 ../libxeus.so.1.0.0 /usr/lib64/libzmq.so /usr/lib64/libcrypto.so /usr/lib64/libuuid.so -lpthread 
make[2]: Leaving directory '/builddir/build/BUILD/xeus-0.23.1'
/usr/bin/ld: cannot open output file test_kernel: Is a directory
collect2: error: ld returned 1 exit status

It seems to be trying to create a test executable at an already existing directory.

It also looks like neither Travis nor AppVeyor CI enable this flag.

Don't install Jupyter kernel specs during test build

This is not particularly nice (especially with --sys-prefix), and doesn't work on builders. At the very least, this is something to be done at install, not build (but I understand it's a bit difficult as it's for tests.)

Jupyter supports something like JUPYTER_PATH which could be used to add a kernel spec without installing it system-wide.

Xeus does not build without GTEST

Building xeus master from sources does not work without GTEST

cmake -D CMAKE_INSTALL_PREFIX=$CONDA_PREFIX ..

outputs:

-- xeus version: v0.23.1
-- xeus binary version: v1.0.0
-- Performing Test HAS_LTO_FLAG
-- Performing Test HAS_LTO_FLAG - Success
-- CMAKE_CXX_FLAGS: 
-- CMAKE_CXX_FLAGS: 
-- Forcing tests build type to Release
CMake Error at /home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR
  GTEST_MAIN_LIBRARY)
Call Stack (most recent call first):
  /home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /home/martin/miniconda3/envs/xeus-python/share/cmake-3.15/Modules/FindGTest.cmake:197 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  test/CMakeLists.txt:79 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/martin/github/martinRenou/xeus/build/CMakeFiles/CMakeOutput.log".

Fails to find ZeroMQ

CMake Error at CMakeLists.txt:51 (find_package):
  By not providing "FindZeroMQ.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "ZeroMQ", but
  CMake did not find one.

  Could not find a package configuration file provided by "ZeroMQ" (requested
  version 4.2.3) with any of the following names:

    ZeroMQConfig.cmake
    zeromq-config.cmake

  Add the installation prefix of "ZeroMQ" to CMAKE_PREFIX_PATH or set
  "ZeroMQ_DIR" to a directory containing one of the above files.  If "ZeroMQ"
  provides a separate development package or SDK, be sure it has been
  installed.

ZeroMQ's build doesn't normally install .cmake files.
ZeroMQ project officially recommends to use .pc files:
zeromq/libzmq#3212 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.