araq/malebolgia

Malebolgia creates new spawns. Structured concurrency. Thread pools and related APIs.

malebolgia's Issues

needs docs

Control of non-blocking threads on the background (start and cancel)

Any ideas to add support for spawning of children threads with non-blocking main thread?
Currently implementation reminds me async, which blocks main thread with awaitAll.

With using threadpool I could send message to child thread and break out of the endless loop using Channels.
For example, recording audio/video stream or endlessly perform periodical checks on the background.

I've managed to spawn thread without awaitAll but I can't stop it with abort() or panicStop() apparently because of joinThreads.

needs CI

Error: type mismatch

Using the first demo from the readme and compile with nim c -d:ThreadPoolSize=8 -d:FixedChanSize=16 dfs.nim results in:

.nimble\pkgs2\malebolgia-0.1.0-532e5b660dac9c98355f51d6704962fa326d77fb\malebolgia.nim(138, 15) Error: type mismatch
Expression: invoke(item.t, item.result)
  [1] item.t: Task
  [2] item.result: pointer

Expected one of (first mismatch at [position]):
[2] proc invoke(task: Task)

Win11, Nim1.9.3, Nimble install malebolgia and Nimble install malebolgia@#head

Blocking main thread can cause low thread utilization

When the main thread takes longer than most worker threads, it leads to low thread utilization, which is unfortunate.

import malebolgia

import std/[os, atomics]

var
    threadpool = createMaster()
    busyThreads: Atomic[int]
busyThreads.store 0

let mainThreadId = getThreadId()

proc findStartPositionsAndPlay() =
    atomicInc busyThreads
    echo "Started. Busy threads: ", busyThreads.load

    sleep(1000)
    if getThreadId() == mainThreadId:
        sleep(10_000)

    atomicDec busyThreads
    echo "Finished. Busy threads: ", busyThreads.load

threadpool.awaitAll:
    for i in 1..10000:
        threadpool.spawn findStartPositionsAndPlay()

ThreadSanitizer Error when running tests

This error pops up when running the tests with threadsanitizer. command: nim c --cc:clang -d:useMalloc -t:"-fsanitize=thread" -l:"-fsanitize=thread" -d:nosignalhandler -d:release -g test1

LLVMSymbolizer: error reading file: No such file or directory
==================
WARNING: ThreadSanitizer: data race (pid=13808)
  Write of size 8 at 0x7fff7d6bf1d8 by main thread:
    #0 pthread_cond_destroy <null> (test1+0x8dd45) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #1 deinitCond__coreZlocks_u20 /home/antonisg/build/Nim/lib/core/locks.nim:65:2 (test1+0xefbc2) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #2 eqdestroy___OOZsrcZmalebolgia_u18 /home/antonisg/code/malebolgia/src/malebolgia.nim:14:2 (test1+0xefbc2)
    #3 main__test49_u37 /home/antonisg/code/malebolgia/src/malebolgia.nim:13:3 (test1+0xefbc2)
    #4 NimMainModule /home/antonisg/code/malebolgia/tests/test1.nim:18:2 (test1+0xefe37) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #5 NimMainInner /home/antonisg/code/malebolgia/src/malebolgia.nim:36:2 (test1+0xefe37)
    #6 NimMain /home/antonisg/code/malebolgia/src/malebolgia.nim:47:2 (test1+0xefe37)
    #7 main /home/antonisg/code/malebolgia/src/malebolgia.nim:55:3 (test1+0xefe37)

  Previous read of size 8 at 0x7fff7d6bf1d8 by thread T241:
    #0 pthread_cond_signal <null> (test1+0x8d805) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #1 signal__coreZlocks_u25 /home/antonisg/build/Nim/lib/core/locks.nim:73:2 (test1+0xee912) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #2 taskCompleted__OOZsrcZmalebolgia_u92 /home/antonisg/code/malebolgia/src/malebolgia.nim:39:2 (test1+0xee912)
    #3 worker__OOZsrcZmalebolgia_u177 /home/antonisg/code/malebolgia/src/malebolgia.nim:113:4 (test1+0xee912)
    #4 threadProcWrapDispatch__stdZtypedthreads_u105 /home/antonisg/build/Nim/lib/system/threadimpl.nim:71:2 (test1+0xedca3) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #5 threadProcWrapStackFrame__stdZtypedthreads_u95 /home/antonisg/build/Nim/lib/system/threadimpl.nim:100:2 (test1+0xedca3)
    #6 threadProcWrapper__stdZtypedthreads_u81 /home/antonisg/build/Nim/lib/system/threadimpl.nim:106:2 (test1+0xe8983) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)

  Location is stack of main thread.

  Location is global '??' at 0x7fff7d6a0000 ([stack]+0x1f1d8)

  Thread T241 (tid=14050, running) created by main thread at:
    #0 pthread_create <null> (test1+0x67206) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #1 createThread__stdZtypedthreads_u60 /home/antonisg/build/Nim/lib/std/typedthreads.nim:246:106 (test1+0xe8acb) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #2 setup__OOZsrcZmalebolgia_u226 /home/antonisg/code/malebolgia/src/malebolgia.nim:119:94 (test1+0xef54a) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #3 atmdotdotatssrcatsmalebolgiadotnim_Init000 /home/antonisg/code/malebolgia/src/malebolgia.nim:140:2 (test1+0xef54a)
    #4 PreMainInner /home/antonisg/code/malebolgia/src/malebolgia.nim:19:2 (test1+0xefe32) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e)
    #5 PreMain /home/antonisg/code/malebolgia/src/malebolgia.nim:31:2 (test1+0xefe32)
    #6 NimMain /home/antonisg/code/malebolgia/src/malebolgia.nim:46:2 (test1+0xefe32)
    #7 main /home/antonisg/code/malebolgia/src/malebolgia.nim:55:3 (test1+0xefe32)

SUMMARY: ThreadSanitizer: data race (/home/antonisg/code/malebolgia/tests/test1+0x8dd45) (BuildId: f600fc0c8e8708fdf1ca7711971442600b2f1f8e) in pthread_cond_destroy
==================
ThreadSanitizer: reported 1 warnings

bad codegen with cpp, works with c

import std/[enumerate]
import malebolgia

func threadFunc(a: int): int =
  a + a

proc run(inputs: openArray[int]): seq[int] =
  result = newSeq[int](inputs.len)
  var m = createMaster()
  m.awaitAll:
    for i, item in enumerate(inputs):
      m.spawn threadFunc(item) -> result[i]

echo run([1, 2, 3, 4, 5])

$ nim --skipUserCfg --skipParentCfg --skipProjCfg c -r thisfile.nim
@[2, 4, 6, 8, 10]

$ nim --skipUserCfg --skipParentCfg --skipProjCfg cpp -r thisfile.nim
/home/jack/.cache/nim/malebolgia_cpp_d/@mmalebolgia_cpp.nim.cpp: In function ‘tySequence__qwqHTkRvwhrRyENtudHQ7g run__malebolgia95cpp_u6(NI*, NI)’:
/home/jack/.cache/nim/malebolgia_cpp_d/@mmalebolgia_cpp.nim.cpp:334:240: error: use of deleted function ‘tyObject_Master__F3e20AfgTBHcMGUUxuJL1g& tyObject_Master__F3e20AfgTBHcMGUUxuJL1g::operator=(tyObject_Master__F3e20AfgTBHcMGUUxuJL1g&&)’
  334 | s/experiments/malebolgia_cpp.nim");        m = createMaster__OOZOOZOOZOnimbleZpkgs50Zmalebolgia4549O51O494551b54b56b48565757c5448515748dd5048525351c53f55a50a484953515450cf4851Zmalebolgia_u25(T3_);
  569 |         return result;
      |                ^~~~~~
/home/jack/.cache/nim/malebolgia_cpp_d/@m..@s..@[email protected]@spkgs2@[e-mail]:105:8: note: ‘tyObject_Master__F3e20AfgTBHcMGUUxuJL1g::tyObject_Master__F3e20AfgTBHcMGUUxuJL1g(const tyObject_Master__F3e20AfgTBHcMGUUxuJL1g&)’ is implicitly deleted because the default definition would be ill-formed:
  105 | struct tyObject_Master__F3e20AfgTBHcMGUUxuJL1g {

Version:

Nim Compiler Version 2.0.2 [Linux: amd64]
Compiled at 2023-12-15
Copyright (c) 2006-2023 by Andreas Rumpf

git hash: c4c44d10df8a14204a75c34e499def200589cb7c
active boot switches: -d:release

package "[email protected]" with checksum "3b6b8b0899c60390dd20453c5f7a2a015362cf03"

Set the number of threads at run time

While the ThreadPoolSize property can be set during compile time, it would be neat to be able to do that during runtime, to avoid being required to compile different versions for machines with different number of cores.

Maybe making the setup proc public and allow passing a thread num argument to it could be a solution.

Edit: On second thoughts, it would also be nice to specify the pool's size per master instead of globally, but I see that this would require a more substantial change and also would change the semantics of the library significantly.

`Warning: =destroy (dest.value) can raise an unlisted exception: Exception [Effect]` displayed when running README's example

Hello, I would like to thank you for maintaining the project.

Describe the issue
I would like to report this because a WARNING was output when I executed the README below.
https://github.com/Araq/malebolgia?tab=readme-ov-file#mutable-parameters-and-data-sharing

Step to Reproduce

1. nimble init
2. Copy https://github.com/Araq/malebolgia?tab=readme-ov-file#mutable-parameters-and-data-sharing example code
3. nimble build

Expected behaviour
WARNING is not output.

Actual behaviour
WARNING is output as follows.

% nimble build 
  Verifying dependencies for [email protected]
     Info:  Dependency on malebolgia@any version already satisfied
  Verifying dependencies for [email protected]
   Building Sample/Sample using c backend
/Users/admin/Scripts/Nim/Sample/src/Sample.nim(15, 4) template/generic instantiation of `awaitAll` from here
/Users/admin/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(244, 5) template/generic instantiation of `checkBody` from here
/Users/admin/Scripts/Nim/Sample/src/Sample.nim(16, 6) template/generic instantiation of `spawn` from here
/Users/admin/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(177, 47) template/generic instantiation of `toTask` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/tasks.nim(132, 20) template/generic instantiation of `isolate` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(37, 14) template/generic instantiation of `=destroy` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(27, 6) template/generic instantiation from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(29, 13) Warning: `=destroy`(dest.value) can raise an unlisted exception: Exception [Effect]
/Users/admin/Scripts/Nim/Sample/src/Sample.nim(15, 4) template/generic instantiation of `awaitAll` from here
/Users/admin/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(244, 5) template/generic instantiation of `checkBody` from here
/Users/admin/Scripts/Nim/Sample/src/Sample.nim(16, 6) template/generic instantiation of `spawn` from here
/Users/admin/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(177, 47) template/generic instantiation of `toTask` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/tasks.nim(132, 20) template/generic instantiation of `isolate` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(37, 14) template/generic instantiation of `=destroy` from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(27, 6) template/generic instantiation from here
/Users/admin/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(29, 13) Warning: `=destroy`(dest.value) can raise an unlisted exception: Exception [Effect]

Environment

• OS: macOS Sonoma version 14.2.1
• Nim: Compiler Version 2.0.2 [MacOSX: amd64]
• nimble: v0.14.2 compiled at 2023-12-24 13:54:41

Additional context
A similar issue has been reported with Nim. Is this an issue on Nim's side?

• nim-lang/Nim#23129

Sorry for my lack of understanding, but if this issue is scheduled to be resolved in Nim, I will close this issue.

malebolgia.nimble

All four releases have the same malebolgia.nimble with the same version number. According to the nimble docs it should have been named package.nimble. When installing nimble nags about the structure not being right.

# Package

version       = "0.1.0"
author        = "Araq"
description   = "Malebolgia creates new spawns. Experiments with thread pools and related APIs."
license       = "MIT"
srcDir        = "src"


# Dependencies

requires "nim >= 1.9.3"

Add support for timeouts

how to feed commands to external process and catch output

i just found your library and i think it can solve my problem : i tried to start a process (a chess engine) in Nim and feed commands to it (stdin) and catch its output (stdout) .. i managed to get this working somehow but my solution seems primitive and incomplete, see https://forum.nim-lang.org/t/10613

when i execute the process 'stockfish' in terminal, the program returns its 'bestmove' but my Nim script doesn't output this last line !? I read about Nim threads, threadpools, async, await, spawn but it i don't find good examples and so it's all too difficult for me ..

can anybody give a (simple) example? Just feed commands to an external process and catch its output? It seems Malebolgia can do this easily, but i have no clue .. eventually i want to build a GUI app which uses the output info of such chess engine to calculate things and output a graph, while the engine is still running; eg. the engine process also has to listen to a "stop" command ..

Malebolgia must transfer exceptions

This would be particularly elegant to implement with a new proc system.stealCurrentException().

tfib_male.nim crashes on the M1

Let's hope helgrind can reproduce it.

Malebolgia must support function with return values

Design: Do not wrap the result in a flowvar, instead declare it as unsafe and users should not access the storage of the result until after the barrier is complete.