DataFrame.vstack() deadlocks in asyncio background task

[benkay86]

DataFrame.vstack() deadlocks in asyncio background task

[benkay86]

Note, this appears to actually be a bug (or a footgun) in Python. The polars-free code below deadlocks if you comment out the nonlocal count line. The reproducible example code above works (no deadlock) if you add nonlocal df to the top of the worker coroutine. Please feel free to close this issue.

import asyncio

async def arange(count):
    for i in range(count):
        yield(i)
        await asyncio.sleep(0)

async def main():
    # Initialize number.
    count = 0

    # Define a queue of data to process.
    limit = 3
    queue = asyncio.Queue(limit)

    # Define a worker task to do the processing.
    async def worker():
        nonlocal count
        while True:
            # Get the next item to process from the queue.
            item = await queue.get()
            # Increment the count.
            count = count + 1
            # Tell the queue we are done with this item.
            queue.task_done()

    # Launch some asynchronous worker tasks to process the queue.
    tasks = set()
    for _ in range(limit):
        tasks.add(asyncio.create_task(worker()))

    # Add some items to the queue.
    async for item in arange(100):
        # Put each item in the queue.
        await queue.put(item)
    
    # Wait until the queue is processed.
    await queue.join()
    # Then cancel the worker tasks.
    for task in tasks:
        task.cancel()
    for task in tasks:
        try:
            await task
        except asyncio.CancelledError:
            pass
    
    return count

if __name__ == '__main__':
    count = asyncio.run(main())
    print(count)

[stinodego]

In that case I'll close this issue. Thanks for looking into it.