arena-ai/arena

A place to evaluate public models

arena's Issues

Fix problems

• Evaluation should not slow response time
• The interface should not fail with anthropic/mistral
• The interface should be more dense and legible for events
• Evaluations should be represented nicely
• Fix bugs

Write a small PII removal post with faker (evaluate the impact of faker with LLM eval)

Included in this release:

• Presidio+Faker integration
• Nicer event presentation with main request highlighted
• Better dark mode support by the logo
• Improve the README.md of the public project
• Parametrization from the app vs client

Use json serialization as the default for celery

Make crud operations async

Do PII substitution with Faker

When the message is very long, the PII removal process can be slow... provoking timeouts

ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
  |     await route.handle(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
  |     await self.app(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     response = await func(request)
  |                ^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app
  |     raise e
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app
  |     raw_response = await run_endpoint_function(
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
  |     return await dependant.call(**values)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/app/app/lm/api/routes/chat_completion.py", line 209, in chat_completion
  |     return await ArenaHandler(session_dep, current_user, chat_completion_request).process_request()
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/app/app/lm/api/routes/chat_completion.py", line 75, in process_request
  |     async with create_task_group() as tg:
  |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "/app/app/ops/computation.py", line 128, in evaluate
    |     async with TaskGroup() as task_group:
    |   File "/usr/local/lib/python3.11/asyncio/taskgroups.py", line 145, in __aexit__
    |     raise me from None
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
      |     yield
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
      |     resp = await self._pool.handle_async_request(req)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
      |     raise exc from None
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
      |     response = await connection.handle_async_request(
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
      |     return await self._connection.handle_async_request(request)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
      |     raise exc
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
      |     ) = await self._receive_response_headers(**kwargs)
      |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
      |     event = await self._receive_event(timeout=timeout)
      |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
      |     data = await self._network_stream.read(
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
      |     with map_exceptions(exc_map):
      |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
      |     self.gen.throw(typ, value, traceback)
      |   File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
      |     raise to_exc(exc) from exc

      | httpcore.ReadTimeout
      | 
      | The above exception was the direct cause of the following exception:
      | 
      | Traceback (most recent call last):
      |   File "/app/app/ops/computation.py", line 114, in call
      |     return await self.op.call(*args)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/app/app/ops/masking.py", line 54, in call
      |     analysis = await analyzer.analyze(AnalyzerRequest(text=input))
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/app/app/services/masking.py", line 45, in analyze
      |     response = await client.post(
      |                ^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
      |     return await self.request(
      |            ^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
      |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
      |     response = await self._send_handling_auth(
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
      |     response = await self._send_handling_redirects(
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
      |     response = await self._send_single_request(request)
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
      |     response = await transport.handle_async_request(request)
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request
      |     with map_httpcore_exceptions():
      |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
      |     self.gen.throw(typ, value, traceback)
      |   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
      |     raise mapped_exc(message) from exc
      | httpx.ReadTimeout
      +------------------------------------
    | 
    | During handling of the above exception, another exception occurred:
    | 
    | Traceback (most recent call last):
    |   File "/app/app/lm/api/routes/chat_completion.py", line 78, in set_content
    |     message.content, _ = await replace_masking(message.content).evaluate(session=self.session)
    |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/app/ops/computation.py", line 136, in evaluate
    |     raise RuntimeError(f'The computation failed. A dump is written there {name}')
    | RuntimeError: The computation failed. A dump is written there /tmp/dump_1719929996.82759.dot
INFO:     10.244.2.11:35978 - "GET /api/v1/users/22 HTTP/1.1" 403 Forbidden

Add the ability to download events in a file

Fix bugs and add doc

Fix:

• parquet download bug?
• Event count display colors

Add:

• Docs