lttlabsoss/markbench-tests

Home of test harnesses used in LTT Labs MarkBench

markbench-tests's Issues

[Bug/Question] #2

Environment

- MarkBench Test Version: 1.0.0
- OS:  Windows 11 22H2
- GPU: RX 6650M
- CPU: Ryzen 7 6800H

What happened?

I am trying to make a harness for Furmark but I am having some issues.

python .\furmark_runner.py --benchmark_mode "GPU stress test" --resolution "1920x1080" --resolution_scale "2x" --anti_aliasing "4x MSAA" is the command that I am running and all it does it open FurMark and then doesnt do anything.

This is my YMAL Manifest :

friendly_name: "FurMark Benchmark"
executable: "FurMark.exe"
process_name: "FurMark.exe"
disable_presentmon: false
hidden: 1
output_dir: "results"
options:

• name: resolution
  type: select
  values: ["1280x720", "1920x1080", "2560x1440", "3840x2160"]
• name: benchmark_mode
  type: select
  values: ["GPU stress test", "Burn-in test", "Benchmarking"]
• name: resolution_scale
  type: select
  values: ["1x", "2x", "3x", "4x"]
• name: anti_aliasing
  type: select
  values: ["Off", "2x MSAA", "4x MSAA", "8x MSAA"]

This is my python script: (Copy and Paste from your github since I am new to this and learning)
import os
import sys

Default installation directory and executable name

DEFAULT_FURMARK_DIR = "C:\Program Files (x86)\Geeks3D\Benchmarks\FurMark"
EXECUTABLE = "FurMark.exe"

Construct absolute path to FurMark executable

ABS_EXECUTABLE_PATH = os.path.join(DEFAULT_FURMARK_DIR, EXECUTABLE)

Check if FurMark executable exists

if not os.path.isfile(ABS_EXECUTABLE_PATH):
raise ValueError('No FurMark installation detected! Default installation expected to be present on the system.')

Extract command-line arguments passed to the script (excluding script name)

args = sys.argv[1:]

Construct the command to execute FurMark

command = f'"{ABS_EXECUTABLE_PATH}" '
for arg in args:
command += arg + ' '

Remove trailing whitespace

command = command.rstrip()

Execute the command

os.system(command)

File name is furmark_runner.py

Relevant log output

No response

Anything else?

No response

[Bug/Question]

Environment

- MarkBench Test Version: 1.0.0
- OS: Win 11 22H2
- GPU: RX 6650M
- CPU: Ryzen 7 6800H

What happened?

When running python .\msikombuster.py --test vkfurrytorus --resolution "1080,1920" -b true , it will launch with iGPU instead of dGPU. What do I need to do to ensure that it always launches with the correct GPU? When going into device manager and disabling the iGPU, only then will it launch with the dGPU. Please help :P

Relevant log output

No response

Anything else?

No response

Feature Request: Add a tokens per second evaluation with Local Large Langage Models

As recent threadripper videos have referenced, there are some chips and gpus that are targets at AI Engineers.

It would be nice for there to be a standardized test harness for evaluating Hardware for LLM usage.

Here's a bit of background on the goals and objectives of evaluating an LLM:
https://www.baseten.co/blog/llm-transformer-inference-guide/

I will note this is a complex and ever-evolving field and fully understand if this is closed without comment.

I also acknowledge this may not be something you want to be wholly responsible for and instead call to a service that does this.

[Bug] poetry unable to install pyyaml

Environment

- MarkBench Test Version: commit 34ddbefaeda3192926c1f0459f790848e07f859a
- OS: Win11
- GPU: Intel Arc A770
- CPU: amd 5600x

What happened?

When running poetry install with a completely fresh python 3.11.5 install it complains about pyyaml (6.0) not supporting PEP 517 builds and suggesting to run pip wheel --use-pep517 "pyyaml (==6.0)" but after running that nothing changes.

Relevant log output

Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pyyaml (6.0)

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel

  running egg_info
  writing lib\PyYAML.egg-info\PKG-INFO
  writing dependency_links to lib\PyYAML.egg-info\dependency_links.txt
  writing top-level names to lib\PyYAML.egg-info\top_level.txt
  Traceback (most recent call last):
    File "C:\Users\hello\AppData\Roaming\pypoetry\venv\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
      main()
    File "C:\Users\hello\AppData\Roaming\pypoetry\venv\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\hello\AppData\Roaming\pypoetry\venv\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
      self.run_setup()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 288, in <module>
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
      dist.run_commands()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\dist.py", line 989, in run_command
      super().run_command(command)
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
      cmd_obj.run()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 318, in run
      self.find_sources()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 326, in find_sources
      mm.run()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 548, in run
      self.add_defaults()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\command\egg_info.py", line 586, in add_defaults
      sdist.add_defaults(self)
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\command\sdist.py", line 113, in add_defaults
      super().add_defaults()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\command\sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<string>", line 204, in get_source_files
    File "G:\Temp\tmpss1jx_vg\.venv\Lib\site-packages\setuptools\_distutils\cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources


  at ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\chef.py:147 in _prepare
      143│
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│
      146│             if error is not None:
    → 147│                 raise error from None
      148│
      149│             return path
      150│
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyyaml (6.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyyaml (==6.0)"'.

Anything else?

No response

[Suggestion] get_app_install_location may return empty erroneously

Environment

- MarkBench Test Version: Latest
- OS: N/A
- GPU: N/A
- CPU: N/A

What happened?

For various reasons, including importing an existing steam library, Uninstall entries for steam apps may not be written to registry. This may cause get_app_install_location to return empty erroneously

A better flow would be the following:

1. Get Steam Install location from "\Software\Valve\Steam\SteamPath"
2. Get Library Folders and their apps from SteamPath + "\steamapps\libraryfolders.vdf"

I am not great with python but I can attempt to make a pull request if wanted.

Relevant log output

No response

Anything else?

No response

[Bug] Documentation error for linking CODE_OF_CONDUCT

Environment

No response

What happened?

Problem

When viewing the CONTRIBUTE.md you provide a link to your CODE_OF_CONDUCT.md file however this link is incorrect.

This link points to: https://github.com/LTTLabsOSS/markbench-tests/blob/main/blob/master/CODE_OF_CONDUCT.m
The branch master doesn't exist, it should point to: https://github.com/LTTLabsOSS/markbench-tests/blob/main/CODE_OF_CONDUCT.md

Expected resolution

Change to https://github.com/LTTLabsOSS/markbench-tests/blob/main/CODE_OF_CONDUCT.md

Relevant log output

No response

Anything else?

No response

[Question] Are there plans to open source markbench itself?

First of all: Thank you for open sourcing what you have! It's wonderful to see the work being shared.

I was wondering if there was intention to open source the markbench tool itself. Having reviewed the code in this repo I would love to contribute, but without markbench being available it will be very hard to ensure changes don't break compatability. At the very least it would be great to get some kind of documentation, or test cases, which help make sure that interface isn't broken.

Again, thanks for open sourcing this repo. I look forward to what else your team has in store. :)

[Feature Request] Consider releasing precompiled / pre-packaged executable

For more general users who aren't familiar with programming or even the command line, a single executable file would simplify the launch process for these people.

There are a few solutions to this, such as:

• Use the pyinstaller program to generate the executable. It may take a bit of work to get the proper files and dependences packaged into the executable, but it is relatively straightforward.
• Use the py2exe program. Similar to pyinstaller
• Use the Nuitka program to compile and package the code. Whereas the prior two solutions package the python code and an entire python runtime into a single executable, Nuitka instead compiles portions of the code down to C for a decent gain in performance (depending on the workload). This may take more work than either of the other solutions, but it'll make the overall program performance much better on top of being packaged into an executable.

You may also consider creating GitHub Actions to automate this compiling / packaging / releasing process.

[Question] Is it possible not to use AH game in Labs Testing at all?

Environment

No response

What happened?

Hello,

My question is mainly about the game selected, and not to the program itself.

I noticed a while back that atomic heart was one of the games selected for benchmarking, and I didn't know where to better forward my question, so I hope that this is the most appropriate one.

AH is a very controversial game due to where it is being developed and where the revenue from this game goes, example:
https://www.denofgeek.com/games/atomic-heart-controversy-explained-boycott-protests-ukraine/

For me and my Ukrainian friends who watch various LTT channels, it was always sad to see this game being used (and effectively indirectly promoted) for benchmarking.

I don't ask to have a political stance here or anything like that. But please consider that out of all the recent games that were released this one is probably the only one that is that controversial, and I understand that you didn't know this part (it was only briefly mentioned a couple of times on TechLinked).
Replacing this game for benchmarking would be something huge for your Ukrainian audience.

I just wanted to give you a bit of context from one slice of your audience.

Thank you

Relevant log output

No response

Anything else?

No response

[Bug] CsGo is no longer available

Environment

No response

What happened?

cs 2 replaced csgo

Relevant log output

No response

Anything else?

No response