prismarinejs/mineflayer-test-api

An automated testing library for Mineflayer and Mineflayer plugins with CI/CD.

mineflayer-test-api's Issues

Load all test files from folder.

Is your feature request related to a problem? Please describe.

Setting up a new test should be as simple as creating a new file within the test folder. Tests should be automatically loaded and executed from a single command within the CI pipeline.

Describe the solution you'd like

A load that retrieves and runs all test files within a folder.

Describe alternatives you've considered

Manually loading and executing each test file.

Additional context

N/A

Use this in mineflayer tests

Move all tests to worker threads.

Is your feature request related to a problem? Please describe.

Having all tests run on the main thread can heavily distort timing and cause lag. This has the potential to make tests fail because the CPU couldn't keep up with executing time-sensitive commands.

Describe the solution you'd like

Each test should be moved to its own worker thread and executed in groups to prevent overlapping.

Describe alternatives you've considered

The only other practical alternative is running tests one at a time, but this would be extremely slow.

Additional context

Since each test is fully self-contained and the server is a separate process, running each test on its own thread would have no memory issues.

Build schematic instantly

Is your feature request related to a problem? Please describe.

When working with Mineflayer plugin development, having a complex yet repeatable environment is extremely useful for testing and performance evaluations. Being able to build a schematic from a file on the server is a good way to provide a complex situation to test against.

Describe the solution you'd like

A simple function to load a schematic file and build it on the server.

Describe alternatives you've considered

Issuing lists of pre-defined /setblock commands.

Additional context

This feature request was mentioned in the original Issue thread for Mineflayer test API, here.

Add plugin loading to test server.

Is your feature request related to a problem? Please describe.

When defining tests, there's often a lot of overlap with the needed API on the server. It would be useful to not have to define these actions multiple times. Pushing common server API sets, such as maze generation or spawning mob groups, would be well suited to external plugins that could be loaded in one line of code.

Describe the solution you'd like

In addition to the registerTest function provided, also provide a loadPlugin function that is called on a server when it is initially started in order to create their own utility test plugins that make it easier to write simple and repeatable tests.

Describe alternatives you've considered

Registering APIs manually within each test. But this is suboptimal and slow.

Additional context

This behavior should work very closely with the bot.loadPlugin behavior provided within Mineflayer. Loading a plugin once will register it for all tests.

Add cleanup from mineflayer tests

mineflayer have some good cleanups as part of https://github.com/PrismarineJS/mineflayer/blob/master/test/externalTests/plugins/testCommon.js and https://github.com/PrismarineJS/mineflayer/blob/master/test/externalTest.js

it could be useful to have this here I think

Test Timeouts

Is your feature request related to a problem? Please describe.

If a test happens to stall, there is currently no way to skip the test and continue. This causes the CI to hang for an extended period of time.

Describe the solution you'd like

Tests should automatically fail once if the test has not completed after a certain period of time. (Such as 1 minute) This time should be configurable.

Describe alternatives you've considered

Manually added test timeouts.

Additional context

N/A

When showing test names, the title is listed as '../My Test Name'

• The readme doesn't contain a resolution to my issue
• The example doesn't contain a resolution to my issue

Versions

• node: 14.16.1
• mineflayer-test-api: 1.0.1

Detailed description of a problem

When showing the names of tests in the console, the test paths are generated incorrectly. For example, with a test file named myTest.js containing the test Jump Forward, the test name will be rendered as ../Jump Forward. This is incorrect.

Your current code

const { registerTest } = require('mineflayer-test-api')

registerTest('Jump Forward', async (server, startPos) => {
  throw new Error('Failed!')
}

Expected behavior

The correct test name should be shown as test/myTest.js/Jump Forward

Additional context

N/A

Better Server API synchronization (accounting for Lag)

Is your feature request related to a problem? Please describe.

During the event of an unexpected lag spike, certain server APIs may not fully finish executing before the promise returns. This leads to race conditions when the CI server is running slower than expected.

Describe the solution you'd like

All server API calls should verify that the action has completed before returning. This would likely be done by adding listeners to the bot instance and waiting for the bot to receive the correct events indicated that the action has completed. This could be messages in chat for the case of becoming OP, or waitForChunksToLoad in the case of teleporting.

Describe alternatives you've considered

A simple alternative would be to increase the sleep time of each promise before returning, but this is only a bandaid solution and not a real fix. Plus, this would only increase the executing time of the events needlessly under normal circumstances.

Additional context

Current Server API:

• Make OP
• Teleport