asimmon/ephemeral-mongo

EphemeralMongo is a set of three NuGet packages wrapping the binaries of MongoDB 4, 5 and 6 built for .NET Standard 2.0.

ephemeral-mongo's Issues

Connection Timeout on Linux Fedora 38 (fixed)

Hello,

This is actually a guide for those that are having issues on Fedora 38 and another Linux distro.

If you get a "Timeout" error when running you tests, maybe that are some missing dependencies needed by mongod to run.

How to fix:

After run dotnet restore command the mongob file will be download to your bin.
You can check the dependencies to run this file using following command:

ldd /path/to/mongod

In my case libcrypto.so.1.1 and libssl.so.1.1 were missing.
After a little search I could check that these libs are included in openssl1.1.x86_64 package.

To install openssl1.1:

sudo dnf install openssl1.1.x86_64

You can run the ldd command again and check if everything is fine.
If positive you can run your tests now!

Hope it helps!

.NET 8 Warning NETSDK1206 and not loading version/distribution specific lib *.ubuntu.22.04-x64

I'm using the <PackageReference Include="EphemeralMongo6.runtime.ubuntu.22.04-x64" Version="1.1.3" Condition="$([MSBuild]::IsOSPlatform('Linux'))" /> as mentioned in #36 but my CI pipeline cant find the lib internal files. The pipeline runs on mcr.microsoft.com/dotnet/sdk:8.0 image.

Bellow its the first error message on my pipeline:
/usr/share/dotnet/sdk/8.0.300/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(284,5): warning NETSDK1206: Found version-specific or distribution-specific runtime identifier (s): ubuntu.22.04-x64. Affected libraries: EphemeralMongo6.runtime.ubuntu.22.04-x64. In .NET 8.0 and higher, assets for version-specific and distribution-specific runtime identifiers will not be found by default. See https://aka.ms/dotnet/rid-usage for details.

As far as I understand, .NET 8 won't load this library because it has a specific distribution on its name, which causes a second error message:
System.IO.FileNotFoundException : Could not find mongod in the following paths: '/github/workspace/tests/***.IntegrationTests/bin/Debug/net8.0/runtimes/linux-x64/native/mongodb/bin/ mongod'

Maybe suffixing it with "linux-x64" would make .NET 8 build include this lib correctly for linux, but im not sure. Nobody had this error like me?

Blocks in Github actions?

Hi,

thank you for this awesome library. I was testing it locally and it was working fine. But I have found two issues:

1. If the test does not exist, the process would not close and I had to terminate the process manually. You could check if Mongo is still running on start and reuse it or close the previous process.
2. The build was not running in Github CI anymore. It is a docker build, based on bullseye-slim and .NET 7 and it was just hanging. After an hour the build was still running, so the question is, is there some kind of wait or timeout, that could be configured?

My test: https://github.com/notifo-io/notifo/blob/main/backend/tests/Notifo.Domain.Tests/UserNotifications/MongoDb/MongoDbUserNotificationRepositoryTests.cs

My Fixture: https://github.com/notifo-io/notifo/blob/main/backend/tests/Notifo.Domain.Tests/Fixtures/MongoFixtureBase.cs

Spelling issue in options.StandardOuputLogger

StandardOuputLogger => StandardOutputLogger

Mongod won't start on CentOs

MongoRunner.Run() method doesn't start the mongod process on linux CentOS.

OS Version: Linux 3.10.0-1160.76.1.el7.x86_64
Error from std error:
mongod: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

Searching for this error brings me to this stack overflow question. Unfortunately none of the solutions posted here can be applied in my case because this is occurring on a CI server.

If a fix is not possible then the documentation should at least be updated with supported OSes and versions.

EDIT: Also I tried Mongo2Go but it has different issue.

Support MongoDB 7

MongoDB 7.0 just came out https://www.mongodb.com/blog/post/4-big-reasons-upgrade-mongodb-7-0. It requires the version 2.20 or higher for the driver first. They claimed massive query performance improvements.

Could not set executable bit for mongod

One of my coworkers is encountering this exception when trying to use this library. It's probably something in their environment, but am flagging it just in case.

System.IO.IOException
Could not set executable bit for '{test project path}/bin/Debug/net6.0/runtimes/osx-x64/native/mongodb/bin/mongod'
   at EphemeralMongo.Core.FileSystem.MakeFileExecutable(String path)
   at EphemeralMongo.Core.MongoRunner.RunInternal()
   at EphemeralMongo.Core.MongoRunner.Run(MongoRunnerOptions options)
   at BaseMongoCollectionFixture..ctor() in {project path}/BaseMongoCollectionFixture.cs:line 49

This machine worked fine with the latest Mongo2Go. We're all using .NET 6. They upgraded to the latest version of the SDK just in case. This library is working fine for other developers and in GitHub Actions.

P.S. thanks for putting this out there! MongoDB 6 support in our tests helps us feel confident about code using newer aggregation features.

Binaries doesnt get downloaded in linux-64 in CI

Hello ,

I have a dotnet project building on CI pipeline in linux OS. However the binaries folder does not exist in runtimes\linux-x64\native folder which is causing the tests to fail with the below error
System.ComponentModel.Win32Exception : An error occurred trying to start process '/opt/****/workspace/Build_NetCore_MSV/source/app/test/app.Test/bin/Release/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod' with working directory '/opt/****/workspace/Build_NetCore_MSV/source/app/test/app.Test/bin/Release/net6.0'. No such file or directory

I was able to see binaries in win , linux when I run locally. Please help

Temp data files are filling up the server.

We have configured to use EphemeralMongo within our unit tests and now are seeing an accumulation of associated temp database files. I don't see an option to clean these up after the instance is terminated, so have I missed an option or are there any examples of automating this? Otherwise this would appear to be a problem as our server has been experiencing disk space issues and swapping to EphemeralMongo appears to have vastly increased them.

[Request] Update EphemeralMongo6.runtime to newest (e.g. MongoDB 6.0.4)

Hey,

we are currently testing it with Ubuntu 22. I have some issues that maybe https://jira.mongodb.org/browse/SERVER-62300 could fix. Would it be feasible to release an update of the EphemeralMongo6.runtime packages? From what i have seen v0.1.13 is on the following versions:

Regards and thanks for the work!

Thomas

Linq ExpressionNotSupportedException : SingleOrDefault

I have a nested array called Documents in my collection.
This query works in production. While writing additional tests using ephemeral I start getting NotSupportedException
```
var projection = Builders.Projection.Expression(b => b.Documents.SingleOrDefault(i => i.Id == documentId));
return await _collection.Find(GetFilterByKey(botId)).Project(projection).FirstOrDefaultAsync();

Stack Trace: 
MethodCallExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, MethodCallExpression expression)
ExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, Expression expression)
ExpressionToAggregationExpressionTranslator.TranslateLambdaBody(TranslationContext context, LambdaExpression lambdaExpression, IBsonSerializer parameterSerializer, Boolean asRoot)
LinqProviderAdapterV3.TranslateExpressionToProjectionInternal[TInput,TOutput](Expression`1 expression, IBsonSerializer`1 inputSerializer, AstSimplifier simplifier)
LinqProviderAdapterV3.TranslateExpressionToFindProjection[TSource,TProjection](Expression`1 expression, IBsonSerializer`1 sourceSerializer, IBsonSerializerRegistry serializerRegistry)
ExpressionProjectionDefinition`2.RenderForFind(IBsonSerializer`1 sourceSerializer, IBsonSerializerRegistry serializerRegistry, LinqProvider linqProvider)
MongoCollectionImpl`1.CreateFindOperation[TProjection](FilterDefinition`1 filter, FindOptions`2 options)
MongoCollectionImpl`1.FindAsync[TProjection](IClientSessionHandle session, FilterDefinition`1 filter, FindOptions`2 options, CancellationToken cancellationToken)
<>c__DisplayClass48_0`1.<FindAsync>b__0(IClientSessionHandle session)
MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
IAsyncCursorSourceExtensions.FirstOrDefaultAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
BotDataStore.GetBotDocument(Guid botId, Guid documentId) line 412
UpdateDocumentFromBotTestIntegration.Handler_UpdateDocument_ReturnsResponse() line 74
GenericAdapter`1.BlockUntilCompleted()
NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
AsyncToSyncAdapter.Await(Func`1 invoke)
TestMethodCommand.RunTestMethod(TestExecutionContext context)
TestMethodCommand.Execute(TestExecutionContext context)
<>c__DisplayClass1_0.<Execute>b__0()
BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)

Problem using Aggregate -> Limit on EphemeralMongo5 1.1.2

I have some code that does a Limit as part of an Aggregate.
On 1.1.2 - I get

System.MissingMethodException
Method not found: 'MongoDB.Driver.IAggregateFluent`1<!0> MongoDB.Driver.IAggregateFluent`1.Limit(Int32)'.

0.1.3 -> Works
1.0.0 -> Works
1.1.0 -> Fails
1.1.2 -> Fails

Sample source code:

  var customerEvents = await mongoDbContext.CustomerCollection.Aggregate()
      .Match(filterLastThreeMonths)
      .Sort(sortByCreated)
      .Limit(500)
      .ToListAsync();

I know there other ways I could have written this code, but this is a simplification from what I'm actually using.

It used to work on Mongo2Go, but I'm trying to migrate away from it.

The IDE cannot copy the mongod.exe to the output dir

Hi,

I'm facing an issue that might be related to the following behavior mentioned in the issue: "Windows Defender Firewall" want to block communication of testhost.exe

• EphemeralMongo also uses .NET Process class to start mongod.exe from your project output directory.

If I run the integration test end to end for the first time, it works perfectly. But if force the execution to end in the middle of the test (Shift + F5), the mongod.exe is left running. Then if I make any change in the test code and try to run the test again, the IDE can't copy the mongod.exe file to the output directory, so I can't run the tests.

This is happening because the mongod process is still running.

Integration Tests using Ephemeral Mongo fail in aws/codebuild/standard:6.0

The latest AWS codebuild images (that come packaged with Net6.0 SDK) are built on Ubuntu 22.04, which has deprecated libssl1.1

EphemeralMongo[0]
      /codebuild/output/BroadcastServicesApi.UnitTests/bin/Release/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

We have been able to work around this limitation by manually installing libsssl1.1 before running our tests.

We haven't checked Ubuntu 22.04 based runners for other build systems.

"Text file busy" on relaunch

The first time i launch my tests, the package works perfectly.
But then if i relaunch my tests i get:

/usr/lib/dotnet/sdk/6.0.116/Microsoft.Common.CurrentVersion.targets(4812,5): error MSB3027: Could not copy "/home/user/.nuget/packages/ephemeralmongo6.runtime.linux-x64/1.0.0/runtimes/linux-x64/native/mongodb/bin/mongod" to "bin/Debug/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod". Exceeded retry count of 10. Failed.  [/home/user/Desktop/Sources/Project/source/back/Project/Project.PublicBo.Api.Tests/Project.PublicBo.Api.Tests.csproj]
/usr/lib/dotnet/sdk/6.0.116/Microsoft.Common.CurrentVersion.targets(4812,5): error MSB3021: Unable to copy file "/home/user/.nuget/packages/ephemeralmongo6.runtime.linux-x64/1.0.0/runtimes/linux-x64/native/mongodb/bin/mongod" to "bin/Debug/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod". Text file busy : '/home/user/Desktop/Sources/Project/source/back/Project/Project.PublicBo.Api.Tests/bin/Debug/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod' [/home/user/Desktop/Sources/Project/source/back/Project/Project.PublicBo.Api.Tests/Project.PublicBo.Api.Tests.csproj]

Improve logs when cannot start mongo

When the service cannot start, by default there is no logs only a timeout exception with a generic message.

System.TimeoutException : MongoDB connection availability took longer than the specified timeout of 30 seconds. Consider increasing the value of 'ConnectionTimeout'.

It would be nice to capture the mongo process logs and display them in the exception message. Then, it could discard the logs once the service is ready. Also, it doesn't check if the process exited while waiting for it to start.
I think this improvement would simplify debugging.

System.IO.FileNotFoundException : Could not find mongod in the following paths

Getting the following error when building on Ubuntu 22.04 in our Docker CI environment:

System.IO.FileNotFoundException : Could not find mongod in the following paths: '/builds/lunsjkollektivet/internal/backend/src/Tests/Tests.Common/bin/Debug/net6.0/runtimes/linux-x64/native/mongodb/bin/mongod'

I am using the package references mentioned here: #36 (comment)

The commands that i'm running when testing are as follows:

export NUGET_PACKAGES="$(realpath ./.nuget)/"
dotnet restore ./Tests/Tests.Common/Tests.Common.csproj --packages ./.nuget
dotnet test ./Tests/Tests.Common/Tests.Common.csproj --no-restore --collect:"XPlat Code Coverage" --logger "trx;LogFileName=test-result.trx;MethodFormat=Class;FailureBodyFormat=Verbose" --results-directory "./.testresults"

ARM64 is not supported

ARM64 will not be supported by default. The two main usecases for using EphemeralMongo are:

• Integration testing - mosly on CI agents which use x64 architecture by default,
• Local development environment - most developers use x64 architecture.

You can still manually download MongoDB binaries for ARM64, only use the EphemeralMongo.Core library, and specify the binaries directory path using options.BinaryDirectory.

Adding ARM64 will increase the total size of downloaded packages. As of today, all OS/CPU architecture specific runtime packages are downloaded even though you only need one. That's because NuGet does not support conditional dependencies on both the target framework moniker (TFM) and a runtime identifier (RID). See related issue #2.

When this will be supported by NuGet, we'll be able to add new OS/CPU architecture specific runtime packages. Until then, ARM64 will not be supported by default.

Two copies of files from the "ephemeralmongo5.runtime.win-x64" package in the publish directory

After publishing the app, there are two copies of files from the 'ephemeralmongo5.runtime.win-x64' package in the output directory.

Reproducing steps

Environment:

• OS: Windows 11
• .NET SDK: 8.0.100

1. Create a new project and install the EphemeralMongo5 package.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>netcoreapp3.1</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="EphemeralMongo5" Version="1.1.3" />
    </ItemGroup>

</Project>

2. Publish the project for win-x64

> dotnet publish -o .\publish -r win-x64

Actual behavior

There are two copies of attached files in the output directory.

Files structure:

• .\publish
  • .\runtimes\win-x64\native\mongodb
    • .\bin
      • mongod.exe
      • mongoexport.exe
      • mongoimport.exe
    • .\community-server
      • LICENSE-Community.txt
      • ...
    • .\database-tools
      • ...
  • mongod.exe
  • mongoexport.exe
  • mongoimport.exe
  • LICENSE-Community.txt
  • ...

Expected behavior

Only one copy of the files in the output directory.

Additional

If publish the app for other OS, the error is happened.

dotnet publish -o publish -r linux-x64

C:\Program Files\dotnet\sdk\8.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ConflictResolution.targets(112,5): error NETSDK1152: 
Found multiple publish output files with the same relative path: 
  C:\Users\MyUserName\.nuget\packages\ephemeralmongo5.runtime.linux-x64\1.1.3\runtimes\linux-x64\native\mongodb\community-server\THIRD-PARTY-NOTICES, 
  C:\Users\MyUserName\.nuget\packages\ephemeralmongo5.runtime.linux-x64\1.1.3\runtimes\linux-x64\native\mongodb\database-tools\THIRD-PARTY-NOTICES, 
  C:\Users\MyUserName\.nuget\packages\ephemeralmongo5.runtime.linux-x64\1.1.3\runtimes\linux-x64\native\mongodb\community-server\version.txt, 
  C:\Users\MyUserName\.nuget\packages\ephemeralmongo5.runtime.linux-x64\1.1.3\runtimes\linux-x64\native\mongodb\database-tools\version.txt.

Doesn't work on Linux systems without OpenSSL 1.1

OpenSSL 1.1 is going EOL in September and is being gradually removed from Linux systems:

1. Ubuntu 22.04 only has OpenSSL 3
2. Gentoo switched to OpenSSL 3 last week
3. Arch switched to OpenSSL 3 in Nov 2022

mongod bundled in EphemeralMongo simply doesn't start on such systems:

$ ~/.nuget/packages/ephemeralmongo6.runtime.linux-x64/1.1.0/runtimes/linux-x64/native/mongodb/bin/mongod 
/home/marat/.nuget/packages/ephemeralmongo6.runtime.linux-x64/1.1.0/runtimes/linux-x64/native/mongodb/bin/mongod: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

.NET 8.0 CI compatibility

Hi, we're just upgrading our .NET projects to 8.0 and while EphemeralMongo tests still work locally, in the CI pipeline which uses mcr.microsoft.com/dotnet/sdk:8.0 image, it fails with the following error:

Error Message:
#18 69.43    System.TimeoutException : MongoDB connection availability took longer than the specified timeout of 30 seconds. Consider increasing the value of 'ConnectionTimeout'.
#18 69.43   Stack Trace:
#18 69.43      at EphemeralMongo.MongodProcess.StartAndWaitForConnectionReadiness()
#18 69.43    at EphemeralMongo.MongodProcess.Start()
#18 69.43    at EphemeralMongo.MongoRunner.RunInternal()
#18 69.43    at EphemeralMongo.MongoRunner.Run(MongoRunnerOptions options)

Any thoughts on why that could be happening when it worked without issues with .NET 7.0?

Optimize NuGet package downloads for a specific operating system

EphemeralMongo use a dedicated runtime NuGet package for each operating system (linux-x64, osx-x64 and windows-x64).

However, when using EphemeralMongo4, EphemeralMongo5 or EphemeralMongo6, the three OS-specific runtime packages are downloaded and copied to the build output, which takes time and a lot of space. The ideal solution would to only download the relevant runtime package for the consuming operating system.

Unfortunately, there is no viable solution today to optimize the download process.

1. It is not an option to download MongoDB and the tools at runtime, there could be network issues, some people use proxies and firewalls, we would have to implement a reliable cache.
2. .NET does not provide (yet) a way to use conditional NuGet dependencies on both the target framework moniker (TFM) and a runtime identifier (RID).

There are actually a lot of opened issues in the .NET GitHub repository about option #2, so it might be possible in a future version of .NET:

• NuGet/Home#1660
• NuGet/Home#5862
• NuGet/Home#10571
• dotnet/runtime#49137
• dotnet/sdk#4552

Using runtime.json is also not an option because it is deprecated, undocumented, and was created for internal .NET Core NuGet packages.

Until then, we'll stick with the RID-specific runtime package.

"Windows Defender Firewall" want to block communication of testhost.exe

Hi,

i am testing the migration from Mongo2Go to EphemeralMongo. I am getting a "Windows Defender Firewall" block when using it. This does not seem to happen when using Mongo2Go. Also when i decide to ignore this message or cancel it; it does not have any negative effect on the testrun.

Any idea were this could come from?

Regards

Thomas