Use DuckDB amalgamation instead of shared libraries

[marcboeker]

Use DuckDB amalgamation instead of shared libraries

[begelundmuller]

I'm happy to hear that. We'd be happy to submit a PR for you to test and will try to prioritise it within a few weeks.

[marcboeker]

Hi @begelundmuller,

thank you for reaching out and thanks for you kind words. Go DuckDB is made possible by many different contributors.

The idea sounds great to me. Especially if we can drop an external dependency. I think we have to test this and see if it works on all major the platforms (macOS, Linux and Windows).

Would it be OK for you, if you submit a PR and we test drive it?

Thanks in advance!

[begelundmuller]

This has now been merged: #40

The PR also includes pre-compiled static libraries inspired by https://github.com/rogchap/v8go. This means that on supported platforms, you get static binaries without having to wait for DuckDB to compile from source.

[LasseJacobs]

@begelundmuller could you explain why having a single source file makes it easier to statically link? I thought this would not matter since in both cases (single vs multiple source files) you end up with a single archive file (.a)? But indeed I am unable to statically link the libduckdb_static.a file duckdb produces when I build it from source, while the archive file produced from the single file links statically without a problem.

I would like to understand why this is the case.

[begelundmuller]

Hey @LasseJacobs, you're right that simply having one source file instead of multiple source files does not itself make a difference to our ability to compile from source or statically link an archive.

The DuckDB amalgamation is convenient because DuckDB is an extensive project with many source files and dependencies. Their build process requires CMake and might change at any time. The point of the amalgamation is to reduce this complexity to a single file with no dependencies that can be compiled using just a C++ compiler.

So by using the amalgamation, we both reduce build complexity and get a "contract" with the DuckDB project that's less likely to break. In fact, the amalgamation is published as part of every release, so we just pull it from their release artifacts.

Now about the static library, you're right that DuckDB outputs a libduckdb_static.a file when you compile it from source. I actually investigated using that instead of building the static libraries from the amalgamation. However, it turns out that libduckdb_static.a doesn't include DuckDB's dependencies, which are compiled as separate archive files (you can find them in the build output if you search for .a files). You have to statically link all of them (in the right order) to make it work – which would tie us into DuckDB's current build setup, which is likely to change.

Therefore, it felt safer to just compile the amalgamation to a static library ourselves (it's not that complicated). If DuckDB ever decides to publish static libraries as part of their release artifacts (currently they only publish shared libraries), I would probably reconsider this.

[LasseJacobs]

@begelundmuller very clear now! Thanks.