Split docker actions based on visualization target

[matsonj]

Split docker actions based on visualization target

[matsonj]

@gregwdata I am getting the following error when running make docker-run-evidence. Do you mind taking a look at this?

cd analyze && mkdir -p data_catalog
cp -r /workspaces/nba-monte-carlo/data/data_catalog/* /workspaces/nba-monte-carlo/analyze/data_catalog
cp: cannot stat '/workspaces/nba-monte-carlo/data/data_catalog/*': No such file or directory
make[1]: Leaving directory '/workspaces/nba-monte-carlo'
make[1]: *** [Makefile:43: evidence-build] Error 1
make: *** [Makefile:50: evidence-visuals] Error 2
make: *** [Makefile:54: docker-run-evidence] Error 2

[matsonj]

Interestingly it looks like the files created in make docker-run-pipeline are not available in subsequent docker run. Which means I think I can fix it.

[gregwdata]

Yeah, that’s the “ephemeral” nature of containers for you. Two approaches to consider are - daisy-chain all the commands in one docker run command; or mount a folder on the host with -v flag so the pipeline results from the container are persisted to the host, then mount that folder into the evidence-running container.

…

On Wed, Dec 21, 2022 at 1:36 PM Jacob Matson ***@***.***> wrote: Interestingly it looks like the files created in make docker-run-pipeline are not available in subsequent docker run. Which means I think I can fix it. — Reply to this email directly, view it on GitHub <#57 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AS7ZCGIWNYC72RJZQWUY5RDWONEZ7ANCNFSM6AAAAAATF5AQDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[matsonj]

good tip on the -v flag, I might split up with that.

I can get the entire thing to run with this make script:

docker-evidence-test:
		docker run \
		--publish 3000:3000 \
	 	--env MELTANO_CLI_LOG_LEVEL=WARNING \
		--env MDS_SCENARIOS=10000 \
		--env MDS_INCLUDE_ACTUALS=true \
		--env MDS_LATEST_RATINGS=true \
		--env MDS_ENABLE_EXPORT=true \
		--env ENVIRONMENT=docker \
		mdsbox make pipeline evidence-visuals

However - now I get a new error, which has to do with running npm run dev -- --host inside a devcontainer inside docker.

Error: getaddrinfo ENOTFOUND true
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)
make[1]: Leaving directory '/workspaces/nba-monte-carlo'

I think we have progressed to an error that is basically "networking is hard" which is fine, but also annoying.

[gregwdata]

If you're still stuck at that error, I was able to get past it after a bit of trial and error. Looks like the --host flag expects the host IP after it. So running npm run dev -- --host 0.0.0.0 in the container had it serving up an Evidence page. The page (and terminal otuput) threw a bunch of Missing Database Credentials errors, as the next piece of the puzzle to figure out.

BTW another Docker debugging tip that was helpful in rapid iteration, is to access the shell of a running container with
docker exec -it containername /bin/bash, where containername is from the NAMES column of docker ps output.
Or starting a fresh container with docker run -it -p 3000:3000 mdsbox /bin/bash with whatever other --env flags you need to throw in.

[matsonj]

Ok really good tip on opening the terminal - i was trying to do that and couldn’t figure it out (googling wrong terms, but seems super obvious in hindsight).

Anyway, now getting this error after I rebased with master.

/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /workspaces/nba-monte-carlo/analyze/node_modules/duckdb/lib/binding/duckdb.node)
Error: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /workspaces/nba-monte-carlo/analyze/node_modules/duckdb/lib/binding/duckdb.node)
    at Module._extensions..node (node:internal/modules/cjs/loader:1302:18)
    at Module.load (node:internal/modules/cjs/loader:1081:32)
    at Module._load (node:internal/modules/cjs/loader:922:12)
    at Module.require (node:internal/modules/cjs/loader:1105:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/workspaces/nba-monte-carlo/analyze/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Module._compile (node:internal/modules/cjs/loader:1218:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)

It looks like my docker image might be missing GLIBC_2.29, so I’m going to see if I can install it and have another go!

[matsonj]

Ended up using a different docker image which had what I needed:

45cfb03