Connection error when creating a dataset

[jitendrawbd]

Connection error when creating a dataset

[ohaibbq]

Can you post the docker run configuration you are using when starting the emulator as well as any logs from the container? Are you mapping a host port to the container port?

We are using a similar setup and can create datasets just fine.

python
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import bigquery
from google.auth.credentials import AnonymousCredentials
from google.api_core.client_options import ClientOptions


client_options = ClientOptions(api_endpoint="http://0.0.0.0:9050")
client = bigquery.Client(
"local",
client_options=client_options,
credentials=AnonymousCredentials(),
)
dataset = bigquery.Dataset('local.test_dataset')
dataset = client.create_dataset(dataset)
print(dataset)
>>> from google.auth.credentials import AnonymousCredentials
>>> from google.api_core.client_options import ClientOptions
>>>
>>>
>>> client_options = ClientOptions(api_endpoint="http://0.0.0.0:9050")
>>> client = bigquery.Client(
... "local",
... client_options=client_options,
... credentials=AnonymousCredentials(),
... )
>>> dataset = bigquery.Dataset('local.test_dataset')
>>> dataset = client.create_dataset(dataset)
>>> print(dataset)
Dataset(DatasetReference('local', 'test_dataset'))

docker run -p 9050:9050 --rm ghcr.io/goccy/bigquery-emulator:latest /bin/bigquery-emulator --project=local --log-level=info
[bigquery-emulator] REST server listening at 0.0.0.0:9050
[bigquery-emulator] gRPC server listening at 0.0.0.0:9060
2024-03-28T17:41:34.742Z	INFO	server/middleware.go:53	POST /bigquery/v2/projects/local/datasets	{"query": "prettyPrint=false"}

[jitendrawbd]

Thanks for the response. I am using the below docker run command
docker run -it ghcr.io/goccy/bigquery-emulator:latest --project=local --log-level=info

The only log I see in the container is below

[bigquery-emulator] REST server listening at 0.0.0.0:9050
[bigquery-emulator] gRPC server listening at 0.0.0.0:9060

The error I get is below

[ohaibbq]

You'll need to map a Docker host port to the emulator container port via docker run -p 9050:9050 when running the emulator in order for the client to access it on the local 0.0.0.0 network.

[jitendrawbd]

Thanks for the response! I did try with port mapping too. Getting the same error
docker run -p 9050:9050 ghcr.io/goccy/bigquery-emulator:latest --project=local --log-level=info

P.S.: I am using python 3.12.2 version

[ohaibbq]

Are you running your pytest command locally or in a separate Docker container? If you are running it in another docker container you may need to run it in the same Docker network.

I'd recommend going over this page and have a look at the example they lay out with redis-server and redis-cli.

https://docs.docker.com/network/#container-networks

[jitendrawbd]

I am running the pytest locally

[ohaibbq]

The reproduction case you provided does not fully encompass what might be going wrong as it works just fine for me.

Are you able to use port mapping successfully with other containers? Perhaps Docker doesn't have permission to mount ports? Does the bq tool work when the emulator is running?

bq --api "http://0.0.0.0:9050" query --nouse_legacy_sql --format prettyjson --project_id local 'SELECT TRUE;'

[jitendrawbd]

Not sure why, but I had to use http://localhost:9050 instead of http://0.0.0.0:9050 to get this working. I used all the above docker commands with connection to localhost to get it working. Thanks for the help!