ark-kun/google_cloud_mlflow

Experimental MLflow plugin for Google Cloud Vertex AI

google_cloud_mlflow's Issues

Model deployed in default us-central1 not using location config

Config file specifies 'us-east1' however model is uploaded to 'us-central1' and endpoint created in 'us-east1' causing a deployment error.

---

deploy_name = f"{model_name}-{model_version}"
dest_image_uri = f"us.gcr.io/databricks-vertex-endpoint/mlflow/{deploy_name}"
 
config = dict(  destination_image_uri = dest_image_uri,
                location = "us-east1",
                description = "MLFlow connector used to deploy",
                # wrong_param = None,
                min_replica_count = 1,
                max_replica_count = 2
             )
 
 vtx_client = mlflow.deployments.get_deploy_client("google_cloud")
 
deployment = vtx_client.create_deployment(
    name = deploy_name,
    model_uri = model_uri,
    config = config
)

Creating Model
Create Model backing LRO: projects/697856052963/locations/us-central1/models/8949131846654885888/operations/2064981311973490688
Model created. Resource name: projects/697856052963/locations/us-central1/models/8949131846654885888
To use this Model in another session:
model = aiplatform.Model('projects/697856052963/locations/us-central1/models/8949131846654885888')
Creating Endpoint
Create Endpoint backing LRO: projects/697856052963/locations/us-east1/endpoints/3315036353837662208/operations/2206342704458104832
Endpoint created. Resource name: projects/697856052963/locations/us-east1/endpoints/3315036353837662208
To use this Endpoint in another session:
endpoint = aiplatform.Endpoint('projects/697856052963/locations/us-east1/endpoints/3315036353837662208')
Deploying Model projects/697856052963/locations/us-central1/models/8949131846654885888 to Endpoint : projects/697856052963/locations/us-east1/endpoints/3315036353837662208
NotFound: 404 Model `projects/697856052963/locations/us-central1/models/8949131846654885888` is not found.

Plans to support MLFlow 2.x?

With MLFlow having released their big 2.x upgrade, this tool no longer properly installs. Any intention of upgrading the tool for MLFlow 2.x

For context, 2.x was first released last October:
https://github.com/mlflow/mlflow/releases/

_build_serving_image doesn't actually pushes the built image

While reading the source of this project, came across this block of code in the _build_serving_image function. Due to that return in the first line of the quoted segment, the rest of the code (that should do the pushing the image to the registry) is unreachable. I presume it's not intended?

google_cloud_mlflow/google_cloud_mlflow/_mlflow_model_gcp_deployment_utils.py

Lines 207 to 222 in 014e3ba

	return destination_image_uri
	_logger.info("Uploading image to Google Container Registry")
	client = docker.from_env()
	result = client.images.push(destination_image_uri, stream=True, decode=True)
	for line in result:
	# Docker client doesn't catch auth errors, so we have to do it
	# ourselves. See https://github.com/docker/docker-py/issues/1772
	if "errorDetail" in line:
	raise docker.errors.APIError(line["errorDetail"]["message"])
	if "status" in line:
	_logger.debug(line["status"])
	container_image = client.images.get(destination_image_uri)
	pushed_image_uri_with_digest = container_image.attrs["RepoDigests"][0]
	_logger.info("Uploaded image: %s", pushed_image_uri_with_digest)
	return pushed_image_uri_with_digest

Update to client definition to include 'deployments'

Update

FROM
client = mlflow.get_deploy_client("google_cloud")

TO
client=mlflow.deployments.get_deploy_client("google_cloud")

DockerException when running inside Databricks ManageMLFlow cluster

• Cluster: 9.0 ML (Spark 3.1.2, Scala 2.12)
• GCP Service Account has proper permissions
• Client successful set via client=mlflow.deployments.get_deploy_client("google_cloud")
• Model successfully trained and model registered using tutorial here https://docs.databricks.com/applications/mlflow/quick-start-python.html#automatically-log-training-runs-to-mlflow

Error on this command

deployment = client.create_deployment(
    name="mlflow_on_gcp",
    model_uri=model_uri) 

Error Details

INFO:google_cloud_mlflow._mlflow_model_gcp_deployment_utils:Project not set. Using <project> as project
INFO:google_cloud_mlflow._mlflow_model_gcp_deployment_utils:Destination image URI not set. Building and uploading image to gcr.io/<project/image>
INFO:google_cloud_mlflow._mlflow_model_gcp_deployment_utils:Building image
2021/08/31 03:57:03 INFO mlflow.models.cli: Selected backend for flavor 'python_function'
2021/08/31 03:57:05 INFO mlflow.models.docker_utils: Building docker image with name gcr.io/<project/image>
DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) ```