[Serve] Allow Serve apps to not be exposed over HTTP

[JoshKarpel]

[Serve] Allow Serve apps to not be exposed over HTTP

[edoakes]

@edoakes question: above I've only described how it would work for serve.run. I haven't looked at the code yet, but I assume that it'll need be to allowed in more places, like from a static Serve config file?

We can do the same in the config file:

   route_prefix: null

[JoshKarpel]

@edoakes question: above I've only described how it would work for serve.run. I haven't looked at the code yet, but I assume that it'll need be to allowed in more places, like from a static Serve config file?

[JoshKarpel]

Great news! As far as I can tell, this actually already works, and is sort-of documented here:

ray/python/ray/serve/schema.py

Lines 979 to 981 in e4cb2fe

	"and app information has not yet fully propagated in the backend; or "
	"if the user explicitly set the prefix to `None`, so the application isn't "
	"exposed over HTTP. Routing is done based on longest-prefix match, so if "

And indeed, passing route_prefix=None to serve.run or setting route_prefix: null in a Serve config file does the right thing:

import time

import requests
from ray import serve

@serve.deployment()
class D:
    async def __call__(self, *args, **kwargs):
        return "Hello, world!"


d = D.bind()

if __name__ == "__main__":
    serve.run(d, route_prefix=None)
    time.sleep(5)
    print(requests.get("http://localhost:8000/-/routes").json())  # {} 
    print('sleeping...')
    time.sleep(100000)

proxy_location: EveryNode

http_options:
  host: 0.0.0.0
  port: 8000

applications:
  - name: d
    route_prefix: null
    import_path: serve:d

It looks like the join point between the controller state and what the proxy ends up seeing is

ray/python/ray/serve/_private/application_state.py

Lines 344 to 361 in 7e315fe

	if deployment_info.route_prefix is not None:
	config = deployment_info.deployment_config
	self._endpoint_state.update_endpoint(
	deployment_id,
	# The current meaning of the "is_cross_language" field is ambiguous.
	# We will work on optimizing and removing this field in the future.
	# Instead of using the "is_cross_language" field, we will directly
	# compare if the replica is Python, which will assist the Python
	# router in determining if the replica invocation is a cross-language
	# operation.
	EndpointInfo(
	route=deployment_info.route_prefix,
	app_is_cross_language=config.deployment_language
	!= DeploymentLanguage.PYTHON,
	),
	)
	else:
	self._endpoint_state.delete_endpoint(deployment_id)

, and it implicitly handles deployments without route_prefixes, not caring whether they are ingress deployments or not.

Perhaps this was done for gRPC support? So that gRPC ingress didn't also imply an HTTP ingress route?

ray/doc/source/serve/advanced-guides/grpc-guide.md

Lines 141 to 142 in b3b79d2

	`route_prefix` is still a required field as of Ray 2.7.0 due to a shared code path with
	HTTP. Future releases will make it optional for gRPC.

It's even tested!

ray/python/ray/serve/tests/test_http_routes.py

Lines 74 to 86 in b074a2f

	def test_deployment_without_route(serve_instance):
	@serve.deployment(route_prefix=None)
	class D:
	def __call__(self, *args):
	return "1"
	serve.run(D.bind(), route_prefix=None)
	routes = requests.get("http://localhost:8000/-/routes").json()
	assert len(routes) == 0
	# make sure the deployment is not exposed under the default route
	r = requests.get("http://localhost:8000/")
	assert r.status_code == 404

The main thing that's wrong seems to be the typing of the route_prefix argument - I'll take care of that, and poke around for any docs adjacent to the route_prefix that should discuss this.

[edoakes]

Great news! As far as I can tell, this actually already works, and is sort-of documented here:

ray/python/ray/serve/schema.py

Lines 979 to 981 in e4cb2fe

	"and app information has not yet fully propagated in the backend; or "
	"if the user explicitly set the prefix to `None`, so the application isn't "
	"exposed over HTTP. Routing is done based on longest-prefix match, so if "

And indeed, passing route_prefix=None to serve.run or setting route_prefix: null in a Serve config file does the right thing:

import time

import requests
from ray import serve

@serve.deployment()
class D:
    async def __call__(self, *args, **kwargs):
        return "Hello, world!"


d = D.bind()

if __name__ == "__main__":
    serve.run(d, route_prefix=None)
    time.sleep(5)
    print(requests.get("http://localhost:8000/-/routes").json())  # {} 
    print('sleeping...')
    time.sleep(100000)

proxy_location: EveryNode

http_options:
  host: 0.0.0.0
  port: 8000

applications:
  - name: d
    route_prefix: null
    import_path: serve:d

It looks like the join point between the controller state and what the proxy ends up seeing is

ray/python/ray/serve/_private/application_state.py

Lines 344 to 361 in 7e315fe

	if deployment_info.route_prefix is not None:
	config = deployment_info.deployment_config
	self._endpoint_state.update_endpoint(
	deployment_id,
	# The current meaning of the "is_cross_language" field is ambiguous.
	# We will work on optimizing and removing this field in the future.
	# Instead of using the "is_cross_language" field, we will directly
	# compare if the replica is Python, which will assist the Python
	# router in determining if the replica invocation is a cross-language
	# operation.
	EndpointInfo(
	route=deployment_info.route_prefix,
	app_is_cross_language=config.deployment_language
	!= DeploymentLanguage.PYTHON,
	),
	)
	else:
	self._endpoint_state.delete_endpoint(deployment_id)

, and it implicitly handles deployments without route_prefixes, not caring whether they are ingress deployments for not.
Perhaps this was done for gRPC support? So that gRPC ingress didn't also imply an HTTP ingress route?

ray/doc/source/serve/advanced-guides/grpc-guide.md

Lines 141 to 142 in b3b79d2

	`route_prefix` is still a required field as of Ray 2.7.0 due to a shared code path with
	HTTP. Future releases will make it optional for gRPC.

It's even tested!

ray/python/ray/serve/tests/test_http_routes.py

Lines 74 to 86 in b074a2f

	def test_deployment_without_route(serve_instance):
	@serve.deployment(route_prefix=None)
	class D:
	def __call__(self, *args):
	return "1"
	serve.run(D.bind(), route_prefix=None)
	routes = requests.get("http://localhost:8000/-/routes").json()
	assert len(routes) == 0
	# make sure the deployment is not exposed under the default route
	r = requests.get("http://localhost:8000/")
	assert r.status_code == 404

The main thing that's wrong seems to be the typing of the route_prefix argument - I'll take care of that, and poke around for any docs adjacent to the route_prefix that should discuss this.

Ah yes I believe you're right, I vaguely recall a conversation with @GeneDer about this when he was adding gRPC support :)

Let's clean up the API ref and document this then!