Benchmark use of libosrm over osrm-routed

[vroom-project]

Benchmark use of libosrm over osrm-routed

[jcoupey]

First results using 50 instances for each size among 20, 50, 100, 250, 500, 750, 1000 and 1500 jobs. Only the cumulative timings (in ms) for actions that are somehow related to OSRM calls are reported (solving time is omitted).

Size	osrm-routed			libosrm			Diff (%)
	L	R	L + R	L	R	L + R	L	R	L + R
20	987	779	1766	1916	1114	3030	+94.12	+43.00	+71.57
50	2438	1507	3945	3712	2081	5793	+52.26	+38.09	+46.84
100	5271	2400	7671	5818	3241	9059	+10.38	+35.04	+18.09
250	15553	4630	20183	12956	5915	18871	-16.70	+27.75	-6.50
500	41607	7791	49398	26961	9452	36413	-35.20	+21.32	-26.29
750	78246	10678	88924	42864	12458	55322	-45.22	+16.67	-37.79
1000	124068	13340	137408	60573	15320	75893	-51.18	+14.84	-44.77
1500	248006	18468	266474	105432	20528	125960	-57.49	+11.15	-52.73

L stands for loading, which includes setting the table request and firing it but also other things that differ:

• with libosrm, creating osrm::EngineConfig and osrm::OSRM objects and going through the matrix using osrm::json::Arrays;
• with osrm-routed, parsing the request result with rapidjson and going through the matrix using rapidjson objects.

R stands for routing, which includes setting the route request and firing it but also other things that differ:

• with libosrm, turning the (huge) osrm::json::String to a rapidjson::Value for output;
• with osrm-routed, parsing the request result with rapidjson and moving the route geometry.

In case anyone wants to reproduce the experiment:

• build OSRM at tag v5.4.1.
• build VROOM at 7def96a
• use this France extract
• generate benchmark files from the setup script at https://github.com/VROOM-Project/vroom-scripts/tree/master/benchmark/libosrm.

[jcoupey]

Quick comments.

• the gain in the L phase that includes the table request is the most significant when instances size grow, which makes sense as the size of the data being serialized within osrm-routed and then parsed with rapidjson is O(n^2).
• I suspect the difference in the R phase doesn't reflect the libosrm and osrm-routed performance as it is probably very much impacted by passing around the route geometry at https://github.com/VROOM-Project/vroom/blob/develop/src/loaders/libosrm_loader.h#L141-L144