wenjinfeng/tbs

Characterizing Commodity Serverless Computing Platforms

Our paper: https://onlinelibrary.wiley.com/doi/10.1002/smr.2394

1. In qualitative analysis, we explore and summarize the various characteristics described in the official documentation of these serverless computing platforms. These characteristics specify the inherent restrictions of different aspects involving developing, deploying, and executing functions, which may result in fatal failures if developers do not comply with these restrictions. To this end, we construct a taxonomy with respect to the need-to-consider information from three aspects,i.e., development, deployment, and runtime. Such a taxonomy can help developers better understand the supported characteristics of serverless computing platforms to facilitate further development practice.

2. In quantitative analysis, we explore the actual runtime performance of these serverless computing platforms from multiple dimensions, in order to help developers select an appropriate platform based on their application features and improve applications' performance with tuned configurations. Indeed, the overall perceived performance of a serverless application may mainly be influenced by three kinds of latency, i.e., the startup latency of initiating the function instance, the execution latency of running the function, and the scheduling latency of waiting for serving by available instances when the number of requests dramatically increases.

2.1 First, we quantitatively analyze how can programming languages, memory sizes, and package sizes influence the startup latency on different serverless platforms. Startup latency severely affects the responsiveness of serverless applications and may limit the adoption of serverless computing under various applications. (The related code is in the folder "startup latency and resource efficiency/src/main/java/cn/edu/pku/".)

Python, Node.js, Java functions with different memory sizes

Python functions with 128 MB of memory under the various number of third-party packages (Pillow, Numpy, OpenCV)

2.2 Second, we quantitatively measure the applications' actual runtime performance to compare the underlying resource efficiency of different serverless platforms with a set of well-designed benchmarks. We categorize these benchmarks into two types,i.e., microbenchmarks and macrobenchmarks. Microbenchmarks consist of a set of simple workloads focusing on specific resource consumption, such as CPU, memory, network, disk IO, etc. Macrobenchmarks consist of a set of real-world representative applications, e.g., multimedia data process, MapReduce, machine-learning-based serving, which need to utilize various system resources. (The related code is in the folder "startup latency and resource efficiency/src/main/java/cn/edu/pku/".)

performance for microbenchmarks, e.g., CPU-bound workloads, memory-bound workloads, and diskIO-bound workloads

performance for macrobenchmarks, e.g., video processing, machine learning serving

2.3 Finally, we quantitatively compare the concurrency performance of different serverless computing platforms, i.e., how they perform when dealing with multiple requests due to different auto-scaling features and inherent concurrency limits. A coming request may be throttled if no available function instances can handle it, which results in non-negligible latency. Meanwhile, we try to analyze the potential causes influencing concurrency performance by analyzing and inferring their scalability strategy and load balancing from a black-box perspective. (The related code is in the folder "concurrency performance" for four serverless computing platforms.)

We create 20 python-based serverless functions with the same configuration and code but different function names [f1, f2, ..., f20] and invoke each fi with 10 x i concurrent requests. Meanwhile, we also explore concurrency performance under various memory sizes.

We use the runtime information (e.g., VMs and function instances) collected by our collection function in TBS, in order to analyze potential causes. Finally, we suppose request scheduling strategies for these serverless computing platforms. More details can be found in our paper.