1. Introduction
AWS Lambda, the leading Function-as-a-Service (FaaS) platform, prices execution as the product of allocated memory and execution time [1]; the platform also scales the available CPU capacity proportionally to the memory setting [2], so that both execution time and per-invocation cost depend non-linearly on a single configuration parameter. The memory value that minimises a chosen cost function for a particular workload cannot be derived analytically and must be identified by empirical measurement on the target platform. Such measurements are however obscured by the right-skewed distribution of execution times, garbage-collector pauses and the hardware heterogeneity of a multi-tenant environment [3].
Established benchmarking toolkits (SeBS [4], ServiBench [5]) rely on samples of 100–500 invocations per configuration — appropriate for profiling, but prohibitive when the invocation budget is a binding constraint, as is typical for automated configuration-search tasks. Tools targeting configuration tuning (AWS Lambda Power Tuning [6], COSE [7]) focus on the search algorithm and leave the statistical procedure for evaluating a single configuration unspecified. The objective of the present work is to define and justify a measurement protocol that yields a statistically reliable estimate of the central tendency of execution time at the smallest defensible sample size.
2. Problem Statement
Let an FaaS workload be parameterised by a single configuration variable m denoting allocated memory. The observed execution time d deviates from the unobservable true value τ(m) by an additive noise term ε whose distribution F(·) is unknown and, as documented in [3], pronouncedly right-skewed:
d = τ(m) + ε, ε ~ F(·).
The task is to construct an estimator τ̂(m) from a sample of k independent invocations that is simultaneously resistant to outliers from the long right tail of F, parsimonious with respect to k, and reproducible across independent experimental runs on identical configurations.
3. The Proposed Protocol
The protocol consists of three complementary components addressing distinct sources of measurement bias.
3.1. JIT pre-execution. Prior to the workload of interest, the function executes a short arithmetic loop of 10 000 iterations. This reliably triggers the optimising JIT compiler of the runtime and ensures that the measured fragment runs in fully optimised code, eliminating a variability component unrelated to the configuration parameter under study.
3.2. First-invocation discard. Any invocation following a configuration change is served by a freshly initialised execution container; the resulting cold-start overhead may exceed the steady-state execution time by an order of magnitude. The first invocation is therefore discarded, and measurement begins with the second, which is guaranteed to execute in a warm container.
3.3. Median-based aggregation. A further k = 10 warm invocations are performed and the sample median τ̂(m) = median(d₁, ..., d₁₀) is taken as the estimator. The median is preferred over the arithmetic mean because of its 50 % breakdown point: the estimator is unaffected by up to four extreme observations within a sample of size ten, which is structurally sufficient to absorb the right-tail outliers reported in [3]. The choice k = 10 is a deliberate compromise: smaller samples expose the estimator to individual outliers, while larger ones grow the invocation budget proportionally and improve precision only as 1/√k. The total cost of evaluating one configuration is eleven invocations — one to two orders of magnitude below [4, 5], which makes the protocol suitable as a building block for iterative configuration-search algorithms.
4. Conclusions
A sample-efficient measurement protocol for FaaS performance evaluation has been proposed. The combination of JIT pre-execution, first-invocation discard and median aggregation over k=10 warm invocations removes the two systematic asymmetric components of FaaS latency measurement (cold-start overhead and JIT-tier transient) and yields an estimator that is robust to right-tail noise while consuming only eleven invocations per configuration. The protocol is intended for sample-constrained estimation of the central tendency of execution time within automated configuration-search procedures; profiling tasks that require characterisation of distributional tails fall outside its applicability and remain better served by conventional protocols [4, 5]. Empirical validation on production FaaS workloads and integration into a search-based configuration framework constitute the next stage of the authors’ work.
References
1. Eismann S., Scheuner J., van Eyk E., Schwinger M., Grohmann J., Herbst N., Abad C. L., Iosup A. Serverless Applications: Why, When, and How? IEEE Software. 2021. Vol. 38, No. 1. P. 32–39. DOI: https://doi.org/10.1109/MS.2020.3023302.
2. Configuring Lambda function memory : AWS Lambda Developer Guide. Amazon Web Services. URL: https://docs.aws.amazon.com/lambda/latest/dg/configuration-memory.html (accessed: 19.05.2026).
3. Wang L., Li M., Zhang Y., Ristenpart T., Swift M. Peeking Behind the Curtains of Serverless Platforms. Proceedings of the 2018 USENIX Annual Technical Conference (USENIX ATC ’18). Boston, MA, USA : USENIX Association, 2018. P. 133–146.
4. Copik M., Kwasniewski G., Besta M., Podstawski M., Hoefler T. SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing. Proceedings of the 22nd International Middleware Conference (Middleware ’21). Québec city, Canada : ACM, 2021. P. 64–78. DOI: https://doi.org/10.1145/3464298.3476133.
5. Scheuner J., Eismann S., Talluri S., van Eyk E., Abad C. L., Leitner P., Iosup A. Let’s Trace It: Fine-Grained Serverless Benchmarking using Synchronous and Asynchronous Orchestrated Applications : preprint. arXiv. 2022. arXiv:2205.07696. URL: https://arxiv.org/abs/2205.07696 (accessed: 19.05.2026).
6. Casalboni A. AWS Lambda Power Tuning : open-source state machine tool. GitHub repository. 2024. URL: https://github.com/alexcasalboni/aws-lambda-power-tuning (accessed: 19.05.2026).
7. Akhtar N., Raza A., Ishakian V., Matta I. COSE: Configuring Serverless Functions using Statistical Learning. IEEE INFOCOM 2020 — IEEE Conference on Computer Communications. Toronto, ON, Canada : IEEE, 2020. P. 129–138. DOI: https://doi.org/10.1109/INFOCOM41043.2020.9155363.
|
Голосування 
Ця робота ліцензується відповідно до Creative Commons Attribution 4.0 International License
Знайшли помилку? Виділіть помилковий текст мишкою і натисніть Ctrl + Enter