A new version of this entry is available:
Loading...
Article
2024
Guidelines for using financial incentives in software-engineering experimentation
Guidelines for using financial incentives in software-engineering experimentation
Abstract (English)
Context: Empirical studies with human participants (e.g., controlled experiments) are established methods in Software Engineering (SE) research to understand developers’ activities or the pros and cons of a technique, tool, or practice. Various guidelines and recommendations on designing and conducting different types of empirical studies in SE exist. However, the use of financial incentives (i.e., paying participants to compensate for their effort and improve the validity of a study) is rarely mentioned
Objective: In this article, we analyze and discuss the use of financial incentives for human-oriented SE experimentation to derive corresponding guidelines and recommendations for researchers. Specifically, we propose how to extend the current state-of-the-art and provide a better understanding of when and how to incentivize.
Method: We captured the state-of-the-art in SE by performing a Systematic Literature Review (SLR) involving 105 publications from six conferences and five journals published in 2020 and 2021. Then, we conducted an interdisciplinary analysis based on guidelines from experimental economics and behavioral psychology, two disciplines that research and use financial incentives.
Results: Our results show that financial incentives are sparsely used in SE experimentation, mostly as completion fees. Especially performance-based and task-related financial incentives (i.e., payoff functions) are not used, even though we identified studies for which the validity may benefit from tailored payoff functions. To tackle this issue, we contribute an overview of how experiments in SE may benefit from financial incentivisation, a guideline for deciding on their use, and 11 recommendations on how to design them.
Conclusions: We hope that our contributions get incorporated into standards (e.g., the ACM SIGSOFT Empirical Standards), helping researchers understand whether the use of financial incentives is useful for their experiments and how to define a suitable incentivisation strategy.
File is subject to an embargo until
This is a correction to:
A correction to this entry is available:
This is a new version of:
Other version
Notes
Publication license
Publication series
Published in
Empirical software engineering, 29 (2024), 5, 135.
https://doi.org/10.1007/s10664-024-10517-w.
ISSN: 1573-7616
ISSN: 1382-3256
New York : Springer US
Other version
Faculty
Institute
Examination date
Supervisor
Cite this publication
Krüger, J., Çalıklı, G., Bershadskyy, D., Otto, S., Zabel, S., & Heyer, R. (2024). Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering, 29(5). https://doi.org/10.1007/s10664-024-10517-w
Edition / version
Citation
DOI
ISSN
ISBN
Language
English
Publisher
Publisher place
Classification (DDC)
000 Computer science, information and general works
Original object
University bibliography
Standardized keywords (GND)
Sustainable Development Goals
BibTeX
@article{Krüger2024,
doi = {10.1007/s10664-024-10517-w},
author = {Krüger, Jacob and Çalıklı, Gül and Bershadskyy, Dmitri et al.},
title = {Guidelines for using financial incentives in software-engineering experimentation},
journal = {Empirical software engineering},
year = {2024},
volume = {29},
number = {5},
}