Abstract
Large language models (LLMs) have been touted to enable increased
productivity in many areas of today’s work life. Scientific research as
an area of work is no exception: the potential of LLM-based tools to
assist in the daily work of scientists has become a highly discussed
topic across disciplines. However, we are only at the very onset of this
subject of study. It is still unclear how the potential of LLMs will
materialise in research practice. With this study, we give first
empirical evidence on the use of LLMs in the research process. We have
investigated a set of use cases for LLM-based tools in scientific
research, and conducted a first study to assess to which degree current
tools are helpful. In this paper we report specifically on use cases
related to software engineering; specifically, on generating application
code and developing scripts for data analytics and visualisation. While
we studied seemingly simple use cases, results across tools differ
significantly. Our results highlight the promise of LLM-based tools in
general, yet we also observe various issues, particularly regarding the
integrity of the output these tools provide.