Empirical Study on Spreadsheet Quality: Case of Serbian SMEs

This paper presents the results of a research which involved analysis of spreadsheets collected from 40 Serbian SMEs. Based on related research, the collected spreadsheets were analysed for presence of errors and assessed in terms of quality criteria, defined in line with the literature review. The final results are consistent with related research and point to various aspects of spreadsheet use that should be improved by the respondents for the purpose of reducing risk associated with development and use of spreadsheets.


Introduction
Spreadsheets are amongst most frequently used commercial software by a majority of organizations. Records, reports, charts, various analyses used in operations and decision making are most likely to be developed or adapted using a spreadsheet software. The fact that export into Excel files has become a standard feature of business software has only contributed to the popularity of spreadsheets.
Owing to a fast learning curve of wide range of functionality, an average user quickly becomes adept at using spreadsheets. However, due to their ease of use, careless users are usually unaware of error-proneness of different spreadsheet applications.
The records on negative consequences of inappropriate development and use of spreadsheets have grown immensely over the last years ("The European Spreadsheet Risks Interest Group," n.d.).
This paper provides an overview of a subset of results from a more comprehensive research on spreadsheet use. Operational spreadsheets gathered form Serbian SMEs were analysed with the intent to estimate the degree to which they comply with the defined quality criteria and test the claim prevalent in related research on the discrepancy between the actual proportion of spreadsheets containing errors and the end users' estimates of this proportion. After a literature review, based primarily on a previous research by Raković (2014), the paper continues with research results, followed by concluding remarks.

Literature review
The proportion of human error is similar among different areas of human activities. Panko (2007) presented the data on the degree of accuracy of mechanical, simple tasks, as well as more complex actions in text editing, programming and spreadsheet development (Table 1) comes to mechanical actions (data entry or cell selection), the degree of accuracy of text, program code and spreadsheets ranges between 99.5% to 99.8%. However, with formulas (that is, more complex actions), this degree drops by several percent. In Table 1, presenting Panko's results, complex thought is signified with C, whereas a greater number of complex thoughts is designated as "the product", and signified with C n . At the level of a more complicated document, program, worksheet or workbook, the degree of accuracy declines to a 0%. In other words, more complex texts, programmes and spreadsheets will, most certainly, contain errors. Source: Panko, 2007 In a 2013 paper, Panko and Port (R. R. Panko & Port, 2013) systematised the results of studies concerned with spreadsheet errors conducted after 1995 they deemed most significant. Their results are presented in Table 2. With the intent to achieve a greater quality of description of errors present in spreadsheets, Panko and Halverson (R. Panko & Halverson, 1996) proposed new indicators to complement the Percentage of Spreadsheet Models Containing Errors-Number of Errors per Model and Error Magnitude (indicates how the error affects the model outputs). However, the proposed indicators did not become widely accepted, which is why the authors later introduced the term Cell Error Rate (CER), based on a programming concept of Fault per thousand lines of (noncomment) source code (fault/KLOC). Almost 15 years after its introduction, Panko and Aurigemma (2010) realised that the term was used ambiguously among researchers, which rendered comparison impossible. While some researchers compared the number of errors with a total number of cells containing formulas, others used the total number of filled-in cells as the denominator. Therefore, Panko and Aurigemma (2010)   Source: Panko & Aurigemma, 2010 In the research concerned with the percentage of erroneous cells revived by R. Panko (2005), CERF ranged from 4.3% to 21.0%, while CERV ranged from 1.1% to 11.9%.
The risk of spreadsheet errors is closely related to their creators' overconfidence, which is a manifestation of the gap between their selfperceived and actual knowledge and skills (M. Grant, D. Malloy, & C. Murphy, 2009;Takaki, 2005). Inadvertence of errors was corroborated by results of several studies (Caulkins, Morrison, & Weidemann, 2008;R. Panko, 1998). Panko (2009) reports that errors were discovered in spreadsheets developed by 86% respondents, whereas only 18% of respondents expected their spreadsheets to contain errors.
Reaction in occurrence of errors and minimisation of other risks associated with the use of spreadsheets is the primary focus of frameworks for spreadsheet development and use. Based on research on spreadsheet errors, inferred guidelines, positive experiences, and best practices in spreadsheet development (Baker, Powell, Lawson, & Foster-Johnson, 2006;Bewig, 2005;Colver, 2010;Dunn, 2010;Ferreira & Visser, 2012;Grossman & Özlük, 2004;Kulesz & Zitzelsberger, 2012;Powell, Baker, & Lawson, 2008b, 2008aPowell, Baker, Lawson, McDaid, & Rust, 2009;Read & Batson, 1999;Weber, 2006), as well own research results (Rakovic, 2014), Rakovic (Raković, 2014; 2019) developed a framework for spreadsheet development and use (Figure 1). Guidelines for the design, implementation and documentation phases proposed by the framework were used as a basis for defining criteria for assessment of collected spreadsheets, results of which are presented in the next chapter. With its particular emphasis on the significance of spreadsheet documentation, the aforementioned framework suggests that each spreadsheet contains an additional worksheet titled "Documentation", with the following information:

Research results
A total of 40 spreadsheets were collected from micro, small and medium Serbian businesses. The collected spreadsheets were not specially developed for this research, but rather operational, used by the respondents in their everyday tasks. The spreadsheets were not trivial, but at a sufficient level of complexity to be considered a Product (C n ), referenced in Table 1.
Each respondent also submitted a questionnaire which, among other things, served to gather the information whether the respondent expected errors to be discovered in their spreadsheets. 90% of respondents expected their spreadsheets to be free of errors.
The following criteria for spreadsheet assessment were defined, in line with the framework proposed by Raković (2014) Analysis of "physical" characteristics of collected spreadsheets indicated that the majority of workbooks contained no more than 3 worksheets (Figure 2), up to 1000 filled-in cells (Figure 3), and up to 100 unique formulas ( Figure  4).   The proportion of compliance with criterial 1 through 18, expressed in percents, is presented in Figure 5. Note: Compliance with criteria Crit 5 and Crit 13 could be tested on only 5 spreadsheets, while compliance with Crit12, Crit14 i Crit15 could be tested on 25, 23, and 14 spreadsheets, respectively.
Formula arguments were positioned above and/or left from the formula in all analysed spreadsheets (Crit 17). Criteria with a high degree of compliance also included Crit 8-Presence of table heading rows (97.5%) and Crit 16-Dependent formulas positioned as close as possible (90%). Ranges were named in three out of five spreadsheets they were used in (Crit 5, 60%).
None of the analysed spreadsheets contained any form of documentation.
85% of analysed spreadsheets contained errors, with the total number of errors varying from 1 to 3650 (original and copied errors). The following errors were detected: use of constants in formulas, references to a non-existent cell, division by zero, use of text as formula argument, errors in VLOOKUP function arguments, and references to workbooks not supplied. The proportion of erroneous cells to all nonempty cells (CERA) was 2.18%, to cells containing numbers and formulas (CERV) was 3.12%, while the proportion of erroneous cells to a total of cells containing formulas (CERF) was 7.25%. It was not possible to compute CERN and CERT.

Conclusion
The research results show similarity with a number of related studies. The proportion of spreadsheets containing errors in total analysed spreadsheets (85%) is close to the average value among related researches (84%), while computed values of CERF (7.25%) and CERV (3.12%) are lean towards the lower brackets of reference ranges (CERF: 4.3%-21%, CERV: 1.1%-11.9%).
According to questionnaire data, only 10% of the respondents acknowledged the possibility of their spreadsheets containing errors. The discrepancy between their estimates and the actual number of spreadsheets containing errors (85%) points to overconfidence among the respondents.
Assessment of quality of analysed spreadsheets clearly suggests that frameworks and best practices are not employed to guide correct development and to ease the use of spreadsheets, and in turn reduce spreadsheet-related risks. On average, analysed spreadsheets received positive STRATEGIC MANAGEMENT, Vol. 24 (2019), No. 2, pp. 058-064 assessment in only 4 out of 18 criteria. The greatest reason for concern is the prevalence of negative assessments in relation to criteria directly linked to spreadsheet errors and spreadsheetrelated risks: non-use of data validation, complex formulas, use of constants in formulas, unprotected formulas, worksheets, workbooks, etc.SM