Towards a Deeper Understanding of the CEFR Principles in Grammar Competence Testing

This paper focuses on various perspectives and challenges language teachers face while assessing language learners’ proficiency level and relying on standardised placement tests in compliance with the CEFR. The aim of this study is to obtain a more critical insight into language testing in terms of the relationship between a test form and test score; to provide an interpretation of the CEFR-based proficiency levels; and to address the inclusion of the CEFR scheme in the real-life, academic language environment. Through a comparative, qualitative analysis, it examines the achievements of a group of respondents, undergraduate students majoring in English, recorded in a standardised pen-andpaper test focusing on their written language competence to determine their CEFR language proficiency levels. It also analyses errors, and compares the results obtained at different levels within the CEFR context, providing a theoretical background to the role and place grammar has in language teaching and learning. The main findings of the research point out the commonest grammatical errors found at the B2 and C1/2 levels, also revealing inconsistent and unexpected types which contribute to a deeper understanding of the complexities language assessment implies. This paper presents results which identify the areas that could be improved in terms of testing grammatical competence within the CEFR framework.


The CEFR and Grammar Assessment
The Common European Framework of Reference for Languages was published in 2001, and describes "learners' ability in terms of speaking, reading, listening and writing at six reference levels" (Cambridge ESOL, 2011, p. 6). The authors of the CEFR intended to create a framework, which could not only serve as help or as a guide for all parties involved in language education, but also as a 'tool' for various purposes.
The Framework points out the importance of various competencies language learners need to develop so that they can deal effectively with different communicative situations (Council of Europe, 2001, p. 9). In order to achieve this, the Framework defines an exhaustive list of competencies: general competencies (including declarative knowledge, skills, and know-how and ability to learn) and communicative language competencies (linguistic, sociolinguistic, and pragmatic). Linguistic competencies are mainly of primary concern for language teaching and learning, and they include: lexical, grammatical, semantic, phonological, orthographic, and orthoepic competencies. According to the Framework, the range of lexical and grammatical competences is rather detailed, including lexical elements, such as fixed expressions, phrasal idioms, fixed frames, and phrases, single words, etc., and grammatical (morphology, syntax, and semantics) (Council of Europe/ALTE, 2011).
What makes the CEFR a valuable document are the clearly stated, detailed descriptors and guidelines for a vertical progress through all language skills. However, even though grammatical competence is given attention in the Framework, it is defined rather broadly, and its authors do not provide descriptors which explain learners' grammatical competence for each reference level. Their attention is limited to a list of grammatical categories and parameters (Council of Europe, 2001, p. 114) and a rather general description of grammatical accuracy expected to be attained at each reference level. The status of grammar-related descriptors within the CEFR document has remained almost unchanged even within the supplementary document (Companion Volume with New Descriptors, from 2018), aimed at enriching the initial document content.
Despite the fact that the CEFR analyses various aspects of language education, it seems that testing and assessment have received most attention. However, the way the CEFR document refers to assessment has resulted in various interpretations and has given rise to rather opposing views, among which are those endeavouring to determine how to assess grammatical competence.
Defining grammatical knowledge has provoked much debate among linguists for decades, and it has yielded much discussion and research. Language teachers undeniably emphasise the importance of the role grammar plays in language learning and testing, and explicit grammar testing is still an integral component of internationally-recognised English language tests, such as IELTS, TOEFL, TOEIC, CAE, CFE, etc. (Bodrič, 2016, p. 163); however, what has changed considerably over the years are the aspects of grammar selected to be tested and the ways to test grammar (Purpura, 2005, p. 4).
As much as the CEFR is praised, there are a number of critics (Deygers, Gorp, & Demeester, 2018;Fulcher, 2012) who have a different view of the values of the CEFR, its application and, particularly, the testing and assessment that claim to be linked to it. One of the limitations refers to the lack of exactness and, consequently, the levels it prescribes cannot be taken as standards, and are of a mere descriptive nature (Deygers et al., 2018, p. 2). On the other hand, there is a completely opposite view, which criticises the Framework for having a negative impact on language learning due to the insistence on standardisation. There are also serious objections relating to the involvement of factors other than purely educational and professional, and they refer to the context which goes beyond language and learner-focused interest, making the CEFR language-independent (Harsch & Martin, 2012) and frequently very difficult for interpretation (Papageorgiou, 2010, as cited in Deygers et al., 2018. Furthermore, a number of theorists claim that one of the most prominent drawbacks of the CEFR is its lack of empirical nature (Fulcher, 2012), and that it also unequally addresses language skills (Staehr, 2008), and does not totally adhere to the principles arising from language acquisition theories (Alderson, 2007, as cited in, Deygers, Carlsen, Vilcu, & Zedler , 2017 2).

Research Methodology
The research includes the analysis of a corpus of tests taken by a group of 22 undergraduate students majoring in English, aged 19-20 years. The main purpose of this study was to test their grammatical competence at the end of their second year of studies so that the language teachers could use the results in order to determine their CEFR language proficiency level. The test was also meant to serve as a diagnostic tool which could help the teachers plan and organise lessons as regards teaching materials and further activities for the next academic year so as to make language learning the most effective for the students.
The test respondents took represents a standardised, pen-and-paper test -Oxford Placement Test 2 (Allan, 2006) -and it has been used as one of the instruments with the potential for identifying the CEFR levels. It contains 100 items belonging to the scope of grammar, all of which are presented in MC form (pick and choose option), and the allotted time for its completion is a maximum of 50 minutes (it is worth noting that the section of the Oxford Placement Test 2 pack involving Listening comprehension has not been used). The items are divided into six sections, all but the very last one being context-providing portions of language material -addressing different general knowledge topics. This includes multiple writing forms -there are descriptive, biographical parts, but also a letter format section. Although the mentioned sections of tasks are regarded (and named) as Grammar Part 1 and Grammar Part 2 -they are not arranged in a graded order -both parts contain more and less demanding items.
Test 2 also comes with a diagnostic Key and Levels chart -so it helped the authors of this study check the answers easily, grade the tests, and, at the same time, determine their students' reference levels. The grading scale served as a framework for the analysis of errors students made followed by a description and comparison of errors. These errors were classified into typical and less frequent, and the obtained results were used to provide a deeper insight into the nature of grammar assessment, including the analysis of the discriminatory value of certain items.

Analysis of Students' Performance in the Placement Test
According to the formal scale provided by the test designers, the students achieved the following scores: 9 students performed at the B2 level, and 13 of them were either at the C1 (7 students) or C2 levels (6 students).
The analysis of the results obtained from these tests, and the comparison of errors made at the same level showed that there are errors made by all students or the majority of them, which we could consider as typical for the specific level. Apart from these typical errors, we will also point out the errors made by some of the students at certain levels, which were taken into consideration in our analysis because they show some unexpected tendencies and could have indicative implications for language planning policies. Prior to illustrating them, it should be noted that the term "error", throughout this paper, is used to denote a lack of knowledge. And the paper does not introduce the differentiation between errors and mistakes.

Errors Made at the B2 Level
The analysis of test results, which were evaluated to be at the B2 level, shows that there is a pattern of repetition of the same errors in some grammatical categories in all test results, or the majority of them, which we refer to as typical errors. Other errors, though not that frequent, are also studied and analysed in this work for the purpose of further analysis and implied directions of investigation.
-Typical errors all students made at this level are: a. the use of questions tags with the adverbs "hardly" and "rarely" (e.g., He's rarely been away for this long before, has he?). b. the use of the gerund after prepositions and the form "be used to" (e.g., Children seem to find computers easy, but many adults aren't used to working with microtechnology.). c. the uses of the Perfect forms. The use of the Past Perfect was problematic for all students at this level. Half of them recognised its use in separate sentences which had either another past action or the prepositional phrase by + time (e.g. By 1956 he had joined Santos and had scored in his first game). However, all students showed a lack of knowledge of how and when to use the Past Perfect in context -within a text that demanded a deeper understanding of the sequences of actions. d. Similarly to the above-explained typical mistake, the use of other verb forms present in longer contexts was also problematic, especially if the sequence of tenses was present in sentences. None of the students at this level could properly decipher which tense is used in a longer text for which congruence the understanding of the sequence of tenses is needed. The analysis of their tests shows that they made rather illogical combinations, which illustrates a lack of understanding of the use of tenses in a broader context. Nobody answered correctly any of the sentences that contained verbs in tenses with a reference point in the past -for example: It was the first time that professional teams had played for a world title. -Typical errors the majority of students (over 70%) made at this level are the following: a. the use of non-finite verb forms: -the use of the bare infinitive after the verb make (in the sense of to force somebody to do something), e.g. His father made him practice every day.
-after phrasal and prepositional verbs, especially after the verb to look forward to.
-the difference between the use of the gerund and infinitive after the verb go on, e.g., Italy, who won, went on to win the 1938 final.
-the use of the gerund after it's no use. b. the use of verb forms: -be going to for past predictions, e.g., He thought he was going to be able to play in the finals in Sweden.
-the use of The Present Perfect after the clause It's the first time. c. the use of the relative pronoun who after a proper noun denoting a country, but in the sense of a team from that country (e.g., Italy, who won, …) d. the use of the Perfect Infinitive after modal verbs to refer to the past, e.g., It's the first time I've lived with anybody before, but I might have guessed what would happen.
e. the omission of the article the in front of most (meaning majority, not a part of the superlative form), e.g., Climate is important in most people's lives.
f. the use of the indefinite article after quite (but they all recognised its use after such), e.g., The history of the World Cup is quite a short one (they chose the answer where the noun phrase does not contain the indefinite article), g. the use of the causative have, e.g., ...then the authorities would have needed to have the original World Cup replaced.
h. the use of the active and passive forms in sentences. Students did not recognise whether the subject of these sentences is the doer of the action or the object of its active counterpart, e.g., In 1957 he was picked for the Brazilian national team.
i. the use of question tags in shortened forms, e.g., We'd better not delay reading this any longer, had we? At this level, the majority of students also made errors if a question tag was added to complex sentences, e.g., So you think he'll be back before November, do you?
j. would rather -at the level of recognition of the use of the bare infinitive after the form, e.g., Some people would just rather not have anything with computers at all.
-Other Errors Made at the B2 Level -some students made other errors that could not be described as typical for this level, but nevertheless occurred in more than 20% of analysed test results, and they are as follows: a. the use of the Present Perfect with the adverbs for and since, e.g., Football has been played for over a hundred years.
b. the omission of articles in front of abstract nouns, e.g., In Madeira they have good weather almost all year.
c. the use of quantifiers in negative sentences, thus, some students opted for answers such as There aren't no... e. the difference in meaning between little and a little, e.g., Even now there is little we can do to control the weather.
f. the use of the plural form of nouns, e.g., He was only ten years old (half of them opted for He was only ten years.) g. the use of the modal should after the verb suggest, e.g., She suggested we should share the house and share the costs.
h. the difference between as and like in simple examples, e.g., Learning a computer language is not the same as learning a real language.
h. the use of the infinitive of purpose, e.g., In hot countries people wear light clothes to keep cool.
i. the use of the Future Perfect Continuous, e.g., At the end of this month we'll have been living together for a year and a half.
j. the use of the Past Simple of the verb to be in the sentence He was born in 1940. Instead of was, some of them opted for the Present Simple form is.

Errors Made at the C1 Level
Even though the number of typical errors made at this level is smaller than at the B2 level, there are still noticeable patterns of errors students made at both C levels. The total number of students who performed at the C level is 13 and, according to the scale, 7 students scored at the C1 level, with the other 6 at the C2 level.
-Typical errors that all students made at this level are the following: a. the use of question tags after sentences which contain the adverbs hardly and rarely. b. the use of tenses within a text, with the focus on the sequence of tenses c. the use of the gerund after prepositions -The analysis showed that the majority of students made the following errors: a. the use of the relative pronoun who in the sentence Italy, who won... b. the use of question tags in complex sentences c. the use of the Perfect Infinitive after modals d. the use of a preposition + gerund after a noun, e.g., Everyone has difficulty in learning (they opted for the plural form of the noun and the infinitive). e. the use of the infinitive/gerund after the verb to go on.
-Other errors students made at this level Although the number of students who made other errors at this level is less than 50%, which could not be referred to as typical, the errors made in their tests are pointed out in this analysis because they belong to the lower level of grammatical difficulty, and therefore will be discussed later in this work. These errors are as follows: a. the use of the article the in front of most, when most refers to the majority of, and not a part of the analytical comparison (most people's lives) b. the use of the active and passive forms -at the level of recognition whether the sentence is active or passive, which influences the choice of verb c. the use of the gerund after the phrase it's no use d. the use of the modal should after the verb suggest e. the basic use of the Future Perfect Continuous

Errors Made at the C2 Level
The results showed that 6 students performed at the C2 level and, unlike with the previous levels, there is no example of an error that was recurrent in all students' tests. Still, there is a clear pattern of errors that could be found in the majority of these tests, and they are as follows: a. the use of tenses in a text, with the focus on the sequence of tenses b. the use of question tags after the sentences which contain the adverbs hardly and rarely c. the use of the gerund after prepositions d. the use of the Present Perfect after It's the first time e. the use of the article the in front of most, when most refers to "the majority of", and not a part of the analytical comparison (most people's lives) f. the use of the Past Perfect in a text g. the use of the infinitive/ gerund after the verb to go on.
Students also made some other mistakes at this level that occurred in more than one test, and are taken into consideration in this analysis because they could be significant for further studies and language teaching implications, which will be discussed in the nextsection. These errors are the following: a. the use of the active and passive forms -at the level of recognition whether the sentence is active or passive, which influences the choice of the verb b. the use of the gerund after the phrase it's no use c. the use of the infinitive to indicate purpose d. the use of the modal should after the verb suggest e. the basic use of the Future Perfect Continuous f. the use of the relative pronoun who in the sentence Italy, who won...
If we compare typical errors the students made in their placement test, we can notice that there is a clear pattern of their occurrence at all levels -from B2 to C2. These errors concern primarily the use of verb forms, the category in English which is the most developed and, at the same time, the most complex. They therefore pose difficulties for students in deciphering subtleties and various forms and meanings.
The comparative analysis of tests shows that the following grammatical aspects were the most problematic for students, and resulted in typical errors including: a. the use of tenses in a broader context b. the use of the perfect forms (present, past, and future) c. the use of question tags after the sentences which contain the adverbs hardly and rarely, and after sentences with the shortened form of auxiliary verbs. d. the use of non-finite verb forms. Students showed a lack of knowledge in terms of the use of the infinitive or gerund to complement other verbs/ nouns/ phrases. e. the use of articles when there are similar forms which go along or without any article. f. the use of the active or passive forms in contexts where students are asked to recognise whether the subject of the sentence is the doer of the action or is the object of the active sentence.
In this analysis, we will compare the typical errors students made at levels B2 to C2 with the descriptors in the CEFR. What is most striking is the consistency in errors the students made at the C levels. According to the CEFR document, the language user at the C2 level is proficient, and in terms of grammatical competence and accuracy "maintains consistent grammatical control of complex language, even while attention is otherwise engaged (e.g., in forward planning, in monitoring others' reactions)" (Council of Europe, 2001, p. 28). The language at the C1 level is also considered to be that of a proficient user who performs at a high level, and "consistently maintains a high degree of grammatical accuracy; errors are rare and difficult to spot" (Council of Europe, 2001, p. 28). According to the CEFR at the C1 level, a user also has "good grammatical control, occasional slips or non-systematic errors and minor flaws in sentence structure may still occur, but they are rare and can often be corrected in retrospect" (Council of Europe, 2001, p. 28).
The analysis of the typical errors the C1 or C2 learners made in this placement test shows that they occur in the sentences which illustrate grammatical categories, structures and uses taught at the lower levels, as presented in coursebooks and grammar books which rely on the principles and descriptors stated in the CEFR. The most common errors students made were in the two parts of the test that contained a broader context, or, as it is stated according to the CEFR document, language in use.
Another issue which arises here is: if students do not recognise a complex grammatical structure containing verb forms affected by the sequence of tenses, how can we expect them to use it effectively and correctly in communicative situations? According to the CEFR, if learners want to participate in communicative situations, they "draw upon a number of competences developed in the course of their previous experience" (Council of Europe, 2001, p. 101). The qualitative aspects of spoken language use describe a C1 language user as someone who "consistently maintains a high degree of grammatical accuracy, errors are rare" (Council of Europe, 2001, p. 114), while a C2 proficient language user "maintains consistent grammatical control of complex language" (Council of Europe, 2001, p. 114). The findings obtained through this study do not comply with these statements since there are instances of lack of consistency of grammatical control.
The errors students made at the C1 and C2 levels do not support the explanation of propositional precision of how well leaners need to perform at these levels, i.e., "convey finer shades of meaning precisely by using, with reasonable accuracy, a wide range of qualifying devices", or "give emphasis, differentiate and eliminate ambiguity" (Council of Europe, 2001: p. 129).
The CEFR describes B2, C1 and C2 language users as independent language users. According to the CEFR descriptors for grammatical accuracy, the B2 learner "shows a relatively high degree of grammatical control" (Council of Europe, 2001, p. 114). A B2 learner also "does not make errors which lead to misunderstanding" (Council of Europe, 2001, p. 114). However, the findings shown in the section describing the errors made at the B2 level are not congruent with this claim, because these types of errors were recurrent in many test results at this level.
What also strikes as incongruous with regard to the CEFR descriptors is the interference of the students' mother tongue, which occurs in several instances at all levels, although "noticeable mother tongue influence" (Council of Europe, 2001, p. 114) is referred to only at B1 and lower language levels. Recurrent errors illustrating this interference are as follows: He is born in 1940. (Instead of He was born); There aren't no easy ways (Instead of There are no easy ways), It's no use to try (Instead of It's no use trying), In some countries is dark all the time (Instead of In some countries it is dark all the time), He was only ten years (Instead of He was only ten), and many examples referring to the sequence of tenses.
The analysis of errors and comparison of their occurrence at all of these three levels reveals another oddity relating to the application of the CEFR principles to the interpretation of the test results. Namely, the following is stated in the CEFR document: "Each level should be taken to subsume the levels below it on the scale" (Council of Europe, 2001, p. 36), and it is assumed that a language user whose competences are assessed to belong to a certain level is expected 'to be able to do "whatever is stated" at the lower level' (Council of Europe, 2001, p. 36).
However, the analysis of the results obtained from the corpus composed of students' placement tests shows that this principle is violated in some cases, and that students of lower language levels performed better at some grammatical categories and aspects than others whose final score was higher. The findings show that some items from the placement test have a lower discrimination index, and these examples are as follows: a. the use of quantifiers -differences between a little/ little, the use of no with negative verbs ('there aren't no' and the adverb hardly ('hardly no') -made by the C1 and C2 students, and not described as typical for the B2 level b. the use of used to for past habits (none of the students made this mistake at the B2 level) c. the comparative form of the adjective warm ('more warm' -an error none of the students made at B2 level) d. the error in the use of between/ among was made only at the C2 level e. the use of the bare infinitive after would rather was made by two students at C2 level, and nobody made this mistake at the C1 level.

Pedagogical Implications
The typical errors described for each reference level can be used as a resource to support teachers while creating materials and organising grammar instruction. The findings of this study also indicate that we cannot rely only on the test results alone -we need to employ other criteria and elements in grouping our students and curriculum planning -if possible, to test both their receptive and productive skills. All these decisions should be based on the aim of learning the language. This placement test, as well as the grading scale, can be helpful, but not necessarily the only source of information relevant for creating various language policies.
The results obtained from this analysis also demonstrate that language teachers should not assume anything -even if students' language competence is assessed to be at a proficient level they still make banal, unexpected, and even illogical errors, but these still exist, and should be taken into consideration.
The analysis of test results according to which the grammatical knowledge and performance of students were classified at either the B2 or C 1/ 2 levels shows that the errors made at these levels can be very similar, and belong to the same grammatical categories. The difference in the number of points which classified their performance at either the B or C levels can sometimes be attributed to sporadic examples, isolated categories, and do not contribute to a clear referential line. Therefore, in order to determine what direction of improvement to take to help students consolidate their knowledge and competencies at an assigned level and confidently move up to the higher level, all parties involved in the creation of language policies, institutions, language teachers and learners, test and material designers should be aware of these complexities, and be ready and flexible to adapt to their students' abilities and needs.

Conclusion
In this paper, we analysed and compared the results students achieved in a standardised placement test in accordance with the CEFR principles. The analysis of the most common errors students made and the interpretation of the results point to the complexity of the nature of testing and assessment, and multidirectional aspects both language teachers and learners need to be aware of.
A deeper understanding of the CEFR principles will help a language instructor assess the test that he/she plans to use. These tests can be viewed as a helpful, supportive tool, and a starting point that provides guidance to both language teachers and learners. However, language teachers need to have a critical eye while using formal tests in order to analyse the applicability of these tests in their teaching contexts, and not to take them for granted.
The findings of this study show that the CEFR grammar competence descriptions are applicable in the analysis of many of the students' test results. However, the discrepancies shown above also support the interpretation that the CEFR should be used as a guide (Council of Europe, 2001), and that other aspects should be included in the language descriptions. As Schneider and Lenz point out, apart from the scaled descriptions, other descriptors can be useful (Schnider & Lenz 2019, p. 41-49), especially in contexts with specific groups of learners who have specific learning goals. The results obtained from this study point out the weakness in applying grammatical knowledge, and areas that can be improved at the levels B2, C1 and C2.
The reseach described in this paper contributes to the complex area of language testing with its methodology of intersecting and comparing results obtained at different levels, revealing what grammatical items need further attention in test designing. These results also showed that for more comprehensive and far-reaching conclusions about the nature of testing, a wider corpus of both tests and respondents needs to be incorporated in some further investigation on this subject.