The effect of web-based classroom response system on students learning outcomes: Results from programming course

The purpose of this study is to examine the effect of using active learning techniques and its ability to increase learning outcomes and students̀ attitudes towards active engagement during programming lectures. The primary objective of the research presented in this study was to compare the learning outcomes resulting from the use of two active learning techniques: web-based classroom response system (CRS) and class discussion. In this study, a mixed method research was used. For the purpose of quantitative research, a pre-test/post-test method was used to assess learning outcomes and anonymous survey to evaluate students’ attitudes toward their active engagement. To obtain the qualitative data, five open-ended questions, for each group (experimental and control group) were included in the survey. The conclusions indicate that the use of web-based CRS and class discussion respectively increase learning outcomes, facilitate learning programming and positively affect students attitude toward programming course. However, the group which used web-based CRS had better learning outcomes than the group which had a class discussion, and also the students’ attitude toward web-based CRS is more affirmative compared to class discussion.


Introduction
Teaching programming has been facing numerous dilemmas since it was introduced to high schools.From deciding which programming language and which platform is optimal for learning, as well as what is the quantity of information that the students can acquire to enable them to solve practical problems through the forms of programming languages. 1,2In previous research on pedagogical practices in programming education, researchers find that the traditional way of teaching is no longer essential to students learning (in programming courses) and suggest the implementation of a teaching approach that involves delivering theoretical and practical content together in the lecture environment. 3,4he rapid growth of information and communication technologies (ICT) has influenced every aspect of our life.Contemporary ICT components like Smartphone, broadband Internet, and Wi-Fi networks have changed the way we seek information and learn.In order to keep up with the progress, schools have begun to invest in new technologies.Nowadays, ICT equipment, such as various types of computers, projectors, smart board etc., represents an essential part of a 21 stcentury classroom 5 .The integration of new technologies in the classrooms opens new possibilities for the teaching and learning process 6 and provides educational institutions an opportunity to increase student enthusiasm and enhance learning outcomes 7 .Technologies such as classroom response system (CRS) are getting more popular among teachers due to their effects on student learning performance.The traditional way of questioning in class allows only a limited number of students to answer a question keeping in that way interactivity between teachers and students at low-level.In that case, all those shy and less motivated students stay inactive.Therefore, teachers do not have a proper feedback during the lecture.A CRS is an instructional technology that opens up opportunities for all students to be involved in the activity in class and learning process through digital dialogue.
In the previous studies, CRS was compared only with traditional lecturing method regarding learning outcomes and students' attitudes towards their usage.For this reason, the primary objective of this research was to compare learning outcomes resulting from the use of two active learning methods: webbased CRS and class discussion and to examine students` attitudes towards their active engagement during programming lectures.For testing the significance of the obtained results, the t-test was used.

Background and Literature Review
"I hear, and I forget.I see, and I remember.I do, and I understand.
(Asian proverb)" According to Keyser 8 , every teaching method that makes students actively involved represents the process of active learning.These techniques put the learner in the center of the learning process in such a way that he is no longer in a passive receiving position but an active factor of the learning activity who directs his own learning in the learning process.The main role of the teacher in a process of active learning is to facilitate learning and to be in the position of learner along with his student 9 .There are many active learning techniques, such as cooperative learning, problem-solving exercises, class discussions, casestudy methods, computer-aided instructions, etc.The techniques that will be used depending on the level, objectives, and subject of the course 10 .
Class discussion is one approach to active learning.By implementing class discussion the teacher is able to listen to students' responses; ask students to elaborate on their thinking by providing explanations, evidence, or clarifications; suggest additional probing questions like: "What makes you think that?Or Please, give an example from your experience", etc.; invite others to react and respond to ideas by providing alternative viewpoints, agreements or disagreements.The students involved in class discussion gain knowledge in a meaning-ful manner.A dialogue with a classmate is a useful way to motivate students to learn, participate and exchange ideas with instructors and classmates 11 Another active learning technique with growing popularity among teachers in colleges and universities are classroom response systems (CRS) or "clickers".According to previous research on this topics, CRS has several benefits over traditional teaching.It can improve interactivity in the classroom 12 , support collaborative learning 13 , enhance student learning 14 , and promote the acquisition of advanced reasoning skill 15 , help students to better understand the lecture 16 , increase student engagement and classroom interaction 17 , assess student learning efficiency on an ongoing basis, facilitate peer learning and help students recognize gaps in their own learning 18 .
The rapid growth of ICT influence the replacement of electronic devices i.e. clickers with smartphones and other personal mobile devices and promote the use of web-based CRS.Integration of mobile devices in the class could lead to enhancement of individual and group learning outcomes 19 and can enhance students learning experience when using mobile devices for educational purpose 20 Knowledge and skills in programming are one of the in-demand professions in the 21 st -century global economy.That is why so many young adults recognize programming as a promising profession.Learning to program is difficult and programming courses often have the highest dropout rate 21 .In some cases, the difficulty with learning programming comes from the inability of the teachers to teach these skills to the learners using only abstract concepts. 22asically, there are two main approaches to teaching programming.One approach is oriented towards problem-solving, considering programming to be an application of skills in problem-solving.Another approach is to base learning of programming in the context of code and system development 23 .There is no single answer which approach is better because different learners have different learning styles and needs 24 .According to Shulman 25 : "There are no single most powerful forms of representation, the teacher must have at hand a veritable armamentarium of alternative forms of representation, some of which derive from research whereas others originate in the wisdom of practice".
Many research was done in this field in the past couple of decades, but there is still no consensus on what is the most effective way to teach programming.Most of the schools are using traditional way to teach programming which consists of lectures, take-home assignments and perhaps demo sessions where model solutions to the exercises are shown 26 .A review of the literature concerning courses in programming shows that among traditional methods of delivery of instruction, the lecture is more dominant in use compared to other forms of instruction.In the traditional way of teaching programming, learners are passive listeners.To overcome this problem educational institutions combine lectures and laboratory exercise in order to build learners skill-based knowledge.However, there are limitations of lecture-type instruction that have been reported in several studies.These limitations include challenges in students' motivation, knowledge assimilation, difficulties in sustaining attention and concentration of learners, and adaptability of instruction.Laurillard 27 states that learners should be given the opportunity to articulate their understanding and to construct the knowledge based on how the learner sees and understands the concept.In order to improve the classic lecture, interactivity between participants is a must.
Teaching techniques affect the teaching 28 .Incorporating new technology in process of teaching programming can increase teacher efficiency in the classroom and open up qualitatively different modes of classroom interaction 29 .Many types of research have been done in order to show the positive effect of using technologies in the teaching process and encouraging students to actively participate in process of learning 30 .One of the most frequently used technologies is computer-based presentations.However, they have several limitations and one of the major is that its use could easily turn learning process into teacher-centered instead of student-centered.To overcome this problem many researchers suggested using additional technologies i.e.Computer Presenter 31 , Smartboard or interactive whiteboard 32 33 34 and CRS or Clickers 35 36

Research problem
The main purpose of introductory programming course in high schools is to provide learners with basic knowledge of programming terminology and understanding of its usage in coding a program.Furthermore, such course includes an understanding of algorithms, data, programming codes and static concepts.For the beginners (learners with little or no previous knowledge), some concepts may be difficult to understand.Instead of continually applying the learned concepts in practical exercises, many beginners only do the actual practical tasks when the first formal assessment is required.In such situation the finer details of the concepts could be forgotten, resulting in their inability to successfully complete the assignment.Bed results could then lead to disappointment and disaffection with the course.
However, the lecture is still the main method of delivering theories and new concepts in programming courses mostly because of efficient use of resources, fewer costs and access to the largest number of students.Many studies have criticized the traditional lecture format as it lacks interactivity, does not fully encourage active learning and does not take into account individual needs 37 38 .Other major problems include poor attendance and students' perception of lectures as boring, irrelevant and time-consuming 39 .
Active learning approach takes into consideration the ability of the learners to acquire their knowledge through interaction.Furthermore, these techniques include exploring personal attitudes and values, engaging the student in critical thinking, and encouraging student participation through giving and providing feedback 40 .These strategies also encourage students to reflect on their experiences.Researchers have found active strategies to be more effective for retention of knowledge and student engagement 41 .
The question, "Do the active learning techniques used during the programming lectures affect students' success and attitudes towards the course?"constitutes the problem addressed in this study, with its sub-problems listed as follows: (i) Is there a difference with respect to students success between the experimental and control groups' regarding the use of active learning techniques?
(ii) Is there a difference in experimental and control group attitudes toward their active engagement in programming lectures?(iii) What are the opinions of an experimental and control group of students following the intervention process (i.e.implementation of adequate active learning method)?

Design
A mixed method was used in this study to assess the use of active learning techniques during the class of programming.A mixed method design is an approach that combines both quantitative and qualitative research methods to understand a research problem.Its use provides comparative results and increases the validity of the study 42 .For the purpose of quantitative research, a pretest-posttest method was used to assess learning outcomes and anonymous survey to evaluate students' attitude towards their active engagement in programming lectures and overall satisfaction with the course.To obtain the qualitative data, five open-ended questions, for each group (experimental and control group) were included in the anonymous survey.Before data collecting, both the experimental and control groups were formed.Students and instructor in experimental group used free web-based CRS -Socrative, while the control group used class discussion as an active learning technique.Teaching materials were prepared in advance according to each active learning approach.The learning objectives and task were communicated to both groups, the experimental group (EG) and the control group (CG).In the experimental group, the teacher prepared mini quizzes using web-based CRS that was used 15 minutes after the lecture started.In control group teacher used the same set of questions upon which he initiated class discussion.During the research phase, comparisons were made between the data obtained from the experimental and control groups, before and after the implementation in order to assess learning outcomes.
The research was designed during the winter term of the 2015-2016 academic year.During that phase all necessarily measurement tools and parameters of research were determined.Data collection was performed at "Nikola Tesla" vocational school from Nis, Serbia, during the spring term of the 2015-2016 academic year and it lasted for two weeks.The students were informed of the intervention at the start of the spring term, but without revealing the purpose of the study.The sample includes students from two classes of the third grade enrolled to the programming in C# course.The students from one class were indicated as the experimental group (n=26), and the students from the second class (n=24) were indicated Patton, Q. M. (2015): Qualitative evaluation and research methods (4th Ed.).London, UK: Sage as a control group.Since an experimental design was conducted in the current study, no universe and sample group was created.Both classes were taught in the same semester, used the same textbooks, additional learning materials and had the same teacher.Learning outcomes were measured by taking the score on a test that was performed immediately after the lectures.
Both classes met once per a week for 90 minutes for the course lecture and question sessions.Each student in EG group has a smartphone with Android operating system.Web-based CRS -Socrative (www.socrative.com) was used to collect students' responses (Matthew, 2012).Socrative is free software which only requires an internet connection and smartphones, and no additional equipment and expenses like clickers.Before the lecture, each student installed Socrative Student application from Google Play, on their smartphones.Students in both classes attended the same lessons.During the first week, the topic of lectures was: "Defining and Application of Statistic Functions in C#".During the second week, the topic of lectures was: "Returning Parameters to the Main Program and Functions Side Effects".In each class, the teacher gives a lecture on selected topic and after the 15 minutes, he poses 10 questions related to the topic.Students in EG group uses Smartphone to give an answer to each question.Then teacher downloads all students` answers and reviews them.After that, he forms several groups of students whose answers were different and let them discuss their answers in the group for approximately 15 minutes.After that, the students answer individually the same questions again using web-based CRS but this time they had a chance to change their minds because they have listened to the arguments in the groups.After finishing the test the teacher collect their answers as posttest results.Finally, the teacher provides answers to the questions and resolves the doubts if they still exist.At the end of the lecture, students took a final test.
In CG group after the 15 minutes of the lecture the teacher gave the students a pre-test and asked them to write down the answers.After that, the teacher collected the pretest and initialized class discussion based on students answers.In CG group during the class discussion teacher listened to students responses, asking them to elaborate on their thinking by providing additional explanations, and trying to include as many students as possible in the dialogue by addressing additional probing questions to less active students.After that, he gave them the same test.As in EG, students in CG also had a chance to change their answers.After finishing the test the teacher collected their answers as posttest results and provided answers to the questions and resolved the doubts if they still existed.

Data collecting and Processing
Data collection process lasted for two weeks.In this study, two data collection methods were used to collect the quantitative data.The pre-test/post-test method was used to evaluate the effect of implemented active learning tech-niques on students learning outcomes, and the anonymous survey was used to evaluate students' attitudes toward active learning techniques that have been used during the class of programming.In the survey, five open-ended questions, for each group (EG and CG) was included to collect qualitative data.The questions were: Considering the last 2 weeks: (i) what frequency would you recommend the professor to use the web-based CRS (for EG) i.e. class discussion (for CG) (always, more than now, with the same frequency, sometimes, but less than now, never.Explain)?(ii) Have you felt more involved in this lectures by using web-based CRS (for EG) / class discussion (for CG)? (iii) Do you participate in collaboration with peer/class discussion?(iv) Which were the obstacles that you had using web-based CRS/ during class discussion?(v) Overall, please rate your experience of this course compared to the previous way of teaching?
The same test was used for pretest and posttest.It consisted of various types of questions ranging from multiple choice questions, true/false to short answers.An example of these questions is shown in figure 2. To measure overall test reliability a Kuder-Richardson formula (KR-20) was used.For final test KR-20 value was 0.78, its average level of difficulty was found to be 0.41, and its average distinctiveness was found to be 0.38.
In order to evaluate students' attitudes toward active learning techniques that have been used during the class of programming, a survey was created.A survey used five Likert-type questions (five-point scale) and five open-ended questions for qualitative research.

Findings
To evaluate the effect of using the selected active learning technique on learning outcomes in the classroom, learning outcomes were assessed, before and after the implementation of the active learning technique.Pretest scores from both groups were subjected to statistical analysis to ensure that both groups are comparable in their pre-class knowledge.The unpaired t-test was used to compare pre-test scores.The overall pretest score averages at the end of the second week, for the experimental group (who were using web-based CRS) and control group (who were using class discussion), are presented in Table 1.No statistical difference was found (p = 0.73).The great percentage of students from EG group agreed that web-based CRS helped them to become active class participants (91%), be aware of the level of their knowledge (94%) and allowed them a better understanding of lecture (74%).This technique also stimulated communication with a classmate (91%) and facilitated learning of programming (78%).
Comparing to EG, 54% of students in CG agreed that class discussion helped them to become active class participants, 76% became aware of the level of their knowledge, 54% think that class discussion allowed them to better understand the lecture; 58% of students in CG think that class discussion stimulated communication with a classmate while 54% of students think that class discussion facilitated learning of programming.
For qualitative research, five open-ended questions were included in the survey.Student opinion regarding the first question: "What frequency would you recommend the professor to use the web-based CRS (for EG) i.e. class discussion (for CG) (always, more than now, with the same frequency, sometimes, but less than now, never.Explain)?"To assess the overall attitude towards the frequency of using proposed active learning technique, comments were read and each student answer was categorized.The results are shown in Table 4.As is shown in Figure 3, students have an affirmative opinion toward active learning techniques since none of them, in both groups, didn`t write down the answer: never.Students in EG recommended the greater frequency of use than the students in CG.From these answers, it can be concluded that positive effect upon using active learning technique occurred when students communicated actively to help one another to understand, get immediate feedback from the teacher, and be able to answer the question anonymously.

Discussion
According to the findings, when each active learning technique was introduced, the programming course became more attractive to students.Students learning outcomes were increased in both groups, but the greater increase is evident in the experimental group, where students used web-based CRS than in control group, where students used class discussion.
As for students' attitudes towards the active learning technique used, there is a significant difference in students' attitudes in experimental and control group.Students that used web-based CRS found themselves more actively involved, more aware of their knowledge and better understood the lecture than the students that used class discussion.Furthermore, according to students' opinion, web-based CRS engaged them in collaboration more than class discussion.
Regarding the third sub-problem of the study, students have very affirmative opinions of using active learning technique in comparison with the traditional way of teaching.The written comments revealed that students in both groups agreed that active learning techniques increased their attention to the lecture, enhanced interaction during the learning process and provided students immediate, real-time feedback.
Students were also found to believe that collaboration with peers is important because it can broaden their personal understanding and enable them to be more active.But there is a significant difference between these two active learning technique, web-based CRS and class discussion in the context of students' active participation.Percentages of students in control group, actively involved in the class discussion are less than of those in the experimental group.This leads to the conclusion that class discussion didn't manage to involve all students in the discussion and those shy, unsure of the correct answer or less motivated students and students who take more time to compose an answer stayed resistant.On the other hand, CRS enables all students to give their answers to questions independently and anonymously.As a result, a student that otherwise wouldn't be involved in learning becomes an active participant 43 .This is consistent with the findings of Coca and Slisko 44 , who found that web-based CRS and smartphones are feasible tools that can facilitate active learning in the classroom and increases the level of student participation.It can be concluded that students that used web-based CRS appreciated the ability to stay anonymous, not to expose themselves and not to be humiliated in case of giving a wrong answer.So it was easier for them to express their thoughts and generally to participate in the learning process.From this point of view, it is reasonable to conclude that web-based CRS generates students` more affirmative attitudes towards their active engagement in programming lectures than class discussion.But, on the other hand, this students' behavior is quite opposite to their behavior on social networks like Facebook, Twitter, etc., where they constantly share their attitudes, opinions, and thoughts.It seems that it is easier for them to share personal information than opinions about a learning subject in the classroom.There are opportunities here for further research into the reasons why students feel free to exchange their attitudes, opinions and thought in the social media sphere rather than in the classroom.
Despite the desire to stay anonymous, students in an experimental group like working in a group.They tend to remember lectures that require interaction with peers 45 .Biggs and Tang 46 confirmed that dialogue and argument are valuable learning opportunities.However, Herrmann 47 suggests that working in a group may rather affect the students' behavior without affecting their motive to learn.On the other hand, Coca and Slisko found that learning in a group affects not only students' learning behaviors but also affects students' motives to learn, which are essential to the quality of the students' approaches to learning.
Compared to the class discussion, the web-based CRS increased student enjoyment during the lecture.Students' find that technique amusing, funny, looking like a game and its implementation during lectures motivated them to attend the class.According to Stowell and Nelson 48 , it might not be the experience of enjoyment (or any other emotion) that mediates the benefits of web-based CRS, but rather the enhanced cognitive processing (attention) associated with it.
The written comments also suggest that some students prefer the teacher to evoke the correct answer rather than be engaged in conversation and confront their own misconceptions with those of peers.It seems that for all those students pointing out to misunderstandings hardly made any sense.They expect more feedforward from a teacher in order to understand the complex topic in programming.According to Dulamă and Ilovan 49 , students need their professor's feedforward to improve their learning process and increase the quality of education.Some limitations should be considered when interpreting the study's results.One limitation here is the size of the classes in the sample.The sample size in this study was 50 students from two classes.There is a requirement to repeat this research in larger groups.A second limitation of the study is a short period of two weeks.Future work should explore attitudes of students who have used webbased CRS and class discussion over a longer period of time (over, 1 or 2 years) to see whether these have changed over time.
This study supports researchers cited earlier in this article, in suggesting that both web-based CRS and class discussion can be a very useful tool in delivering pedagogical outcomes, such as classroom engagement, involvement, participation, and enjoyment.From the students' perspectives, web-based CRS has some added value since it enables students to stay anonymous so there is no humiliation in getting a question wrong.It is possible that more positive attitudes towards web-based CRS than to class discussion could lead not only to students' greater involvement, but also to better learning outcomes for students, but this has not been firmly established by research.While this active learning technique is useful, some degree of caution is required, because teachers' primary concern is to enhance the student's development through effective learning.

Conclusion
With an increasing awareness that many students are passive during programming teaching sessions, needs for teaching methods that allow students to become actively involved have increased.Active learning techniques are widely promoted as methods by which students may become more involved in class.This paper focuses on testing the influence of two active learning techniques web-based CRS and class discussion on students learning outcomes and students` attitudes towards their active engagement in programming lectures.The results have shown that after introducing these active learning techniques students' learning outcomes increase, but better learning outcomes were recorded in the group that used web-based CRS than class discussion.
Students welcomed the idea of becoming more actively engaged in lecture, because the understanding of lecture increases, they become more aware of their knowledge since they get immediate, real-time feedback.Students agreed that these active learning techniques facilitate learning of programming, but their attitude toward web-based CRS are more affirmative compared to the class discussion since this technologies evoke positive emotions and enable students to answer to questions without exposing themselves.

Table 1 -
Pretest success scoresResults shown in Table1indicate that pretest score averages of both the experimental and control groups are close to each other, and there is no significant difference between the two groups' pretest scores (p>0,05).So, both groups are equal with respect to pretest scores.At the end of each lecture, both groups had a final test.The unpaired t-test was used to compare overall posttest scores at the end of the second week.The posttest score averages of the experimental group and of the control group are presented in Table2.

Table 3 -
Students attitude scores towards used active learning technique