8discussion.tex

\chapter{Discussion}
\label{chapter:discussion}

This chapter reviews the results as a whole and further analyses the findings compared with the literature. The theoretical implications are discussed in Section~\ref{section:theoretical}, while practical implications are covered by Section~\ref{section:implications} with the added perspective from the instructor covered by Section~\ref{section:instructor}. Section~\ref{section:limitations} discusses the limitations of the evaluation process.


\section{Reflection on Research}
\label{section:theoretical}

In the autumn of 2015, the retention rate in our course was relatively good, with 137 students out of 170 ($\approx 81\%$) completing the course. We cannot attribute the retention rate to the successful design of the exercises although the assignments discussed in this thesis have received good feedback during the course. When students were asked to evaluate the interestingness of the assignments on a scale from one (poor) to six (awesome), the rock-paper-scissors assignment got 4,12 (n=151); the tic-tac-toe assignment 4,48 (n=148); the sound filtering assignment 4,47 (n=148); and the image filtering assignment 4,21 (n=145).

The course design and the effort we put in attending to our students, so that they begin working on the assignments on time and turn them in before the deadline, also has significant effect on the retention rate. In order to pass the course, students are expected to (1) successfully complete all assignments, some of which could be replaced by an exam this year, (2) attend the problem based learning sessions for at least the second and the third assignment round, and (3) return two essays; one at the beginning of the course and one at the end of the course. Most of the students who did not complete the course successfully, either did not start the course at all or dropped out during or after the second assignment round. Some students who failed the course could not meet the requirement of attending the mandatory PBL sessions. When a student is committed to taking the course, she usually finishes it successfully.

Our choice for the use of games and media part of the learning content we want to teach our students. The course is not a prerequisite for any other course, but takes a side-track familiarising students with media computation. Other important learning outcomes of the course include project management, social, and presentation skills. The course does not have a final exam, and we did not measure how well our students actually master these skills. However, the assignments are designed so that students should meet enough of the learning outcomes to pass them successfully.

In the final feedback for the course that, the students are asked to evaluate if their interest toward computer science has increased on a scale from one (completely disagree) to four (completely agree). One hundred and three students gave final feedback at the end of the course in 2015, they gave the average evaluation of 2,82; which was slightly lower compared with the previous year (3,07; n=88). While the decline in the evaluation this year is disheartening, we do not actually know what effect the course had on students and should not be too discouraged about the result as the literature suggests that curriculum will most likely have no effect on students' choice for minor~\cite{guzdial:2013}. Furthermore, the students evaluated that the things they learned during the course will probably be useful for them in the future with grade 3,98 on a scale from one to five and that the course was useful with grade 3,12 on a scale from one to four.

To conclude, this thesis did not contribute to the research on student motivation, because the focus was on the technical implementation and trial of the system. From the perspective of the research on programming environments and web-based programming environments in particular, the thesis presented a unique implementation of a modern system that provides a managed execution environment for students' exercises. The produced GUIs were modern as well as supported interaction and media manipulation. The students enjoyed using the system, and were impressed that their code was doing "something real". Furthermore, the system that is tailored for Scala programming course is a novel one, because Scala is yet not a very common choice for CS1 courses.


\section{Implications of the Results}
\label{section:implications}

Clearly, EDCAT's biggest disadvantages compared with a local trial GUI were with performance times and runtime error messages. Especially the speed constitutes a problem of such scale that it should be solved before the system is considered for further use. In the present trial configuration of EDCAT, the system was run on a virtual server with shared CPU and disk resources. This meant that JVM, as well as \emph{sbt} needed to be started for the compilation of each submission and disk input and output operations were not at their optimal speed.

Regardless of server configuration improvements, the software should be modified so that it supports updates for a time estimate and position in the queue while the code is compiled. This way the page could be automatically refreshed once the compilation is completed. Additionally, limiting the number of submissions per student on the server to only one submission would help to contain the stress on the server.

The problem with runtime error messages is twofold; Viewing the error messages requires opening the browser's JavaScript console and the generated error messages do not correspond to right lines in Scala code. Opening the console is not a big inconvenience but impossible a task if one is not aware of its necessity. The easiest way to overcome this issue is to instruct students on how to access the JavaScript console. A better solution would be to have a console visible at all times alongside the trial GUI.

Because only the generated JavaScript was passed to A+ and the source map was discarded, the error messages did not correctly convey the location of the errors. The source maps could have been included although discarding them contained the size of response files; The response files were on average one megabyte in size and the source maps equally large. If the JavaScript file would have been fully optimised and minified, the equivalent file sizes would have been in the range of two hundred kilobytes for the JavaScript file and six hundred kilobytes for the corresponding source map file. However, the limitations of the compilation server did not allow us to use the full optimisation procedure.

Both the student questionnaire as well as the interviews revealed that there were misconceptions regarding the purpose and use of EDCAT. There were students who expected the system to explicitly indicate if their solution was correct or wrong. This may very well be related to the fact that the \emph{Programming 1} course --- which nearly all students taking the \emph{Programming Studio 1} course are completing concurrently --- uses a grading system that clearly shows which tests the student's solution did not pass. All students did not think of their solution as a part of a complete program but more so that EDCAT was a system their code was passed through to test it.

Students' misconceptions could be easily avoided with comprehensive introduction and instructions to the purpose and use of the system. It could be beneficial to similarly explain broadly how EDCAT works. The general introduction accompanied by a demonstration could be given in the first lecture of the course and there could be an introductory exercise that showcases the operation of the system and does not affect the course grade. Additionally, each trial GUI could have their own instructions and explanation of the operating principle. This way similar obscurities as in the third assignment round, where students did not expect all image filters to be applied to the image, would be avoided.

Better instructions could have been useful also in regard to the submission procedure and grading practices. As some students mentioned, they were annoyed by the need to separately submit code for grading after submissions for trial runs and the fact that A+ did not show their points. Sharing information would be a great first aid to the annoyance, but those issues could also be overcome altogether. The page with the trial GUI could have an option to submit the code for grading so that the student would not need to upload the code twice. Additionally, A+ could be modified to support half and bonus points, or the grading should be adjusted so that there would not be half points and the bonus points would be taken into consideration while setting the grade limits.

Furthermore, the system as is, did not measure up to local GUIs in terms of debugging features. As one of the teaching assistants had noted, some students used EDCAT as a sole measure for running their program code. They definitely encountered difficulties when their code contained errors as they did not have proper tools or skill to solve them efficiently. Additionally, the assignment feedback forms revealed that many students had difficulties with getting started with the assignment and many struggled also with errors. The students should be taught a systematic way to debug their code and those skills should be formally practised throughout the course.

Additionally, browser dependencies caused some minor issues in the second assignment round. Those could have been noticed and avoided by testing the system with different browsers. All in all, the students enjoyed the fact that their code was used with real media content and produced some real program. The web environment and interactivity should definitely be taken a greater advantage of in the future. Allowing the upload of students' own image and sound files for manipulation would highlight more strongly that the program code students write actually performs some real functions.

From the point of view of teaching assistants, EDCAT's potential in remote tutoring should have been supported and emphasised more. The students should be encouraged to utilise the provided forum platform and guided to share links to their submission so assistants could easily see to what students' problems are related. Similarly, the teaching assistants should have been formally assigned to have also remote tutoring duties.

Compared with the local GUIs, using EDCAT was beneficial for teaching assistants' grading tasks especially when the student's solution was correct or nearly correct. With solutions far from correct, the locally run GUI with IDE's debugger would have been more helpful. Furthermore, many teaching assistants ran students' code against a test suite which required to first download students' code. Testing functionality could be integrated as a feature for teaching personnel; The assistants could together define tests that could be ran similarly in browser. All the assistants would have the same tests at hand and the grading procedure could be more uniform.

The responses to the teaching assistant questionnaire revealed that teaching assistants were unwilling to put effort towards using the local GUI even when it was provided. This encourages to further develop online features in order to facilitate teaching assistants' work and allow them to concentrate more on the relevant tasks of helping the students and less on the technical issues that they need to overcome to do their job.

The technical usage data in addition to justifying the blame for long compilation times revealed that students started working on the assignments close to the deadline and did not use that many submissions. There is no reference value to estimate if the number of submissions was uncharacteristically low or data to show if starting late in some cases meant that students did not have enough time to reach the correct solution. However, as the server was slow, the cost of submitting code for a trial run could be high in students' eyes.

It is possible that some students wrote the program code in its entirety before running it with the trial GUI for the first time, which would make debugging more difficult. Bringing the compilation times down and encouraging students to submit code after each functionality would probably benefit most students. The late starting date as such will not be a problem, if the student still has time to finish the assignment before the deadline. However, some students inevitably do run out of time and should be provided with better support to plan the time needed to finish the assignment. As with the debugging skills already discussed earlier, students' study skills are lacking at the beginning of their academic career. However, even if providing support for planning study time in the introductory programming course is useful, it is not the course's role. Nevertheless, there are always stubborn students who either ignore the course staff's kind suggestions or are more interested in other courses or activities. 

Perhaps the most surprising revelation was that students submitted code that did not compile on its own. The students are probably not familiar enough with their development environments as the compilation errors are highlighted by the IDE with red marks. Some compilation errors were caused by inconsistencies with the GUI code, which are fully acceptable. Examining the compilation errors further and during the course would give a chance to address the most common and grave issues could be discussed during the lectures. The errors could also shed some light on how the problem solving process progresses from an erroneous submission to a fully functioning one.


\section{Instructor's Perspective}
\label{section:instructor}

Apart from the already presented practical implications drawn from students' and teaching assistants' feedback, EDCAT affected the work of the course's instructor. The distribution of GUI code to students was simple and easy and did not require any more work compared with the locally ran GUI distribution. Furthermore, the distributed solution proved quite useful as in the second assignment round the assignment and the related GUI code was updated after the assignment had been open for one week.

To monitor that the GUI code does not have any errors, the system was in the beginning configured so that compilation errors were reported by email to the instructor. At first it was nice to follow that students were working on the assignment and it was possible to intervene when the same error occurred multiple times in a short period of times. Usually, this meant that the same student submitted identically erroneous program code many times in a row and could be given instant feedback through A+ by the instructor. After it became apparent that most compilation errors were related solely with sloppiness in the students' work the compilation script was modified and emails would be sent only if the compilation error resulted when student's code was compiled together with the GUI code. Moving the trial execution online meant also that the instructor could follow when students started working on the assignments and encourage those who were putting off the work.

As the instructor is not involved in the computer lab sessions but helps students only through IRC and Piazza, EDCAT greatly facilitated remotely helping students with their problems. It was fast to try out student's code to duplicate the error and easy to compare with the model solution to confirm what caused the error. All in all, EDCAT proved to be useful although it did not support the development of students' own functionality as well as local GUIs. EDCAT was at its best in the first assignment round where the resulting programs were small and simple but its benefits in showcasing that students can produce real programs that use real media files should not be disregarded.


\section{Limitations and Validity}
\label{section:limitations}

This study was limited to evaluating if EDCAT is useful and can provide equal support to local GUIs in an introductory programming course that has a special focus on media programming. As already mentioned, the system is beneficial with assignments that result into interactive and visually rich programs. All assignments are not well suited to work with an online system; For example it might be necessary to limit the networking functions of the environment and file writing or creation is often wise to prevent. 

Only roughly a third of the students in the course responded to the student questionnaire, which is not such good a sample. Similarly, the interviews did not capture a representative group of students. However, both the responses to the questionnaire and the interviews painted quite concise a picture of the main problems related to EDCAT. Some further information about the programming experience of the respondents and the interviewees would possibly have revealed more clearly the different needs novice programmers might have in comparison with the more experienced programmers. However, there was already some indication that the novices and the experienced have a different outlook on EDCAT.

Furthermore, the students had clear misconceptions about the purpose of EDCAT and what part of the system was actually evaluated. The students did not understand that A+ was a separate system that showed them instructions and provided submission features and what was meant with EDCAT was responsible for compilation and returned the trial GUIs. Thus, some responses were disregarded and some responses may have been misinterpreted. However, the results were useful in evaluating and assessing the full submission process.

In addition to the used evaluation methods, observation could have been used to better understand how students used EDCAT. Observations could have revealed usability problems and additional ways to misuse the system, similar to that of using the system as a rudimentary version control system.