NEW: For a prettier blog interface, see the Wordpress version!

Quality review phase

The submission system will be reviewed for quality and ease of use by teachers as they will determine its use in their classes. The first version of the teacher interface is geared toward technical users, so heavier weight will be given to the ability of the system to meet the teachers' requirements for a submission system. Ease of use and more intuitive interfaces will follow later as other potential users are identified.

A sample project with submissions and evaluations will be created beforehand so that teachers can evaluate the most common uses of the system. They will primarily be concerned with ease of use for the students, so student submissions will be demonstrated. Downloading project submissions and tracking activity will also be demonstrated.

Open-ended questions will be asked to gather feedback from the teachers.

Pilot testing

Pilot testing will be done by representative students from my CS21A class. Eric Vidal's CS177 class may also be tapped to provide a review by students who have worked with other submission systems. The pilot test will take approximately an hour and a half. The focus group will be composed of 8 to 12 students. Because this is a tool to organize submissions and promote peer review, care will be taken to select both high-achieving and low-achieving students based on their participation in class and their performance in quizzes and projects.

The students will be randomly assigned to groups of 4 students each. The group members will be assigned the roles of A, B, C, or D. For the peer reviews, A will be partnered with B and C will be partnered with D. For the pilot testing, A will be partnered with C and B will be partnered with D.

The students will be asked to perform the following tasks:

1. Submit a program.

The students will be asked to submit one of their old programs. They will do this in their pilot-testing pairs. While A and C go through the process of submitting a file, B and D will keep detailed notes on how the first student did. These notes will form the basis for the first evaluation.

After 5 minutes, B and D will take over the respective computers and upload their own submissions while A and C take detailed notes. These notes will help us evaluate the system from the perspective of someone who is familiar with its use, and provides an opportunity for students to reflect on how to improve the system.

Program submission should take no more than 5 minutes for the first student and 2 minutes for the second, taking into account their unfamiliarity with the system. Non-completion of the task within the allotted time indicates areas where improvements must be made if this system is to be used for in-class exercises and hands-on examinations.

A short discussion of the process will follow, allowing students who failed to submit files within the given time to raise their concerns and provide feedback.

2. Evaluate a program.

B will then be assigned to review A's program while D is assigned to review C's program. B and D will download the program assigned to them for review while A and C take notes on what they do. B and D will each fill out the project evaluation rubric previously prepared by the teacher.

Because the system is designed to encourage quick peer review and feedback, the evaluation process should take no more than 20 minutes. The division of time between evaluating the actual program and working with the system should be noted. The bulk of the time should be spent on writing suggestions, not figuring out the user interface.

The short timeframe for evaluation during the pilot testing phase allows us to focus on the usability of the evaluation interface and measure the usefulness of quick feedback. When the system is deployed, students will be given more time to review their buddy's program and may be expected to give longer reviews.

3. Read their evaluation.

A and C will then view the evaluation of their own programs as accomplished by B and D respectively while B and D in the pilot-testing pair take notes. The program authors will note any discrepancies between the program they wrote and the evaluations they received. They will also evaluate the helpfulness of the comments and suggestions. This part of the pilot test will take 5 minutes.

4. Browse all the files.

B and D will then browse through the other evaluated files uploaded to the system while A and C take notes. They will evaluate how the reviews help them identify files they can learn from by comparing their use of the reviewed files to their use of the uncategorized files. This should take 10 minutes.

5. Evaluate the system.

In addition to the notes from each step of the process, students will also be asked to evaluate the system through a questionnaire that uses both closed and open-ended questions. See the following section for the questionnaire items. This should take 15 minutes.

Field trial

After suggestions from the pilot test have been incorporated into the system, the system will be tested in the classroom. The teacher will provide each student with a copy of detailed instructions on how to use the system and the opportunity to test the system by uploading a file to an ungraded project. A suggested project is a text file containing a self-introduction which will be evaluated based on its adherence to specifications. This can be assigned as homework.

The teacher will then require the use of the system for a project or hands-on examination. Students will be given two days to complete their evaluations and another two days to modify their programs based on the evaluations. They will then be asked for a brief evaluation of the submission system.

The field trial will take approximately one week.

Questionnaire items

Quality review phase

What other submission systems do you use right now? Please compare submit3 with them in terms of advantages and disadvantages.

Would you consider using submit3 as the project submission system for your classes? If not, what needs to be improved in order for you to consider it?

Would you consider using submit3's peer review features in your classes? If not, what needs to be improved in order for you to consider it? (Please indicate whether you have pedagogical concerns or technical concerns.)

Do you have any other comments or suggestions?

Pilot test

Part 1 for A, B, C, D

Please describe step-by-step how your partner submitted a program. Include any mistakes your partner may have made.

Was your partner able to successfully submit a program within the allotted time? Yes No

Did your partner encounter any difficulties logging on to the system?

Did your partner encounter any difficulties finding the project your partner was asked to submit?

Did your partner encounter any difficulties submitting the files?

Did your partner encounter any other difficulties not mentioned here?

Please compare submit3 with other submission systems you have used for this class in terms of submitting files.

Part 2 for A and C

Please describe step-by-step how your partner reviewed a program. Include any mistakes your partner may have made.

Was your partner able to successfully evaluate a program within the allotted time? Yes No

Did your partner encounter any difficulties finding the project your partner was asked to review?

Please estimate how much time your partner spent on the following activities:

Did your partner encounter any difficulties reviewing the program?

Part 3 for B and D

Please describe step-by-step how your partner read the review of a program. Include any mistakes your partner may have made.

Were there any differences between the program and the review? Describe.

How helpful were the comments and suggestions provided by the reviewer? What did you learn from them?

Do you think that the feedback you get is worth the time spent on reviewing someone else's program?

Part 4 for A and C

How helpful did your partner find the categories and the comments when browsing the programs?

Compare browsing through the files on submit3 with other submission systems you have used in the past.

What did you learn from your classmates' programs?

Overall

Compare submit3 with other submission systems you have used in the past. Describe the advantages and disadvantages of each method.

Would you like to use submit3 as is for your future projects and hands-on examinations? If not, what needs improvement before submit3 can be used in the classroom?

Do you have any other comments or suggestions?

Field trial

Compare submit3 with other submission systems you have used in the past. Are there any particularly important advantages or disadvantages?

Would you like to use submit3 as is for your future projects and hands-on examinations? If not, what needs improvement before submit3 can be used in the classroom?

Do you have any other comments or suggestions?

Previous day | Next day

I'd love to hear about any questions, comments, suggestions or links that you might have. Your comments will not be posted on this website immediately, but will be e-mailed to me first. You can use this form to get in touch with me, or e-mail me at [email protected] .

Page: Instructional Software Design.Evaluation
Updated: 2004-11-21
NOTE: ANTI-SPAM MEASURE NOW IN PLACE. Please answer the following question with the right number in order to send me your comment.
What is two minus one? (hint: one ;) )
Name:
E-mail:
URL:
Comments: