The Optical Mark Reader (OMR) run by the Information Technology Services Help Desk in IACC 150 provides four main areas of service:
The OMR scanner reads marks made by a #2 pencil (pen marks are not recognized) on special answer sheets (generally two-sided). OMR reads through each sheet, taking the responses off both sides of the sheet at once. Therefore, the back of the typical two-sided sheets cannot be used for comments or other written material. Two types of test forms are available:
General Purpose, 5-choice, or 10-choice NCS Answer Sheets may be purchased at the NDSU Bookstore. If you send your students to buy these sheets individually, it is important that you make sure you are explicit in specifying which sheet (blue or green) you are going to use. The two types of forms are NOT interchangeable. If you have any questions after reading this document, please contact the ITS Help Desk at 231-8685.
OMR scans will be accepted at the ITS Help Desk during regular Help Desk hours and will be ran after 4:00 p.m. daily. Users will be notified by phone when they are ready for pick-up.
ITS Help Desk hours are
If users wish to be present while scans are ran, they can come to the Help Desk after 4:00 p.m. on Monday-Friday.
Please note: When picking up scored exams at the Help Desk, you will need to bring your NDSU photo ID or a copy of the confirmation e-mail message for identification purposes. The goal is to prevent anyone from stopping by and picking up a scored exam to which that person should not have access.
Two different scoring methods or types of grading are available to users of the OMR Test Scoring Service at NDSU; Conventional Grading and Weighted Grading. In Conventional Grading, the student's answer sheet is checked against a KEY sheet filled out by the instructor. The student's score is the number of questions answered correctly. In Weighted Grading, the instructor can assign weights of 1 through 5 to each of the items on the exam. If a question was given a weighting factor of 4, it would be worth 4 points on the test. The student's score is computed by summing the weighted values of questions answered correctly. The instructor identifies the type of grading using Question #1 on the INFO sheet.
If this option is chosen, it will be necessary for the instructor to fill out a WEIGHT sheet in addition to the KEY sheet. The instructor must mark the desired weight of each question: A = 1 point, B= 2 points, C = 3 points, D = 4 points, E = 5 points. If a question is not given a weighting factor, a value of 1 is assumed. Note that only integer weights from 1 to 5 are allowed under this option.
Up to five copies of the test results can be printed. Question #2 on the INFO sheet is used to specify the number of printed copies.
An instructor may want to give a test with both objective (multiple choice, true-false) questions and subjective (essay, short answer) questions. The scoring program allows the instructor to add the points earned on any subjective questions to the results of the machine-scored multiple choice questions. To do this, the instructor must determine the number of subjective points earned, mark it on each student answer sheet (right-justified in the SPECIAL CODES field) and mark the maximum subjective points in the SPECIAL CODES field on the KEY sheet.
Results from a Test Scoring run can be output to a tab-delimited text file suitable for import into most spreadsheets. You must fill in Question 6 on the INFO sheet in order to generate this output file. You may provide a unique name for the output file in the NAME field of the INFO sheet. See Filling Out the INFO Sheet and Downloading the Scores File for details.
Examples of the scoring reports generated during a test run are included in this section.
The first report in the printout lists the original order in which the tests were read to provide you with quick access to any test in the stack. This is useful when you want to check individual test sheets for accuracy and/or errors. In other reports the data will be sorted by student name or ID number as determined by Question #4 on the INFO sheet.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 1 | | Original Order of Test Sheets | | | | Read Down Starting from Left | +-------------------------------+ ID ID ---------- ---------- SWAN ALV 47130 61095 1226 67929 77086 2950 20492 2052466 GUSTAVSEN 29457 50296 911 17878 4212 29804 3361 1710 55555 7793 10317 1825 1088 824 91110 89232 1356 56556 20860 7673 30191 6132 8663 96970
Report 2 lists all of the Scoring Options chosen for this run. In addition, the answer key and weights associated with each question are printed.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 2 | | Instructor Information | +-------------------------------+ Department --> CDD Course Number --> Test Date --> / / Grading Rule --> Conventional Order of Summary Report --> by Alphabetic Number of Copies --> 2 Number of Questions --> 50 Total Subjective Points --> 0 Ques Key Weight Ques Key Weight Ques Key Weight Ques Key Weight Ques Key Weight ---- --- ------ ---- --- ------ ---- --- ------ ---- --- ------ ---- --- ------ 1 B 1 11 A 1 21 E 1 31 A 1 41 A 1 2 B 1 12 A 1 22 C 1 32 B 1 42 B 1 3 B 1 13 B 1 23 E 1 33 A 1 43 B 1 4 B 1 14 E 1 24 D 1 34 A 1 44 A 1 5 B 1 15 A 1 25 E 1 35 A 1 45 A 1 6 A 1 16 A 1 26 D 1 36 A 1 46 B 1 7 A 1 17 A 1 27 E 1 37 A 1 47 A 1 8 B 1 18 D 1 28 E 1 38 A 1 48 B 1 9 B 1 19 A 1 29 E 1 39 A 1 49 A 1 10 B 1 20 E 1 30 D 1 40 A 1 50 A 1
Report 3 lists the number of questions, the number of tests graded, the subjective points possible, total points possible, mean, median, mode, standard deviation, and the Kuder-Richardson reliability scores (KR-20 and KR-21). Overall class performance is illustrated with a histogram showing frequencies and percentiles for each score obtained on the exam.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------------+ | REPORT # 3 | | Test Analysis & Distribution | +-------------------------------------+ Number of Tests Subjective Total Standard Reliability Questions Graded Points Points Mean Median Mode Deviation KR-20 KR-21 50 ( 50 Pts) 67 0 50 42.31 44.00 44.00 4.63 0.759 0.711 -- Percentage ----> 100% 84.63 88.00 88.00 9.26 ---- Score ---- Per- Cum Histogram Points Percent centile Percent Freq Freq Each * Stands for 1 Score 27 54.0 0 1.5% 1 1 * 29 58.0 2 1.5% 1 2 * 30 60.0 3 1.5% 1 3 * 31 62.0 5 1.5% 1 4 * 35 70.0 6 1.5% 1 5 * 36 72.0 8 1.5% 1 6 * 37 74.0 10 3.0% 2 8 ** 38 76.0 13 3.0% 2 10 ** 39 78.0 18 7.5% 5 15 ***** 40 80.0 24 4.5% 3 18 *** 41 82.0 32 10.4% 7 25 ******* 42 84.0 38 1.5% 1 26 * 43 86.0 44 10.4% 7 33 ******* 44 88.0 55 13.4% 9 42 ********* 45 90.0 69 13.4% 9 51 ********* 46 92.0 81 10.4% 7 58 ******* 47 94.0 90 7.5% 5 63 ***** 48 96.0 95 3.0% 2 65 ** 49 98.0 98 3.0% 2 67 **
Report 4 lists simple counts of how many students selected each choice, as well as the number of invalid and omitted responses for each question. When the "AND" or "OR" options are used, the number of people answering combinations is displayed, although the actual combinations are not.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 4 | | Frequency Distributions | +-------------------------------+ Ques A B C D E Omit Invalid 1 25 41 1 2 23 44 3 2 65 4 27 40 5 5 62 6 66 1 7 62 5 8 6 61 9 12 55 10 10 57 11 62 5 12 66 1 13 14 53 14 5 62 15 65 1 1 16 63 4 17 51 6 2 8 18 1 9 3 45 9 19 65 2 20 1 4 1 61 21 23 6 38 22 20 2 45 23 4 1 3 59 24 1 2 14 36 14 25 2 2 62 1 26 2 1 1 63 27 2 1 2 62 28 1 5 2 59 29 2 2 28 35 30 1 1 56 9 31 64 3 32 15 52 33 59 8 34 61 6 35 48 19 36 66 1 37 54 13 38 60 7 39 51 15 1 40 64 3 41 67 42 10 57 43 4 63 44 66 1 45 67 46 39 28 47 60 7 48 7 60 49 65 2 50 62 5
Report 5 presents an analysis of the individual items on the exam. The class is divided into thirds based on the overall test scores of the students. For each question, the percentage of students from the upper third selecting each choice is compared to the percentage of those from the lower third who picked that choice. The correct answer is denoted by the letters CR for single correct answers and by the correct combination for combination answers.
A well-written test item should have responses in all or nearly all of the possible choices. When no student in the class picks a choice then it is not a good distractor. The choice had the same effect as if not there. Also, you may want to check that many of the students in the upper third do not flock to any one wrong choice. This often indicates a poorly worded choice or one in which the distinction between the right and wrong answer is not totally clear - even your "good" students were confused on the point covered.
Items should discriminate between the students in the upper and lower thirds of the class. If all items cover the same general topic, then the amount of students in the upper third who correctly answer a question should be greater than or equal to those in the lower third who correctly answer the same question.
The item discrimination index shows this relationship using the following formula: (U - L)/ N where U is the number of students answering correctly in the upper third, L is the number of students answering correctly in the lower third and N is the number of students in a third. A negative score means students who did poorly on the test did better on this item that those who did well -- this item probably needs to be rewritten. A score below .20 suggests there is low discrimination power and the question may need examination. Ideally, you should have a variety of indices covering the .2 to .8 range.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 5 | | Item Analysis | +-------------------------------+ Item Qus A B C D E Omit Invalid Discrimination Index 1 Freq 25 41CR 1 Upper 3rd 18.1% 81.8% Lower 3rd 72.7% 22.7% 4.5% 0.59 2 Freq 23 44CR Upper 3rd 13.6% 86.3% Lower 3rd 59.0% 40.9% 0.45 3 Freq 2 65CR Upper 3rd 100% Lower 3rd 9.0% 90.9% 0.09 4 Freq 27 40CR Upper 3rd 13.6% 86.3% Lower 3rd 77.2% 22.7% 0.64 5 Freq 5 62CR Upper 3rd 100% Lower 3rd 18.1% 81.8% 0.18 6 Freq 66CR 1 Upper 3rd 95.4% 4.5% Lower 3rd 100% -.05 7 Freq 62CR 5 Upper 3rd 86.3% 13.6% Lower 3rd 90.9% 9.0% -.05 8 Freq 6 61CR Upper 3rd 100% Lower 3rd 18.1% 81.8% 0.18 9 Freq 12 55CR Upper 3rd 100% Lower 3rd 50.0% 50.0% 0.50 10 Freq 10 57CR Upper 3rd 13.6% 86.3% Lower 3rd 31.8% 68.1% 0.18 11 Freq 62CR 5 Upper 3rd 95.4% 4.5% Lower 3rd 81.8% 18.1% 0.14 12 Freq 66CR 1 Upper 3rd 100% Lower 3rd 95.4% 4.5% 0.05 13 Freq 14 53CR Upper 3rd 13.6% 86.3% Lower 3rd 18.1% 81.8% 0.05 14 Freq 5 62CR Upper 3rd 100% Lower 3rd 22.7% 77.2% 0.23 15 Freq 65CR 1 1 Upper 3rd 100% Lower 3rd 90.9% 4.5% 4.5% 0.09 16 Freq 63CR 4 Upper 3rd 100% Lower 3rd 86.3% 13.6% 0.14 17 Freq 51CR 6 2 8 Upper 3rd 100% Lower 3rd 40.9% 22.7% 9.0% 27.2% 0.59 18 Freq 1 9 3 45CR 9 Upper 3rd 13.6% 77.2% 9.0% Lower 3rd 4.5% 9.0% 13.6% 54.5% 18.1% 0.23 19 Freq 65CR 2 Upper 3rd 100% Lower 3rd 90.9% 9.0% 0.09 20 Freq 1 4 1 61CR Upper 3rd 100% Lower 3rd 4.5% 13.6% 4.5% 77.2% 0.23 ** PORTION DELETED TO SAVE SPACE ** 46 Freq 39 28CR Upper 3rd 40.9% 59.0% Lower 3rd 68.1% 31.8% 0.27 47 Freq 60CR 7 Upper 3rd 90.9% 9.0% Lower 3rd 90.9% 9.0% 0.00 48 Freq 7 60CR Upper 3rd 4.5% 95.4% Lower 3rd 13.6% 86.3% 0.09 49 Freq 65CR 2 Upper 3rd 100% Lower 3rd 95.4% 4.5% 0.05 50 Freq 62CR 5 Upper 3rd 100% Lower 3rd 77.2% 22.7% 0.23
This report provides detailed information on each student taking the test. The student's name, identification number, weighted percentage score, weighted correct score , number of omitted questions and individual subjective points obtained are displayed. Also, the student's answer to each question on the test is indicated. A $ is printed for each correct answer. If an answer is incorrect, the response chosen is printed. NOTE: For either the five- or ten-choice, single-correct answer exam, an asterisk (*) is printed for each answer which could not be read by the machine. This is usually due to more than one answer being chosen on a single-response test or to stray marks on the page.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 6 | | Individual Scoring | +-------------------------------+ P W _ _ S S N C C O S A O O M U Q Q Q Q Q Q Q Q Q Q Q Q Q Q M I R R I B Q Q Q Q Q Q Q Q Q 1 1 1 1 1 1 1 1 1 1 2 2 2 2 E D E E T J 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 BINKLEY MIKE 29804 82 41 0 0 A $ $ $ $ $ $ $ A $ $ $ $ $ $ $ $ $ $ $ A $ $ DALLAS STEVE 1020 80 40 0 0 A A $ A $ $ $ $ $ $ B $ $ $ $ $ D E $ $ $ $ $ JETSON ELROY 3361 78 39 0 0 $ $ $ A $ $ $ A $ $ $ $ $ $ $ $ $ B $ $ A A $ OPUS P 911 88 44 0 0 A A $ A $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ QUEST JOHNNY 22305 92 46 0 0 A $ $ A $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ RUBBLE BARNEY 6803 94 47 1 0 $ $ $ $ $ $ B $ $ $ $ $ $ $ $ $ $ $ $ $ A $ $ STIMPY 3368 76 38 0 0 A A $ A $ $ B A A $ $ $ $ $ $ $ $ $ $ $ A A $ N A Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q M I 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 E D 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 BINKLEY MIKE 29804 $ $ $ $ $ $ $ $ A $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ DALLAS STEVE 1020 C $ $ $ B D $ $ $ B $ B $ $ $ $ $ $ $ $ $ $ $ $ $ $ B JETSON ELROY 3361 $ $ $ $ $ $ $ $ A $ $ $ $ $ $ $ $ $ $ $ $ $ A $ $ $ $ OPUS P 911 $ $ $ $ $ D $ $ $ $ $ B $ B $ $ $ $ A $ $ $ $ $ $ $ $ QUEST JOHNNY 22305 C $ $ $ $ D $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ A $ $ $ $ RUBBLE BARNEY 6803 E $ $ $ $ $ $ $ $ $ $ $ B $ $ $ $ $ $ $ $ A $ $ $ $ STIMPY 3368 B $ $ $ $ $ $ $ A B $ $ $ $ B $ $ $ $ $ $ $ A $ $ $ B ** PORTION DELETED TO SAVE SPACE ** Blank Denotes Omitted Answer $ Denotes Correct Answer * Denotes Invalid Answer
This report may be used for posting results of the test. The student identification number sorted in ascending order is printed on the left side of the page. Student names are placed on the right side of the page or you may choose to have them omitted. To the right of each student identification number is the unweighted number of questions answered right, wrong and omitted, the subjective points obtained, the weighted points and weighted percentage.
Summary Report for Claim Number 641 Data Scanned on 1-14-96, Report Generated on 02APR96 +-------------------------------+ | REPORT # 7 | | Student Summary | +-------------------------------+ Student Summary: CDD Student Obj. Obj. Obj. Subj. ---- Score ---- ID-Number Right Wrong Omits Points Points Percent Means>>===> 42.3 84.6 3361 39 11 39 78.0 JETSON ELROY 911 44 6 44 88.0 OPUS P 3368 38 12 38 76.0 STIMPY 6803 47 2 1 47 94.0 RUBBLE BARNEY ** PORTION DELETED TO SAVE SPACE ** 1020 40 10 40 80.0 DALLAS STEVE 29804 41 9 41 82.0 BINKLEY MIKE 22305 46 4 46 92.0 QUEST JOHNNY
The Test Scoring System at NDSU consists of an Optical Mark Reader scanner, software enabling the OMR to write scanned data to a file, and custom-written SAS programs running on a UNIX host which read the raw data files from the scanner and scores them. There are four different kinds of sheets which comprise the stack of sheets representing a separate Test Scoring run. These include (in order from top to bottom of the stack) the INFO sheet, the WEIGHT sheet, the KEY sheet, and the STUDENT sheets. The INFO, KEY, and STUDENT sheets will be part of every run, whereas the WEIGHT sheet is optional. Completed stacks should be taken to the ITS Help Desk in IACC 228 for processing.
<tt> Example: INFO CHEMISTRY XYZQ Claim Number: 8 Scores filename: XYZQ.008 Example: INFO AG ENG Claim Number: 777 Scores filename: ENG.777 </tt>
<tt> A B C D E F G H I J K L M N O 1234567890 RUBBLE BARNEY 81.0 17 21 17 4 0 0 CHEMISTRY 111 1 1 1 3 </tt>
If any correct answer is to be worth more than one point, you must complete a WEIGHT sheet. This procedure is often used when you want to weight correct multiple choice questions more than true-false questions.
The students MUST use only #2 lead pencils (pen will NOT work) and fill in their answer sheets as follows:
DO NOT FOLD, STAPLE, PUNCH HOLES IN, or TEAR TEST SHEETS as they may not feed through or be read properly by the scanner. Also, DO NOT RANDOMLY MARK THE SHEETS with lead or ink that has a carbon content as this may also cause the scanner to improperly read the sheets. If a sheet can not be read correctly, it will be placed across the other sheets but will not be scored. OMR reads the sheets starting with the TOP sheet, so the sheets must be placed in the following order: The INFO sheet on the top, underneath it the WEIGHT sheet (optional), the KEY sheet, and finally, on the bottom, the STUDENT ANSWER sheets. Assemble the student answer sheets. All sheets must be facing up, with the name grids aligned. If subjective questions are used, the instructor must mark each student answer sheet with the number of points that the student earned on the subjective question(s). Align this score in the SPECIAL CODES grid so that the last or right-most digit ends up in the "P" column.
If you have filled out questions 1-6 on the INFO sheet properly and your exams have been scored, then you should be able to locate a copy of the scores output file on the anonymous ftp server at NDSU. Remember the algorithm used to name your scores file uses the last word from the name field on the INFO sheet, appends a period, which is followed by the claim number ; e.g., CDD.456 where the alphabetic portion is all in upper case. It is important for you to know the naming algorithm so that you can obtain a copy of your file. The ftp directory where the scores files are stored has the permissions set so that users cannot list the available files. This is designed to provide a modest amount of security so that not just anyone is able to look at the scores files. Anonymous ftp can obtain copies of the files if you know the names in advance. The remaining steps describe an anonymous ftp from the NDUS host ftp.nodak.edu. You should also be able to perform an anonymous ftp from a PC running WSFTP or from a Macintosh using Fetch. However, when you get to the point where you normally see a list of available files, there will be none listed. You can still tell your program to get the appropriate file by specifying the exact filename (be careful about the upper case letters) and it should be able to find it and download it.
A program providing simple frequencies, percentages, means and standard deviations for each question is available with the Optical Mark Reader. This program is useful for summarizing data which has been encoded on the scanner sheets under the five-choice, single answer option. Follow the steps below to utilize the SURVEY capabilities of OMR.
OMR sheets represent an easy way to collect data and convert it into an electronic format for research purposes. The READ program associated with the OMR scanner provides a convenient interface for this purpose. To just read your sheets into a text file, do the following:
Data from the sheets read by the Optical Mark Reader are placed in a file accessible via anonymous ftp. To access the data, follow the same steps as described for Downloading the Scores File from exams, except replace the ftp directory SCORES with RAW (be sure RAW is entered in upper case letters). A partial record from such a file is shown below followed by a list of fields in order of occurrence. The first four fields are inserted automatically by the scanner. Information from the forms begins with the fifth field.
5-Single 064 0003 08-26-96 FLINTSTONE FRED 1 07 05 31 96 0123456789 020310 2 5 3 2 1