ASSESSMENT REPORT—ENGLISH MAJOR

Academic Year 2007-2008

 

Brief Summary of Assessment Plan

The assessment plan for the English major includes the following Goals and Outcomes:

              Goal #1: Knowledge of Literary Works

                 Outcomes:

                 Students should be able to

• Identify major authors and literary works within the Anglo-American tradition
• Identify major authors and literary works within other literary traditions
• Position literary works within cultural and intellectual history
• Compare and contrast literary works within cultural and intellectual contexts

 

Goal #2: Literary Analysis

                 Outcomes:

                 Students should be able to

• Classify literary works by genre
• Identify the elements of literature

3.  Read critically and creatively            

                            4.  Interpret literary works from an individual perspective

              Goal #3: Major theories of language/literature studies

                 Outcomes:

                 Students should be able to

• Explain major theories of language and literature
• Differentiate among major theories
• Apply critical theory to literary works

 

Goal #4: Communication

   Outcomes:

   Students should be able to

• Write in genres typical of the field
• Exhibit accuracy in writing conventions
• Contribute to oral discussions
• Use appropriate technologies

 

Goal #5: Research Methods

   Outcomes:

   Students should be able to

• Locate appropriate resources
• Evaluate the usefulness/credibility of resources
• Synthesize information from various sources
• Integrate research with literary analysis

 

The English faculty chose items from three of the goals above to assess for 2007-2008: Goal #1 (Outcomes 1-4), Goal #4 (Outcomes 1-2), and Goal #5 (Outcomes 1-4).  These assessments build upon the previous year’s assessment of all of the outcomes of Goal #1 and the first two outcomes of Goal #5.  Please note that the Outcomes assessed in Goal #5 apply to the location and appropriateness of resources rather than the application of research to writing.  These new assessments focus on the application of literary knowledge and interpretation to writing and the use of resources in students’ papers. 

 

Assessment Methods

The Goals mentioned above were assessed in the following manner:

             

Goal #1 (Outcomes 1-4)

The English faculty chose to administer the ETS Major Field Examination: Literature in English to English majors nearing graduation.  Nine seniors took the online version of the examination under controlled circumstances in the Testing Center in March, April, and October 2008. The course ENG-L495 Senior Seminar, a required capstone seminar for seniors, served as the starting point for identifying students, informing them about the examination, and scheduling test times. Students not in L495 were contacted individually. Students took the examination in a two-hour period using the ETS guidelines and were proctored by  Testing Center and faculty personnel. Students were scheduled where possible in groups of five, but those students who could not attend the scheduled times were allowed to schedule individually as per the guidelines of the ETS examination online. All students received their total score and subscores immediately upon completion of the exam. They also received information on national percentiles for individual scores. The statistical reports included in a later section of this report contain information for the one cohort that was closed in 2008 (nine students). 

The ETS Major Field Examination: Literature in English contains 150 multiple-choice questions covering major works of poetry, prose, and drama in British and American literature. Some questions require the student to read a passage or a complete short work and answer questions on genre, elements of literature, literary history, and interpretation.  A small portion of the examination covers works of literature from outside the Anglo-American tradition.

The departmental reports provided by ETS include

 

• Departmental Roster (individual scores and subscores for each student in the cohort)
• Departmental summary of total scores and subscores (compiled for the cohort)
• Mean scores for eight assessment indicators (compiled for the cohort)
• Departmental Demographic Summary (compiled for the cohort)

 

The examination provides subscores in the following four areas: Literature 1900 and Earlier, Literature 1901 and Later, Literary Analysis, and Literary History and Identification.  In addition, the report of assessment indicators provides a mean percent correct score for each cohort for the following eight areas: British Literature Pre-1660, British Literature 1660-1900, American Literature to 1900, British and American Literature 1901-1945, Literature in English Since 1945, Literary History, Identification, and Literary Theory.

This examination is a nationally prepared examination which provides comparisons to students and departments across the United States.  It is intended to assess student knowledge at the undergraduate level, and, unlike the Graduate Record Examination, is not meant as an entry examination for graduate school.

The CTLA at Indiana University Kokomo subsidized and ordered the examination.  The chairperson of Humanities (who is also the coordinator of English) scheduled and administered the examination and analyzed the results.

Our benchmark for our majors is to meet or exceed the individual national mean and median for all departments using the examination and for the department as a whole to meet or exceed the national institutional means and medians for the total score, the four subscores, and the eight assessment indicators.

Goal #4 (Outcomes 1-2)

Goal #5 (Outcomes 1-4)

To assess both Goal # 4 and Goal #5 listed above, the department continued to use its previous assessment tool of requiring all English majors in ENG-L495 Senior Seminar to provide a portfolio of three papers from their previous English courses: a literary interpretation, a research paper incorporating sources and using the MLA documentation system, and a third paper from ENG-L371 Critical Practices. These papers provide evidence of students’ ability to write in genres typical of the field, exhibit accuracy in writing conventions, locate appropriate sources, evaluate the credibility and usefulness of sources, synthesize information, and integrate research with literary analysis. These papers also provide information about applying critical theory to literary works; however, the department is not expressly evaluating that outcome this year. It is likely that this outcome (Goal 3, outcome 3) will be assessed either through these portfolios or directly with other outcomes of Goal 3 in ENG-L371. The English faculty developed a rubric by which to assess the six areas expressed in the outcomes above. An English faculty member read the portfolios and assigned each area a score of Good, Fair, and Inadequate (a score of 1-3 with 3 being the highest).  A mean was determined for each portfolio.

Description of Assessment Results

Results for the ETS Major Field Test: Literature in English:

Official scores for the nine students who took the exam and who are within the closed cohort are attached.  These scores occur within a range of 120-200 and were based on seniors taking the exam nationally from August 2005 to June 2008.  Table 1 below shows the percentile results for our nine students (all seniors) based on the national data:

 

Table 1. Individual scores and percentiles

 

187	95%
185	95%
171	80%
165	65%
158	50%
152	40%
146	30%
143	20%
130	5%

 

Tables 2 and 3 below show the mean and median scores for individual student scores (aggregated):

Table 2. Individual scores—means and medians

 

IU Kokomo mean	National mean	IU Kokomo median	National median
160	155	158	155

 

These individual scores represent an improvement of 3.8 points in the IU Kokomo mean and 3.5 points in the IUK median over AY 2006-2007.  These scores place IU Kokomo in the 65th percentile nationally in terms of individual student mean scores, an improvement from last year’s score in the 55th percentile. These scores exceed the departmental goal of having a departmental mean score that equals the national average. These individual scores fall just short of meeting our benchmark for Goal #1 that 60% of our students score at or above the national average (55% of our students met our benchmark). We believe that 60% is still a realistic goal.

In addition to comparisons of individual student scores to national individual student scores, the following tables provide information about the overall institutional scores nationally for the total scores, the four subscores, and the eight assessment indicators.

Table 3. Institutional mean score (total score)

 

IU Kokomo mean	National Institutional Percentile
160	65%

 

 

 

 

 

 

Table 4. Institutional scores (four subscores)

IU Kokomo means	National Institutional Percentile
Subscore 1:      59	60%
Subscore 2:      60	65%
Subscore 3:      60	65%
Subscore 4:      56	50%

Please see the previous discussion for the identification of the four subscore areas. These scores represent gains in both Subscores 2 and 3 (55.7 to 60 and 45% to 60% in Subscore 2; 55.7 to 60 and from 45% to 65% in Subscore 2), a slight loss in Subscore 4 (from 56.3 to 56 and from 55% to 50%), and a drop in Subscore 1 (from 61.4 to 59 and from 70% to 60%). Subscore 4 (Literary History and Identification) was our lowest-scoring area as a whole this year, but students did significantly better as a group in Literature 1901 and Later and Literary Analysis.

Table 5. Institutional scores (eight assessment indicators)—Cohort 1 (11 students)

IU Kokomo means	National Institutional Percentile
Indicator 1:     60	80%
Indicator 2:     48	50%
Indicator 3:     61	40%
Indicator 4:     60	75%
Indicator 5:     64	50%
Indicator 6:     45	35%
Indicator 7:     52	70%
Indicator 8:     57	80%

 

Please see the previous page for information on the areas covered in each assessment indicator. All eight indicators either equaled or surpassed last year’s means.  The lowest area was again Literary History. Students appear to be having the most difficulty with 17th to 19th century British and American literature and with literary history. Students did well with very early British literature, 20th century British and American literature, identification, and literary theory.

Certain indicators (such as Indicator  8 Literary Theory) are highly contingent upon students’ having completed ENG-L371 Critical Practices, a course they are often taking simultaneously with ENG-L495. The English faculty will be examining these scores to determine whether any curricular change is warranted.  The findings appear to indicate that the curriculum needs to focus more on literary history, particularly in the 18th and 19th centuries.

Results for the Portfolio Evaluations:

Results from the seven student portfolios (of nine; two were not submitted) from our assessment are listed below:

 

 

Table 7. Portfolio Evaluation Results (from highest to lowest on a 3-point scale)

 

Student	Portfolio Average
Student #1	3.0
Student #2	3.0
Student #3	3.0
Student #4	2.8
Student #5	2.7
Student #6	2.5
Student #7	2.3

 

Some student portfolios were difficult to evaluate because they did not contain clear examples of research writing.  This may indicate that students were not clear about what needed to be included in the portfolio or that students are not universally doing research writing in their courses. This year’s portfolios clearly showed papers that more intentionally worked with literary theory, probably as a result of students’ completing the requirement of ENG-L371 Critical Practices, which only became a requirement in fall 2006. The portfolio results showed that most students were performing in the Good to High Fair range (see rubric below). This information should help the faculty determine a benchmark for individual student performance.

 

English Major Assessment Rubric (Goals 4 and 5)

Circle the evaluation that best describes the overall portfolio.

 

Goal 4.1 Write in genres typical of the field	Goal 4.2 Exhibit accuracy in writing conventions	Goal 5.1 Locate appropriate resources	Goal 5.2 Evaluate the usefulness/credibility of resources	Goal 5.3 Synthesize information from various sources	Goal 5.4 Integrate research with literary analysis
Good	Good	Good	Good	Good	Good
Fair	Fair	Fair	Fair	Fair	Fair
Inadequate	Inadequate	Inadequate	Inadequate	Inadequate	Inadequate

 

 

Using Assessment for Program Improvement

 

As indicated in the ETS results (both total scores and subscores), some of our students have a good general knowledge of literature from all periods, do well at literary analysis, and have a good general knowledge of literary theory.  Other students performed at a lower level. An encouraging sign is that student have risen over the last two administrations of the exam.  The exam also provides valuable individual information to students because they receive their scores immediately after completing the test, along with information about national norms for individual scores.  The faculty will also be able to use this information to focus more on literary history and to provide courses that fill gaps, especially in 18th-century literature.    

Our results from the portfolio evaluation indicate that, while faculty are providing students with a wide range of experiences with resources, we might benefit from targeting the integration and documentation of resources that are central to the field and making certain that particular courses within the curriculum include explicit instruction and exercises pertaining to these abilities. Next year’s goals include adding Goal #3 for assessment and establishing a benchmark for the portfolios.

Dissemination of Results

 

This report has been distributed to resident English faculty, to the Dean of Arts and Sciences, and to the Assessment Committee. The complete assessment report is posted on the Arts and Sciences website, and a summary paragraph with a link to the complete report is posted on the CTLA website.  Students receive scores on the ETS examination immediately after completing the test.  Students may also access the assessment report at the SOAS website.