Psychology Assessment Report Summaries

Link to all Psychology Assessment Reports

2013-2014

Link to Full Psychology Assessment Report 2012-2013; 2013-2014 Plan

2012-2013

Link to Full Psychology Assessment Report 2012-2013; 2013-2014 Plan

2011-2012

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology (Psychological Perspectives, Research Methods, Statistics, Biological Bases, Learning/Cognition, Developmental, Personality/Individual Differences, Abnormal, Clinical/Counseling, & Social Psych). The group performed at or above our 50^thpercentile benchmark (compared to the national sample of other Psychology Majors) on all 10 topic areas.  When looking at individual, rather than group performance, our first benchmark is that 75% of individuals will achieve 50^thpercentile or higher on the four outcomes representing coursework that all psychology majors must take. These are psychological perspectives, research methods, statistics, and developmental. The benchmark of 75% of individuals achieving the 50^thpercentile or higher occurred only for psychological perspectives.  Strong performance on identifying characteristics of psychological perspectives was also assessed using a 20 statement quiz from P459 History & Systems. Our students met a benchmark of 75% of individuals achieving at least 75% on this perspectives quiz as well. However, on a measure of students’ understanding of advantages and disadvantages of the different perspectives, only 60% of students achieved acceptable ratings (though the benchmark was to have 75% of students with acceptable ratings). This latter result may be due to assignment instructions, which will be revised prior to re-assessment.
  
  The second individual benchmark is to have 75% of students achieve 50^thpercentile or higher on at least one of the subtests representing our two course cluster areas. This benchmark was achieved for both Cluster 1 (personality, social, health, abnormal) and Cluster 2 (learning, cognition, neuroscience). Although overall our students are doing well with content on the ACAT, the continued concern is with research methods, statistics, and developmental. Planned revisions to our research methods curriculum which involve going from one research methods course to a two-course sequence (with students ideally taking the lower level methods before statistics and the upper level methods after statistics) may help with both research methods and statistics performance. The new sequence of courses will begin Fall, 2013; thus it will be a number of years before we will see the effects of this curriculum change in our assessment data. Preliminary review of the ACAT developmental questions suggests this subtest may not align well with our program’s learning goals for the Lifespan Development class.

Goal 6: Writing in Psychology, Outcome 1 Communicate Ideas.

• Since last year, a new rubric for evaluating this outcome was created and pilot tested. Preliminary results showed students performed best on Sufficiency and Quality of Evidence, next best on Focus and on Sentence Structure and Mechanics, and worst on Audience Awareness and Style and on Organization, Flow, and Transitions.

Link to Full Psychology Assessment Plan for 2011-2012; 2012-2013 Plan

2010-2011

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology. This year, student scores met benchmarks on only three topic areas (Experimental Design, Physiological, and Learning/Cognition). We believe the poor performance this year was due to a series of computer problems during the test. During the administration of the test, most students received pop-up boxes reporting errors, and we believe this error caused problems with the scoring of the test, as some of our stronger students performed very poorly. However, this year we also assessed knowledge base in our 400-level History and Systems course using our own series of mastery tests of basic concepts from the General Psychology course and textbook. We are pleased that we achieved benchmark in 5 of the 10 topic areas. This provides support that students are developing a knowledge base in research, statistics, learning, psychological disorders and treatment, and social psychology and came close to achieving benchmark in cognition. The four topics where results were the farthest below benchmark related to biological bases (most likely because quite a few of these students had not taken Neuroscience and because the mastery quiz included sensation and perception items and we do not have a sensation/perception course), development (which students take very early in their program and most likely have forgotten much of the content), and Personality/Motivation (again, quite a few students had not taken the Personality course and we do not have a Motivation course. We would like to hire an additional psychologist with background in biological bases who could offer either Sensation/Perception or Motivation. Doing this would also allow us to restructure the major requirements such that students are more likely to take at least one biologically focused psychology course; at present, students are able to avoid the only one we can offer (i.e., Neuroscience).

Goal 2: Research Methods, Outcome 5 Advanced Level (Design basic studies to address psychological questions.).

• Students in our Senior Seminar course write research proposals for which they develop their own hypotheses based on psychological literature and design studies to test those hypotheses. They receive instructor feedback on a first draft and turn in a final draft that is used for assessment. Although some of the most simple components (e.g., having a hypothesis and stating it positively, having most of the background logically connected to the research question or hypothesis, including a plan for data analysis, and having chosen all scholarly sources) were addressed by all or nearly all of the subset of students, most students had spotty performance on the more difficult components (e.g., stating the hypothesis clearly, correctly identifying the research design, choosing a research design that tests the hypothesis, and choosing appropriate statistics.) Many did not identify a research design at all, most were not able to state their hypothesis clearly, and most were able to provide operational definitions that were only “somewhat” clear. We also noticed that students were trying to emulate the very complex designs and statistics of studies they were reading in Senior Seminar instead of using the more basic “tools” they learned in their methods and statistics classes; tools they should be better prepared to use well. We have decided to revise our research curriculum to include two courses, one lower level and one upper level, so we can provide more opportunities for students to learn and practice these skills. Currently we have only one methods course (lower level) and believe students need more opportunities to learn and practice applying this content and set of skills. 

Goal 6: Writing in Psychology, Outcomes 3 and 4 (Use APA style documentation that is appropriate to the assignment and Use APA style formatting that is appropriate to the assignment).

• Final drafts of research proposals written by students in P457 Senior Seminar were used for this assessment. Individual students seemed to be inconsistent in their mistakes; with different sets of students performing well vs. poorly in different areas. As a whole, they performed best on: citing when appropriate to do so, using appropriate conventions for paraphrases vs. direct quotations, and putting references in alphabetical order. They also performed best on: using correct order of elements and using correct margins and line spacing. However, they had trouble with correct use of APA style in parenthetical citations and providing complete information in correct APA style for references. This suggests they have learned the more basic rules, but do not follow through consistently in applying the more detailed rules.

Goal 7: Quantitative Literacy, Outcome 2 (Select appropriate statistics).

• Final drafts of research proposals written by students in P457 Senior Seminar were also used for this assessment. We found that most did not give enough information regarding level of measurement (nominal, ordinal, etc.) for us to be able to tell whether or not the statistics chosen were appropriate for the data and that many were choosing inappropriate statistics that were well beyond their own knowledge and skill levels. In other words, students were not making use of the usual statistics they learn about in an undergraduate statistics course and were instead haphazardly selecting much more sophisticated statistical procedures because the background articles they read as part of the Seminar had used such procedures. In the short-run, we plan to make our expectation of using the statistical tools they learned in K300 Statistics very explicit in future Senior Seminars. In the longer run, we plan to discuss other assessment methods for this outcome to see if they perform better in the context of a different type of assignment or assessment method.

Link to Full Psychology Assessment Report for 2010-2011; 2011-2012 Plan

2009-2010

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology (Psychological Perspectives, Research Methods, Statistics, Biological Bases, Learning/Cognition, Developmental, Personality/Individual Differences, Abnormal, Clinical/Counseling, & Social Psych). The group performed at or higher than our 50^thpercentile benchmark (compared to the national sample of other Psychology Majors) on 8 of the 10 topic areas.  History & Systems and Personality were the exceptions at 46% and 41%, respectively. When looking at individual, rather than group performance, our first benchmark is that 75% of individuals will achieve 50^thpercentile or higher on the four outcomes representing coursework that all psychology majors must take. These are psychological perspectives, research methods, statistics, and developmental. However, only 55% of individuals achieved the 50^thpercentile benchmark in these areas. The second individual benchmark is to have 75% of students achieve 50^th

Goal 2: Research Methods, Outcome 5 Design Basic Studies at the Advanced Level

• We piloted a new assessment rubric for this outcome on a subset of research proposals from Senior Seminar with the primary purpose of determining if the rubric was useable with senior-level work. We revised the rubric extensively because faculty evaluations using the original version varied a great deal. We will try out the new rubric next academic year.

Goal 5: Ethics in Psychology, Outcomes 1 (Recognize necessity of having ethical standards/acting ethically), 4 (Understand what it means to be ethical in practice), and 5 (Apply ethical principles in either research or practice).

• Students in our P390 Helping Skills course completed a 12-item, multiple-choice quiz designed to assess these three outcomes. Students achieved benchmark of 75% of students correctly answering 75% of items for each outcome. In fact, modal individual score was quite high at 92%, indicating students have a good understanding of the need for ethical guidelines in practice as well as what those guidelines are and how they apply.

Goal 7: Quantitative Literacy, Outcome 2 (Select Appropriate Statistics).

• Eight exam items relevant to selection of appropriate statistics from PSY K300 Statistics were used to evaluate this outcome.  The benchmark of at least 75% of students in the class getting a 75% or higher on this set of items was met; students are learning how to select appropriate statistics at an acceptable level in our program.

Link to full Psychology Assessment Report for 2009-2010

2008-2009

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology. We have seen encouraging improvement on this goal this year, although we still have several areas in which to improve. Students met our benchmarks for both of the sets of Cluster courses that are part of our major requirements: (Learning, Neuroscience, and Cognition) and (Personality, Social, Health, and Abnormal) as well as for Lifespan Development. Students performed below benchmark standards on knowledge about psychological perspectives, research methods, statistics, and social psychology, though we are starting to see improvement on psychological perspectives as more students take the now-required P459 History and Systems course. For both biological bases and social psychology, which are offered every other year, we have noted consistent patterns of performance relating to the number of students who have taken the class and when those classes are offered relative to the ACAT (e.g., students do better if they have taken the courses nearer the time of the ACAT). For statistics, we plan to investigate whether students are taking statistics earlier than they used to (resulting in a loss of information), and for both statistics and research methods we plan to look for ways we can help students carry these areas of knowledge through their other coursework, and to review the ACAT questions to determine their correspondence to our goals for the statistics and research methods classes. 

Goal 5: Ethics in Psychology, Outcomes 1 (Recognize necessity of having ethical standards/acting ethically), 4 (Understand what it means to be ethical in practice), and 5 (Apply ethical principles in either research or practice).

• Students in our P390 Helping Skills course helped us pilot-test a new measurement instrument for the ethics goal by taking a short multiple-choice quiz designed to evaluate the three ethics learning outcomes identified above.  All students achieved at least a 70% on this quiz; performance met benchmarks for Outcomes 4 and 5, but not for Outcome 1 (most likely because there were only two items on the quiz for this outcome). Students seem to come away from this course generally appreciating the need for ethical standards, and equipped to recognize practices which violate ethical standards (though there was some confusion about which specific standards were violated by particular actions given in example-type questions). This preliminary data indicates need to add more items to the quiz for Outcome 1 and illustrates the importance of discussing specific ethical standard violations in detail during the course and having students pay attention to that detail.

Goal 2: Research Methods, Outcome 3 Design Basic Studies to Address Psychological Questions

• Students in our P211 Research Methods course design and conduct a survey research project as a class and then each student writes an individual research report based on the project.  Students perform best on writing a research question, purpose, or objective and on writing a hypothesis appropriate to the research method (though they still do not meet benchmark performance criteria for the latter). Students particularly struggle with clearly specifying the hypothesis, writing a hypothesis that is appropriate to the literature review, and writing clear operational definitions, we have seen improvement in making the hypothesis appropriate to the literature review despite revisions to course assignments and course content emphasis. We have made as many changes as we can to the course without eliminating time devoted to other course content (which would be inappropriate) or increasing the number of credit hours (thus increasing instruction time). We plan to assess this same outcome at an advanced level via the research proposals students write in Senior Seminar to see if performance improves after students have matured as scholars and have been exposed to additional psychology coursework and more practice reading research in psychology. 

Goal 8: Career Development & Planning, Outcome 1 Demonstrate Basic Knowledge of Careers in Psychology

• Students in P199 Planning Your Psychology Career take a short quiz that directly assesses students’ knowledge of career paths as well as what is needed to either prepare for Bachelor’s level jobs or prepare for graduate education for those careers that require additional credentials. Students who complete P199 do very well (approx. 89%) on this quiz; we are pleased with achievement on this outcome, though students still struggle with understanding the difference between Ph.D. and Psy.D. degrees. This year, the psychology faculty agreed to shift P199 from a course required for all majors, to a highly recommended, but optional course. This was in response to several issues (e.g., students taking the course late in their program when it seemed less relevant for them, a lot of effort for a one credit course, high levels of non-attendance for a significant number of students).

Link to full Psychology Assessment Report for 2008-2009

2007-2008

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology. Students met our benchmark for performance on knowledge of content areas represented by our course cluster of P325 Learning, P326 Neuroscience, and P335 Cognition and came close to meeting our benchmark for the course cluster of P319 Personality, P320 Social, P303 Health Psychology, and P324 Abnormal. However, students performed below benchmark standards on knowledge about psychological perspectives, research methods, statistics, and lifespan development. We started a new requirement (beginning with newly admitted students in Fall, 2006) that students take a P459 History and Systems course which should soon start paying off in improved performance on psychological perspectives (none of this year’s test-takers had that course) and may help improve performance on lifespan development, methods, and statistics as the course includes review of a number of topics in psychology. The performance on methods and statistics represents a decline in performance from the previous year.

Goal 2: Research Methods, Outcome 3 Design Basic Studies to Address Psychological Questions

• Students in our P211 Research Methods course design and conduct a survey research project as a class and then each student writes an individual research report based on the project. Students performed best on writing a research question appropriate to the research method. Students continue to struggle with achieving benchmark performance on clearly specifying the hypothesis, writing a hypothesis that is both appropriate to the literature review and appropriate to the method, and writing clear operational definitions, we have seen improvement in making the hypothesis appropriate to the literature review (primarily because we have added an assignment that focuses specifically on this issue.

Goal 8: Career Development & Planning, Outcome 1 Demonstrate Basic Knowledge of Careers in Psychology

• This year, we piloted a new assessment tool (a short quiz) that directly assesses students’ knowledge of career paths and what is needed to either prepare for jobs one can get with a Bachelor’s degree in psychology or what is needed to prepare for graduate education for those career paths that require additional credentials. This quiz was given to students in our one-credit-hour P199 Planning Your Psychology Career course. Results indicate that by the end of the course, students have a very strong understanding of employment issues related to psychology. Students’ knowledge about IU Kokomo faculty recommendations regarding taking practica vs. independent research, and about differences between psychiatrists and psychologists and between Ph.D. vs. Psy.D. graduate degrees needs improvement.

Senior Exit Survey Results

• Students in our Senior Seminar course complete a survey about their experiences in the psychology program. Students continue to report strengths of the program as being: the faculty and variety of courses as well as writing and learning APA style. Students also reported that the program was very or extremely effective in helping them learn content of psychology, critical thinking, and application of concepts.

Link to full Psychology Assessment Report for 2007-2008

2006-2007

Goal 1: Knowledge Base in Psychology

• Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test (ACAT), a nationally normed, multiple-choice test which includes questions from 10 different topic areas in psychology. Because we had only four students in the course this year, we combined this year’s data with that from the previous year’s students. Students met our benchmark for performance on knowledge of content areas represented by our P211 Research Methods and our K300 Statistics courses (very close to meeting benchmark) as well as our benchmarks for knowledge of content areas from our course cluster of P325 Learning, P326 Neuroscience, and P335 Cognition and from our course cluster of P319 Personality, P320 Social, P303 Health Psychology, and P324 Abnormal. However, students performed below benchmark standards on knowledge about psychological perspectives and lifespan development. We started a new requirement (beginning with newly admitted students in Fall, 2006) that students take a P459 History and Systems course which we hope will help improve performance on psychological perspectives. Low performance on lifespan development content may be due to the fact that this 200-level course is taken very early in their program by most psychology majors.

Goal 2: Research Methods, Outcome 3 Design Basic Studies to Address Psychological Questions

• Students in our P211 Research Methods course design and conduct a survey research project as a class and then each student writes an individual research report based on the project. Students continue to perform best on writing a research question and a hypothesis appropriate to the research method. Students continue to struggle with achieving benchmark performance on clearly specifying the hypothesis, writing the literature review, writing a hypothesis that is appropriate to the literature review, and writing clear operational definitions. However, we made changes to the course this year regarding choices of project topics, instruction on writing the review, and a new assignment on making the hypothesis match the literature review, and we have seen corresponding improvements in performance. Overall pattern of performance improved by 30% from last year.

Goal 8: Career Development & Planning, Outcome 1 Demonstrate Basic Knowledge of Careers in Psychology

• Students in our one-credit-hour P199 Planning Your Psychology Career course self-reported improvements from beginning to end of semester in their familiarity with types of jobs available at the psychology Bachelor’s degree level and their familiarity with the types of careers that graduates from our program have attained.

Senior Exit Survey Results

• Students in our Senior Seminar course complete a survey about their experiences in the psychology program. Students reported that the program was very or extremely effective in helping them learn content of psychology, including research methods and ethics, and in helping them improve their critical thinking and communication skills.

Link to full Psychology Assessment Report for 2006-2007

2005-2006

Goal 1: Knowledge Base ACAT results

Students in our Senior Seminar course took the Psychology Area Concentration Achievement Test. Students scored above our 50 th percentile benchmark in the following topic areas of psychology: abnormal, clinical/counseling, developmental, experimental design, human learning/cognition, physiological, social, and statistics. Students scored below our 50 th percentile benchmark in history and systems (a course our students currently are not required to take) and in personality (also not a required course, plus our course places less emphasis on names of researchers and theorists than does the test). Based on these results, we have changed major requirements such that all students will take a history and systems course.

Goal 2: Research Methods, Outcome 3
Design Basic Studies to Address Psychological Questions results

Our students performed best in the areas of writing a research question or purpose and writing a hypothesis that is appropriate to the research method used. In an effort to improve students’ ability to write an appropriate literature review and to write a hypothesis that is appropriate to that literature review, we have modified an assignment so they now have to think and write about the relationship between the background literature and their hypotheses early enough in the semester to get feedback from the instructor before they write their research reports. These results are based on the survey research reports written by students in our Methods of Experimental Psychology classes.

Career Planning and Development results

Students taking our Becoming a Psychology Major course reported that from beginning to end of the semester, they improved their ability to gather information about psychology careers, increased their knowledge of what those careers entail, and increased their interest in earning a minor. Self-assessment assignments reveal students are aware that procrastination will keep them from achieving their goals, but they are NOT aware of how to improve. In the short-term, we plan to focus a class period on time management, with the possibility of a workshop on time management and procrastination avoidance in a future semester.

Senior Exit Survey results

The majority of students taking this survey reported that the psychology program was very or extremely helpful in helping them learn the content of psychology, including research methods, and in helping them develop their critical thinking. Students reported that the psychology program’s greatest strength is the high quality of the faculty.

Link to full Psychology Assessment Report for 2005-2006