Case Study

Juried Assessments at Three Unique Schools

The Juried Assessment tool in Tk20 makes it easy for multiple reviewers to evaluate student performance based on observation, coursework, portfolios, or other artifacts. Juried Assessments may be used to anonymously assess student work that is housed both in and out of Tk20. There are many benefits to using this functionality, such as determining whether instructors are using an assessment tool in a similar fashion or determining whether a rubric is truly assessing what you intend for it to assess. Many institutions have implemented Juried Assessments in general education to determine whether students have learned similar concepts across courses and sections. In this case study, the University of St. Thomas, California State University Channel Islands, and Texas State University will share how they use Juried Assessments on each of their campuses.

A Case Study by Sarah Smith, Business Analyst at University of St. Thomas, Dr. Marie Francois, Director, University Experience Program and Professor of History at California State University Channel Islands, and Yaoling Wang, User Service Consultant at Texas State University.

Facilitating Collaboration at the University of St. Thomas

Juried Assessments provide opportunities for faculty to collaborate on assessment. Teamwork is evident in the Modern and Classical Languages department which uses Juried Assessment to assess speaking, writing, and cultural reflection. The assessments take place in a proctored language lab. For the speaking assessment, students are given a series of written prompts and contextualized images. Students are asked to respond to the prompts by speaking in their language of study. An audio file with each student’s response is created. For the writing and cultural reflection assessment, a written prompt requires spontaneous writing by the student without access to online translation tools or assistance from tutors.

The Juried Assessment process includes sending a Tk20 assignment through courses to the students. Students log into Tk20 and upload the appropriate file. A simple rubric is attached to the assignment allowing the instructor to check off whether the student completed the assignment or not. Next, the Juried Assessment is created to randomly assign jurors and associate the student’s file with the appropriate rubric. St. Thomas faculty members have found Tk20 Juried Assessment to be very easy to use. Upon logging into Tk20, the assessor clicks on a link which displays the rubric on one side of the screen and the associated file can be opened on the other side. Since the speaking assessment is an audio file, the juror listens to the audio while completing the rubric. If the assessment involves written information, the juror can view the written submission on one side of the screen and complete the rubric on the other side.

Last semester, Modern and Classical Languages held an “Assessment Party” to make the assessment more enjoyable for faculty. At the Assessment Party, faculty were trained on how to complete the assessments and then had time to complete some or all of their assessments. A sense of camaraderie was evident as the faculty discussed the rubrics and possibilities for making improvements. Juried Assessments engage many faculty in assessment which creates opportunities for ongoing collaboration and camaraderie.

Tk20 is an excellent tool for facilitating Juried Assessments and establishing assessment processes. As Ashley Shams, Modern and Classical Languages assessment committee chair, said, “Another positive from the Tk20 experience for me is that it will ease the transition between me and the next committee chair. I have peace of mind knowing all our documents are stored there and that the next ‘committee generation’ can easily access it.”

Validating Outcomes-Based Gen Ed at CSU Channel Islands

Juried Assessments serve as a validation tool with faculty as CSU Channel Islands transitions to an outcomes-based General Education (GE) platform and as part of research studies on curriculum design. Assessment rubrics are drafted to align with the GE outcomes and need to be vetted with student work from across lower and upper division courses from multiple disciplines. The process begins with a face-to-face meeting with a group of interdisciplinary faculty using anchor papers to apply the rubrics, and often revise the rubrics. This also serves as a norming session to calibrate scoring (the rubrics are on a 4 point scale). A Juried Assessment is set up through Tk20 where faculty use the revised rubric on additional student papers. The system is intuitive enough that most faculty just get an email about accessing their Tk20 account and do the assessments without any additional training. At the follow-up meeting, the Juried Assessment report from Tk20 is used to see how close we are and how suited the rubric is to our needs. Depending on the result, it is applied to another batch of papers or finalized. This process is repeated with each new rubric to ensure its validity as an assessment tool.

Channel Islands also uses Juried Assessments relating to a research study on First Year Experience redesign. The same Critical Reasoning assignment, tied to a GE rubric, is assigned in sections of the old course (UNIV 110) and the new course (UNIV 150). Blind assessments of all the essays in both groups are executed through the Juried Assessment tool (over 200 essays are assessed by 2 reviewers each). Assessment data is triangulated with pre and post scores on the Watson Glaser Critical Thinking diagnostic, which aligns well with criteria on the GE critical reasoning outcome rubric. In the future, Channel Islands plans to use Juried Assessment to sample coursework assessed by faculty for a General Education Program review of outcomes achievements, both for aggregate measures of student learning to inform continuous improvement of teaching and learning, and as quality control on faculty assessments.

Assessing Career Readiness at Texas State University

Juried Assessment is used as a part of the Southern Association of Colleges and Schools (SACS) Quality Enhancement Plan (QEP) entitled PACE: Personalized Academic and Career Exploration at Texas State University. The PACE plan is being presented to SACS as a part of the reaffirmation of accreditation process. The purpose of PACE is to help students clarify their career goals in the early years after they enter the university, and assist them in developing and implementing their academic plan to meet their career goals.

One of the important components of PACE is University Seminar (US1100), a freshmen level course, that is required of all freshmen. In order for students to demonstrate their accomplishments in US1100, they need to complete three assignments, which include 1) self-assessment of career readiness, personalities, interests, and reflections on self-assessment; 2) research future career scenarios by conducting informational interview with professionals in a potential field, and reflection upon the interview; 3) development of their academic and co-curriculum plan, completion of a resume to help them develop their professional images and communication skills. Through expanded curriculum in academic and career planning, students will have the opportunity to explore their personal interests, abilities, and values in conjunction with various academic majors and related career options. At the end of each semester, a PACE assessment committee is formed by the University College to assess a random sample of US 1100 students’ work.

The assessment process starts with sending three assignments via three course binders to all US1100 course sections at the beginning of each semester. Instructors use three respective rubrics to assess students’ submissions on the course level. At the end of the semester, the three course binders are attached to a portfolio, which is configured for Juried Assessment. Last fall, 12 jurors assessed almost 1000 students’ portfolios. The process went smoothly. They all proclaimed that Tk20 is very easy to use. It made the process simpler, especially with the capability of assigning a tie-breaker. Presently, PACE is the only program that is using juried assessment at Texas State University, but quite a few other programs have expressed interest in using it for other assessment purposes. Texas State is looking forward to expanding the use of juried assessment.

 

As evident by these institutional stories, Juried Assessments can build collaboration and camaraderie among stakeholders, can help determine the validity and reliability of assessment tools, and can be applied to an assortment of scenarios at colleges and universities.