State and school officials, students getting first-hand look at computer testing

They didn't always understand exactly what information was being sought.

Not having access to a pencil and a piece of paper was overwhelmingly their most common complaint.

And, to no one's surprise, almost to a student, the pupils in Gretchen Yorke's Mesa High School 11th-grade English class barely batted an eye at the fact that, for the first time, the test was being administered via computer program.

It is exactly this type of feedback - in addition to how the students fared on the test questions themselves - that school and state officials are seeking from small-scale trials of new computer-aided assessments that will be used starting in 2014-15.

"This is all part of us trying to be prepared for the roll-out of the new assessments," said the California Department of Education's Deb Sigman, personally overseeing Monday's classroom trial at Mesa.

"The idea of these trials is to see how kids engage with the process and respond to the assessment items - that's what we'll be looking at as we move forward with the development of the assessments," said Sigman, in charge of the CDE's assessment and accountability division.

The assessments, being developed by a state consortium known as SMARTER Balanced, are aligned to the new common core curriculum standards in English and mathematics, which state officials are attempting to bring to classrooms by next fall. As a member of the consortium's executive committee, Sigman is helping to oversee development of the new assessment system, ultimately being designed as a computer-adaptive model - one in which the computer program adjusts the level of each question based on an individual student's answer to the preceding question.

Monday's trial, however, was simply computer-aided, meaning the assessment is conducted on a computer rather than the standard paper-and-pen multiple choice test.

Mesa, in the Sacramento-area San Juan Unified School District, is one of 500 schools in 23 SMARTER Balanced member states participating in the assessment trials, continuing through next month.

Classes in grades four, seven and 11 will be given between 60 and 90 minutes to complete the trial assessment, which features a mixture of selected-response, constructed-response and performance task essays, and technology-enhanced items.

The American Institutes for Research, which is administering the trials, will use the student responses to evaluate and test automated scoring strategies, including an evaluation of the open-source item scoring software being developed for the consortium by AIR.

But both Sigman and San Juan Unified officials said they'll be forwarding other suggestions to program developers based on student comments as well as their personal observations.

For example, said Sigman, two female students in the Mesa junior class said they preferred the pen-and-paper tests, pointing perhaps to a need for accommodations for "more tactile" learners or those will less technological experience.

A majority of the students also agreed that being able to use scratch paper to work out math problems or jot essay ideas down would have been helpful during the trial. Officials on Monday said they weren't sure if that option had been considered but were going to pass it on to the consortium for discussion.

Several students said it was sometimes unclear what information the question was seeking, and several others said they thought the trial assessments were more difficult than the currently-used Standardized Testing and Reporting (STAR) assessments.

Few, if any, had difficulty maneuvering through the computer program. In fact, noted Sigman, most skipped through the instructions, intuitively figuring out the process as they went.

Mesa was one of four San Juan Unified schools participating in the trial. The others included an elementary school, a middle school and an independent charter school.

While officials have known for several years that the new assessments were going to require technological changes or additions at many schools, the test trials are bringing that reality home.

"Mesa has a robust wireless system but many schools don't," said Donna O'Neil, director of San Juan's accountability and evaluation department. The district has a small team beginning to "map out" a plan for transitioning existing technology to mesh with the new assessments.

"Schools are going to have to think about their tech purchases," O'Neil added, pointing out that with K-8 schools having to assess all third through eighth graders, computer capacity and bandwidth are among the top concerns.

But in addition to funding issues, those decisions are also complicated by the pace at which technology advances, said Mesa principal Rick Messer.

Computers are already being made smaller and smaller, with more and more users moving to the more mobile tablet- or notebook-style. While the new assessment program isn't designed to be taken on these types of computers, the school officials on Monday said they've been told it soon will be.

Still, said Messer, "We don't know what the world's going to look like in three years," he said.

Research from the trial will inform the development of more than 30,000 items and performance tasks for the operational administration of the Smarter Balanced assessment system in the 2014-15 school year.

Smarter Balanced will conduct a Pilot Test of the assessment system beginning in February 2013. A broader Field Test will be administered in spring 2014.