The following is an opinion article from Moak, Casey and Associates. A printer-friendly version of this article is included below as a pdf attachment.
Apples, Oranges, and Inside
Baseball: Key Components to Understanding 2017 STAAR Results
The
Texas Education Agency's recent release of statewide 2017 STAAR results seems
to have generated more heat than light about student achievement: Did
performance improve, decline, or stay the same compared to prior years? In each of the three newspaper articles listed
below, the reporters attempt to explain what they believe is a “drop” in STAAR scores
when the 2017 results are compared to the 2016 results. Their work provides classic examples of how "apples-to-oranges"
comparisons and the need to know technical details – “inside baseball" –
affect accuracy and simplicity in reporting. Each of these issues is discussed in more
detail below.
·
July 7, 2017: Dallas
Morning News - Texas students losing ground on STAAR
tests https://www.dallasnews.com/news/education/2017/07/07/texas-students-losing-ground-staar-tests
·
July
10, 2017: Drops in Texas STAAR scores raise questions — and testing angst. http://www.mystatesman.com/news/drops-texas-staar-scores-raise-questions-and-testing-angst/60anViKo3ipmhKYTQKJ7QN/
·
July
13, 2017: Houston Chronicle - Newest STAAR exam results add up to difficult
math problem for HISD, local districts http://www.chron.com/news/education/amp/Fewer-Texas-Houston-students-pass-state-s-11286107.php
Apples-to-Oranges
Most
people are simply unaware that trying to compare the 2016 STAAR results to the
2017 results is like comparing “apples-to-oranges.” To understand that the two
years’ results are not comparable requires one to know, remember, or even find information about the multiple
changes made to the 2017 STAAR tests which render their results different from
the prior year’s results. That is, one
has to know a little “inside baseball.”
Commissioner
Morath’s January 30th blog outlines a list of changes that TEA made to the
STAAR program to address “multiple issues
on multiple fronts.” The specific changes for 2017 that were listed in his blog
are shown below, and provide evidence as to how the 2017 tests differ from the
2016 tests.
- Shortening the grade 4 and grade 7 writing
assessment (from a two-day) to a one-day administration.
- Designing and implementing shorter test
blueprints for the 2017 administration of STAAR in grades 3–8.
- Redesigning the three English end-of-course
blueprints for the spring 2017 administration so that the reading sections
no longer contain short answer questions.
- Incorporating changes to the Algebra I curriculum
to correspond to the end-of-course assessment.
- STAAR
A and STAAR L were administered for the final time in December of 2016. Beginning with the spring 2017 administration,
the embedded supports that were available on these assessments are now
individual embedded online accommodations on STAAR for eligible students.
Information about these embedded supports and other accommodations, or Designated
Supports, can be found on the Accommodation Resources webpage. (emphasis added)
- Accommodations policies were revised so that more
of the decisions regarding the appropriateness of an accommodation for an
individual student are made at the local level. In most cases, accommodations eligibility widened
significantly. (emphasis
added)
And
that’s not all of the “inside baseball” one needs to know! In the past, TEA
provided different paper versions of the STAAR tests. Regular STAAR, STAAR L,
STAAR A, STAAR M (which was discontinued in 2015), STAAR Alternate (which was
discontinued in 2015) and STAAR Alternate 2 were unique test versions, each designed
to assess the learning of specific groups of students. (For more information about
unique STAAR test versions, please refer to TEA http://tea.texas.gov/Student_Testing_and_Accountability/Testing/)
TEA
changed the testing program with the Spring 2017 STAAR administration. Gone are
the different paper test versions; instead, students who in previous years took
a different test version, now are required to log into a computer to take an
online version of STAAR that contain a variety of accommodations, or designated
supports, which are “changes to materials
or procedures that enable students to access learning and testing.” TEA’s Accommodation Resources webpage lists three
main categories of accommodations and/or designated support options for
students:
1.
Accessibility Features – these are NEW features beginning with the Spring
2017 test administration. “These are
procedures and materials that are allowed for any student who needs them.”
2.
Designated Supports – which requires ARD, LPAC, and 504 committee approval before a
student may access these supports
3.
Designated Supports Requiring TEA Approval – before a student may access these supports
Therein
lies part of the problem. TEA’s press release did not mention that embedded support test results were included in
the 2017 statewide STAAR summary reports – which is quite different from how results
were reported in prior years. The agency did
not include a footnote on the statewide summary reports to alert readers to
this fact: the summary report titles look identical to prior year summary report
titles. TEA did not publish a separate
summary report showing the number of students who took an embedded support test
and the percent who passed that test. Without a little inside baseball –
knowing these substantive changes, or where to find the information about them
– most people could not realize that comparing the 2016 to the 2017 STAAR results
was largely inappropriate, just like comparing apples to oranges. (Appendix A
shows the apples-to-oranges comparison.)
Red Apples-to-Green Apples
Unfortunately,
it is impossible to prepare the most appropriate comparison between the two
years’ results with the current, publicly available data. To create a true
“apples-to-apples”, year-over-year (YOY) comparison, the agency must release
the embedded support test results separately so that recalculations of the most
accurate YOY comparisons can be made.
However,
MCA can (and did) create a “red
apples-to-green apples” YOY comparison by aggregating the 2016 STAAR, STAAR L, and STAAR A statewide test results and then
calculating the one-year change relative to the 2017 STAAR with embedded
support test results. MCA finds that if this type of modified comparison were
used (which we acknowledge is still not the most statistically accurate comparison),
the headlines would be very different from what was reported using the
apples-to-oranges comparison. When comparing “red apples-to-green apples”:
- Reading for all
grade levels, except grade 7, declined slightly. 7th grade reading
improved by three percentage points. And instead of a 7 percentage point
drop in 4th grade reading and a 6 percentage point drop in 8th
grade reading as was reported, scores dropped 5 and 3 percentage points,
respectively.
- Math scores increased
at all grade levels. The increases ranged from a one percentage point gain in 7th
grade to five percentage point gains in both grades 5 and 8.
- First-time EOC
testers improved on 3 out of the 5 subjects, including Algebra I, which
increased by five percentage points. Performance in the other 2 subjects (Biology
and English II) remained relatively flat.
(Appendix A also shows the red apples-to-green apples comparison.)
What else is important to know?
It
was surprising to see that the number of students testing in 4th
grade this year increased dramatically from the number tested in 2016. More than 13,000 additional students tested in
4th grade reading (and math and writing) in 2017 than were tested in
2016 (as shown in Appendix B).
What
we do not know is how many students
took an embedded test and how they performed as a group. We don’t know who /
which students took an embedded test. We don’t know if there is an
over-representation from one particular student group or not. And we also don’t
know how or even if this significant increase in the number of students has impacted
teaching, learning and student performance on STAAR.
Until
TEA publishes the full complement of facts about the 2017 embedded support test
results, opinions will continue to vary as to what the YOY change in STAAR
performance mean on a statewide basis. Cypress-Fairbanks ISD, and all other
districts that are treating the 2017 STAAR with embedded support tests as new
baseline results (as per the Houston Chronicle article), provide some
much-needed light on the very heated debate about student performance in 2017. Others may wish to emulate these exemplars. In
addition, districts would be wise to prepare for a much larger group of 5th
grade students in SY 2017-18, who may need a variety of new designated supports
and accelerated instruction to be successful in the upcoming school year.