Rapid Hub Analytics: A Focus on Continuous Improvement
By Paul LeMahieu
Carnegie has made considerable
progress on developing an information system that integrates real-time
faculty and student data with institutional records to inform continuous
improvement as we address problems of educational practice and
constantly improve our ability to serve developmental mathematics
students at community colleges.
For example, we are designing a system
that will incorporate institutional records going back to 2008 on the
longitudinal performance of cohorts of students designated for
developmental mathematics at each of the 30 colleges participating in
our community college mathematics pathways
initiative. These data constitute a baseline for understanding
institutional performance over time, for establishing college-by-college
improvement targets, and for exploring the antecedents and conditions
of performance going forward. Building off of this, these two pathways —
Statway™ and Quantway™
— draw additional detailed data from weekly faculty facilitator calls,
faculty forum discussions, two-minute faculty and student surveys
reported on a lesson-by-lesson basis, supplemental background data on
students, dispositions toward success and math tests of fundamental
concepts, follow-up student surveys, data feeds from the out-of-class
digital platform that include student homework and other out-of-class
activities, common topical and end-of-module assessments, as well as
end-of-course assessments and course grades. It is all data for the
improvement of teaching and of the materials.
Analytics is now prototyping continuous
data feed reports to faculty on their classroom context and individual
student progress. Identifying places where rapid interventions might
occur is one driver in our overall plan for advancing student success.
Specifically, analytics address four broad purposes:
- To inform faculty so they can more effectively address the needs of students – Routine reports describing the “classroom ecology,” including students’ language background, interest and engagement in mathematics, and entering levels of basic mathematics skills can inform faculties’ curricular and instructional planning from the very outset of the course. Predictive models developed through analytics can forecast students’ needs (including real time indicators of engagement or disassociation) to enable faculty to tailor interventions and support services for individual students.
- To identify needed improvements in the instructional system – Data from numerous sources, especially those that track student engagement and achievement, but also including frequent faculty and student feedback, are analyzed to determine where instructional improvements may be needed. This includes possible changes to the instructional kernel itself — curricular materials, the out-of-class platforms and assessments. It also includes identifying areas of faculty knowledge and skill that may require attention in the advancing quality teaching strand.
- To test proposed changes as potential improvements – As modifications to the instructional system are proposed, they are prototyped and rapidly tested using the techniques and methodologies of improvement research to ascertain whether these changes can be warranted as improvements. In this manner, Carnegie aims to ensure effective implementation across locations and contexts.
- To examine and test for impact and accomplishment of the whole effort – The data that are gathered to enable improvement also allow us to examine the overall impact of the Pathways initiative. We are now in the process of developing value-added estimates for each classroom, college and sub-group of students and will compare these against benchmarks established this first year and against results for similar students not enrolled in the pathways in each college. We intend to continue to track the success of students post-pathway through the following academic year. Understanding “what happens next” for students is another key to improving what we do with them while they are actually enrolled in a pathway.
Our overarching goal is to enhance the
capacities of participating faculty to engage in systematic program
improvement. We pursue this both for the sake of their fullest
participation in the Statway™ and Quantway™ Networked Improvement
Communities and to develop the overall capacity of member institutions
to use the approaches, tools, and techniques of quality improvement to
enhance the diverse program implementation efforts in which they may
engage. Ultimately, the biggest payoff of all may reside there.
————
This post was written with contributions from Carnegie colleagues.
RECENTLY ON THE PATHWAYS BLOG
Lesson Study the Carnegie Way: Sharing Ideas for Continuous Improvement
Some of the News Fit to Print
ABOUT K-12
FEDS AIM TO SPARK FRESH THINKING ON SCHOOLING
As the private sector works faster and more boldly to churn out next-generation technology and embrace cutting-edge practices, the U.S. Department of Education and its partner federal agencies are ramping up their efforts to bring more spark and innovation into elementary and secondary schools. Under President Barack Obama, the administration has updated education technology and broadband plans that seek to set a national vision, launched a competition to reward school districts and nonprofits for innovative ideas, and started a pilot project to allow federal money to pay for mobile devices to put digital learning within reach for more students. The article is in Education Week.
RANDI WEINGARTEN: STOP THE TESTING OBSESSION
Educators and policy-makers from twenty-three nations gathered in New York this week for the second International Summit on the Teaching Profession, hosted by the Organization for Economic Cooperation and Development. The purpose of the summit was to identify effective reforms for improving teacher quality. Notably, the research paper released in conjuntion with the event showed that compared to the United States, other nations put little faith in student test scores as a measure of teacher quality; the phrase "value-added," for example, never appears in the 103-page report. Instead, top-scoring nations like Finland and China have focused on improving training before teachers enter the classroom, and on making education a more attractive career choice by providing teachers with opportunities to participate in curriculum writing, group lesson planning and other professional activities alongside other adults. American Federation of Teachers President Randi Weingarten attended the summit. Here are her reflections on what the United States can learn from international education reform efforts, which she also had the opportunity to witness firsthand on a recent trip to Shanghai, Japan and Singapore. The interview has been consensed and edited for clarity. The article is in The Nation.
CLIMATE OF DISRESPECT FOR TEACHERS GETS WORSE
Gregory Michie writes in the The Washington Post’s The Answer Sheet blog: Just when you think the climate of disrespect for teachers can’t get any worse, it does. The Chicago Tribune’s editorial board has now urged Illinois parents to demand that the state emulate New York City (and Los Angeles) by making individual teachers’ “value-added” ratings available for public scrutiny. For those not versed in the latest trend in educational accountability, “value-added” formulas are complex statistical calculations that attempt to isolate a teacher’s impact on a student’s growth — as measured by gains on standardized test scores. In essence, value-added measures try to show how much of a student’s test score increase can be chalked up solely to the influence of their teacher. If that sounds tricky and imprecise, that’s because it is — and there’s no research to show that value-added models have done anything to help teachers improve, or to help kids learn.
ABOUT HIGHER ED
LESS REMEDIAL INSTRUCTION IS THE GOAL
In the California State University system, which accepts the top third of high school graduates, most freshmen have to take basic composition courses or review algebra in classes that don't count toward a degree. In 2010, 57 percent of CSU freshmen required remediation in English or mathematics. The rate is higher in the California Community Colleges system, where about 85 percent are unprepared for college-level math and 70 percent are unprepared for college-level English. As state resources dwindle, education officials are searching for ways to hasten students through the system anddevote resources to those who will earn degrees. Yet unprepared students are at greater risk for staying in school longer or dropping out. The article is in the Sacramento Bee.
STATE AND LOCAL SPENDING ON HIGHER EDUCATION REACHED A NEW 25-YEAR LOW IN 2011
As if anyone associated with public higher education needed a reminder, 2011 was a lousy year for higher-education finance. A new report from the State Higher Education Executive Officers confirms just how awful it was: State and local money for higher education fell to a quarter-century low for the second consecutive year, while enrollments continued their climb to record highs. From the beginning of the recession, in the 2007-8 fiscal year, through the 2011 fiscal year, college enrollment increased nationally by 12.5 percent, to 11.5 million students, the report says. But state and local appropriations have decreased by $1.3-billion over the same period. The article is in The Chronicle of Higher Education.
ABOUT HIGHER ED
TRYING TO ASSESS LEARNING GIVES COLLEGES THEIR OWN TEST ANXIETY
The Collegiate Learning Assessment, launched in 2000, has brought rare scrutiny to higher education. Until now, colleges have been largely exempt from the accountability movement sweeping through public elementary and secondary schools yielding the No Child Left Behind law and other initiatives. In a landmark study published last year, sociologists Richard Arum and Josipa Roksa used the test to measure collegiate learning in the nation. Using data drawn from a sampling of public and private colleges, they shook the academic world with a finding that 36 percent of students made no significant learning gains from freshman to senior year. The article is in The Washington Post.
DROPUTS ‘UN-COLLEGE’ PATH TO SUCCESS
For many people, attending college is part of the American dream, a gateway to success and a good job. But, as many recent college graduates are learning the hard way, dreams don't always become reality. It didn't take a diploma for twenty-year-old Dale Stephens to figure that out. Disillusioned with college because he felt he was not getting the skills he wanted, he dropped out and started "UnCollege," a social movement that challenges the notion that college is the best path to success. The article is from Voice of America.
FIRST-YEAR STUDENTS’ FINANCIAL CONCERNS ON THE RISE
Today's freshmen are more focused on the financial benefits of a college education than were their counterparts four decades ago. Freshmen now are also more racially and ethnically diverse, harbor higher expectations for the college experience, and are increasingly interested in pursuing graduate degrees. Those are among the findings of the Cooperative Institutional Research Program, part of the Higher Education Research Institute at the University of California at Los Angeles, which will soon issue a wide-ranging examination of 45 years of responses to its annual Freshman Survey. The article is in The Chronicle of Higher Education.
ABOUT K-12
NEW TEACHERS GETTING READY TO BE GRADED ON CLASSROOM WORK
Anyone completing a teacher-education program and seeking a teaching license in Wisconsin will have to demonstrate their skills through the Teacher Performance Assessment, a portfolio-based assessment. Teaching candidates will have to submit lesson plans, reflections of their work and a video of their classroom interactions with students as part of the Web-based program. The article is in the Milwaukee Journal Sentinel.
TRAINING OF TEACHER EVALUATORS EXAMINED
Formal training of the principals and other observers conducting teacher evaluations is a complex, necessary, and often overlooked component of the systems, concludes a new paper written by the experts who oversaw the training and scoring of thousands of teachers' lessons, as part of the Bill & Melinda Gates Foundation's Measures of Effective Teaching Project. Such training, including the certification and testing of observers, will help to ensure that judgments of teacher practice are valid and reliable for the purpose of professional development and other decisions, they assert. The post is from Education Week’s Teacher Beat blog.
FEDS AIM TO SPARK FRESH THINKING ON SCHOOLING
The digital divide will be an enormous hurdle to overcome as the Obama administration pushes for national adoption of e-textbooks for all students by 2017. This month, the administration is expecting to convene a group of CEOs in the digital-publishing industry to jump-start the effort. "Education is one of the most important challenges we face as a country. Part of the solution is new technologies: ...digital devices, digital textbooks," said FCC Chairman Julius Genachowski during a national town hall meeting last month that was focused on digital learning. "It will require educators, companies, parents, everyone in the ecosystem to come together to agree on a goal and work hard to get there." The article is in Education Week.
AT ELITE COLLEGES, TOO MUCH HUBRIS?
Carnegie Board member and LaGuardia Community College President Gail Mellow writes in a letter to the editor in The New York Times: There is a potent antidote to Andrew Delbanco’s fears that our most prestigious colleges’ elitism encourages smugness among their students. If each Ivy League college committed to accepting transfer students from community colleges for 1 percent of their junior classes, privileged students would begin to encounter students from the other 99 percent. Research confirms that low-income and minority transfer students, after being transformed by a community college education, graduate at the same or higher rates than students who began at the same college. Making transfers a part of the elite colleges would provide living proof that intelligence, drive and achievement are not the sole province of students born with good fortune, but are as alive as the American dream among strivers at community colleges.
posted Mar 14, 2012 10:10 am
Daily News Roundup, March 14, 2012
Some of the News Fit to Print
ABOUT HIGHER ED
THE HIGHER EDUCATION MONOPOLY IS CRUMBLING
Kevin Carey writes in The New Republic: The historic stability of higher education is remarkable. As former University of California President Clark Kerr once observed, the 85 human institutions that have survived in recognizable form for the last 500 years include the Catholic Church, a few Swiss cantons, the Parliaments of Iceland and the Isle of Man, and about 70 universities. The occasional small liberal arts school goes under, and many public universities are suffering budget cuts, but as a rule, colleges are forever. I think that rule is going to change, and soon.
BIG JOB, BIG PROBLEMS
The top job for an enormous swath of American higher education is opening up. It offers the potential to play a lead role in determining the success of the national college “completion agenda,” as well as a laundry list of problems daunting enough to intimidate even the most ambitious of applicants. At the end of August, when Jack Scott steps down as chancellor of California’s community college system, his successor will take on a budget with a hole of at least a half-billion dollars in this cycle alone. The new chancellor will also arrive amid a heated debate over priorities as the system, which will turn away an estimated 200,000 students this year, wrestles with a plan to for the first time restrict access intentionally and give first dibs to students who are most likely to succeed. The article is in Inside Higher Ed.
ABOUT K-12
TEST DRIVING A PILOT EVALUATION SYSTEM
Education officials agree that test scores, alone, are not a sufficient way to rate a teacher. That is why New York State is using its $700 million federal Race to the Top grant to develop a new teacher evaluation system in which test scores will count toward 40 percent of a teacher’s rating. The other 60 percent will come from observing teachers at work. New York City is considering using a new framework for evaluating teachers designed by Charlotte Danielson for that 60 percent. The article is in The New York Times.
POLL FINDS MOST VOTERS FAVOR RELEASE OF TEACHER RATINGS
A majority of New York City voters approve of the public release of ratings for thousands of public school teachers, even though a plurality of voters believe that the ratings are flawed, a new poll has found. The poll, released early Wednesday by the Quinnipiac University Polling Institute, found that 58 percent of voters approved of the release of the ratings, known as teacher data reports, while 38 percent disapproved and 5 percent were undecided or did not answer. The poll was conducted less than two weeks after the teacher rankings were made public by the city. The article is in The New York Times.
KHAN ACADEMY CREATOR TALKS ABOUT K-12 INNOVATION
Salman Khan, a graduate of the Massachusetts Institute of Technology and the Harvard Business School, was working as a hedge fund manager when he began posting videos on YouTube six years ago to tutor young family members in math. That led to the 2008 creation of the Khan Academy, a nonprofit organization that has built a free, online collection of thousands of digital lessons (nearly 3,000 of them created by Mr. Khan himself) and exercises in subjects ranging from algebra to microeconomics. Education Week Staff Writer Lesli A. Maxwell recently interviewed Mr. Kahn about the evolution of the academy and its potential for changing K-12 education.
INDIANA OVERHAULS HOW IT EVALUATES TEACHERS
Indiana is in the midst of a massive education reform effort that includes the creation of vouchers, increasing the number of charter schools and adopting a new system to hold schools accountable. For the first time, that includes taking control of failing schools away from districts. It could be argued, however, that no reform measure will have more impact in the classroom than a new state program that is being piloted in six Indiana school districts and is about to be unleashed in every school across the state. Beginning next school year, the way teachers are evaluated—in essence, graded—will change dramatically. The result, reformers predict, is that large numbers of bad teachers will be tossed out, good teachers will be rewarded, and teacher quality will be raised in classrooms across Indiana. The article is from the Hechinger Report.
WHEN TEACHERS ARE PUBLICLY QUESTIONED
Emery Petchauer blogs in Diverse's The Academy Speaks: How the United States decides what a “good teacher” is and what we do with that information has gained national attention in the past two weeks. The controversy involves “value-added” measurements of teacher effectiveness, which evaluate teachers based upon their students’ one-time standardized test scores. A number of scholars and writers have set the proverbial record straight in the past week, including Stanford’s Linda Darling-Hammond and EdWize education wire. These articles reiterate the shaky practice of value-added teacher evaluations; very appropriately scold the Post, Mayor Bloomberg, and the NYC Department of Education for such a practice; and offer a more accurate portrayal of Pascale Mauclair, who is a perfectly fine teacher by all reasonable standards. What has not been addressed with this issue to sufficient depth is the following question: How do these types of attacks affect students who wish to become teachers? This is not a rhetorical question. I spend almost every day with these students educating them to be teachers, so I know the answer.
ABOUT HIGHER ED
REMEDY FOR REMEDIAL COURSES
It's dispiriting to open a community college’s course catalog and see page after page of course descriptions on math fundamentals, including fractions and percentages. Massachusetts can cut down on this problem by testing students’ readiness for college work while they’re still in high school - and sparing them from spending their savings or financial aid on remediation courses that don’t even count toward graduation requirements. This editorial is in The Boston Globe.
COMMUNITY COLLEGES IN LINE FOR FEDERAL LARGESSE
California, home to a quarter of the nation's community college students, could reap huge benefits from President Obama's $8 billion plan to pair local businesses and schools. Bay Area community colleges are expected to compete for the money aimed at teaching students the skills they need to fill job opportunities in their regions. The article is from the San Francisco Chronicle.
