Students in rural schools in China are falling behind. The average rural student has 1/10th of the chance that an urban student does of attending university. Almost all urban children attend high school; only 37% of rural children do.
Beyond this rural-urban gap, features of the rural education system that focus resources on high-achieving students exacerbate substantial gaps between high- and low-achieving students in rural areas—even within the same schools. Junior high dropout is a huge problem (for children and their families, of course, but also for China—which is producing a cohort of students that will not be ready for future economic change). One of the number one causes of dropout is the perception by students that they are being treated unfairly or not getting the attention of their teachers.
Much of the burden for why students from China’s rural schools perform poorly may fall on teachers. Studies from developed and developing countries, including China, consistently show that teachers are one of the (if not the) most important factors affecting student achievement. If teachers do not help students raise their levels of learning, students may have little hope of doing so on their own.
Despite their importance, teachers in rural schools in China often lack strong incentives to help students learn. Even when teachers do have incentives to help students learn, these incentives may lead them to concentrate only on the highest-achieving students, possibly at the expense of lower-achieving students.
Surprisingly, almost nothing is known about how to incentivize teachers in rural China to help their students. Every single company (outside of education) in the world has a performance pay scheme in place to incentivize good performance. It is the hallmark of good human resources management. However, no work has been done to understand or institute effective performance pay policies in rural schools. Even with the new Teacher Performance Pay Policy of 2009, policymakers simply asked schools to implement performance pay with no clear directions on how to do so.
The overall goal of this project is to provide guidance for policymakers on how teacher performance pay should be designed to most effectively improve student achievement.
We seek to answer four questions:
- Can teacher performance pay improve student achievement?
- What method of linking teacher performance to student achievement is most effective?
- Can performance pay be designed to benefit all students?
- Does the size of teacher performance pay matter?
REAP tested different teacher performance pay designs to evaluate which one worked best for students.
In order to answer these questions we conducted the largest randomized teacher performance pay policy experiment ever performed in China!
We looked at different designs of teacher performance pay that are most relevant for rural schools to see which one works best. And our next step is to inform policy regarding teacher performance pay so that the substantial resources being dedicated can be used most effectively to the benefit of all rural students.
Policy Experiment Design
One of the strengths of our team is in designing policy experiments. In a policy experiment, teachers are randomly assigned to different groups (and receive different performance pay contracts). This approach—a randomized controlled trial (RCT)—is a gold standard in the research world. The randomized assignment means that we can be sure that any change in student achievement over the course of the study is due to the performance pay design. The first group is called a “Control.” Teachers in this group do not receive any performance pay from us.
· The second group received a performance pay design called “Levels,” where teachers are rewarded based on how well their students do at the end of the school year. This is generally how teacher performance pay is done in China (when it is done at all).
· The third group received a “Gains” design, which rewards teachers based on how much their students improve. It more accurately reflects teacher effort and may therefore be a stronger incentive.
· Unfortunately, a “Gains” design can lead teachers to focus more on middle to high-achieving students and ignore low achievers. Low-achievers may not improve as quickly, for example. A fourth group received a design called “Pay-for-Percentile,” which attempts to balance this out by making every student count in the determination of teacher rewards.
Within each of these groups, we then gave half of the teachers a “large” incentive worth around 2X monthly salary on average and the other half of teachers a “small” incentive worth about 1X monthly salary on average.
How we assigned schools to groups is shown in the table below:
|Teacher incentive treatment||X. Large incentive payout||Y. Small Incentive payout|
|A. Control||A. 52 schools|
|B. Levels incentive||BX. 26 schools||BY. 28 schools|
|C. Gains incentive||CX. 26 schools||
CY. 30 schools
|D. Pay for percentile incentive||DX. 26 schools||DY. 28 schools|
With this experimental design we can find the most effective performance pay design in rural areas. We can answer questions like:
- Is any type of teacher incentive effective in improving student performance (by comparing group A with the other groups)?
- Which method of linking student test scores and teacher pay is most effective (by comparing groups B, C, and D)?
- How do effects on low- and high-achieving students differ?
- Does the size of the incentive matter (by comparing group X with group Y)?
Design and Pretesting of Teacher Incentive Materials
After designing our policy experiment, we needed to know how to explain each of these performance pay designs to teachers in a way that they could understand. To do so, we conducted interviews with over 80 teachers and principals across three different counties of rural China to understand their experience of performance pay.
Armed with this information, we worked with a group of designers to create a set of standard training materials to train teachers about different incentive pay designs. For example, we created a short booklet that detailed what a performance pay contract entailed and how (for each group), their performance pay would be calculated. We created a short quiz at the end of the booklet to help teachers recall the key points in the contract.
Still not satisfied with the quality of our product, we then trained a team of enumerators to go to three schools with our booklets, where we tried out our training materials with dozens of teachers and asked them for their feedback on how to make them clearer.
Teacher Training on Incentives
Once we created and constructed materials for each of the different performance pay designs, we were ready to invite teachers to attend our training session.
REAP involved government officials throughout each step of the evaluation, so that the best program can be scaled up.
We worked closely with our government partners in the Yulin and Tianshui Prefectures (in Shaanxi and Gansu Provinces) to invite teachers to attend our trainings. Teachers from different groups were, of course, trained separately. We trained roughly 30 enumerators to help us manage the transportation, lodging, and logistics of a training conference for 243 teachers.
One of the most important aspects of our training was our commitment to keeping government leaders involved—our goal is, after all, to influence policy. We brought in the top-ranking education bureau official from both Yulin and Tianshui Prefectures to attend our training. In his words: “we have been trying to understand how to do performance pay for years! We absolutely want to know the results from this study.”
At the end of the school year, we needed to return to these schools to see how much the students were learning.
To do so, our first task was to create tests that could actually reflect student achievement. This is one of the most important tasks of an evaluation. We spent four months working with specialists in test construction and poring through national curriculum standards. We came up with a group of 100 test items that students were supposed to learn. We piloted this set of questions with 800 students. We used the results to calibrate our “true” test… one that we were confident could actually be used to tell us how much students were learning.
We recruited, trained, and organized roughly 260 enumerators to travel to rural China and track down our sample students. Over the course of two weeks, these enumerators visited 200+ schools located all across rural China, surveying and testing roughly 20,000 students.
We are still in the process of analyzing our results. Here is what we have so far:
Finding 1: Only “Pay-for-percentile” incentives had a large (statistically significant) impact on average student achievement.
On average, teacher incentives based on average student achievement (levels) and the change in student achievement (gains) are ineffective. Pay-for-percentile incentives, however, had a large impact on average student achievement. In fact, teachers have told us that pay-for-percentile feels more fair. Each student has equal potential to improve, and teachers feel like they can care for them equally.
Finding 2: “Gains” incentives led teachers to focus on middle to high achieving children while “Pay-for-percentile” improved scores for all students.
Incentives based on gains improved performance among middle to high achieving students more than low achieving students. Pay-for-performance incentives improved the scores of low- and high-achieving students equally. “Levels” incentives (or the type of incentive that China has used in the past—when they used incentives) did not work for anyone.
Finding 3: Increasing the size of incentives by 100% did not increase the impact on student achievement.
Increasing the size of potential rewards for teachers from approximately 1X their month salary to 2X their monthly salary had no additional impact on student scores. Small and large incentives were equally effective. This is a result that—while a bit surprising—has been found in other contexts—inside and outside of China. Incentives may be as much as a way to “acknowledge” hard work and achievement as they are as a material incentive.
Our results have important implications for how Teacher Performance Pay Policy is implemented in China:
1. We show that incentives can significantly improve student achievement when implemented correctly.
2. We show that design matters: not only is the way incentives are usually done ineffective, incentives can be designed to benefit all children
3. Incentives don’t need to be large; resources can be saved by making them just “big enough”
Following a meeting REAP held for policymakers in China presenting the findings of this project, the prefectural government in Tianshui (in Gansu Province) has asked for REAP's help in rolling out a "pay for percentiles" design for teacher performance pay across the prefecture on a county by county basis. We have put together a three-year plan for the implementation of the new program and look forward to working with policymakers to make this a reality.