Blog Special Analysis: Beyond 'College Value Ratings' Part 2

President Obama announced in August that the Department of Education would be creating the Postsecondary Institution Rating System (PIRS), a new rating system for colleges. The Department of Education issued a request for ideas on how to design and implement the PIRS. This series of blogs posts is adapted from the comment we plan to submit. Last week, we presented our recommendations for the new federal college rating system. Today, we’ll detail how we constructed our system.

Creating the Ratings

Category Selection

Access, affordability, and outcomes, the three categories suggested by the President, provide a strong foundation for this rating system. The access and affordability categories match two primary goals of federal financial aid: to enhance equality of opportunity and to make college more affordable. Outcomes can gauge the results of federal financial aid spending.

Metric Selection

In selecting metrics, we considered several issues:

First, we did not want a single year of data to have an outsized influence on a college’s rating. That could lead to wild swings from year to year that might not reflect a college’s long-term performance. To temper this volatility, we used a three-year average for many of the metrics – long enough to blunt temporary disturbances and short enough to allow recent performance to carry substantial weight.

Second, even if a college performed badly on one metric, we wanted to encourage and reward improvement. With that in mind, in three of these metrics we used absolute performance and change-over-time in the same metric. This allowed college performance to be gauged on an absolute level and to account for recent improvements or declines.

Third, with so many different types of institutions, any single set of metrics would not be equally appropriate for all colleges. To account for this, we dropped each college’s lowest score in the categories of access and affordability. For outcomes, there were not enough data to justify dropping any scores.

There is enough information in U.S. Department of Education databases to introduce reasonable measures of performance in the access and affordability categories. We selected four metrics in each.

Two outcome measures have readily available data, but three others (marked below with asterisks), have no available data. Since the U.S. Department of Education has specifically asked for “data elements not currently available to the Department or other federal agencies but that could be collected,” we have included these three in the list below, because as these data become available, they should be incorporated into the ratings.

Here are the metrics we chose:

1. Access

Percent of undergraduate students receiving Pell Grants
Change in percent of undergraduate students receiving Pell Grants
Ratio of low and middle income aid recipients to all aid recipients
Change in ratio of low and middle income aid recipients to all aid recipients

In the context of federal financial aid, access primarily refers to the participation of financially disadvantaged students. The first two access metrics are based on enrollment of Pell Grant students. Since Pell Grants are primarily awarded to students from low-income families, these two metrics gauge the extent to which a college is enrolling low-income students. The other two access metrics expand the income range to include those students from middle-income families (families with incomes under $75,000). Colleges were rated against each other on each metric in their percentile ranking; the higher the percentile, the better the rating.

Consider two, four-year public colleges, the University of Maryland and the University of Virginia.

At the University of Maryland, approximately 19 percent of students receive Pell Grants, putting Maryland in the 8th percentile among four-year colleges (92 percent of four-year colleges have a greater share of students receiving Pell Grants). Between 2009-10 and 2011-12, Maryland saw a 1 percentage point average increase in the share of students receiving Pell grants (the 28th percentile). An average of 47.4 percent of Maryland’s students were from low- or middle-income families (15th percentile), up an average of 2 percentage points a year (80th).

At the University of Virginia, approximately 12 percent of students receive Pell Grants, putting the university at the 2nd percentile (98 percent of four-year colleges have a greater percentage of students receiving Pell grants). The typical yearly increase in the percent of students receiving Pell grants was 0.5 (the 21st percentile). An average of 46.9 percent of students were from low or middle income families (14th percentile), down 2 percentage points a year (20th percentile).

Maryland’s percentiles are higher than Virginia’s on every metric. This does not mean that Maryland is a better college than Virginia, but it does indicate that the University of Maryland does more than the University of Virginia to provide college access to financially disadvantaged students.

2. Affordability

Net tuition
Change in net tuition
Borrowing per student
Change in Education and Related Spending per student

The second primary goal of federal financial aid is to make college more affordable. The most direct measure of affordability is net tuition (published tuition minus grant aid). The first two affordability metrics focus on net tuition. Another indicator of affordability is debt per student; colleges that require more borrowing tend to be less affordable than others. The fourth affordability metric is a measure of the change in educational spending at each college. Since this spending must be covered by the students, state government, or philanthropy, this measure gives an indication of whether there is likely to be upward or downward pressure on tuition in the future.

Consider two four-year private non-profit colleges in New York City, Columbia University in the City of New York and New York University.

Columbia had typical net tuition of $23,600 (29th percentile), which was rising by an average of $2,100 a year (29th percentile). Borrowing per student was a little under $2,400 (87th percentile) and the change in Education and Related Spending per year was -$5,200 (95th percentile).

New York University had a typical net tuition of $29,000 (27th percentile), which was rising by $1,700 per year (31st percentile). Borrowing per student was $3,500 (78th percentile), and the change in Education and Related Spending averaged -$4,900 (94th percentile).

These two universities have similar affordability profiles. Net tuition was a bit higher at New York University; at Columbia, net tuition was rising by a slightly greater amount. Students at New York University borrowed more; Columbia was cutting expenses faster. The similar affordability profiles do not mean that the two schools are equal in every respect or that they provide the same value. All it means is that both schools have similar performance in keeping college affordable.

3. Outcomes

Actual learning minus expected learning*
Actual earnings minus expected earnings*
Actual social spillovers minus expected social spillovers*
Actual graduation rate minus expected graduation rate
Actual default rate minus expected default rate

Many consider outcomes to be the most important of the three categories; it is also the category with the least available data. What students learn in college is crucial, yet there is little publicly available information on the subject. Data about the jobs and incomes of a college’s graduates are also vitally important to most incoming students and their parents, but reliable figures are rarely available (though there is progress on this front). Social spillovers (positive effects on others caused by a graduate’s college experience) likewise lack data. We do have data for two other important outcomes: graduation rates and student loan default rates.

Using regression analysis, each college’s expected graduation and default rates were compared to their actual rates. For each metric, the percent of undergraduate students receiving Pell Grants and the percent of undergraduates who attend part time were used as control variables. Ideally, other control variables would be used as well (particularly those that account for incoming-student achievement and preparation), but this data is not available. Therefore, this analysis demonstrates the technique for calculating outcomes metrics rather than providing the final rating.

To illustrate, consider the Massachusetts Institute of Technology (MIT). Using the percent of students receiving Pell Grants and the percent of students who attend part time as control variables in a regression for all four-year universities leads to a predicted graduation rate for MIT in 2010-11 of 67 percent. Its actual graduation rate was 93 percent, so MIT had a much higher graduation rate than expected, which would lead to a high rating on the graduation rate metric. (Some of MIT’s over-performance is likely attributable to the superior academic preparation and achievement of the students MIT enrolls. Including control variables for prior preparation and achievement, would give a more accurate predicted graduation rate, but such data is not yet widely reported.)

Most readers can stop here, but if you are really interested in the nuts and bolts of the calculations, feel free to continue on. The rest of this post details the methodology used to calculate each of the metrics, the conversion into percentiles, and the aggregation into category and overall ratings.

To be included in the ratings, a college had to:

be degree-granting.
participate in Title IV financial aid programs.
have had at least one student each year from 2008-09 through 2011-12.

Calculating Raw Metric Values

Access

Percent of undergraduate students receiving Pell Grants.
- Calculated as the average of the percent of undergraduate students receiving a Pell grant in 2009-10, 2010-11, and 2011-12.
Change in percent of undergraduate students receiving Pell Grants.
- Calculated as the average of the change in percent of students receiving a Pell grant from 2010-11 to 2011-12 and 2009-10 to 2010-11.
Ratio of low and middle-income aid recipients to all aid recipients.
- Calculated as the ratio of aid recipients from families making $75,000 or less to all aid recipients in 2009-10, 2010-11, and 2011-12. (Note: Aid recipients in this context are defined as “full-time, first-time degree/certificate-seeking undergraduate students paying the in-state or in-district tuition rate” who were “awarded any Title IV Federal financial.”)
Change in ratio of low- and middle-income aid recipients compared to all aid recipients.
- Calculated as the average of the change in the ratio of aid recipients from families making $75,000 or less from 2010-11 to 2011-12 and 2009-10 to 2010-11. (Note: Aid recipients in this context are defined as “full-time, first-time degree/certificate-seeking undergraduate students paying the in-state or in-district tuition rate” who were “awarded any Title IV Federal financial.”)

Affordability

Net tuition
- Calculated as the average of net tuition for 2009-10, 2010-11, and 2011-12. (Note: Net tuition was calculated as in-state tuition and fees minus average grant aid per student calculated as the percent of first-time full-time students receiving a type of grant times the average grant aid among recipients. This value was multiplied by negative one.)
Change in net tuition
- Calculated as the average of the change in net tuition from 2010-11 to 2011-12 and 2009-10 to 2010-11. (This value was multiplied by negative one.)
Borrowing per student
- Calculated as total federal undergraduate loan volume divided by full time equivalent undergraduate enrollment for 2009-10, 2010-11, and 2011-12. (This value was multiplied by negative one.)
Change in Education and Related Spending per student
- Calculated as the average of the change in education and related spending for 2008-09 to 2009-10 and 2009-10 to 2010-11. (Note: Education and Related Spending is equal to instructional and student services spending plus the education and related share of overhead spending times the sum of institutional and academic support spending. The formula differs slightly for for-profits due to combined spending categories.) The education and related share of overhead spending is instructional and student services spending divided by the sum of instructional, student services, research and public service spending. This value was multiplied by negative one.

Outcomes

Actual minus expected learning* (Insufficient data precluded calculation.)
Actual minus expected earnings* (Insufficient data precluded calculation.)
Actual minus expected social spillovers* (Insufficient data precluded calculation.)
- Note: Since the U.S. Department of Education has specifically asked for “data elements not currently available to the Department or other federal agencies but that could be collected,” we have included the three listed above, because as these data become available, they should be incorporated into the ratings.
Actual minus expected graduation rate
- Calculated as the average of the graduation rate residual for 2008-09, 2009-10, and 2010-11.
- Note: The residual is the college’s actual graduation rate minus the graduation rate predicted based on a regression controlling for the percent of students receiving a Pell grant and the percent of students who are part time.
Actual minus expected default rate
- Calculated as the average of the default rate residual for fiscal year 2009 and 2010.
- Note: The residual is the college’s actual default rate minus the default rate predicted based on a regression controlling for the percent of students receiving a Pell grant and the percent of students who are part time. This value was multiplied by negative one.

Calculating Metric Percentiles

The raw metric values were converted into percentiles. With a higher percentile being considered better, it was necessary to multiply some of the raw metric values by negative one to ensure consistency. In addition, if a college was missing the data necessary to calculate its raw metric scores, it was given the lowest percentile score (since ties received the same score, this value varied based on how many colleges were missing data).

Compiling Category and Overall Ratings

The percentiles were used to calculate category and overall ratings. Access and affordability ratings were calculated as the average of each college’s three highest scoring metrics in those categories (the lowest score in each of those categories was dropped). Outcome rating was the average of the two calculated outcome metrics (no scores were dropped).

The overall rating was then calculated as the average of the access, affordability, and outcome category metrics.