Why is the score trend getting higher on the GMAT

[tomslawsky]

10 years ago, a 700 was ~98th percentile. Not too long ago, 700 was 95th percentile, now it is ~90th percentile. Over the same time, the average score has risen from ~500 to 538-ish. The sub scores of math and verbal are slifing up, too. If you look at the LSAT as a benchmark, the average score in the past 10 years has gone from 149 to 150.5-ish, and is much more stable. I feel that the value of getting the once vaunted 700+ score is being cheapened and the GMAT might be losing respect as a tough test. Anyone else want to weigh in on this?

[StaceyKoprince]

Interesting topic. I'm not sure why they've been allowing the scoring range to skew. From a technical standpoint, they can put the scoring levels wherever they want and keep them there - it's basically a scaled (forced curve) score. The only real information being conveyed here is the distinction between different test-takers, and that's taken care of with the percentile ratings.

So, I don't know, but I do know that the 3-digit scores are an artificial construct. If I wanted to, I could reset the whole scale tomorrow to be 1 to 99, and your score is also equal to your percentile level. Or we could make it 10 to 1 million. So the short answer to "why is it changing" is: the people who score the test want it to, for some reason. That's all. :)

[tomslawsky]

Thanks for replying, but I'm not sure you hit on the crux of what I'm asking. Or, may be I am misunderstanding how the GMAt is scored. The way I understand it- a 700 GMAT is indicative of a certain skill level. Whether that 700 was 10 years ago or last year, a GMAT score, (much like an IQ score) SHOULD indicate the SAME skill level- an apples to apples comparison. So, there are really only 3 reasons for the percentiles at a particular score to shift:

1) The GMAT is getting easier and the testing "IQ" is about the same

2) The GMAT has the same level of difficulty and the test taking population is shifting and becoming more bright.

3) People are cheating

I tend to think that for the past 15 years or so, investment banking has lured many of the countries best and brightest into finance type jobs on Wall Street, draining them away from law and medicine, causing a shift to the right in the test taking population's IQ. I also believe that cheating in foreign countries is a problem.

It just seems to invalidate the GMAT to some extent (my opinion) that the test percentiles per score are shifting dramatically. I am planning on taking the exam again and hopefully changing careers to be a standardized test tutor and I am amazed at the level of pure brain power it is taking to bump my quant score from mid-high 40's to 51. The reason I chose to attack the GMAT first is that because of the level of abstract math, it seems to be the hardest of the SAT/GRE/GMAT trio. Recently, I bench marked myself against a GRE PowerPrep quant and scored an 800 with three errors and 5 mins to spare. The day before, my GMAT Powerprep quant score was a 46 and a 48 was ~85th percentile or 1 ~SD above the mean. I always assumed that the aptitude of the two test pools was about the same. WRONG!!!!


To sum up my understanding in of score/percentiles "GMAT Critical Reasoning Lingo":

Score = absolute measure of skill

Percentile = measure of skill relative to testing pool

To refer to the two or mix them interchangeably would be illogical. That is, unless the scores were perficaly distributed and that distribution was static.

Thinking of it this way is interesting to me, as you can measure certain things like:
the change of the test taking pool while holding all other factors constant- a-la partial derivative. I'm sorry if my questions and comments are above the nature of this board, but I am REALLY trying to disect this exam from all angles and obtain a true nuts and bolts understanding of the exam. In the end, that will make me a better teacher of the exam- assuming I can break the 98%tile barrier that is.....

[StaceyKoprince]

Okay. No, a 700 does not necessarily indicate the same skill level over a long period of time. The percentile ranking indicates the same skill level over time, but the 3-digit score assigned to a percentile ranking can change (and has changed) over time. They don't hold a 700 as an inviolate score level indicative of the same thing always in the same way as, say, IQ tests say that 100 is always "average."

If the 90th percentile 10 years ago was 650 and the 90th percentile today is 700 and the 90th percentile 10 years from now becomes 42... then 650, 700, and 42 would represent the same skill levels (during their respective periods of time). But, of course, schools aren't trying to compare scores from 10 years ago to scores today.

Within a few years, minor changes of scaled score to percentile ranking do occur as a result of a number of things. One of those things does have to do with a shift in the test-taking population:
(1) more people are studying, and a more sophisticated level of preparation material is available, meaning that more people are better prepared to take the test
(2) a broader pool takes the test, including many non-US students who tend to have better math schooling in general, meaning that the level of math knowledge across the population of test-takers has generally increased over the past decade or two

There's probably some other, less savory stuff going on as well, including some cheating, though the test-writers are pretty good at preventing that on the whole.

The test writers still need to maintain the scaled score to percentile conversion as much as possible over shorter periods of time (several years), because the schools compare scores among applicants who have taken the test over the past few years but who are all applying right now. So, as the test-writers try to maintain the 3-digit scale as much as possible over a few years, the test-takers are simultaneously preparing themselves better. Over time, that means the test-writers have to shift the 3-digit scale somewhat - and that can add up to a lot, over long periods of time. (But the test-writers aren't concerned with big shifts over long periods of time. The scores expire after 5 years.)

[tomslawsky]

I almost feel that there are "competitive forces" at work here. The ETS publicly offers a free "conversion calculator" to convert a GMAT score to an equivalent GRE score. The GMAC counters by "shifting" the GMAT curve, thus rendering the conversion invalid. That's what I would do. I just wish the GMAC would shift the curve to the left rather than to the right, as it seems the value of a 700+ score is dropping precipitously.

[StaceyKoprince]

Ah, interesting idea! The GRE is trying to compete with the GMAT, and I'm sure the GMAT folks are responding somehow (though I don't know exactly how, of course).

[tomslawsky]

Right, but whatever their rational, I wish the ETS wouldn't mess with the sanctity of the 6-sigma model. It seems to me that the GRE, GMAT, LSAT (and prob MCAT, DAT, Etc) all were set up to have a center score, representing the average IQ of the test taking population. These center scores represented an IQ of ~115:

GRE center- 1,000, Standard Deviation of ~200
GMAT center- 500, Standard Deviation of ~100
LSAT center- 150, Standard Deviation of ~10

Now all of the scoring systems were designed to capture the "6 sigma range", which captures ~100% of all test takers and discriminates between them. For the GMAT, this should represent a center score of 500 Plus, Minus 3 SD, or a total scale of 200-800.

What is happening now is that the curve is becoming REALLY skewed. Whereas a 600, was intended to be average + 1SD, or ~85th percentile, now 600 is not even 60th percentile. A 700 score "should" represent the average + 2 SD, or ~97.5%, but now is 90% and will probably dip into the 80's soon. Why the ETS wold do this? It bothers me because it really cheapens the exam and must, by definition, become an easier exam as the ceiling is lowered. Even if the ceiling is not being lowered, the exam is becoming easier in the ranges of 300-700. I feel this cheapens the exam. A score of 700 is soon going to become common place as more than 10% of the test takers will reach this plateau. Then, the "700" threshold will soon become 730+, which, I already believe it is for white, male, non minority applicants. Although, I am shocked at how many MBA "consultants" completely ignore the race issue, despite there being amazingly different admissions standards of GMAT score based on race. In fact, when someone asks "chances" based on GMAT score, the FIRST question that a competent consultant SHOULD ask is "which box will you be checking on your application". If one scores a 670, but is an unrepresented minority female, this score may as well be a 750 in the eyes of Stanford or Wharton. GMAT based race discrimination is for another discussion, though. Maybe someday people who offer admissions advice will stop ignoring the subject.

Back from my rant ;) The LSAT doesn't seem to have this center score drifting issue. the average score is pretty much set within the realm of statistical insignificance. LSAT driven shifting averages in my opinion represent a combination of statistical noise coupled with slightly changing test population IQ levels. :

http://www.phil.ufl.edu/ugrad/whatis/LSATtable.html

Just a thought.

[StaceyKoprince]

I've only been working with the GMAT for about 15 years now, but during that time at least, the SD was never 100. I think the SD has pretty consistently been around 30 points for a long time. Back when I first took the test in the mid-90s, 600 was not the 85th percentile. So I'm not so sure it's the case that the 100-point-SD thing was once true for the GMAT.

As to why they set things up this way in the first place? Who knows? I certainly don't. The thing that matters at the end of the day, though, is not the three-digit score but the percentile associated with that score. They could change the scoring scale to A through Z and that wouldn't change - the percentile is what matters.

FYI, by the way - ETS doesn't make the GMAT (it used to, but it lost the contract about 4 years ago). GMAC owns the test and manages the algorithm; ACT writes the test items.

Also, I agree that there are also long-standing racial (and gender) biases in standardized tests. As you said, that's a discussion for another post. :)

[tomslawsky]

Thanks Stacey. Just one point, the current GMAT Standard deviation of the means are as follows:

Quant: Average-35.8, SD 10.6
Verbal: Average- 28, SD 9.0

Overall: Average- 539, SD 120.5

An overall score of 670 = Average +1.0 Standard Deviation of the mean, 790= Average +2.0 SD of the mean. Therefore, the test functionally has a lower ceiling than originally designed.

Also, I read somewhere that in the mid 80's, the average score was ~480. I need to find that reference though.

http://www.mba.com/mba/thegmat/gmatscor ... gmeans.htm

[RonPurewal]

@ Stacey,

I've only been working with the GMAT for about 15 years now, but during that time at least, the SD was never 100. I think the SD has pretty consistently been around 30 points for a long time.

you guys are talking about different standard deviations.

the one that you're talking about (the ~30 point one) is actually the standard error inherent in ONE INDIVIDUAL STUDENT's scores.
i.e., if you took ONE student (who is not smack up at the 200 or 800 end of the scale), and had him/her repeat the test under identical conditions** a large number of times, then you would witness a SD of approximately 30 points in the resulting bell curve.

the one that tom is talking about (the 100 point one) is the standard deviation in the ENTIRE POPULATION of gmat scores.
i.e., if you look at the entire bell curve of EVERYONE's gmat scores, you'll see a standard deviation close to 100.
this is in fact the entire genesis of the 200-800 scaled score. the spread of exactly 600 points is modeled after a "six sigma" bell curve, from ~3SD below the mean at the left to ~3SD above the mean at the right.
it's also the reason why other standardized tests will normally (heh, normally) have total score ranges that are nice multiples of six. for instance, the lsat is scored on a scale of 120-180 (weird, don't ask me why) -- again, a perfect 6SD spread if the SD is 10 points.

--

**(from a practical standpoint this is impossible, since "identical conditions" would necessarily imply that the student's knowledge would fail to increase as a result of the repeated administrations)

[RonPurewal]

@ Tom

I am shocked at how many MBA "consultants" completely ignore the race issue, despite there being amazingly different admissions standards of GMAT score based on race.
...
In fact, when someone asks "chances" based on GMAT score, the FIRST question that a competent consultant SHOULD ask is "which box will you be checking on your application". If one scores a 670, but is an unrepresented minority female, this score may as well be a 750 in the eyes of Stanford or Wharton.

i've done admissions consulting for various types of graduate schools, and, yes, of course i've given advice in context -- where "context" includes sex and race in addition to all the usual stuff (work experience, etc).
it would be absurd not to include that sort of advice, since business schools have "effective quotas". in other words, they don't actually have numerical quotas -- since such quotas would probably be struck down by any non-radical-left judge (see the Bakke vs. U of Michigan case) -- but they have, essentially, a "target distribution". in other words, they have a platonic ideal of what they want the demographic profile of the incoming class to look like, and that certainly includes proportions by race and sex.
the ultimate result is that you aren't really competing against the entire applicant pool - you're basically just competing against people like you.
it's almost like trying out for a football team: you have to try out for a specific position, and you couldn't care less about the quality of the recruits at other positions. if you're a running back, you're just competing against the other running backs. it's irrelevant whether the kickers are slower than you.
if you think of your position as "white male, aged X, in work field Y", this analogy goes surprisingly far.

most schools just posit "diversity" as the reason for these policies, although some types of schools argue on a more solid footing. for instance, medical schools justify their diversity efforts, among other ways, on the grounds that doctors of ethnicity X are substantially more likely to return to practice in neighborhoods that consist predominantly of ethnicity X; that's pretty solid reasoning whether you support social-engineering initiatives or not.
i know less about business schools' justifications for their policy (beyond the vague and constantly changing term "diversity").

--

re: admissions consultants
if you are correct that other admissions consultants don't ask that question, it's probably because they don't usually have to!
admissions consulting is almost always face-to-face, so, except in cases where the admissions consultant can't determine the race of the applicant, the question is unnecessary.

in any case, it's not the place of this forum to discuss the ethics of such quotas, whether formal or informal, although we can certainly discuss the objective consequences of those quotas.

[RonPurewal]

Stacey -

Okay. No, a 700 does not necessarily indicate the same skill level over a long period of time. The percentile ranking indicates the same skill level over time, but the 3-digit score assigned to a percentile ranking can change (and has changed) over time.

i don't agree with this.

there are two possible reasons for the observed discrepancy:

(1) the percentile is held to a constant skill level, and the scores are being changed for some other reason.

(2) the score is held to a constant skill level, and the percentiles are changing as a result of changing demographics.

you are taking side #1, which rests on the highly questionable assumption that the demographics and preparation of the GMAT pool is constant.
i.e., your statement that "The percentile ranking indicates the same skill level over time" can only be true if BOTH of the following are true:
* the inherent skill level of the testing pool doesn't change over time;
* the preparation level of the testing pool doesn't change over time.
the first of these is almost certainly not true -- the skill level of the average MBA applicant has risen significantly in the last 5-10 years, by several objective measures -- and the second is demonstrably false, in light of the explosion of test-prep sales and revenue in the last 5-10 years: "test prep" has gone from a relative rarity, existing primarily in large urban areas, to a nearly universal phenomenon.

so it's much more likely that #2 is the case: the numerical score of 700 represents roughly the same performance, but the percentiles have decreased because there are now better -- and better prepared -- test takers.

this is made even more likely by the fact that this exact situation (#2) is the express directive of GMAC. i.e., they must create a test on which a four-and-a-half-year-old 700 is comparable to a brand-new 700, because otherwise the five-year validity of scores would be unethical.

[tomslawsky]

Stacey and Ron,
I can tell by the quality of someone's post whether they put thought and care into generating it. Both of you have done a great job, thank you both. I will put this topic to rest now, at least as far as this particular forum goes.
Tom

PS- Ive read a ton about Bakke vs. U of Michigan. It's great reading for people who care about the current legal tug a war going on between affirmative action proponents and those who think that the USA should function as the constitution mandates us to!

[StaceyKoprince]

Oh, Ron, I see what you're saying. I should have been more precise in my language. When I said "same skill level over time," I meant "same relative skill level over time." That is, a 90th percentile test-taker is "better" than 89% of test takers and "worse" than the other 9 percent and that doesn't change - that's always what "90th percentile" represents.

[JinJJa]

This is a most excellent set of posts with much brain power and analysis.

I think Tom brings up a very good point which has been documented in BusinessWeek articles. GMAT cheating is rampant in foreign countries. I even heard of a famous Chinese GMAT taker who has a photographic memory and he would post every question verbatim on a Chinese language only internet site. They call it "GWD" GMAT or something like that. Note, I am not Chinese and can't read Chinese.

I think the GMAT scores are skewed mainly because of the GMAT cheating. The chief evidence comes from the overly skewed nature of the quant scores. I scored Q45 and yet that was only 74% and my V41 was 92%. And I think I am much better at math than verbal. One reason is that quant questions are much easier to memorize and later posted online for the GMAT cheaters.

To Ron and Stacy, how hard is it to improve from 710 to 740?

Here is my strategy. I am going to take a good guesses on about 30% of the questions from 1-27 to conserve time to average about 1.6 minutes per question. Then, I am going to focus on getting the last 9 questions correct with the extra time that I have since they all count towards my score. Do you find this to be a winning strategy?

Thanks for replying, but I'm not sure you hit on the crux of what I'm asking. Or, may be I am misunderstanding how the GMAt is scored. The way I understand it- a 700 GMAT is indicative of a certain skill level. Whether that 700 was 10 years ago or last year, a GMAT score, (much like an IQ score) SHOULD indicate the SAME skill level- an apples to apples comparison. So, there are really only 3 reasons for the percentiles at a particular score to shift:

1) The GMAT is getting easier and the testing "IQ" is about the same

2) The GMAT has the same level of difficulty and the test taking population is shifting and becoming more bright.

3) People are cheating

I tend to think that for the past 15 years or so, investment banking has lured many of the countries best and brightest into finance type jobs on Wall Street, draining them away from law and medicine, causing a shift to the right in the test taking population's IQ. I also believe that cheating in foreign countries is a problem.

It just seems to invalidate the GMAT to some extent (my opinion) that the test percentiles per score are shifting dramatically. I am planning on taking the exam again and hopefully changing careers to be a standardized test tutor and I am amazed at the level of pure brain power it is taking to bump my quant score from mid-high 40's to 51. The reason I chose to attack the GMAT first is that because of the level of abstract math, it seems to be the hardest of the SAT/GRE/GMAT trio. Recently, I bench marked myself against a GRE PowerPrep quant and scored an 800 with three errors and 5 mins to spare. The day before, my GMAT Powerprep quant score was a 46 and a 48 was ~85th percentile or 1 ~SD above the mean. I always assumed that the aptitude of the two test pools was about the same. WRONG!!!!


To sum up my understanding in of score/percentiles "GMAT Critical Reasoning Lingo":

Score = absolute measure of skill

Percentile = measure of skill relative to testing pool

To refer to the two or mix them interchangeably would be illogical. That is, unless the scores were perficaly distributed and that distribution was static.

Thinking of it this way is interesting to me, as you can measure certain things like:
the change of the test taking pool while holding all other factors constant- a-la partial derivative. I'm sorry if my questions and comments are above the nature of this board, but I am REALLY trying to disect this exam from all angles and obtain a true nuts and bolts understanding of the exam. In the end, that will make me a better teacher of the exam- assuming I can break the 98%tile barrier that is.....