Scoring Accuracy?

[PistolPreet]

Hello:

Thanks for the product. It really has been helpful. I just have a question about the accuracy of the scoring.

I took one test and got 19 out of 37 (equated to 47) on the Quant section and 28 out of 47 (42) on the Verbal section and was scored a 720. I took another exam and again scored a 19 out of 37 (47) on Quant and a 29 out of 47 (51) on Verbal and was scored a 770.

I know the difficulty of questions matter a lot, but it seems hard to understand how getting 47 and 48 out of 84 would still equate to the 99th percentile.

Can you explain? I am trying to best gauge my level of preparedness.


Thanks Manhattan GMAT!

[ayang]

Preet,

From what has been released by GMAC, their algorithm functions such that one can get a substantial proportion of questions wrong and still wind up with a very high score. At higher levels, this means that one is also getting a substantial proportion of difficult questions correct as well. Our test simulates this algorithm, in that it is not necessary to get all or even most of the questions correct in order to achieve a high score.

As I think it may be helpful, I've also cribbed from an earlier posting as follows:

We’ve been monitoring the scoring closely. So far, our data is showing that the highest score a student gets on one of our practice tests overshoots the score on the real GMAT by about 10-15 points. We haven’t yet made an adjustment to the algorithm because we haven’t yet confirmed this is statistically significant. We are looking to make an adjustment, though, as soon as we can confirm the level of the inflation.

The standard deviation of the difference (between the practice test score and the real GMAT score) is 50-60 points. We would like to reduce this and are taking steps to do so, but note that for many reasons, a practice test can never perfectly predict the GMAT. For one thing, the GMAT itself has a standard error of 29 points - meaning that only 2/3 of the time would you expect to get within ~30 points of your "real" score. Also, we write simulated questions and use a simulated algorithm - neither of which can we calibrate perfectly to the real GMAT. Finally, YOU are a big variable. When a student takes a practice test - even the GMATPrep, which does use the real algorithm and real questions - the student knows that it’s practice. The fact that the real GMAT counts is a factor that no practice test can simulate, naturally.

Our algorithm mimics the algorithm of the real GMAT but is not exactly the same, true. GMATPrep does use the real algorithm, but the pool of questions does not seem to be anywhere as deep as that of the real GMAT. The practical effect of this "shallowness" of the pool is that high performance deviates more from the theoretical 50-60% right that pure IRT (item response theory, the theory behind the algorithm of the GMAT which we understand quite well) would predict. In simpler terms, if the pool is shallow, the algorithm cannot give you "tough enough" questions while still meeting the constraints of content breadth and format breadth. Thus you would expect to get more right on a shallow-pool GMATPrep than you would on the deep-pool real GMAT. Our pool is deeper, which leads to a closer fit to the 50-60% right that pure IRT predicts.

We will continue to invest time and resources into perfecting our algorithm and our question pool. We believe we already have the best test out there besides the GMAT Prep (which of course uses real questions & the real algorithm), but we won’t rest on our laurels.

All that said - the actual score is the least important piece of information from any practice test. You should neither be elated by a high practice test score nor deflated by a low one. You should use practice tests to do two things: (1) identify relative strengths and weaknesses, and (2) gain experience with the format of the test.

The takeaway - the fact that you got a 780 on the CAT is a great sign, but you should not take it as indicative of what you'll score on the real GMAT beyond the fact that you're performing very well on our practice test. Continue to do your best to identify relative weaknesses in preparation for the real test. The only perfect data remains the real GMAT itself.


I hope that this is helpful to you Preet. Let me know if we can provide any other info or assistance. All of the best, - Andrew

[anishs]

Andrew,
Thanks for all the responses provided here - Your SC book is absolutely fantastic and the RC/CR book is quite good too!

I am concerned about the accuracy/consistency of the MGMAT test scores.

Here are my test results from the first 5 CATs
MGMAT1: 710
MGMAT2: 700
MGMAT3: 780
MGMAT4: 780
MGMAT5: 800

The Quant sections on the first 2 tests were significantly longer and harder than the last 3. Additionally, the verbal sections seem to get shorter and shorter.
I was able to complete the final one with 15 min to spare.

I am a bit sceptical because I am aiming for a 750+ score and do not want to go in to the actual GMAT with my timing/expectations off.

Are there structural differences between the first 2 and next 3 tests or did I just come across questions that i wasted time on?

Can you please let me know if i'm only being paranoid or there is reason to worry?

thanks,

Anish

[StaceyKoprince]

I'll let Andrew handle this one since you guys have already started a dialog but just wanted to mention that he will likely want to know the dates you took those 5 tests. We adjusted our algorithm in late April, so the dates can help him to better assess your performance.

[anishs]

Thanks for the prompt response - the tests were all taken last week, May 9th to May 14th

[StaceyKoprince]

Ah, I see. In that case, there's unlikely to be much statistical difference in the scores. Tests taken in such a tight timeframe are essentially the same b/c you haven't had time to improve in between.

I'm curious about your impression that the later tests were easier. I'm going to ask our algorithm guru to take a look at this thread, but in general, the later tests use the identical algorithm. The one thing I can think of is that you're scoring at such a high level, the test might have "maxed out" the hardest questions (it is not allowed to give you a question it has already given you for the first 6 tests). However, this should also have resulted in a lower score, since the "points" earned via the algorithm are tied to the difficulty level of the question - so I'm guessing this is not what happened, since your score went up. Perhaps you just feel a lot more comfortable taking the test!

Also, have you taken a GMATPrep test yet? You should do that ASAP. Andrew explains below that it will also have a wider standard deviation than the official test, but it is still a great tool to gauge your progress.

[anishs]

Thanks Stacey,
I havent taken either of the GMATPrep tests as yet, but i attempted the PowerPrep tests and got similar results - 730 and 770 (However, a lot of the questions on those are form the Official Guides)

Kaplan tests totally blow me away though - I've gotten scores of 770, 730, 680 and 630! Scaled scores that would usually result in a 700 on the GMAT or MGMAT give you a 630 on Kaplan ... so i'm not quite sure of what to think of them.

I'm taking the final MGMAT today and the GMATPrep's and the GMAT in the next 4 days - will let you know how it goes!

Anish

[StaceyKoprince]

People I've spoken with on another forum I moderate consistently say that Kaplan tests score too harshly. Those folks say that you should add 50-100 points to a Kaplan score to get a better approximation. (I haven't tested this myself - but it seems to be the consensus among a wide swath of the GMAT-prep land.)

[anishs]

Stacey/Andrew - Thanks for all your help in the forums.
I know you guys use actual results to recaliberate the software - so here goes:

MGMAT1:710
MGMAT2:700
MGMAT3:780
MGMAT4:780
MGMAT5:800
MGMAT6:800

GMATPrep1:750
GMATPrep2:770

GMAT: 760

thanks!
Anish

[StaceyKoprince]

Excellent job - congratulations! And thanks for the data; we really appreciate it.

Well, you've got at least one part of the qualification to become one of us now. Should we send you an application?

[anishs]

sure - why not. What are the other qualification requirements?
I think you'll find my email address under my profile - login: anishs

[StaceyKoprince]

You're also required to have extensive teaching experience (you don't need to be a certified teacher, but you have to know how to manage a classroom and how to convey information cleanly and concisely to your students). We also only hire about 10-20% of the people who do have a 99th percentile score and teaching experience - so that stuff, by itself, is not enough.

Part of the initial interview is teaching (I had to teach a question to an actual GMAT class when I interviewed while the founder of the company and the class's regular teacher sat there and critiqued me ). Typically there are several interviews, some in person and some over the phone.

Then there is the training process, which typically takes several months and involves 75-100 hours of unpaid training. (We don't want people applying just because we pay a lot - we want to make sure our teachers really want to teach!) At the end of the training period, you have to undergo a couple of "teachbacks" in which some combo of the founder, CEO, curriculum director, and other instructors are your "class" and will throw anything and everything at you to make sure you know what you're doing. IF you make it through that, you will be assigned your first class. To keep the job (not just after your first class but any class you teach), you have to average better than a 4 out of 5 on the end-of-course student surveys (among other metrics).

It's kind of a grueling process, but those who make it through are very happy. Our voluntary turnover is zero. (That is, some people don't make it through the process or are let go at some point - but those who are great enough to continue to work for the company pretty much never leave.)

If the teaching position still sounds appealing to you, let me know and I'll put you in touch with the right person to start the process. (FYI I'm leaving for a 2-week vacation later today, so I'll respond to you when I get back if I don't see your reply before I leave. Or one of my fellow instructors trolling the boards can respond - hint hint.)

Good luck!

[anishs]

Stacey,
Thank you for your very comprehensive response! I didn't think the trainers/instructors went through such extensive training and had to adhere to such strict standards!

Hopefully this forum will help current GMAT students in making an informed choice about selecting their training providers.

Unfortunately, with my current workload I don't believe I will be in a position to make time commitments of that magnitude.

Hope you're having fun at your vacation.

thanks,

Anish

[MBA Applicant 2007/8]

Anish,

First Congrats on your fantastic performance. I was wondering if you
could share some insight on your test preparation in terms of:

1) Study Strategy to Master Timing
2) Study Strategy to Master Content ( Resources - Books,

Any additional tips (daily plan, etc) to master Number Prop,
Word Translation type problems

I already took the GMAT and scored a 700 but i wanted to boost
my score by another 40-50 points.
I understand that this forum may not be the right place to post this but
for someone who is scoring 650s and 660s in MGMT CAT exam and
aspiring for a 750 in the real exam, I couldnt help but post this question
to you.

[anishs]

MBA Applicant,

I think for the kind of improvement you are looking for, your questions are quite generic. You are already at a pretty high level and a daily plan/study strategy are personal choices that may or may not work for everyone.

You may want to take more CAT tests and identify any areas that you are particularly weak in and then use the Manhattan texts for content clarification.

After that, timing is just practice, practice and more practice.

Apologies if I wasn't as specific as you wanted me to be, maybe one of the instructors reviewing the board may be more helpful.

thanks,

Anish