2005-05-23

CISSP Practice Exams: Buyer beware

As I've said in earlier posts, in the interests of being unbiased, I try to avoid commenting on products, services, or companies directly unless it applies to a specific point. This article will be an exception.

Earlier today, I received an email forward from a coworker of mine who is studying for the CISSP. On Saturday, he spent around $100 on a practice exam produced by Boson, and his experience is worthy of note. Sean is an experienced, skilled, and knowledgeable IT professional whose opinions I respect greatly, and while I have never used the product he refers to, I have no reason to doubt the validity of his complaints. The entire contents of his email are as follows:
---------------------------- Original Message ----------------------------
Subject: CISSP practice exam a huge let-down
From: "Sean Wilkerson" <sean@xxxxx.com>
Date: Sat, May 21, 2005 4:06 pm
To: support@boson.com
--------------------------------------------------------------------------

Boson Support,
I am in the midst of studying for my CISSP exam, which I am due to take
in three weeks. In preparation for this exam, I have taken numerous
practice exams, including those offered directly by ISC2 (the
organization who makes, and hosts the CISSP). I did some side-by-side
analysis of your exam features vs. cram-session's and decided to go with
yours. This morning, I purchased the three exam pack which you feature
for the CISSP, and then took exam one of the series. I was miserably let
down by the content, grammar, and structure of your exam, which I found
to be counterproductive, and distracting. After about 20-30 questions
in, I realized that the problems with your exam were not limited to the
rare case of a bad question, but were throughout the entire question-set.
As this point, I started taking notes as to the major complaints I have
with your product, which I would like to share with you here.

##
- The UI continually messes up, by not showing entire question. To see
the entire question, you have to frequently adjust the size of the
application window. Even if the window is maximized, you still don't see
it all, without adjusting the window slightly, which results in the rest
of the text suddenly appearing. This is a glitch, flat out. I am using
Windows XP SP2, which should be supported. Being a security
professional, I also have my system entirely patched (with the latest MS
patches), firewalled, anti-virus protected, and have NO malware
installed on my system as detectable by any of the several tools I use.
- I find your questions to be vague and confusing. They are not clear or
specific (as the real CISSP questions are). I have done enough research
for the CISSP to know, what types of questions to expect on the exam and
what you provide is not it. I found that your questions were not even
remotely similar to the sorts of questions I will see on the exam. The
fill-in-the-blank non-sense, the questions about vendor specifics (see
below), the failure to use the actual terms you were describing in the
question, were all symptoms of this problem. This issue is exacerbated
by the bad grammar, (see below).
- Incredibly poor grammar throughout the test in both the questions and
answers, though mostly the questions. Lots of simple mechanics
mistakes. Extremely poor editing. This is INCREDIBLY distracting. I
found myself mentally correcting the exam's grammar, rather than
concentrating on the content. This is not a failure of me, the test
taker, but of the test content provider and editing staff.
- The CISSP uses the same format for every question. Specifically, there
is a question, with four multiple choice answers, to which the test-taker
should choose the one (read 1) choice which best answers the question.
Your test had questions with anywhere from four to six possible answers.
Furthermore, many of the questions required more than one answer. If the
intention is to prepare a customer for the CISSP, than this is
counterproductive.
- There are too many questions on proprietary software and OS
platforms. The CISSP is software and OS agnostic, so a well-written
practice exam should be as well. Being intimately familiar with MS
Windows, for example, is not a requirement of either being a CISSP or a
security professional, and should therefore not be on a practice test
designed to prepare one for the CISSP exam.
##

I am not usually the one who speaks out, or complains about trivial
things, but I feel this is non-trivial. I the $99.48 I paid for these
this morning was a waste of money. Additionally, the time I spent this
morning both taking your exam, and writing this e-mail, has done nothing
to help me prepare for, or pass the CISSP exam, but has instead giving
you critical feedback which will *hopefully* help you to improve your
product.

I have already un-installed your software from my computer, and plan to
never use it again, as I see no benefit.

Please get back to me soon and explain how you will honor the quality of
your product and customer service.

Respectfully,
Sean Wilkerson
========================================================================
I'd like to thank Sean for his permission to reproduce this informative email. I know how thoroughly and meticulously he researches everything, so I'm certain that from all the information available publicly, this looked like a good exam. The only recommendation I can make to avoid this situation is to talk with people who have taken both the practice exam and the CISSP itself before spending money on any practice exam.

3 comments:

Mike Helmick said...

ah... this is another reason that I say pick up a college degree rather than get certifications.

Of course I'm also soon to be making a living by those going for college degrees...

Anonymous said...

Not that this is so much related to the context above but more to the first comment. I actually disagree with your degree philosophy. I have certifications coming out of my ying yang but have no degree. I am surrounded by people that have degrees. I got my job based on experience; most got it because of the degree. We have to update our certs periodically but I have never been asked to obtain a degree. Besides, any time we hire a college grad my time is wasted teaching them how things really work as opposed to the classroom. Get your certs, only get the degree if you need it.

Hilbert said...

Hello! I read this article! Big thanks to author, very interesting. Write more.