To all recruiters & employers Listen Up!
Online testing services for software developers are useless, please stop using them. These exams do a great disservice both to the developers and the prospective employers who rely on them.
Lest you think that maybe this is a "sour grapes" rant, I can assure you that I have taken these types of exams many times during my career, almost always scoring in the high 90s. In spite of my scores I would still recommend banning their use for the following reasons...
Reason #1 - Not all people are honest! There is a significant group of people who cheat on the exams, they have books open, help files available or they simply copy and paste answers into a live instance of visual studio or SQL server and test each answer to find the correct one. I can't entirely blame them, they are fighting for an opportunity to get a job and these questions can be pretty obscure, not to mention that most developers don't bother committing to memory things that can easily be "looked up".
Reason #2 For those of us who who don't cheat we now risk having a good solid score of 88% looking terrible to employers in comparison to scores of those in group 1 above.
Reason #3 - Over the years I have found "many" questions on the exams that had wrong answers marked as the correct ones. I have had to call the makers of the exams in almost every instance to help them "correct" an answer or explain why their answer was not correct.
[update 6/13/2009 - I once again had to take a Prove-IT exam. I found two incorrectly marked answers, one on the VB.Net exam and another on the SQL 2008 exam. I have spoken with "Prove-IT" and they scrubbed the answers and agreed with me that two of them were "incorrect". They assured me they would fix the exam, but they were "unable" to correct my scores! I am currently working that out between Prove-IT and the recruiter. I also asked Prove-IT whether they were planning to send out letters to past exam takers who might have received lower scores than they were entitled to...I have yet to receive a response]
>[update 6/24/2009 - Prove-IT (Kenexa Inc.) finally agreed to update my scores after several discussions where they tried to convince me that they had no access to the data ...hmm. I would STRONGLY encourage anyone taking an exam to copy the questions and possible answers and check them for accuracy after you complete the test. While waiting for this "correction" I was, of course, unable to submit my scores to clients and to my knowledge, Kenexa did NOT notify other exam takers of the possible scoring mistakes.]
Reason #4 - Many people simply don't "test" well, even though they are excellent developers. Why should they lose an opportunity to get a job for lack of a skill that isn't required in the job?
Reason #5 - Maybe the most important reason ... even if you don't cheat and even if you get 100% correct it is NO indicator that you are a good programmer, only that you have a good memory. Think of it like giving prospective writers of your biography an English grammar and spelling exam. Just because the writer gets a good grade doesn't imply that he is a good writer, it means simply that he is unlikely to make many grammatical or spelling mistakes, but the writing itself could easily still be boring, dull or incomprehensible!
Recruiters, please stop...if you want to weed out people who are incompetent or you want to know the technical knowledge level of a programmer then interview him yourself or work with someone who has the technical expertise to interview candidates in person and stop relying on easy, cheap and inaccurate testing services.
Online testing services for software developers are useless, please stop using them. These exams do a great disservice both to the developers and the prospective employers who rely on them.
Lest you think that maybe this is a "sour grapes" rant, I can assure you that I have taken these types of exams many times during my career, almost always scoring in the high 90s. In spite of my scores I would still recommend banning their use for the following reasons...
Reason #1 - Not all people are honest! There is a significant group of people who cheat on the exams, they have books open, help files available or they simply copy and paste answers into a live instance of visual studio or SQL server and test each answer to find the correct one. I can't entirely blame them, they are fighting for an opportunity to get a job and these questions can be pretty obscure, not to mention that most developers don't bother committing to memory things that can easily be "looked up".
Reason #2 For those of us who who don't cheat we now risk having a good solid score of 88% looking terrible to employers in comparison to scores of those in group 1 above.
Reason #3 - Over the years I have found "many" questions on the exams that had wrong answers marked as the correct ones. I have had to call the makers of the exams in almost every instance to help them "correct" an answer or explain why their answer was not correct.
[update 6/13/2009 - I once again had to take a Prove-IT exam. I found two incorrectly marked answers, one on the VB.Net exam and another on the SQL 2008 exam. I have spoken with "Prove-IT" and they scrubbed the answers and agreed with me that two of them were "incorrect". They assured me they would fix the exam, but they were "unable" to correct my scores! I am currently working that out between Prove-IT and the recruiter. I also asked Prove-IT whether they were planning to send out letters to past exam takers who might have received lower scores than they were entitled to...I have yet to receive a response]
>[update 6/24/2009 - Prove-IT (Kenexa Inc.) finally agreed to update my scores after several discussions where they tried to convince me that they had no access to the data ...hmm. I would STRONGLY encourage anyone taking an exam to copy the questions and possible answers and check them for accuracy after you complete the test. While waiting for this "correction" I was, of course, unable to submit my scores to clients and to my knowledge, Kenexa did NOT notify other exam takers of the possible scoring mistakes.]
Reason #4 - Many people simply don't "test" well, even though they are excellent developers. Why should they lose an opportunity to get a job for lack of a skill that isn't required in the job?
Reason #5 - Maybe the most important reason ... even if you don't cheat and even if you get 100% correct it is NO indicator that you are a good programmer, only that you have a good memory. Think of it like giving prospective writers of your biography an English grammar and spelling exam. Just because the writer gets a good grade doesn't imply that he is a good writer, it means simply that he is unlikely to make many grammatical or spelling mistakes, but the writing itself could easily still be boring, dull or incomprehensible!
Recruiters, please stop...if you want to weed out people who are incompetent or you want to know the technical knowledge level of a programmer then interview him yourself or work with someone who has the technical expertise to interview candidates in person and stop relying on easy, cheap and inaccurate testing services.
Posts
9 comments:
Agreed re cheating; the potential for unfair results (vs cheaters) and weakness vs illustration of real, hands-on experience.
However, they can be useful for self-assessment (private) as they can high-light areas one is weak in.
For example - I switch between C++, Java & C# so from job to job tend to get rusty on whichever I'm not using.
Agreed with you I have MCITP SQL 2005 Server and 12 years of experience Why the recruiter told me take the Prove it exam the exam is ridiculous and some questions were tricking. Why need compare my MS certification vs exam created for third.
Waknuk,
Good point, I hadn't thought of using them for self-assesment, but then again I'd have to pay to take them and I still wouldn't have any confidence that their answers were correct.
I completely agree with your post. I don't think that Brainbench and Prove IT are good assessors of hands-on experience, and a high score does not necessarily mean that the person is a good software designer or efficient coder. Trust me - I have experienced situations where candidates scored high on Brainbench exams (master level) but they couldn't design and develop a very simple application, or they wrote inefficient code to solve a problem.
I have taken a few Brainbench exams when they offered them for free, and I have also found errors in their tests and brought it to their attention. I have also noticed that quite a number of the questions are about really obscure topics that most actual developers don't commit to memory. For example, in the industry, most .NET developers will interact with SQL Server or Oracle databases, and they may interact with Access databases. The test that I took had a high amount of questions about interacting with MySQL and Progress databases.
What's frightening is some employers are using Brainbench and Prove It as screening tools for their applicants. All due respect to HR folks, but this typically happens in corporations/companies where HR wields a lot of power with hiring candidates, and the HR staff has ZERO clue about information technology and what type of employee an IT department actually needs.
I just took a proveIt test for Java for a local placement service and can firmly attest that it was not only worthless it was functionally wrong. One question used the word bool in code instead of boolean, another question used elif for java! Have I missed something all of these years?
ProveIt should be sued out of existence.
The exam makes little sense as all the developers I know including myself utilize Google, the vendors web site and other online and textual references when developing. Why would someone memorize all these things?
I have taken the SQL and ASP.NET tests and like a previous comment, a lot of the questions were obscure and irrelivant. I have been in development for 15+ years and learned all the languages hands on instead of taking classes. I personally feel cheated by these kinds of tests. I have worked with other developers who may be "book" smart but can't get anything done, or do it so badly I have to go behind them and fix it. Microsoft comes up with new technologies and hiring managers think they must have someone who is an expert and knows everything about everything. My best companion is Google. I admit I don't know everything, but that's why there is search engines, blogs, and examples out there.
Dear Readers,
Would you agree that there ARE VERY good developers out there who don't test well ?
Brainbench is a way of assessing knowledge for some technologies. That said would you agree that some Brainbench tests are "closer to reality" than others in assessing what the "real world" is needed ? (and this can be hard to gauge on a local, much less national or global basis).
What I have seen is that Brainbench is using images (image files placed in the web page of the test) of Code in in a number of tests which prevents the test-taker from "pasting" the code to an IDE like SQL Server or Visual Studio.
Brainbench has helped me "prove" a basic level of skill in some instances and a higher level some cases and this has been beneficial.
The single best opportunity I see in IT Services is that people are looking for employees, contractors and vendors who *keep promises while delivering what was promised at a guaranteed level of investment for the Clients.
These people DO NOT CARE (in most cases) what your degree or certifications are COMPARED to what you can do for them. HOWEVER they *may* view degrees, certifications and other things as indicator of a person's willingness to refresh their skills and maintain some currency.
Short of hooking someone up to a lie detector (which can register false results) some type of screening (whether "live" in person/on the phone or some type of testing) will always be done.
My congratulations are to the people who 1) ask technical and maybe business questions of a technical person during the interview and b) ask for "on the spot" / "do it now" samples of work.
However would you agree that in MANY cases this does not happen and the employer or business still may need to validate the person's skills in SOME WAY.
The ideal interview to me (for a worker who *bills by the hour*) is 1) a live demonstration of the candidate's skills, and 2) a lie detector test whether or not they are telling you the truth (for everything they have told you) and 3) an scenario where the candidate
COULD DEMONSTRATE a minimum of very basic social, language and communications skills (I am sure the scenarios for this would be many given the many good minds who read this blog).
Would you agree that if some of these things (and I'm sure others) occurred in every interview for every technical person the cost of hiring for good technical people could be considerably lower ?
Our company has a simple approach.
We don't ask our Clients to invest/pay by the hour.
We deliver what is promised based on a written agreement by the date promised, and with the investment/payment level promised.
That way the Client gets the #1 thing they care about which is the result they were promised - that to them, is the best way of showing anyone that your certification has value.
Thank You.
Tim
I guess this test practice is not about our qualifications but rather to make sure that we are willing to follow corporate instructions no matter how stupid and inefficient they may be. Just like drug test - they don't care if a potential employee is a heroin addict or an alcoholic because those things are not detectable after a few days but if they said "don't smoke pot" we simply must obey.
Same thing here.
Post a Comment