Paul Kimmel says tests are impersonal, impractical, and inefficient for evaluating potential. Worst of all, they promote laziness in personnel departments that use them as the determining factor for hiring.
Standardized online tests are bunk, and the companies that use multiple-choice tests as a sole determinant when hiring people are likely staffed with very lazy people. I dont say this just because I am a poor test taker, and I dont say this just because my 40-year-old brain is no longer nimble and good at memorizing facts. (However, it is full of too many useless facts already.) Standardized online tests as a sole means of screening people is problematic and is no better at determining potential for success than grades in college. (Many a C- student millionaires and leaders will agree with me hereBill Gates, Michael Dell, George Bush, and John Kerry, included.)
A True Story
A local peer of mine recently wrote to lament a college student he hired as a developer. From his letter, he clearly was impressed by her academic record, the fact that she had earned a scholarship to an excellent university, and that she seemed articulate and bright. By all accounts, she was a superstar waiting to shine. This employer also mentioned that his technical abilityread, his ability to screen her real skillswas somewhat wanting. What was the result?
This academically successful person is failing miserably at her job. Her peers are saying shes in over her head. Her boss is seeking advice from several corners, including researching her precise academic curriculum in detail, and doesnt know how to proceed. Yes, maybe the boss needs a refresher on hiring strategies, but thats not why I wrote this.
This young person should have been a star. What happened? The answer is that grades arent everything. Grades reflect an ability to take tests and remember facts, but many classes arent based on critical thinking and problem solvingthey are based on timed fact recollection. This suggests that test scores may be an element for evaluating potential but not the only oneand probably not the most important one. Actual accomplishments are the best means of evaluating potential.
Another True Story
I generally refuse to take online skills tests. One reason is that I dont test well. (I am not sure if I ever have.) Another reason is that more than one of these online testing companies has asked me to help write their tests for money and, since I dont agree with them as a good screening device in general, I declined to accept their offers.
Recently, however, an agency asked me to take a C# test for a prospective customer. Being a little curious, I decided to take the test. This particular test was 40 multiple-choice questions; the time for each question was three minutes; and you could use online materials and books in the time allotted. I took the test using only extemporaneous recall. Guess what? I scored in the 53 percentile. Out of 7,000 tests takers, I scored as well as about half.
Granted, had I employed Google, Visual Studios help documentation, and the compiler, I probably would have scored somewhat higher, but what was really tested?
One way to look at the results is that I got half the questions rightabout 20 out of 40so I know 50 percent of the facts. Say there are 10 million facts, then according to the test results, 3,500 people and I know 5 million of them. Thats pretty good. But what good are facts alone?
Here is another problem. In practice, no time limits exist for solving problems in softwarefor the most part. If one spends an hour or two finding a fact or working out a problem, no one cares. If one is stuck after an hour and reaches out to online columns like mine or emails friends who may know, then the problem ultimately is solved. Who really cares what the compiler switch is for range checking when a 10-second search of the help documentation will provide the answer?
Building software is about problem decomposition, solution composition, tenacity, problem solving, good tools, organization, and having the money and time to finish. Facts are the least of anyones problems. Seventy-five percent of all projects in our industry dont fail because of an ignorance of facts; they fail because of poor planning, flagging budgets, or lousy analysis, designs, and specifications. On a team of 540 people, the team members can rattle enough facts off the tops of their heads to fill an encyclopedia, yet projects routinely fail.
The Best Interview I Ever Had
Maybe its a cliché, but the best test I ever took was an interview in Redmond for Microsofts National Practices Team (part of Microsoft Consulting). To get to a Redmond interview, one has to pass a phone screen followed by a phone interview. If you pass the first two interviews, then Microsoft invites you to Redmond where everyone who will be on your team subjects you to a very rigorous oral examination. These interviews last about a day and can be brutal (even though everyone is congenial).
During these interviews, you are asked about facts. You are instructed to think out loud, and if you dont know something then youre allowed to think and discuss the problem critically. You are asked to solve problems like why are manhole covers round? How would you move Mt. Fuji? How would you go about counting all the stars in the sky? The following was my question:
Given three barrels (one marked apples, one marked oranges, and one marked apples and oranges) that are all mislabeled, you can pull only one piece of fruit from one barrel. How would you go about determining the proper labeling of the barrels?
(The answer can be had in a Sesame Street song: [singing] one of these things is not like the other.)
During the course of the morning, I was asked hundreds of facts. But I was also asked about my accomplishments, best and worst projects, hobbies, who I was as a person, what I had done, and what Id like to do yetnot a single multiple-choice question.
To get hired at Microsoft (at least for the job I was interviewing for), one had to get a unanimous thumbs-up. I got a thumbs-up from everyone who interviewed me but one developer. He said I seemed too much like a manager, which from developers is not a compliment. My secret belief is that he wasnt impressed that I didnt know what string-interning was. (Damn! Foiled by a fact again.)
The worst part about the string-interning question is that it was a pre-screening question. Id been asked the same question during the pre-screen interview, and I forgot it when asked during the Redmond interview a week later. (String interning is how strings are stored internally to conserve memory and is why strings are immutable in .NET.) The truth: during the pre-screen I determined that string interning was beyond my control and a truly irrelevant fact except for the .NET compiler writers, so I immediately discarded the factoid.
There are so many factoids that unloading most of them daily is a necessary survival skill just to keep some room in ones brain. Facts are one of the reasons I write books. I write all the stuff I know at a given time in a book, and then I proceed to forget most of it. If I need the fact again, I pick up one of my own books and search the index. In this way, my books become the permanent overflow storage of my brain, and it is an added bonus that I get paid to store the overflow.
Testy About Testing
I cant tell you whether you should take a brain strain test or not. If its a pre-condition of a job that you want, then you have to take it. If the score is a big determinant as to whether you get a person-to-person interview and you dont get in, dont feel bad. The most important thing to remember is grades, pedigree, and other people can't tell you whether you will succeed or not (at least in the USA)only you decide that.
I have to admit I am a little disappointed that I didnt score higher than everyone else on that particular test, but thats probably my ego talking. I am wise enough to know that I am not the smartest person on the planet and definitely not the best test taker. I used to believe I had a practically photographic memory, but that was many winters past.
I will definitely return to my insistence on not taking tests nor traveling for onsite interviews. (I work with too many companies to take tests and travel for onsite interviews as pre-conditions for collaboration.) The reason wont be just poor test-taking skills. The truth is that by using the resources available during the test, my score would be improved noticeably. I wont be taking tests because they are impersonal, impractical, inefficient for evaluating potential, and they promote laziness in personnel departments. When I work for a company, they probably wont want me to be lazy, and I dont want them to be lazy either.
How do you feel about the use of standardized testing as a screening tool for tech jobs? Post your opinion in the CodeGuru Forums.
About the AuthorPaul Kimmel
has written several books on object-oriented programming and .NET. Check out his upcoming book UML DeMystified from McGraw-Hill/Osborne (Spring 2005). Paul is also the founder and chief architect for Software Conceptions, Inc., founded 1990. He is available to help design and build software worldwide. You may contact him for consulting opportunities or technology questions at firstname.lastname@example.org
. If you are interested in joining, sponsoring a meeting, or posting a job then checkout www.glugnet.org
, the Web page of the Greater Lansing area Users Group for .NET.