October 22nd, 2000
I remember it well. It was a Wednesday. I was in my office when Theo came in and started stammering about some kind of serious consumer relations problem with our robots. I calmed him down and coaxed it out of him.
"The handyMan, er ... uh, seems to be disobeying its owners, sir."
"How's can that be?" I asked.
"The most common complaint is shopping. An owner, will, uh, tell the handyMan to go buy a six pack of Pepsi and it comes home with Coke. Or they'll ask for, in one case, Yoplait yogurt and the handyMan comes home with Dannon. No one, uh, none of us knows why yet."
"Have you done any additional testing in the factory?"
"Mmm, yes sir. Everything checks out perfectly. In the meantime, massive complaints are piling up, so we thought we'd better let you know."
"Okay, Theo. Thanks. I'm on it."
This was not good news. We had built myRobot, Inc. into a multi-billion dollar corporation in twenty months, and our first product, "vacuumVeronica," had become the hottest selling consumer product since the Nokia VideoPhone five years ago. We were rolling in cash. Early this month, we had consumers waiting with breathless anticipation for "handyMan," our next generation robot. Unlike Veronica which was house bound, handyMan could get out and around. The handyMan system was designed to do just one thing, but do it very well.
And so, two weeks into shipping handyMan, I was not at all pleased to hear about massive consumer complaints. Something like this could ruin a young company like ours, just now forming its public image. I took a deep breath as I walked out of my office. Time to start asking questions.
The first person I wanted to talk to was Carol Strassman, my fiancée and cofounder of myRobot, Inc. Carol is very cool. She's slim, has long blonde hair and wears cute horn-rimmed glasses. She's very intellectual. I don't know a smarter woman. Or a sexier one.
When I walked into engineering, she was talking to our production chief, Jeremy Thorpe.
"Hey guys." I tried to appear calm and relaxed.
Carol turned and smiled. "Hi! What's up?"
"Have your heard about the consumer complaints?"
"Sure," she said suddenly looking just a little more nervous than I would have expected. "It's nothing to be worried about. I'll handle it."
"I'm sure you will, but I'd feel better if we get together with marketing and consumer support and get to the bottom of this very quickly."
Carol glared at me in a way that I had never seen. "I said, it's no big deal right now. It's probably just a first run product glitch. Let me work on it."
I wasn't accustomed to Carol acting like that and something seemed not quite right. Now the little man in my head was telling me that getting Jeremy alone with her here would be tough, so I shrugged and started to walk away. "We'll see," I said calmly.
Time to reconsider. Sometimes, when there's a problem with a product, starting at the top is not a good idea. There's too much money involved to get a straight answer. Besides, when a manager reaches a certain level, they stop thinking like an engineer. I decided to try another approach.
Jeremy tends to work late, so I went back to my office and messed around with e-mail and other things until very late in the afternoon. When I returned to Engineering, about six o'clock, sure enough, Jeremy was working on the handyMan GM prototype, the one that was sent to production. He had the main bus pulled out, and there were wires everywhere. The thing you need to know about Jeremy is that he's a genius in robot design but just a little clumsy when it comes to tools. I approached him, clearing my throat.
"Hey, Jeremy. How's it going?"
"Not good," he said peering quite literally into the guts of the robot.
I paused and let that sink in a little. I was growing fond of oodles of cash and ... my new Porsche Boxster. I didn't like the puzzled look on the face of my genius friend.
"Care to tell me what's going on?"
"Steve, we have a very simple system here. As you know, a supervisor system manages eyes, ears and coordinates several simple subsystems. The super manages a voice recognition unit to store information about and then purchase certain products. The purchase decision and financial transaction subsections are trivial. The navigation services are a little more complex, but just a little. Basically, all we need to do is make sure the robot doesn't get run over by a truck as it uses its GPS, mobility, hearing, and pattern recognition systems to walk to the store. I just don't understand why the purchase decision unit is going haywire."
"What about the fallback code? When the store is out of the requested item..."
Jeremy reflected for a second. "In that case, the robot has a set of user selectable second and third choices. If none of the choices are available, then the robot skips that item. It should never buy something not on its list."
Suddenly, I had a hunch. I hadn't forgotten all my engineering. "Are all choices stored in flash memory?"
"Of course," Jeremy said continuing to tinker.
"What's the data path between the flash memory and the supervisor system?"
"Huh? Minimal. The voice recognition unit writes to flash memory and the transaction system reads from it."
"Could supervisor code override the flash memory?"
Jeremy looked up and stared at me for a long time. He blinked. He rubbed his chin. Then he rolled his eyes and looked at the ceiling. Then he rubbed his chin again. "What are you saying, Steve?"
"I don't know. You tell me."
Jeremy squinted as he looked at me. He spoke slowly, deliberately. "What you're asking me is ... can a software update of the core OS ... the supervisor ... be written so as to alter the flash memory data after user input"
I sat down on a high stool and pondered.
"Yep. That's what I'm asking."
"Let's find out," Jeremy said and winked at me. "I know a fast way. Here. Hold this."
Jeremy handed me his voltmeter and reached into the motherboard with a small screwdriver. Of course, I have known Jeremy for a long, long time, ever since graduate school in Alabama. I knew his techniques tended to be, shall we say, rather blunt at times. So I quickly rolled off the chair and dove under the bench just as a giant electrical surge ripped through the robot.
The small explosion almost toppled the robot and stung my ears.
When I crawled out from under the table, I saw Jeremy on the floor on the other side of the robot. He was sitting upright, looking a little dazed, and brushing some pieces of solder off his shirt. He looked up at me and smiled.
"Steve, I believe the answer to your question is ... yes."
The next morning, we had a very early meeting. The evening before, I had asked Theo from Customer Support, Jeremy from Engineering, and our newly acquired corporate attorney, Tom Bellows to be there. I knew that Carol didn't usually come into work until 9:00 am, so I called the meeting for 8:15 am.
I asked Theo to give us a brief report.
"Well, it looks like this. We've shipped 2,127 handyMan models in the last two weeks. We've had complaints from 1,014 of those customers, and all from the earliest shipped units. The, um, complaint is that the unit buys unauthorized products or overrides their initial instructions. As of right now, 702 of them want their money back ... immediately."
"What have we told them?" I asked.
"Nothing much. We've simply said that the units are thoroughly tested before shipment and we believe they are functioning correctly. We have suggested that the customers may not be clearly communicating their requests to the handyMan."
"Do we really believe that?" I said, looking at Jeremy.
Jeremy cleared his throat and wiggled in his seat. "Before the units leave here, they go through a two day burn in. We ... our computers ... give each unit about six thousand different voice instructions, in fourteen accents and do a diagnostic dump of the bar codes they will search for in the store. If any unit fails any test, we would stop production. That has not yet happened."
I stopped to think for a second. "How do we do firmware upgrades?"
Jeremy continued. "Every Sunday morning, at 0400 hours local time, the units dial into our computer to see if there are any updates."
"Who manages that software update database?"
"Uh ... Carol Strassman." Jeremy said, looking at me with a squint. A thoughtful, worried squint. The room was very quiet for a few seconds. It was all starting to sink in.
"Theo," I said, "Send an e-mail to every customer. Tell them we may, repeat, may have found a problem with the handyMan units and that they should wait until this Sunday's firmware update."
Tom Bellows interrupted. "I don't think it's a good idea to admit any product liability here. I recommend we just do the update and say nothing."
I was annoyed. Long ago, I had decided that I would run my company the right way. "Tom, our customers have a defective unit," I insisted firmly. "An alarming percentage want their money back. We need to tell our customers that we're going to fix it. Otherwise, they'll never really trust the handyMan again. And they'll tell their friends they don't trust it. Then it'll be so much for myRobot, Incorporated. D.O.A."
"But our exposure to litigation is just too great," Bellows said. "I don't recommend it."
"Tom, your advice is duly noted." I looked at my watch. It was 8:45. "This meeting is over. Theo, you have your orders. Everyone. Do not discuss this meeting with anybody else. Period." As they started leaving the room, I remained seated, reflecting. I ran my hand over the massive, smooth walnut table. The sun was shining through our large conference room windows and beautifully lighting up the dark wood on the far end of the table. Next to me, our prototype of vacuumVeronica was in a glass case. She had earned me my first billion. I had worked hard to build this company; I didn't want to lose it.
Jeremy and I went to Engineering and logged on as root to the Sun workstation that manages the firmware updates. What we found was that every handyMan that had reported in so far was given a firmware update version 1.01. We weren't very surprised when we found that the new file contained code to prefer certain products. I was beginning to suspect that these vendors had paid someone in our company to modify the handyMan firmware so that it would buy their product --- at some frequency -- not the one the customer requested. But we couldn't prove who did it.
Jeremy wrote the nasty version 1.01 code to a DVD-RAM disc. Then he replaced version 1.01 with the original 1.0 code, but left it labeled as 1.01. We reset the date stamp of the file to be the same as version 1.01 and padded the file with dummy code that looked like the original code to make the file size the same. Then Jeremy wrote a Perl script to run in the background and log any attempts to read or modify the restored firmware file.
Then we waited.
About the Author
John Martellaro lives in Colorado at 2,800 meters above sea level with a Ph.D. wife and two cats, Nikki and Data. He holds a B.S. in Astrophysics and an M.S. in Physics. His hobbies, include amateur astronomy, downhill skiing, bicycling, and listening to piano solos. His personal Macs are a B&W G3/400 with a flat screen Studio Display and a blueberry iBook.