I’m guessing the woman was a Jewish-American Princess, though I’ve no idea how cold she was on her honeymoon.* She certainly had a frosty look on her face as she did an about-turn and marched back towards me. Maybe not frosty, exactly; more plain scared. For she’d just let slip that Tom Waits had declined the chance to top one of the bills at California’s Coachella Festival. The little nugget had popped out and, as she trotted away, it dawned on her what she’d done.

“Hey,” she said in a palpably anxious voice as she returned, a clammy palm placed on my forearm. “Don’t get me in trouble with that Coachella story, you hear?”

I beg your pardon? You’d think she’d whispered that Waits was cheating on his wife, not that he’d turned down the organizers of a frigging rock festival.What did it matter if people knew? To her, apparently, it mattered a lot. The mild panic on the woman’s face told me something of what I was up against simply in attempting to write a book about Tom Waits.

A few weeks later, Keith Richards sent word from on high that he was happy to talk about working with Waits on Rain Dogs and Bone Machine. But the offer was summarily withdrawn because Tom – or more accurately, “Tom and Kathleen” (Brennan, Waits’ wife) – had been apprised of my request. The apparent perversity of not wishing one of rock’s undisputed greats to go on the record with his love and admiration confounded me. It also got me thinking about the Waitses’ real agenda in stymieing biographers past, present, and future.

At various points during two years of researching Waits’ life and work I had to stop and ask myself, “Do I actually have the right to write a book about Tom Waits?” It’s tough not to personalize the rebuffs, not just from the Waits camp but from certain acquaintances and collaborators.Tough, too, not to see their polite requests that such people not consort with me as covert censorship.

“What do you think they’re afraid of?” friends asked me. Generally what I said was something like: “I don’t know what they’re afraid of. I think they know I’m not Kitty Kelley or Albert Goldman or J. Randy Taraborrelli – or Nick Broomfield or A. J.Weberman or Rupert Pupkin. There’s not a lot of dirt to dig up anyway. So Waits got loaded with Rickie Lee Jones and Chuck E.Weiss. So they defaced lawn jockeys in Bel Air and got into a spat with some cops at Duke’s coffee shop. So what.” Usually I paused before adding: “Actually I don’t think they’re afraid of anything. They just don’t want a book out there that, in some cod-Freudian, ad hominem way, reduces Waits to the sum of his life experiences. And I have some sympathy with that; in fact, I have total respect for such a stance.”

My friend Jeb Loy Nichols reminded me of how the artist Joseph Cornell (one ofWaits’ minor heroes) resisted all attempts at investigation of his life. For the eccentric Cornell – subject of Deborah Solomon’s biography Utopia Parkway – the only thing worse than being misunderstood was … being understood. And as Bob Dylan – one of Waits’ major heroes – once said,“What’s so bad about being misunderstood?”

For decades Waits has played an elaborate game with the media, hiding behind the persona he projects. To the question “Will the real Tom Waits please stand up?” there is no real answer.“Tom Waits” is as much a character created for his fans as it is a real man behind the closed doors of family life. “Am I Frank Sinatra or am I Jimi Hendrix?” he said when I asked him if his persona had ever merged with his actual personality. “Or am I Jimi Sinatra? It’s a ventriloquist act, everybody does one.”

But, I countered, some artists are more honest about it being an act than others. We aren’t supposed to think Neil Young is doing an act. “I don’t know if honesty is an issue in show business,” Waits retorted. “People don’t care whether you’re telling the truth or not, they just want to be told something they don’t already know. Make me laugh or make me cry, it doesn’t matter. If you’re watching a really bad movie and somebody turns to you and says, ‘You know, this is a true story,’ does it improve the film in any way? Not really. It’s still a bad movie.”

Reading this quote again, I think that Waits a) should be right but b) is being disingenuous.He knows full well that fans and critics alike experience rock music as, in some sense, communing with an artist’s soul. Robert Christgau, “dean of rock critics”, called this “the idea that the artist’s persona is their fundamental creation”. Put another way, fans of auteur-artists such as Bob Dylan and Neil Young have long sought to establish a correspondence between their life and their work. Heritage rock mags are predicated on rooting out the “stories”behind albums such as Blood on the Tracks or Tonight’s the Night.There’s an inordinate amount of investment in the notion of the artist as suffering seer or tortured poet.

“I spend my entire time trying to explain to people that I’m a creative writer,” P. J. Harvey – a hardcore Waits fan – told me. “People jump to conclusions, and I can understand it, because if I’m very interested in an artist – whether it’s Neil Young, Bob Dylan, whoever – I want to imagine that those stories are true. But I think also that when I listen to those writers I project my own stories into their songs. And I’d like people to be able to do that with mine.”

Dylan and Young, of course, wrote the book on messing with the preconceptions of fans and critics – of not being the “Bob Dylan” or “Neil Young” that people want them to be. Not for nothing was Dylan the first singer-songwriter model for the young Tom Waits starting out in San Diego.

The games Waits plays with interviewers thwart all attempts to marry his music to his life: like Prince, an artist he loves, he’s too protean to be so easily captured. Moreover, in our age of mass celebrity he refuses to sell himself as a rehab car-crash fuckup. He’s the anti-star who declines to live according to the narrative of sin and redemption that celebrity culture requires. All of which puts a biographer in the invidious position of feeling like a parasite feeding on a resentful host. (In 1999, after years of following his career and writing at length about him, I finally had the opportunity to interview Prince.“Is it truth or is it conjecture?” he asked me about Imp of the Perverse, a book I’d written about him. “What gives you the right to write a book of conjecture about my life?” I had to think about that. I guess I’m still thinking about it.)

I’ve interviewed Waits in person twice, and spoken to him on the phone a number of times. Like most of the journalists who’ve talked to him, I (like to think I) have got along well with him. I’ve been regularly reduced to helpless laughter by his conversation. The first encounter was between albums in downtown New York, the interview a special concession to New Musical Express, who’d made Swordfishtrombones their Best Album of the Year. Not being on the interview treadmill, Waits was easy company, teasing waitresses and talking about the everyday madness of his adopted Manhattan.

The second time, fourteen years later in a diner near his northern California home, Waits was partway into a week of being grilled by the European press, and I got the distinct sense that he’d been ground down by the earnestness of his interlocutors. He loosened up some when we went for a backroads spin in his 1970 Coupe de Ville, but there was a wariness, a fatigue, about him that hadn’t been there in 1985. The struggle to hold on to his privacy in the face of almost cultish fascination with his every move had, I thought, started to tell.

I first contemplated writing a book about him circa 1991. Somewhere in a drawer lies a thin proposal for a tome entitled A Sucker on the Vine: Tom Waits in Tinseltown. Fifteen years later came a phone call from an American editor and a conversation about possible biographical subjects.We stopped at Waits. I said that, as hard as it would be, a serious study of Waits as artist and man had to be attempted by somebody.

Yet nothing quite assuages the guilt a biographer feels in prying into the personal history of such a resistant subject. (“We have a right to know,”Waits mutters with sinister invasiveness on 1999’s hilarious “What’s He Building?”.) I’ve often tried to put myself in Waits’ shoes during the process of researching this book, and try as I might to defend my right to write about a public figure I can understand his distaste for the idea of someone rooting about in his past. “The stories behind most songs are less interesting than the songs themselves,” he said to me in Santa Rosa.“I mean, that’s my opinion. So you tell somebody,‘Hey, this is about Jackie Kennedy.’ And they go, ‘Oh wow.’ Then you say, ‘No, I was just kidding, it’s about Nancy Reagan.’Well, it’s a different song now. In fact, all my songs are about...