Shakespeare, Opera, Pop and a Pet Shop Boy. Rufus Wainwright delights a slightly damp London

The jolly atmosphere at the bar, which is like being backstage at The Baftas, featuring the likes of Mark Gatiss and Helena Bonham Carter, is tempered by a trickle of rain. The odds that it will rain on Rufus’s parade are short; it’s Wimbledon tennis month after all.

Despite the gathering rain, John Walker, a Chelsea Pensioner, takes the stage and cheerily greets us. "Welcome to my home, a home for 300 retired soldiers."

Rufus introduces himself with the line "‘I’d like to dedicate this to the ladies", opening with “Little Sister” from Want Too. There’s no band tonight, just his sonorous voice and piano (and occasionally a troublesome acoustic guitar). When it ends he says “I thought that song matched the English rainy atmosphere.”

However, during third song “Vibrate” something miraculous happens, it stops raining. As if to celebrate, he ends it holding a note for twenty seconds and it’s like you’re in the presence of a modern-day pop version of Maria Callas.

The baroque “Want” with its prescient lyric, “I don't want to make it rain/I just want to make it simple”, is a tear-jerking song about his father - “I really don't want to be John Lennon or Leonard Cohen/I just want to be my Dad” - Loudon Wainwright III, who infamously wrote the song “Rufus Is A Tit Man” about his son’s appetite for breast milk.

The spoken interludes throughout are so wittily ad-libbed you’re half convinced they’re rehearsed. At one point he frets about his broach, concerned that "Maybe it’s a little too Diane Keaton? I think it’s the hair". He decides to take it off, "It’s piercing my Adams apple; I might impale myself and die onstage".

His storytelling reaches its zenith on “The Art Teacher”. The tremulous longing of the melody is mirrored in the lyric, a conversation between a pupil and the object of her desire. Their discussion about art is casual, but to return to Diane Keaton, it’s like her subtitled conversation with Woody Allen in Annie Hall, where the real meaning is to be found in what’s not said.

He introduces "a friend since I was a lonely gay boy in the corner of the club not talking to anyone" and the Pet Shop Boys' Neil Tennant joins him for “Poses”. The poised restraint of Tenants’ voice dovetailing with Wainwright's garrulous drawl is mesmerising, even if Tennant seems to be reading the lyrics from his phone and it ends with their voices locking on the refrain.

Wainwright is currently writing musical interpretations of Shakespeare’s sonnets and plays a stunning version Sonnet 20, which starts with a line that could have been one of his. “A woman's face with nature's own hand painted/Hast thou, the master mistress of my passion.”

An aria from his opera Prima Donna is "about a Diva deciding whether to continue her career or not… It has nothing to do with my life!" Even though it’s sung in French, it sounds so mournful you understand its meaning regardless. He seems to inhabit the narrative, sounding like its abandoned heroine. It’s certainly not Edith Piaf’s “Non, Je Ne Regrette Rien”.

He finishes with “Cigarettes And Chocolate Milk” with the walls of the courtyard beautifully bathed in red as the sun falls. Its waltzing piano riff is playful, despite the lyrics darker tone, “I’m a mess”.

The encores include the Gershwin like "Montauk" and a new song about his sister Lucy who is “Green, yellow, red, just like the Springtime’, but the highlight is his take on Leonard Cohen’s “Hallelujah”. Whereas Jeff Buckley’s version is the sound of a broken heart, Wainwrights’ is exhilarating, as if the question of “How to shoot somebody who outdrew you” is a puzzle to be solved. The flourish in his voice makes it a hymn of devotion rather than an elegy.

Tonight Rufus Wainwright metamorphosed from balladeer to vaudeville performer to operatic diva without skipping a beat. Equal parts raconteur and troubadour, his performance was so enthralling that despite being a little sodden, leaving the venue it felt like the sun had its hat on all along.

• Photo by Jason Williamson. See the full gallery here.