“Man, they’ve got good seafood, reeeeaaallll good seafood” says my cab driver rolling his r’s with relish as we leave the skyscrapers of Manhattan behind and begin to pass through Harlem before heading to the Bronx.

It’s nice of him to finally say something positive, because for the first 15 minutes of our cab ride, he’s been pleasantly but firmly trying to put us off our destination, City Island.

“Don’t you mean Coney?” he’d said when we first got into his cab. “Tourists never go to City Island, that’s just for New Yorkers.”

But we’d insisted it was City Island we were looking for and we weren’t going to change our minds and he finally seems to have come round to the idea to the point where he has started to eulogise the plus points of the tiny bolthole that Manhattanites call ‘the Hamptons of The Bronx’.

City Island is a 2.5km by 1km island that juts out into the Long Island Sound right at the very top of The Bronx and just before New York City turns into New York State. It’s a 30minute cab ride from Manhattan but, in true travel cliché fashion,  may as well be a world, or at least a couple of states away: City Island’s 4,000 residents, small fishing community and fabulous seafood restaurants are much more like being in a small New England village than New York.

The island was originally settled by Lenape Indians before it was bought by an English nobleman Thomas Pell, who subsequently sold the land to Benjamin Palmer in 1761. Palmer had grand plans and, convinced that thanks to the still waters of the Sound he could build a port to rival New York’s, he renamed the island from Minefer’s to make it sound sexier.

Unfortunately, his plans were scuttled by the American Civil War and a period of decline ensued until the late 1800s, when a small community based around fishing and shipbuilding was established. The fishing led to catch and the catch to the restaurants, which soon attracted a crowd of locals and in-the-know Manhattanites.

The main drag has the kind of downtown normally seen on quirky independent films with antique shops, a few taverns, New York’s smallest civic library, an ice-cream parlour and a cracking deli called Papa John’s, but where the island really excels is on the water. There are four yacht clubs and most of the piers have special anchorages – a free docking policy where you can sail up and moor. There are also companies that will take you out into the Sound for a day’s fishing for around $50 (£25) and a kayaking club that rents canoes.

There’s no doubt the island retains a slightly down-at-heel homely feel and has some way to go before becoming a destination in its own right – but it is moving far enough in the right direction to be a calming bolt-hole during a trip to New York.

If you’re going to stay overnight (and after a visit, a late-night return to Manhattan through The Bronx and Harlem may not be the best idea) then check out Le Refuge Inn, the island’s only B&B.

Owned by Jackie O’s favourite chef, Pierre Saint-Denis, its shabby chic fits right in with the island’s atmosphere. It’s an atmosphere that has started to attract the likes of Vincent Pastore of The Sopranos – who’s obviously a fan of reeeeaaallll good seafood

Le Refuge Inn (001 718 885 2478, www.lerefugeinn.com). For more on New York City, visit www.nycvisit.com