Tory minister Baroness Warsi has warned David Cameron against trying to "out-Ukip" Nigel Farage's party in a bid to win back voters.
She claimed that as well as pulling in Tory and Labour voters, a "chunk" of Ukip's support came from the far-right.
And she appeared to endorse Mr Cameron's assessment of some Ukip members as "fruitcakes, loonies and closet racists".
In an interview with Huffington Post UK, she said Ukip's vote "comes from the Tory Party (and) the Labour Party - but I do think there's a chunk of it that comes from the far-right".
The surge in support for Ukip, which could potentially see them take top spot in May's European elections, has been a cause for concern in the Conservative camp and Lady Warsi acknowledged that there had been a change in tone since Mr Cameron's 2006 "fruitcakes" attack.
She said "we're all being much more restrained and much more diplomatic" about Ukip, "but I don't think I could have put it any better than the way the Prime Minister put it when he described them many years ago".
Lady Warsi added: "Nigel Farage is trying to change his party and how successful he is in that will determine whether the label the Prime Minister gave (Ukip), all those years ago, sticks."
The Prime Minister has promised an in/out referendum on European Union membership by 2017 as part of the effort to woo voters attracted by Mr Farage's Eurosceptic party.
But Lady Warsi refused to back calls from the Conservative right to take tougher lines on issues such as immigration and the EU in an effort to win back voters.
"The one thing that no party can do is out-Ukip Ukip to win those voters back," she said.
Lady Warsi, a Foreign Office minister, also offered her assessment of progress in Afghanistan - with combat troops due to come home this year.
She said the UK was leaving Afghanistan "not having achieved that we said we would achieve over a decade and a half ago, but leaving it a much better country than we found".
In December, Mr Cameron was a sked by reporters if the troops would come home with "mission accomplished", the Prime Minister said: "Yes, I think they do. I think they can come home with their heads held high."
He added: "To me, the absolute driving part of the mission is a basic level of security so it doesn't become a haven for terror. That is the mission, that was the mission and I think we will have accomplished that mission and so our troops can be very proud of what they have done."
Lady Warsi, who was replaced as Conservative Party chairman by Grant Shapps, defended her successor over the controversial "beer and bingo" Budget advert.
He was widely pilloried after posting an image on Twitter claiming cuts to bingo and beer taxes would "help hardworking people do more of the things they enjoy".
But Lady Warsi said: "I think some of the attacks on Grant Shapps have been incredibly unfair because he's been made to carry the can of decisions which probably involved a whole load of other people before that poster came out."
Reports have suggested that Chancellor George Osborne had signed off the image, but Lady Warsi said: "Well, I don't know that...but I'm not convinced (Mr Shapps) should carry the can for this."
Lady Warsi said you "need a mix" of people in politics, achieved by " surrounding yourself by people from lots of different backgrounds".
But she said there was a "women problem" in politics, including at the Cabinet table - where there are just five female ministers, adding: "I think it needs to increase."