The leader of one of the largest Tea Party groups says supporters will likely vote for Mitt Romney, but they won't campaign for him.

Judson Phillips, leader of the Tea Party Nation, tells the Daily Caller his group won't coalesce around Romney because they don't think he's sufficiently conservative. In fact, he thinks about a quarter of Tea Party activists won't vote for Romney no matter what.

“While that number will change as we get closer to the election, Romney has a huge problem with the conservative base of the GOP,” Phillips said. “He had better do something about that ASAP or he won’t have to worry about that moving to the middle nonsense. Without the GOP base, he is a lost cause.”

“Most of us,” he added, “are focusing on Senate and House races now.”

Another conservative leader, Tony Perkins of the Family Research Council, struck a similar note after former Pennsylvania Sen. Rick Santorum dropped out of the presidential race this week.

“It’s difficult for us to back a candidate our constituents don’t believe in and aren’t excited about,” Perkins told CNN, suggesting that social conservatives will instead focus their efforts on helping Republicans win control of the U.S. Senate in 2012.