Mr Downer will make an announcement shortly after he returns on Wednesday from the annual Australian American Leadership dialogue in the US, The Sunday Age in Melbourne has quoted the sources as saying.

The move means that another by-election, this time in the Adelaide Hills seat of Mayo, is imminent.

The paper says there are believed to be two main Liberal candidates for the seat: Jamie Briggs, former Prime Mminister John Howard's adviser on industrial relations; and Matt Doman, a former staffer to right-wing power broker Nick Minchin, who was Finance Minister under Mr Howard.

It's believed that Mr Downer is supporting Mr Briggs, while Senator Minchin favours Mr Doman, the paper says.

"The Mayo by-election would be a much more realistic litmus test of how the Liberal Party is travelling than the Gippsland by-election," a source told the newspaper.

Mr Downer told the newspaper through a spokeswoman that he's "tired of the constant speculation", but did not elaborate.