Malaysia's government wants to distance itself from the affair, which is the focus of money-laundering probes around the world, before elections that will be called as soon as this year.
"Private settlement has hit a wall - they are disagreeing on one or two big issues. It looks like the court will have to decide," said a source close to the talks, although he added that an out-of-court settlement was still possible.
An Abu Dhabi-based source said: "The best solution would be an out-of-court settlement, but that doesn't seem likely."
In December, IPIC proposed that 1MDB pay it an initial $1.2 billion to help reach a final settlement but the Malaysian side rejected this, the sources said.
Spokesmen for IPIC and 1MDB declined to comment.
The case is also potentially embarrassing for Abu Dhabi, which is traditionally secretive about its investments.
In July last year, the U.S. Justice Department filed lawsuits seeking to seize $1 billion in assets bought with money stolen from 1MDB, including luxury real estate in New York, Beverly Hills and London, paintings and a private jet.
Former IPIC group executives Khadem al-Qubaisi and Mohammed Ahmed Badawy al-Husseiny were named in the U.S. lawsuits.
Al-Qubaisi and al-Husseiny could not be reached for comment. Abu Dhabi police and government officials declined to comment.
Malaysia and the United Arab Emirates have strong diplomatic ties and state-linked Abu Dhabi firms have invested in Malaysian real estate, banks and infrastructure.
In June 2015, IPIC agreed to lend $1 billion to 1MDB and assume coupon payments on $3.5 billion of 1MDB debt. It also forgave an undisclosed amount of debt that 1MDB owed, in exchange for assets which were not identified.
The tie-up unravelled last year, when IPIC said 1MDB had defaulted on the loan, and the Malaysian firm defaulted on interest payments for two bonds which IPIC had guaranteed.
At the centre of the dispute is a disagreement over how much money 1MDB paid IPIC, the sources said. The Malaysian firm insists that it paid $3.5 billion to a subsidiary of IPIC, Aabar Investments.
IPIC says, however, that the $3.5 billion went to a British Virgin Islands-registered company with an almost identical name, Aabar Investments PJS Ltd, that was not part of the IPIC group.
1MDB, founded by Malaysian Prime Minister Najib Razak in 2009, is the focus of money-laundering probes in at least six countries including the United States, Switzerland and Singapore.
Najib has denied any wrongdoing and in January 2016, an official Malaysian investigation ended, clearing him of any offence.
Since the dispute flared, both Malaysia and Abu Dhabi have sought to repair some of the damage by restructuring their funds. Last year, Malaysia's finance ministry dissolved 1MDB's board of advisers and took over most of its remaining assets.
The Abu Dhabi government last month merged IPIC with another big state investment firm, Mubadala Investment
But any resolution of the dispute may be complicated by the international probes of 1MDB and uncertainty over how much of its assets will be recovered.
(Editing by Andrew Torchia and Adrian Croft) ((email@example.com; +60323338047; Reuters Messaging: firstname.lastname@example.org))