Gifford misses a possible source for this phrase offered by Joyce's friend Byrne in his memoir:
On Wellington Quay, about midway between the bookshops of the brothers Strong, there was another divided shop run by people named Moulang and Goyer, the latter selling pictures, frames, etc. In Goyer's window there were featured Currier and Ives and "Darktown" pictures — including "Fire Brigade", "Cock Fights" and "Dog Fights". One of the last named groups had a legend reading, "She'd have won the money only for the other dog" [...]1
John Francis Byrne, Silent Years: An Autobiography with Memoirs of James Joyce and Our Ireland (1953), p. 20
The phrase turns up in the 1890s, first in literal use:
"My pup", as the Negro wailed, "would have won the battle, if it hadn't been for the other dog."
Literary World (1895), vol. 51, p. 244
That Lawyer McAlister's dog 'Garry' should have got first prize. So he would - only for the other dog.
Observer (Auckland) (1897), 13 November, p. 11
But it quickly came to be used metaphorically:
We are now told with that true sportsmanship which characterises the beaten Britisher that the visitors had the best of the luck in the drawn test matches, but the insurmountable fact remains that they won the only game that was finished, and that over and over again they strongly urged that the matches should in all cases be fought to a finish. In fact, it is the old tale retold. England would have won "only for the other dog".
Freeman's Journal (1899), 7 September
Mr. BLAKELEY. - The party on this side made its position very clear to the people of Australia on 5th May, and prior to that date; but because of the noise of the Win-the-War party, our protests were unheard in a great number of quarters, with the result that we did not get the votes we probably otherwise would have got.
Mr. HUGHES. — You would have won but for the other dog!
Parliamentary Debates: Senate and House of Representatives (Australian Parliament) (1916), vol. 82, p. 332