FWIW, Wikiquote.com calls this "misattributed":

Misattributed by former Defense Secretary James Schlesinger on Fox News.
Patton commanded French troops, the 2nd Armored Division commanded by
Philippe Leclerc, integrated in the Third Army, and had rocky but friendly
relations with the French general. For instance, on August, 15 1944 Patton
wrote in his diary: "Leclerc came in very much excited. He said, among other
things, that if he were not allowed to advance on Paris, he would resign. I
told him in my best French that he was a baby and said I had left him in the
most dangerous place on the front. We parted friends"