and my soul had been poured out in prayer unto my God all the day long for them; nevertheless, it was without faith, because of the hardness of their heartsMormon 3:12
Mormon didn’t have faith in the Nephites. But James said
But let him ask in faith, nothing wavering. For he that wavereth is like a wave of the sea driven with the wind and tossed.
For let not that man think that he shall receive any thing of the Lord.James 1:6-7
Does that mean, that if Mormon prayed in faith, would the Nephites have repented? Maybe. But faith isn’t all that’s required. The Nephites would have to listen to the Holy Ghost’s promptings. Mormon knew that they wouldn’t listen, and that’s why he had no faith.