At higher difficulties Stats of Monsters and other Opponents increase.
It can be seen as the opponents and Monsters are smarter on higher levels, when actually they are not.
As Monsters have higher Stats, they last longer on the battlefield, and thus, have more time to get to the weaker Party Members and reduce their health.
Members with less Health like Magic Users, can therefore be more vulnerable at increased difficulties.
Magic Users need to stay away from Monsters, and at higher difficulties it's more challenging to do so.