Try these
1/ Prop or shaft not balanced or shaft bent
2/ New motor not correctly lined up with the shaft
3/ Motor – shaft coupling out of alignment
4/ New motor not secure in its mounts
5/ Is the rudder loose or have any sidewards play in its shaft
And questions would be:
Does the shuddering occur across the full speed range or just at certain revs?
Is the new motor the same type and size as the old one?
Did the boat suffered any underwater damage or strike anything during the new motor test?
Paul