I've had a problematic Tornado ESC, replacing it with a Marine Viper 25 in instead solved everything in one go. My problem was, early on, poor control, on its last trip, sticking on full forward.
Small props will never cause overheating of either the motor or ESC, over-revving will not be a problem unless the motor is not rated for the voltage that it is running on, and it will be the motor suffering, not the ESC. The ESC heating indicates that it is having its actual (rather than what the label claims) rating exceeded. And in that case, rapid brush wear would be the problem. Not having the spec sheet handy, I don't know off-hand what voltage the speed 600 bb is rated for, but from the name its a 540 with a lowish turn count and thick wires to carry lots of current. 9.6 rings a bell, somebody will know for sure. A very ordinary 540 (but rated for 12 volts) will be perfectly adequate, even more if mated with a 2 blade prop about the same size.
12 volts suggests a SLA battery – this might also not be up to providing what a "Speed" motor wants. Could be that the hot ESC is a side effect – the battery might just want some recovery time.
The boat where the Tornado let me down was second hand, the previous owner told me it went very well with a 12 volt battery and the battery compartment was set up for an SLA. I used a 7 cell NiMH pack instead. The thing woul plane for fun and had a surprisingly long cruise time, probably due to not having to lug lead ballast around.