They are humanoid characters who will do their worst to spoil your garden and must be dealt with.
If you bribe them with chocolate coins, they will leave your garden.
A Ruffian will show up and take away your helpers if they become trapped in your garden.
If there is water in the way of a Ruffian they will "throw up" land to make a bridge and leave the garden.
You can scare them away by whacking them with the upgraded shovel. The Captains Cutlass will keep Ruffians and Professor Pester out of your garden. Mainly they're just nasty creatures....