Broccoli and cauliflower, too late to save?

Went away for 3 weeks and the caterpillars have been having a party on my broccoli and cauliflower, is it too late to save? Should I cut back the worse leaves then spray with Dipel? Just spray everything? Pull out and start again. Suggestions welcome.