Yeah fair enough, i just find if they are on its annoying having to rebait every time you drop a line in, especially if your in a rush getting livies etc.
Havent encountered those Diamond fish where i get my yakkas, but guess they'd behave similarly to sweep. Are you fishing off a wharf? Sometimes if the sweep are a real nuisance ill cast out with a split shot or touch of lead and try and get past/underneath them as the yellowtail sit a touch deeper and arent hugging in as tight to the structure. Could be different circumstances though.