Unfortunately jay I hear the same thing from Sydneysiders all the time. It's also unfortunate that we can't teach common sense. If I was you, I would get myself a tinny and open up your fishing experiences to a whole new world. Not a 5hp either. Something with maybe 25-40hp and that will open a few more areas as you become experienced. As for swimming, I'm not a strong swimmer either, that's what life jackets are for! I regularly go to the shelf, so swimming in from there isn't an option! Lol. Don't let that deter you!
As I said, it's all about common sense! Good luck with your decisions!
Cheers scratchie!!!