Hi Tak:
To my knowledge, the ferry is only at weekends and on public holidays.
It’s become much busier there in past few years, but Saturdays not too bad.
I’ve previously managed to take ferry off the island at around 11am on a Sunday – but that was some time ago, not sure if still possible. (Landed at Wong Shek Pier – from where there are buses to Sai Kung and to Diamond Hill in Kowloon.)
On Sundays, especially, seems most people arrive in big groups: bustling and noisy when they’re around, but once they’ve swarmed on, peace can return to whatever part of the island you’re in.
But of course, for tranquillity, would be best after ferry’s gone on Sat, and before ferry and tour boats arrive on Sun.
There’s timetable for ferries, and contact info, here:
http://www.traway.com.hk/routes.html#tungpingchau
Martin