The house on Penny Lane in McCrae on the Mornington Peninsula collapsed shortly before 9am on Tuesday.
Local resident Tanina Osborne and her husband had just finished their morning beach swim when they heard a "horrible sound".
"I heard this really, really loud rumbling, and like a crash," Ms Osborne told AAP.
"So my husband and I ran across the road and I saw that this house had tumbled down the hill - I've just never seen anything like it."
An advice alert for the incident warns the landslide could continue to move for days.
Multiple nearby homes remained a concern and were being monitored by authorities, State Emergency Service operations manager Chris Gregory said.
"A couple of them were vacant holiday homes and others had residents who were working with councils and other agencies to make sure they have a safe place to go," Mr Gregory told ABC Radio Melbourne.
Ambulance Victoria said an injured person had been taken to Frankston Hospital in a stable condition.
The destroyed house was empty after it had been hit by a smaller landslide about a week before and was being monitored by geotechnical engineering experts.
Authorities have disconnected utility services in nearby properties as they assess the area for potential danger and the Nepean Highway has been closed between McCrae and Dromana.
"We do have a road closure in place and we'll seek to lift that as soon as we possibly can," Mr Gregory said.
Other houses in the area had reportedly been deemed unsafe in the past year, Ms Osborne said.
"I think that entire hill is going to have to be looked at," she told AAP.
"It would be an engineer's nightmare."
Ms Osborne said a water outlet near their daily swimming spot had been gushing into the bay three weeks earlier due to a water main issue on a road above where the landside occurred.
"That could have contributed as well, I don't know," she said.
"I'm not an expert, but you know that whole hill just has a lot going on."
20°C