The wider conflict escalated in 2015 after a Saudi-led coalition of Arab states intervened in an attempt to restore the government's rule.
Saudi-backed forces said the units were being moved to protect Aden and neighbouring governorates, a shift that leaves Houthi fighters in control of a key stretch of coast.
Houthi-run media reported later on Sunday that the group had taken the territory after government-aligned troops pulled back.
The withdrawal ends a weeks-long offensive to secure the area near the waterway. The Houthis seized it in a surprise advance in September, squeezing global oil exports and prompting thousands of people to flee.
Officials cast the redeployment as a choice to hold the capital rather than the coast."This step is driven by the priority of securing the capital, Aden, and neighbouring governorates, reinforcing the front lines and addressing existing threats,"an official said.
The official said that since fighting escalated in August, more than 60% of documented civilian casualties were women and children.
The Bab al-Mandab is a choke point for tankers leaving the Red Sea. With troops shifted to Aden, Houthi media said the coastal ground taken in September was again under the group's control.