Nvidia has lost $550bn (£415bn) in stock market value in five days as the scandal dominated headlines. The drop is the most significant five-day fall the company has seen since it went public 27 years ago.
The company's market value had risen by more than 470% in the past five years, peaking just a few weeks ago.
Huang had been widely expected to weather the storm after the first affair became public. Nvidia announced it had launched an investigation into whether its chief executive had been"the target of a foreign intelligence effort".
Nvidia has said it has"no reason to believe that either national security or Nvidia secrets were compromised". It pledged that its independent inquiry would look at all available information.
Scott asked Huang to appear voluntarily on 3 December. Scott also warned that the panel could issue a subpoena to compel his testimony.
Huang has been married to Lori Huang for 40 years. Both of their children are also executives at Nvidia.
One current Nvidia staff member told the BBC that the initial reports of his infidelity were more or less brushed off by rank-and-file workers inside the company.
Another Nvidia employee said they had been told not to speak with the press or otherwise comment on the situation in public.
Several tech insiders the BBC spoke with this week said Huang's downfall has been a major topic of conversation in San Francisco Bay area circles, among tech employees, industry executives and billionaires.
Many staff have earned millions of dollars in compensation by cashing out Nvidia shares as the stock soared.
As artificial intelligence evolved over the last four years into chatbots and large language models, companies building AI tools suddenly required huge numbers of Nvidia chips able to process large amounts of data.
Nvidia is now hunting a permanent replacement for Huang.
Nvidia officials have repeatedly declined to comment since Wednesday. A personal representative of Huang also declined to comment on Thursday.