Colom, 43, took 49.2 percent of the vote, according to figures from CBS. Hyde-Smith received 48.7 percent, and an independent candidate got 2.1 percent.
Excitement filled the air as Colom thanked supporters who had waited with him for the count to be settled.
"They did not believe a moment like this was possible, but I kept the faith,"Colom said.
The Democrat campaigned aggressively, visiting all of Mississippi's 82 counties.
He appeared to ride a national wave of resentment against Republicans.
Hyde-Smith congratulated Colom in a video posted on X on Wednesday evening and wished him well.