Scholz won the support of 207 MPs in the 733-seat lower house while 394 voted against him and 116 abstained.
This means he missed the required majority of 367 votes by a large margin and the chancellor will now ask President Frank-Walter Steinmeier to dissolve parliament.
Scholz submitted the motion for a confidence vote after the collapse of his centre-left coalition - the only way for a chancellor to trigger rare early elections.
The result of the vote had been widely expected as Scholz's government no longer holds a majority in parliament.
Following months of fierce infighting over economic policy and the 2025 budget, Scholz fired then-finance minister Christian Lindner of the pro-business Free Democrats (FDP) in early November.
The junior partner then exited the three-party coalition, leaving Scholz in a minority government with the Greens.
Addressing parliament ahead of the vote on Monday, Scholz said there were important political issues facing Germany, including long-overdue investments in its ailing economy, that required the public's input.
"This decision is so fundamental that it must be taken by the sovereign itself, by the voters," he told MPs.
Early elections would allow "the citizens to set the political course" for the country.
Scholz has already been chosen as the SPD's top candidate for the upcoming election but his chances for re-election appear slim, with current polls putting his party in third place, behind the conservative CDU/CSU bloc and the populist anti-immigration Alternative for Germany party.
The chancellor used much of his 30-minute address to make campaign pledges to German voters including stable pensions, an increase in the minimum wage and his long-standing refusal to supply Ukraine with long-range Taurus cruise missiles.
Friedrich Merz, the top candidate of the CDU/CSU alliance who is seen as most likely to replace Scholz, accused the chancellor of leaving behind one of the country's greatest economic crises in Germany's post-war history.
The state of Germany's economy, once considered the most stable in Europe, was the main point of contention leading the collapse of the coalition government.
Rising energy prices, high inflation, a decrease in exports and the ailing car industry, struggling with increasing Chinese competition and the transition to electric vehicles have all contributed to stagnating economic growth.
Now that MPs have withdrawn their confidence in Scholz's government, the chancellor will immediately ask President Frank-Walter Steinmeier to dissolve parliament and call early elections seven months earlier than originally planned.
The president then has 21 days to make that decision, a step which is considered a formality as he has already signalled support for the February 23 date proposed by Scholz.
It would only be the fourth time in German history that early elections are held.
Scholz and his ministers, as well as the current lower house members, will all remain in office until newly elected MPs convene for the first time following the election.
As coalition negotiations are often protracted, it is possible that no new government will have been formed by then.
In that case, Steinmeier can request Scholz's government remain in office for the time being - a request they are obliged to accept.
with AP
