Shares of large and mid-sized U.S. banks sharply underperformed the broader market on Wednesday with the S&P 500 Banks Index closing down 2.0% while the benchmark S&P 500 Index fell 0.6% with worries about commercial real estate loans in focus among bank investors.

Investors worried about potential losses among banks from office real estate loans after comments from executives, including Wells Fargo & Co Chief Executive Officer Charlie Scharf and Blackstone President Jonathan Gray at a Sanford C Bernstein investor conference.

Scharf said on Wednesday there will be losses in the office loan sector and that the bank was proactively managing its portfolio while he looked to reassure investors that it is not "overly concentrated" in that area.

Gray talked about "unprecedented weakness" in older office buildings while noting that this segment currently makes up less than 2% of company's equity portfolio in real estate.

"Vacancy is 20-plus percent, rents are declining, companies now are obviously thinking about their space needs in light of remote work and the economic climate that's ahead. Lenders are reluctant to have exposure to office buildings. Buyers are reluctant. Valuations are going down," Gray said, according to a transcript from the Bernstein conference.

But Gray still estimated that "office buildings are about 3% of the U.S. banking system" so the size of losses, "relative to what happened in the housing market 15 years ago is dramatically different."

Rick Meckler, partner at Cherry Lane Investments, a family investment office in New Vernon, New Jersey, said "continued concern over loans made to the office market" hurt bank stocks broadly on Wednesday, citing the Wells Fargo comments.

"The implication is that there are those that will suffer even if Wells Fargo is diversified enough," Meckler said.

Wells Fargo ended the session down 2.9% while Morgan Stanley dropped 2% and Bank of America fell 1.7%. Goldman Sachs and JPMorgan Chase & Co lost 1.3% and Citigroup shares closed down 0.9%. Blackstone shares ended down 0.9%.

Regional lenders came under more pressure with KeyCorp falling 5.9%, as the biggest decliner in the S&P bank index, followed by Zions' 5.6% drop and Citizens Financial's 5% decline.

Also on Wednesday, the Federal Deposit Insurance Corporation said U.S. banks' total deposits declined by a record 2.5% in the first quarter.

Hurting the broader market as well as bank stocks were jitters ahead of a lawmakers vote on a deal to raise the U.S. debt ceiling and unexpectedly strong labor market data that reinforced bets for more Federal Reserve interest rate hikes. (Reporting by Sinéad Carew in New York, Mehnaz Yasmin in Bengaluru; Editing by Nick Zieminski and Richard Chang)