Hoyer's agent, Joe Linta, said negotiations have taken place but an extension "will be a very difficult deal to do. We're always open to talking with the Browns, but we're content to wait and see what happens down the road."
Hoyer is in the final year of a two-year contract that will pay him $1 million this season.
Coach Mike Pettine said earlier this month that Hoyer was ahead of first-round draft pick Johnny Manziel on the depth chart but the gap between them was not great.
Hoyer, a backup for much of his four-year NFL career, was 2-1 as a starter for Cleveland last season before tearing an ACL. He threw for 615 yards with five touchdowns and three interceptions.
The difference between Hoyer remaining a starter and reverting back to a backup role could be as much as $7 million per year.
The Browns open training camp on July 26.