Both.
1) Not necessarily that he breaks, but that he just won't be worth the contract for various reasons. OG is a top 10 defender in the NBA, but only really an average offensive player. Even in this "OMG DPOY!!!" season he's had with us and the Knicks, he grades out as the 36th most valuable player in the NBA according to EPM (similar according to BPM as well). I'd be mildly surprised if his next contract didn't end up starting at ~37 million and scaling from there (ending up at 46-47 million in it's 4th year). That's top 20 money for a guy who tops out at his absolute best as a top 35-40 type player. We've seen less interested, less engaged versions of OG sit around 1.5 EPM, which is a top ~70 player in the league. I don't think you should be paying big money to a guy who doesn't score the ball at a high level that kind of money unless you have your batman & robin bucket getters and the last thing you need is that elite defensive stopper. OG wouldn't be our last piece.
2) The Raptors have never been a high spending team. We've flirted with the luxury tax during the title season, but we generally sit with the big pile of clubs that are over the cap but under the luxury tax thresholds. The NBA is taking steps to contain the highest spending big markets to create a bit more parity, so the Clippers and Warriors actually have to make difficult decisions and can't just throw money at all of their problems. I just don't think OG is a valuable enough player to warrant 1/4 of our cap space with where we are as a team. Like I said, he's not the missing piece here regardless of how nice it would be to have a top defensive wing again. On our contender teams, his role was smaller than it became in the FredScal era. Now that he's left and gone to another good team, his role got reduced again (stand in the corner and shoot). So could we fit OG in? Yeah. But then that's it. Pay Scottie, IQ, Star J, OG and then scraps for the rotation. If I thought that was a title winning combination, I'd be in. It's not imo.
Because 35-45 million can be spent way, way better around those 3 than on OG. If you can't get a bucket when we need it, you're not worth max money. It's really that simple. I'd rather part out what OG does well on multiple good role players than sink all of that money in one really good role player.