I don't think there's any way attention to your horse is "worrying too much."
We bred TBs to be athletes, among the most demanding of sports. Make an athlete lay about with little to do but eat and human or horse, they are going to put on weight. This is stress ... not just weight, but stressful weight, and idleness of mind.
There are lots of things you can do to exercise the horse that doesn't require riding, though riding is nice. Maybe you could think about things to do while you ride out. Teaching much more rein response, for instance, doing heavy conditioning exercises on the ride ... I include, for some of my students horses that have similar problems as you relate ... lots of transition work, downward transitions with strong departures, both rate and gait changes.
Half halts, and circle work builds collection and collection builds condition and burns calories very fast.
This tends to keep joints and tendons strong and healthy. It's the idle time that breaks horses down much more than regular steady work. Just like we couch potato humans.
This work is very good for the rider as well, as it builds core strength for riding that only riding can really do correctly.
Great for balance improvement as well.
On the ride out, stopping to do lateral work, turns on forehand, hindquarters, half pass, shoulder in, all work the horse much more than just straight riding, even at speed. And riding becomes a lot more fun when you an do these things with your horse, and wonderful to do outside the barn or school, in the open.
And just as some human athletes, some horse athletes (TBs for sure) don't exercise their minds quite enough and become slow. Yet if you give them mind challenges they come to love it. Trick training can be quit strenuous and may be something that will engage your interest...it almost always does the horse once he gets the idea.
You do work with behavior event marking and a reward reinforcement system, yes? It's clicker training. And you don't have to stuff the horse with treats.
Have you someone to move your horse to dry lot part of the day or night so he isn't constantly feeding on that lush rich sugar and starch laden pasture?
One last question, what is "chaff," precisely? Maybe he doesn't need that.
Tell me about his hooves ... they can be great indicators of nutritional events in the horse's life. And in the end, for his health, you may have to muzzle part of the day ... and the muzzle need not be totally closed, just enough for slowdown on his grazing. Best wishes, Donald
Love is Trust, trust is All
So say Don, Altea, and Bonnie the Wonder Filly.