Please see annotated attachments. I’m in a 2a zone Florida home initially built as ‘low income’ housing (it shows) in 1998. It’s no longer a ‘newer’ house in the area which has undergone some rapid gentrification. It sat empty for over a year during the housing crisis in 2008 and was purchased by us around 2010. HVAC issues were noticeable immediately (excessively hot upstairs, poor A/C output), but it wasn’t until the return of a racoon in the dormer that eventually encouraged me to cut into the ceiling and see what was going on… it was a disaster and quickly explained a portion of the excessive energy use for cooling (including adding window unit – two local HVAC specialist failures to accurately identify the problem).

The 2nd floor is built ‘between’ the largest spaces of the trusses with a dormer in the middle of the 2nd floor where stair landing is (see photo). Both sides of the dormer are ‘attic’ space over 1st floor, with insulation on the floor, 2nd story walls, and dormer walls. The back side of the house this ‘attic’ space runs continuously through the whole length of the house with HVAC ducts.

Based on reading here and at EnergyVanguard, there are two major things I know need to be done:
1) Windows – though not the highest ROI at the moment, they’re single pane and very leaky
2) I know the environmental envelop has to be moved, bringing HVAC ducts into it. I’m struggling with a few different approaches and looking for input/recommendations:

Moving vents from ceiling to wall in order to bring ducts below peak attic would make a tighter envelope easier except for a bathroom one (it’d end up in the shower), and the dormer (inclined to consider floor in this extreme instance since I know attic is seriously problematic in my setting). The current ducts also are ‘pinched’ (see photos) at corner of 2nd floor ceiling and actual roof sheathing is another reason I think they should move.

Moving the actual ducts inside conditioned space. Here core of my question regarding strategy…

Should I expand the total conditioned space by creating an insulated layer along the rafter lines?

The attic is vented, so I know I can’t do sealed spray foam (aside from expense) so I was planning to tack 1×1 spacer to allow for a continuous vent from the soffits on side of house all the way to peak of the roof, then place some R5/R7 foil backed foam insulation board against it to seal it and offer radiant barrier, layer fiberglass insulation like R30, then another layer of continuous foam board to effectively seal probably 90% of the rafters from thermal bridging potential. This increases total conditioned space quite a bit.

OR

Reinforce, fix, and seal over with foam board (foil backed?) the existing insulation areas, and on the HVAC duct housed side basically shift the current envelop wall to fully enclose HVAC duct work?

I may have read too much on here at this point as I’m probably considering more variables than are necessary!

(Zip file contains additional photos)