Why are the the branches on my potted mango wilting and dying?

Bit of background: I’ve had this potted mango plant I grew from seed for about 3 years in Queensland. When it was about 1, it “died” but sprouted new shoots which were going really well last year, lots of new growth on the tree. This year, I noticed panicles growing for the first time. After some research, I cut some panicles a couple weeks ago to allow for the energy to go to new growth as opposed to fruiting.

But now I’m noticing that the branches are just dying. The branch on the far right was the first to go (it was looking wilty like the other branches are going now). The middle branch is the only one which looks relatively ok and those panicles at the top are new, but now I’m afraid to trim them.

I water regularly and it gets plenty of sun, it’s survived NSW winters these last couple years through positioning/shelter closer to our rental.

Do you reckon the whole tree will die? Have we left it in a pot for too long? Any suggestions would be greatly appreciated as we were so happy last year seeing how well it was going and now unsure what will happen to it. If it does survive we would like to plant it when we eventually get our own house (maybe January).