Help with Kojo-no-mai

Looking for help with this blossom tree. Bought about 4 years ago. The first year it was full of blossoms, and it in leafybstage it was packed. As the years have gone on the blossoms have gotten less and there are lots of bare branches even though its got leaves now.

Is there anything I can do to get it back to its former glory? Prune? Fertilise? Its a south-west facing garden and it gets midday to late afternoon full sun.