Why do these baby limes keep falling of my lime tree instead of growing?

We didn’t get any limes last year, so we’ve been watering and fertilizing it this year in the hopes to get some limes.

There’s plenty of the baby limes, but we keep finding more and more falling to the ground. What are we doing wrong?