Kaffir Lime Fruit

As the fruit of the Kaffir lime is generally not eaten, is there any reason for leaving them on the tree consuming growth? Led to believe that each fruit, on any plant, requires approximately 35 leaves for the fruit to grow/develop.