Preheat oven to 350° and place oven rack in middle position. Prepare two 6X2 inch baking pans by greasing, flouring, and lining with parchment paper.
Mash ripe bananas, this may take anywhere from two to four bananas depending on the size. Measure out one cup of mashed bananas and set aside.
In the bowl of a stand mixer using the paddle attachment or a hand mixer add the softened butter, brown sugar, and granulated sugar. Mix on medium speed for two to three minutes. Add in the eggs and vanilla extract. Mix for another one to two minutes, scraping the bowl down occasionally. Add the mashed bananas and mix until combined.
In a separate bowl add the all purpose flour, baking powder, baking soda, cinnamon, ginger, nutmeg, and salt. Whisk to combine. Add half of the dry ingredients to the wet ingredients and mix on low speed. While the mixer is on pour in half of the buttermilk. Add the remaining dry ingredients and the remaining buttermilk and mix on the lowest speed until just combined. Be careful not to over mix.
Divide batter evenly between two cake pans and bake in the preheated oven on the middle rack for 22-28 minutes. A toothpick inserted into the center should come out clean. Allow the cakes to cool in the cake pans for about five minutes. After that use an oven mitt to help remove cakes and place on a wire cooling rack to cool completely. Once completely cool wrap with plastic wrap and place in the refrigerator for at least an hour to make slicing the layers easier.
Brown Sugar Cream Cheese Frosting
In the bowl of a stand mixer with the paddle attachment or using a hand mixer, add the room temperature butter and cream cheese. Mix on medium speed for about a minute to combine. Add the powdered sugar, brown sugar, cinnamon, and salt and mix on medium speed for two to three minutes. Scrape the bowl down occasionally. Add one to two tablespoons of heavy whipping cream to reach desired thickness and mix on medium high speed for another minute.
Once cakes are cooled and you are ready to assemble cut each cake layer in half horizontally. Stack and frost each layer and frost cake as desired.
Notes
This cake is best served within 1-2 days after baking for optimal texture. Leftovers can be stored in an airtight container at room temperature or refrigerated for 2-3 days. Cake slices can also be frozen and enjoyed later.