Preheat oven to 180°C / 160°C fan / Gas 4. Grease and line a deep 23cm springform tin.
Halve and stone the plums. Mix demerara sugar and cinnamon in a small bowl. Set both aside.
Beat butter and caster sugar for 4–5 minutes until very pale and fluffy. Add vanilla and almond extract.
Add eggs one at a time, beating well after each. Add a spoonful of flour if the mixture curdles.
Sift in flour, baking powder, cinnamon, mixed spice, and nutmeg. Add ground almonds and milk. Fold gently until just combined.
Spoon batter into tin and spread evenly. Arrange plum halves cut-side up on top, pressing each gently into the batter. Scatter cinnamon demerara sugar and flaked almonds over the top.
Bake for 50–60 minutes until deep golden and a skewer inserted into the sponge comes out clean. Cover top with parchment if almonds brown too quickly.
Cool in tin for 20 minutes. Release springform sides. Brush warm plums with apricot jam if using. Dust with icing sugar once cool.