Allow frozen puff pastry to sit at room temperature for 20-30 minutes to soften.
Preheat oven to 400 degrees.
Slice each peach into 16 slices (no need to peel).
Sprinkle peaches with sugar, flour, and cinnamon. Add lemon juice and gently stir until everything is well coated.
Unfold puff pastry sheets. Cut each sheet into 9 squares using a pizza cutter or sharp knife. Arrange puff pastry squares on baking sheets lined with parchment paper.
Place 3-4 peach slices in the center of each puff pastry square.
Brush edges of pastry with egg wash. Sprinkle tart with coarse sugar (optional).
Bake tarts in preheated oven 18-20 minutes in the oven, turning pan halfway through for even baking. Puff pastry should be golden brown.
Allow tarts to cool 20 minutes.
Mix powdered sugar, milk and vanilla. Drizzle glaze over tarts.