Using a fine-mesh strainer, sift the cocoa powder, sugar, salt, and cornstarch together to remove any clumps. Place the dry ingredients in a medium saucepan.

Turn the heat on medium. Using a whisk, slowly pour in the milk and continue to stir to work out any additional clumps. As you stir, the mixture will thicken, this should take between 5-10 minutes.

Once the mixture has reached the consistency of smooth pudding, remove from heat and allow to cool at room temperature. Once cooled, add the vanilla extract and stir. Pour into the popsicle molds and place them in your freezer overnight.

Run the molds under warm water to loosen the popsicles and serve.