Stir together the water, tea, spices, vanilla, and sugar in a medium saucepan. Heat over medium-high and cover until just beginning to simmer. Turn off heat and allow to brew with the lid on for 3-5 minutes.
Strain the liquid, discarding solids. Let cool to room temperature, then refrigerate.
Mix the milks together in a small container until combined. Pour over tea when serving.