Wash and peel sweet potatoes. Cut into 1/2-inch thick slices.
In a large pot, combine brown sugar, white sugar, butter, cinnamon, and nutmeg.
Add sliced sweet potatoes to the pot. Heat on medium, stirring occasionally, until butter and sugars melt.
Reduce heat to low and simmer gently, stirring occasionally to prevent sticking.
When potatoes are tender and sauce has thickened, add vanilla extract.
Remove from heat and let cool slightly before serving.