The calculation method is set in the swap contract. The contract specifies the source of prices and how often they are observed, the factor that annualizes daily moves, and whether the average return is subtracted from each move.
For example, in a contract that counts 260 business days a year, the daily variance calculated from daily moves is multiplied by 260 to give annualized variance.