Model details
SEIS Model
Our COVID-19 prediction model has an underlying simulator based on an elaboration of the classic SIR model used in epidemiology: the SEIS (susceptible-exposed-infectious-susceptible) model. We added an exposed (E) period due to the reported incubation period of COVID-19 during which individuals are not yet infectious. We also modified the last step from recovered (R) to susceptible (S) to account for the possibility of re-infection, a phenomnenon that has already been reported in several regions and by WHO. With that said, we assume that a recovered individual is less likely to be infected again.

To quickly summarize how an SEIS model works, at each time period, an individual in a population is in one of three states: susceptible (S), exposed (E), and infectious (I). If an individual is in the susceptible state, we can assume they are healthy. If they are in the exposed state, they have been infected with the virus but are not infectious. If they are infectious, they can actively transmit the disease. An individual who is infected ultimately either recovers or dies. We assume that a recovered individual’s chances of re-infection is lower, but not zero. We can model the movement of individuals through these various states at each time period. The model’s exact specifications depend on its parameters, which we describe in the next section.

For our implementation, we use a discrete time series where each data point is a day in the simulation. For each day, we have a probability distribution for which an infected individual will transmit the virus, and another probability distribution for which an infected individual will succumb to the disease. These distributions are then convolved with the total existing cases to determine the number of new infections and new deaths per day. For new infections, we multiple the convolution by R0, while for deaths, we multiple the convolution by the mortality rate.

Assumptions
Please see the About page to read about the assumptions in our model.

Data
The sole data source we use is the daily reported deaths from Johns Hopkins CSSE. In addition, we use population data for each state/country to calculate the total susceptible population.

Because the raw data may be noisy, we must first run a smoothing algorithm to smooth the data. For example, if a state reports 0 death on one day and 300 deaths the next day, we smooth the data to show 150 deaths on each day.

Parameters
For our SEIS model, there are basic inputs/parameters that must be set to begin simulation. covid19-scenarios.org developed by the University of Basel provides a good visualization of sample inputs/parameters into a simulation. We chose a set of parameters that we think are important for the accuracy of the simulation. We divide the parameters into two categories:

Category 1: Fixed parameters
Fixed parameters are those that are fixed for this particular COVID-19 epidemic and likely do not fluctuate significantly across countries/states/regions. These include the following:

Latency period
Infectious period
Time between illness onset to hospitalization
Time between illness onset to death
Hospital stay time
Time to recovery
We use various reports and publications to determine the best values for these fixed parameters. Most of these sources reference studies from Wuhan, China, where the epidemic first began in December 2019. While it is possible that the exact values may fluctuate slightly from region to region, we believe that the impacts these fluctuations have are minimal compared to the variable parameters.

Category 2: Variable parameters
Variable parameters are those that may change depending on the country/state/region. Some examples are the following:

R0 - the basic reproduction number
Mortality rate
Imports of positive cases per day
Mitigation effects (i.e. post-mitigation R)
Lifting of shelter-at-home orders (i.e. post-reopening R)
Population
Hospital beds per 1000
Note that variable parameters such as population and hospital beds per 1000 are easily computable from a simple lookup. However, the other parameters are not easily retrievable. We will allocate the majority of our resources towards estimating the most sensible values for these parameters for each region.

To model the effects of shelter-at-home/lockdown orders, we assign an R value for post-lockdown and a separate R value for post-reopening. These R values are unknown ahead of time, and will be learned by the model. We assume that post-mitigation R is less than 1 to account for the decrease in cases, and that the post-reopening R is on average less than the initial R0.

Parameter Search using Machine Learning
As described in the previous section, determining the best values for variable parameters such as R0, the mortality rate, post-mitigation R, and post-reopening R will help us determine how COVID-19 will progress in a region. If we know what the “true” values of those parameters are, we can accurately simulate what is happening in the real world using our SEIS model. To determine these values, we use hyperparameter optimization.

Hyperparameter Optimization
We found that a brute-force search method that iterates through the entire parameter space is the most effective in finding an optimal set of parameters. We use grid search to iterate through the parameter space. So if we have 10 values for R0 and 10 values for the mortality rate, then there are 100 different parameter combinations for those two parameters. To optimize computation time, we prune unrealistic parameters (e.g. R0 > 20).

To measure the error of a parameter set, we use a loss function that minimizes the error between our projected daily deaths and the actual daily deaths. We find that an ensemble loss function that minimizes both absolute daily deaths and total daily deaths works well. For parameters where we do not have the data to estimate (e.g. we do not know the post-reopening R for regions that have not reopened), we consider all values equally, resulting in a wider confidence interval.

While we do not have much out-of-sample data to work with, we try our best to take advantage of the data from countries such as China, Italy, and Iran, whose progression is much further along than regions such as the US.

It is also important to limit the number of free variables. For example, it would be very difficult to try to determine 5 free variables when you only have 20 days of data. Therefore, for some of the variable parameters, we try varying one parameter at a time while holding all other parameters constant. This allows us to improve the signal-to-noise ratio when doing parameter search.

Validation Techniques
For any machine learning model, it is important to minimize overfitting. This model is no exception. In fact, due to the limited amount of data (e.g. if a region reported its first death 10 days ago, we only have 10 data points), it is very easy to overfit if we are not careful. That’s why we have developed a robust validation system that allows us to test various changes in a controlled environment such that overfitting is minimized. We can set our model to run on the first 20 days of data and compare the performance on the next 10 days. We can then repeat this process by running it on the first 21 days and comparing the performance on the next 9 days, and so forth. This allows us to perform cross-validation while preserving the maximum amount of data in our training set.

We use our validation techniques to test out various model-specific parameters and functions. For example, we can try various loss functions (e.g. mean square error, mean absolute error, ratio error) and pick the one with the lowest validation error. We also use validation to determine parameters such as the alpha for exponential decay of data points. We continuously perform validation to ensure that the parameters we select are truly meaningful and predictive, rather than simply being a product of overfitting.

Putting it All Together
For each region, we run our optimizer on the data to score each set of parameters. After some additional validation techniques described above, we aggregate the forecasts generated by each set of parameter, giving higher weights to parameters that have a lower error. This approach was inspired by the Monte Carlo method, allowing us to simulate the future while incorporating the inherent uncertainty. We now have our projections.