Can we tell if the vaccines are working in Minnesota?

After a year of fear, tragedy, waiting (and more waiting), and a furious all-out vaccination campaign, we're all wondering if we'll ever be past our COVID-centric existence.

Many people are looking for a sign that vaccines are making an impact and wondering: How will we know if they're working? Can we see any evidence in Minnesota's coronavirus data?

Short answer: Yes! Longer answer: It's exceptionally complicated.

The basic problem is there's no way to know who would have contracted COVID-19, but didn't, because they have been vaccinated. So if we can't look at individuals, the other option is to look for any declines in COVID spread or case rates that are likely explained by expanding vaccinations.

It's easier said than done, because even if scientists fully understood precisely how easily and in what circumstances COVID spreads — which they don't — there are so many variables that affect how likely someone is to be exposed to the virus, and how sick (or not) they might get.

Another issue is the data we have is far from perfect. None of it existed just over a year ago, and the systems for collecting and distributing it are still developing and inconsistent. And there's a lot of data that governments and health care providers have that they haven't released.

But even beyond those issues, epidemiologists are cautious about overinterpreting what data we do have, with good reason. That's especially true with COVID, which we don't fully understand, not to mention the heightened level of distrust many people feel toward experts prescribing preventive measures like masks and brand new vaccines.

Let's start with the data points most experts do agree on.

The initial focus in most states' vaccine rollout was to first rapidly inoculate health care workers and people living in long-term care facilities. Publicly available data on cases among health care workers in Minnesota is limited, but there is data on cases in long-term care facilities.

If enough people in long-term care have been vaccinated, we'd expect to see case rates falling. And that's what the data show. Since early March, Minnesota has reported fewer than 100 cases among long-term care residents per week, the lowest numbers seen since late July 2020. And long-term care facilities didn't see a spike in cases in April, while most areas of the state were fighting a new wave of infections. (The blip on the week of March 7 is due to a big dump of older, previously unconfirmed cases due to problems with four labs.)

Minnesota's second target for early vaccination was people over the age of 65 and essential workers. Again, we don't have good data for cases among essential workers, but we do have data on cases by age group.

This is a little tricky. While seniors and people with underlying health conditions have been at the highest risk of dying from COVID, they've never made up a huge percentage of the state's overall cases. Younger people have always made up the bulk of cases — they're just less likely, in general, to be hospitalized or die.

So the best way to look for changes in age groups is to focus on cases in that group as a percentage of total cases.

And there's a clear trend. Since January, cases among seniors age 70 and older have made up a smaller percentage of the state's weekly cases than their share of the population.

The trend has been especially durable among Minnesotans over 80, who haven't made up more than 1% of the state's weekly cases since February.

So far, the vaccines seem to be living up to the hype of keeping people from getting seriously ill, according to data collected by HealthPartners.

Dr. Mark Sannes, a leader of the health care network's COVID-19 response, said that as of late April, HealthPartners had just over 1,200 hospitalizations for COVID since the start of the year at the network's eight hospitals, and the group documents every patient's vaccination status.

"Ten people were fully vaccinated out of those 1,200," Sannes said. "We simply have not had very many people hospitalized that have had their full vaccination."

Hospitalizations among older people mirror the decline in case rates among those groups, Sannes said.

"We have seen the number of folks in that older population in the hospital decline significantly really since the onset of vaccination," he said. "Not only in the age of the individuals that are coming in, where we're seeing 55-year-olds now instead of 65-year-olds on average, but deaths have dropped off so much during this four-month period."

With any vaccine, some percentage of people who receive it will still contract the disease. That can happen because a person was exposed to the disease before the vaccine had time to fully take effect, but also in rarer "breakthrough cases," the virus finds a way to gain traction despite the vaccine. While many people may find this scary, it's something experts expect.

The COVID vaccines available in the United States actually have a remarkably high estimated effectiveness compared with other more familiar vaccines — flu vaccines are sometimes as low as 50 to 60% effective, for example. The Moderna and Pfizer vaccines are thought to be about 90 percent effective in the real world, according to the CDC. There's less real-world data on the Johnson & Johnson vaccine, but in clinical trials was found to be 66% effective.

OK, now it gets a little controversial.

All the experts we interviewed agree that the declines in cases in long-term care and among older people in general are likely signs that the vaccines are working. But when you start talking about data at the county level, there's no clear consensus.

Let's start with what we do know.

Since the beginning of vaccinations, some Minnesota counties have been significantly faster or slower than others. Cook County has long been the vaccination champion, with nearly 70% of residents with at least one dose. Meanwhile, Benton County has not yet reached 30%; most other counties are between 38 and 45%.

That means no county, except maybe the overachievers in Cook County, is close to the threshold of herd immunity. That's why most counties, except for some of the state's least populous, haven't seen their case rates plunge to near zero — there are still plenty of people around to get infected.

Still, even short of herd immunity, there does appear to be a difference in case rates between counties that are higher and lower on the vaccination scale.

This was especially apparent during Minnesota's April surge. While case rates went up nearly everywhere, they went up more in counties with lower vaccination rates. This became most apparent in mid-April, right about the time vaccines started becoming more widely available.

And this is where epidemiologists get really itchy.

We can say that counties with higher vaccination rates, on average, have had lower case rates since mid-April, and vice versa. We can say that relationship is "statistically significant" by running a linear regression analysis on the data, using the percent of the population that had completed vaccination 14 days prior, in order to give immunity time to kick in.

"The big picture is that on average more vaccination equals lower rates, that's really clear," said Ryan Demmer, an epidemiologist at the University of Minnesota. "It's basically looking the way you would expect, with at least some kind of noise that you would always expect as well."

That "noise" that Demmer is referring to is that there are a lot of reasons that case rates might vary between counties that have nothing to do with vaccination, such as localized short-term outbreaks, different variants affecting some counties more than others, differences in age and racial demographics and higher shares of workers in vulnerable industries.

And then there are the potential problems that can arise when comparing two variables that might be connected to each other for reasons other than the things you want to measure.

For example, people who were eager to get vaccinated were likely also trying their hardest to observe preventive measures like masking and social distancing. And people working in lower-income jobs that exposed them to more COVID risk may also have had more difficulty in accessing vaccines if it meant taking time off work and driving a long way.

Epidemiologists at the Minnesota Health Department said they don't think looking at the data by county is appropriate, and that there are too many complicating factors to make any judgments using a regression analysis.

"What you found might be true, but we don't feel it's a conclusion you can draw from the methods that you're using," said Kristin Sweet, an epidemiologist and COVID-19 data lead for the health department. "Given the challenges we've had in predicting how disease moves through populations, I would say that right now we don't feel confident in saying there is a relationship over different geographic areas."

Demmer said he agrees with the health department that a regression analysis isn't an ideal tool to look at COVID, but that there aren't well agreed upon alternative methods, either.

"Overall, your goal here is less scientific and more descriptive and informative, which is why I don't see a major problem with showing some descriptive patterns," Demmer said.

Maria Sundaram, a University of Minnesota-trained epidemiologist, said that there are almost always complicating factors in epidemiological analysis, so it's important to be transparent about what assumptions are being made and what can't be accounted for.

"And if our assumptions are wrong, here's how it would change our results," Sundaram said. "Every epidemiological study has limitations."

And there are plenty of caveats and limitations.

First, for example, the relationship between cases and vaccination is stronger in more populous counties than in smaller ones.

That could mean that the urban counties have a larger number of data points and therefore we can trust their results more than numbers from counties with fewer data points to rely on.

Or it could mean that people in urban and rural counties act completely differently. For example they might be better at social distancing and mask-wearing, or limits on gatherings are more strictly enforced there.

Or it could mean that a particular outbreak or a particular variant has led to more infections in a cluster of rural counties, which is skewing the data.

In any of those cases, the trend wouldn't have anything to do with vaccination.

"We have to use all our critical capacities because the world is so messy," said Jessie Shmool, another health department epidemiologist who doesn't think the regression is the right way to analyze regional variations. "Because if we see what we're expecting, what if it's not what's happening? Just because we expect it to be true doesn't mean it's true."

Even counties doing relatively well at vaccination have seen short-term outbreaks that shoot their case rate through the roof. Those have generally been smaller counties, but Itasca County, which has 45,000 residents, has had several weeks of elevated rates despite a better-than-average completed vaccination rate of nearly 40%.

Demmer said it's not surprising that some counties with middle-of-the-pack vaccination rates are still occasionally seeing higher case rates because half the population still isn't vaccinated. He also noted that the data has implications for counties lagging behind, but also for the rest of the state.

"If you have low vaccination rates in your county, your county is likely going to continue to experience higher infection rates and the outcomes associated with it," Demmer said. "And the other issue is that to some degree we're all in this together. So the infection isn't really going to be confined to counties."