How We Made It: The halfwheel Top 25

In a few hours, our top 25 list will be published.

They say you never want to see sausage made, but if you’re curious and want to know how our top 25 list is constructed—here’s 1,500 words and a lot of numbers explaining it.

Spoiler alert: there’s no spoilers for the top 25 itself.

---

No matter who you are, how many cigars you reviewed or how long your list is, every publication’s list inspires a fair amount of criticism—oftentimes more criticism than praise.

I think much of that is because people don’t understand how lists are constructed.

In truth, everyone is probably different. But, I do think explaining the process can help readers understand the results.

---

No process will be perfect, but that doesn’t mean we will stop trying to improve our own list. The efforts for this upcoming list began shortly after the 2014 IPCPR Convention & Trade Show, i.e. July 2014. We identified a few issues and wanted to see if we could solve them.

Our previous process meant that any cigar that debuted in the year and scored in a certain range would be eligible. That range can technically fluctuate, but we use a relatively simple formula: whatever the 25th highest ranked cigar scores—it’s been 91 since 2013—is the basis. So any new cigar that scored 91 or above would be eligible. Our reviewers would then smoke each cigar and come up with a simple list: one to whatever.

THE SAMPLE SIZE

And therein lied our first problem: rarely were there 30 new cigars that scored 91 or above. Each year there’d be a handful of cigars that made the bottom of the list that multiple reviewers thought didn’t belong on a top 25 list. But because the sample size was so small, there was nothing we could do about, unless we wanted a top 17 list.

Patrick Lagreid raised an interesting question at one of those meetings: how important was the top 25 list in comparison to the daily reviews of cigars?

The answer is the day-to-day reviews take precedent, but they weren’t without their own issues. So, we revamped how we scored cigars in an effort to extend the typical range of scores. The goal wasn’t to make the average score necessarily higher or lower, but rather, make the individual scores themselves both higher and lower.

The results seem to have worked, the average score for all cigars published in 2015 was 86.29 compared to 86.88 in 2013 and 86.91 last year. Lower, but only slightly.

Where we really wanted to see the difference is in what we consider to be good cigars, 87-90 point ratings. In 2013, 42.7 percent of all cigars scored between 87 and 90, in 2014 that number jumped to 48.59 percent. This year, only 37.18 cigars scored within that range and yet, the average score was only affected slightly.

While it produced lower scores, particularly a great increase in the amount of cigars scoring in the low 80s, it also meant that we had far more cigars scoring 91 or above. This year there were 38 scores that would have been eligible for our top 25 list, up from 28 last year and 30 the year before. Three more new cigars would have been eligible this year, but they failed to ship in 2015 meaning they are bumped to the 2016 list.

DREADED DECEMBER

While we normally have a fairly easy to follow schedule for reviews, December is the exact opposite for a number of reasons.

October and November are busy months for cigars shipping because so many of the cigars that debuted at the mid-summer IPCPR Convention & Trade Show get delayed because of production issues. Furthermore, there’s another mini-rush starting around Thanksgiving as many factories begin closing in early December and remain closed until after the new year. January is traditionally not a busy buying season in cigars, whereas December is the holiday shopping season, so many companies push the factories to have their new releases arrive in the holiday season versus the slower January and February.

All this means scheduling becomes problematic.

Our list is comprised of new cigars and while those that debut in the latter half of December are eligible for the following year’s list, if we don’t review a cigar that ships on Dec. 1 by Dec. 31, it won’t ever be eligible for a top 25 list.

So for much of December reviewers are smoking a sample or two trying to determine whether the cigar has a chance at being top 25 eligible. If it does, the review is finished. If not, the review gets shelved in December so that other cigars have a chance to make the list.

Combine this with holiday and travel schedules, add in some extra top 25 cigars to smoke and December becomes my least favorite month of work.

THE VIAJE FRIENDS & FAMILY PROBLEM

We decided this year to make a committed effort for the other reviewers to smoke the cigars closer to when they came out to help alleviate the stress of December, but also for another reason.

In February 2012, the original Viaje Friends and Family scored 92 points, enough to make it for top 25 consideration. But then in October of that year, Brooks Whittington reduxed the cigar and scored it 78 points. The cigar was still eligible for top 25 consideration because only the original score counts for that purpose, but it raised an interesting problem: cigars that were reviewed in the early part of the year had a chance to get better or worse. It’s one thing when the original reviewer recognizes the change—they still have the original review as reference point—but for the rest of us who didn’t smoke the cigar until late in the year, we saw a much different cigar than that what Brooks gave 92 points to and he was the first one to admit it.

Cigars that are reviewed in December simply don’t have that chance and because the redux scores would never make a cigar eligible, it’s an inherent advantage they have over those released earlier in the year. Presumably if  a cigar scores 91 or above in December, it’s a good cigar and smoking well at the time. Furthermore, there’s only a handful of weeks for the cigar to naturally get worse before our other reviewers smoke it. Cigars reviewed in February? There could be 10 months for a cigar to go through a sick period and we have no way of knowing if or when that will happen.

So, we made a committed effort to reviewing the cigars within a month of the original review in order to try to avoid these inconsistencies. It didn’t happen in every case for varying reasons including those out of our hands, but each reviewer began evaluating 91 point rated cigars in January of this year, a major change versus years past.

SCORING RATHER THAN RANKING

But you can’t remember everything, not for 12 months.

The idea behind ranking the cigars was that it forced reviewers to avoid ties. One cigar has to be better or worse than the other. That’s great when you are smoking all the cigars in a short window, but not for 12 months. So, we made the decision that each reviewer would rate the cigars individually instead of simply ranking them.

This also helped to better separate the cigars and will allow for some detailed analysis to be made about where there are big gaps in between one ranking and the next.

---

So, for 2015 the process is as follows.

Any cigar that scores 91 or above and debuted in 2015 was eligible. Two cigars are delayed: the Montecristo 80 Aniversairo and Romeo y Julieta Gran Reserva Cosecha 2009, both of those will be eligible for 2016’s list. A third, the Dissident HOME, seems unlikely to ship at all, but will be eligible if it does.

The three reviewers who did not publish the original review filled out a score sheet based off of one sample. Most of the cigars were smoked within a month or so of the original review. In some cases, due to either a delay in the cigar shipping or our issues with acquiring the cigar, this did not take place. Scores were tabulated and the 25 highest scoring cigars were ranked and will be published in order later today.

Even this year, the scores from the four of us as to which was the best cigar aren’t unanimous. That’s what happens when you solicit opinions from multiple people. However, this year is far and away the most sound list we’ve published and that’s solely to the changes we’ve made in how we create the list.