Case Study – Selection Criteria

With the introduction of the NCAA Power Index (NPI) this season, the NCAA changed the selection game in an apocalyptic manner. That’s not to say it was for the worse but it certainly was destructive. Prior to NPI going live, we were able to see simulations run with the past selections versus what NPI would have done. Unfortunately, we do not have the ability to see what the old selection criteria would have done this year in the hands of the Selection Committee. There is just nothing that can be done about that. Well, until I do something about it.

What we will do here is use Ned’s comparison program from past years uploaded with match data so it gives us the old selection criteria. I will then play the part of the Selection Criteria and see which 21 at-large teams they would have selected. The inherent problem with this is my personal bias and philosophy when it comes to the old selection criteria. I will do my best to mimic what the committee would have done. To remind everyone, here is the old selection criteria:

● (Record) Won-lost percentage against Division III opponents.

● (H2H) Division III head-to-head competition.

● (Common Opponent or CO) Results versus common Division III opponents.

● (Ranked Results or RR) Results versus ranked Division III teams as established by the final ranking and the ranking preceding the final ranking.

• Conference postseason contests are included.

• Contests versus provisional and reclassifying members in their third and fourth years shall count in the primary criteria. Provisional and reclassifying members shall remain ineligible for rankings and selections.

● (Strength of Schedule or SOS) Division III strength of schedule*.

• Opponents’ Winning Percentage (OWP).

• Opponents’ Opponents’ Winning Percentage (OOWP).

* See my FAQ Page and take a look at the 2023 Pre-Championship manual if you want a further understanding of the SOS calculation.

At the bottom of this article, I have listed the at-large teams (or Pool C teams). I also have where they differed with Ned’s QWI tool and where they differed with my process. I wrote about QWI and its at-large selections on Sunday.

What else do I need to remind everyone about before we get started? Oh yeah, the process. There is a whole long article written on this in the FAQ that is now obsolete but in a nutshell:

• Each region goes off their regional rankings and “nominates” the top team without a bid to the committee.

• The committee then compares the 10 teams put forth using the selection criteria.

• One team is selected and the next team from that region replaces the selected team.

• Rinse and repeat until we get 21 teams for our at-large bids.

The regional rankings can be found here. Unfortunately, they were updated only to November 12^th so they do not include the conference tournaments. I will not address the sins of the committee here and simply use those rankings for my ordering. (I do have my own listing as of the 17^th and will refer to that at times.) Remember that the regional rankings were simply NPI ordering by region.

Now it’s time to dive into the process and the first thing I’m going to do is take a shortcut. Without trying to delve specifically in the criteria, I want to select teams that are very obvious. When things get tight, you’ll start seeing me write about the criteria more.

For Round 1, I will list the teams, per the regional rankings that needed a bid that would have been up for consideration.

Round 1

These are your top teams for each region reading down from I to X. MIT was probably the selection here. Springfield now replaces MIT for consideration. (See how that works?)

Round 2

UW-Oshkosh is another easy call and gets the bid here. WashU comes in to replace UWO.

Round 3

WashU comes in and lasts just one round before they are given an at-large bid. In comes Millikin.

Round 4

Marietta out of Region VII will break-up the Region VIII love for a bit as they get the next at-large bid. In comes Case Western Reserve (CWRU).

Round 5

Back to Region VIII, and I’m sure it would have bothered some of the committee taking so many teams from one region, but the pick is Millikin. In comes UW-Platteville.

Round 6

At this point, you are probably thinking this selection committee job is pretty easy. So far, you would be right. Another pretty straight-forward selection is Colby so let’s give them their bid. In comes Wesleyan.

Round 7

With a great Record and SOS, I think Gettysburg goes here pretty easily. In comes Salisbury.

Round 8

We’ve got some tough choices here but it’s between teams that will all get bids so I’m still doing my short cut going with UW-Platteville in this spot. In comes UW-Eau Claire. That’s now four Region VIII teams out of the 8. (It should be pointed out that Chicago may have been ranked in the final regional rankings higher than UWEC but I didn’t feel like going to double-check. As I said, we are paying for the NCAA sins since they didn’t update the rankings.) (Narrator’s Voice: Chicago was ahead of UWEC.)

Round 9

CWRU and Salisbury are pretty close in their comparison but CWRU wins it by a smidge (a technical term the NCAA uses that can be found on page 329 of their Rules & Procedures manual or Appendix G, subsection 2r, of that same document). In comes Carnegie Mellon.

Round 10

I will go ahead and take Salisbury here. In comes Franklin & Marshall.

Round 11

As a reminder, here are the teams now under consideration almost halfway through the process:

You’ll notice that William Smith, NYU, Virginia Wesleyan, Gustavus Adolphus and CMS have been under consideration since round 1. You did not get bonus points the longer you wait. You did not have a right to a spot because you had to wait longer. In fact, this spot goes to the newcomer in Franklin & Marshall. I spot checked a few comparisons here and they were hard to beat. In comes Eastern.

Round 12

Looking at the NPI numbers used for selection, Virginia Wesleyan and CMS should be the next two teams in. I’ll compare the two and then spot check the winner against the other teams.

CMS probably gets the slight edge although they have almost identical criteria (CMS has SOS while VWU has Record). It’s the SOS advantage that gives CMS the bump. Eau Claire compares almost equally with CMS and may have a slight advantage depending on how you look at the Ranked Results. Double-checking UWEC against VWU and I see the same slight advantage.

Here’s where we get an alternate universe. The Selection Committee is made up of humans and, of course, Paul Dill. Will they feel comfortable putting in their fifth team from this region with a lot of rounds left to go? Will the other representatives balk at this? Will there be a public school bias in play as the larger public universities are often frowned on being in DIII? I don’t think the selection committee would take Eau Claire here but I’m going to go by the criteria and my best effort at this point. Eau Claire is the pick, bringing in Chicago.

Round 13

CMS has the edge of the other teams, including Chicago, and Region X gets their first team into the tournament. That brings in, everyone’s favorite team, the Southwestern Pirates.

Round 14

We have 8 spots left and so far, things have been pretty easy. Let’s take a second and talk about some of the teams we aren’t talking about. The teams under consideration below the NPI cut-line are Southwestern, Eastern, Gustavus Adolphus and Wesleyan. We know that none of these teams received a real bid. From a selection criteria perspective, these teams have their warts. Southwestern has the lowest SOS of the entire group right now. That means they lose that comparison with every team. (Coupled with poor Ranked Results, they are all but eliminated.) Wesleyan have the same problem with Record. These are two big hurdles to overcome and if it happens, it will happen late in this process. Gustavus Adolphus has a below average (to this group) Record and a mid-SOS. The same is true with Eastern but they are slightly better off than Adolphus. That’s basically 4 of our 10 teams out-of-the running for the next 4 rounds or so.

Of the remaining six teams, I still like Virginia Wesleyan here as a slight favorite over Carnegie Mellon. In comes Centre as the replacement as Region VI gets on the board.

Round 15

From an NPI perspective, every team that was 61 or higher is now in the tournament. It truly is good to see that NPI doesn’t have a team needing an at-large above 61 not be deserving of one based on the old system. Of course, there is still the case of Whitworth (62.48) who won the NWC with an SOS of 0.529. Just for fun, I compared them against William Smith, a team that was one of the last in using NPI and still haven’t been awarded a spot here. William Smith has Common Opponent, SOS and Ranked Results advantages. Whitworth only has Record. Comparing them to Eau Claire (a team in under both selection systems) shows it closer with Whitworth winning Record and Common Opponent. UWEC would still get the edge due to a tougher schedule and a much better Ranked Result. Centre was our last team in using NPI and they would have an easy comparison win over Whitworth using the criteria. Whitworth was truly the Mary Hardin-Baylor and Plymouth State examples in past year’s simulations (i.e., they were undeserving) except for the fact that they won their conference. (Not trying to be mean here but just showing you how bad NPI can get things compared to the old criteria when their 17^th best team overall wouldn’t have received an at-large bid under the old system.)

Back to the matter at hand. We now really get into another alternate universe situation as how the committee views Ranked Results comes into play and no where is it more troublesome than Centre. Centre was 2-2 in Ranked Record. Is that better than 8-9 like Carnegie Mellon had? Is it the same as 9-9 as Chicago had? If I do a direct comparison between Centre and Chicago, Centre has Common Opponent (1-1 versus 1-2) and Record. Chicago has a large advantage in SOS but would need Ranked Results just to tie Centre in the criteria. I’m inclined to believe (hope) that the committee would choose Chicago’s 9-9 record over 2-2 for Centre factoring in 24 matches played against the RPI Top 100 (14-10) versus Centre’s 6-5 record (11 matches total). Ranked Results should be more than record but this is where universes parted ways using the old system. Some just took the record and didn’t look further. In this comparison, even if you gave Chicago Ranked Results, it was a tie. In these cases, the committee would probably have gone to secondary criteria and I think Centre would have taken that. (I purposely didn’t go into secondary criteria. You are welcome.) I believe in my heart that Chicago was better but the criteria probably goes to Centre. It’s a shame they didn’t play late in the season as that…checks notes…would have told us that Chicago beat Centre in the first round of the NCAA Tournament. Of course, hindsight (and future results) were not part of the criteria. (Props to NPI for getting that ordering correct while I think the old criteria would have gotten it wrong.)

Lots to unwarp in Round 15, I guess. The main problem with these teams is that Carnegie Mellon actually compares well to everyone except for Chicago who probably loses out to Centre by the thinnest of margins. If I’m the committee I go Chicago here but that’s another Region VIII team, of course, I’m currently writing this in the universe that doesn’t care about how many teams come from the same region. Chicago is in, replacing them is UW-Stevens Point although they were not regionally ranked. That’s right, we have run out of public Region VIII teams from the rankings. (Not really an issue with NPI because we see all of the scores but it was a headache under the old system, which wasn’t as transparent.)

Round 16

Since I liked Carnegie Mellon last round, I check them against UW-Stevens Point and they are in a virtual tie. I probably lean CMU here and the secondary criteria probably also agrees if there is a tie in the criteria. CMU is the pick with Calvin coming in as the replacement.

One other issue with Stevens Point is they tie Southwestern in the criteria with Southwestern having Record and H2H. UWSP has SOS and Ranked Results. Secondary criteria could go either way. (Southwestern has a better record of the last 25% of matches played while UWSP has a better nonconference SOS.)

Round 17

As I get into the bare bones of this exercise, I realize there is one fatal flaw to my work. I’m using regional rankings based on NPI but comparing teams based on the selection criteria. My work will naturally give an advantage to the NPI teams because they are the only ones being compared due to the old process. For instance, let’s redo the Region X Regional Rankings using just the selection criteria.

I believe Trinity gets the top spot over ETBU (NPI’s choice) due to the H2H win and other “wins” in the criteria. ETBU gets the second spot. It’s actually possible that ETBU would have been placed behind La Verne and CMS (despite their H2H win over CMS). It would have come down to how the Region X Regional Advisory Committee (RAC) favored Ranked Results. To avoid any embarrassment to NPI (such a sensitive entity), I’m going to keep ETBU second but will flip CMS and La Verne. Whitworth drops from 3^rd to 5^th. As for the last two spots, which went to Southwestern and then Whittier that might be right. Or Texas Lutheran should have been there instead of Southwestern or maybe UC Santa Cruz is there after Whittier. It’s a bit messy and would have required a lot of RAC discussion. (This is the main reason NPI will stay because it eliminates all of that annoyingly fun discussion.) Anyway, that’s just one region and those changes don’t change a lot but would (maybe) change who is under consideration right now. We are down to the last 5 teams and this is where the selection system matters. Who is considered next is actually a big deal.

As a reminder, here is who is being considered:

Let me to some anti-NPI spot checking for each region as this article now becomes a universe unto itself.

Region I – I looked at Wesleyan, Plymouth State, Tufts and the real Trinity. Tufts actually would be ranked higher using the criteria so they replace Wesleyan.

Region II – I looked at Springfield, Babson and Brandeis. Springfield has the edge over Babson but Brandeis is 2-2 in criteria with Ranked Results remaining. Brandeis is 5-13 playing a much tougher schedule but both Babson and Springfield have a better Ranked Record. I think the NPI ordering would probably hold for the selection criteria. (I wonder if there is an anti-Brandeis bias from the New England representatives since they play in the UAA? I asked Paul Dill but he told me he couldn’t say because it’s against policy.)

Region III – This region is William Smith and no one else comes close.

Region IV – This region is NYU and no one else.

Region V – NPI has Eastern in front of Swarthmore and it looks like the selection criteria agrees.

Region VI – This region holds to the NPI with Centre and then Bridgewater.

Region VII – Calvin, Ohio Northern and then Alleghany is the NPI ordering and the selection criteria agrees. Calvin gets the Ranked Results decision to push them over ONU.

Region VIII – We don’t have any more public teams in the regional rankings for VIII. The possibles are UWSP, North Central and Edgewood. North Central has the advantage over Edgewood but Edgewood ties UWSP. UWSP has the edge over North Central. The RAC could have gone in any direction here but I think the final regular season NPI standings of UWSP, North Central and then Edgewood would have also been the ordering with the old selection criteria.

Region IX – This region would be between Gust Adolphus, Coe, St Kate, which was the NPI ordering. The selection criteria would most likely have the same order although GAC and Coe tie, but I’d give the secondary criteria edge to GAC.

Region X – I talked about this one above. The big question is who is under consideration and I’m staying with Southwestern because they are my Pirates and this is my article. (There are no wrong answers here and no right answers. I like Southwestern because of the chaos they insert with UW-Stevens Point.)

New teams under consideration (with some backups):

Can you tell that as we get to the end, I’m writing more? This is what happens in the RACs and the Selection Committee. There was more and more discussions and splitting hairs as you got to the end.

NYU is my pick in this round. They either tie or win outright against every other team under consideration. I liked their secondary criteria in the ties as well. In comes Stockton.

Round 18

I have four spots left and from my comparisons in the previous round, I know that William Smith takes out both Tufts and Springfield. They basically have the edge over everyone else left except for UWSP who they tie. This tie could go either way so I’m taking William Smith here. In comes no one of consequence from that region.

Round 19

I have three spots left that went to Springfield, Centre and UW-Stevens Point per NPI. Let me order these based on the criteria and then see how they do against the other teams.

The immediate problem is that each of these teams tie in the criteria leaving Ranked Results as the tiebreaker. This was very common with the old system as head-to-head and common opponent were rare tiebreakers. So, you tell me; who do you like:

Team A: 2-2 Ranked Record / 0.635 RPI / 6-5 versus RPI Top 100 / Best Win CMS / Worst Loss AsburyTeam B: 6-7 Ranked Record / 0.657 RPI / 11-7 versus RPI Top 100 / BW Smith / WL Coast Guard

Team C: 6-10 Ranked Record / 0.648 / 9-11 versus RPI Top 100 / BW UW-Whitewater / WL Edgewood

If you go simply by Ranked Record then that is the order; A, B and then C. If you go by their window and floor then I probably go with Team C. They have a higher ceiling and a higher floor then the other teams and their Ranked Results are hurt by their tough schedule. I would probably go C, B and then A but I’m not going to argue too hard with B, C and then A. Again, this is my article so Team C (UW-Stevens Point) gets the nod over Springfield (Team B) and Centre (Team A).

Looking at the other teams, I start with Calvin. They lose to all three of the above teams so they need some anarchy to have a chance for an at-large.

Moving to Southwestern, they tie UWSP and may have a secondary criteria edge but humans looking at these two teams would say that Southwestern was better in September but UWSP was better the rest of the time. Since this isn’t September, I think the committee would overlook the H2H Southwestern win and give the edge to UWSP. Southwestern also loses the battle with both Centre and Springfield. If they could have won one of those two battles then I think Southwestern gets a bid under the old system.

I have Eastern losing to all three in the criteria battle so things are looking good for NPI.

Gustavus Adolphus is a different story. They probably have an advantage over Centre but lose out to Springfield and Stevens Point. The battle for the last spot is now going to be very interesting.

I did check Stockton but I don’t like their comparisons. That leaves four teams for three spots but we also have to factor in who replaces an at-large team. This bid goes to UW-Stevens Point. In comes North Central.

Round 20

Springfield is the choice but we have to compare them to North Central first. It’s an easy win for Springfield as they have a better Record and SOS. No coming back from that when there is not head-to-head and no advantage in common opponent. In comes Babson.

Round 21

Best case, I’m going to match NPI exactly. Worst case, I’m going to be off by one. I really have four teams in the mix; Centre, Gustavus Adolphus, North Central and Babson. The other six teams could be in the mix if these four prove difficult. I really hope that is not the case.

North Central is out. They lose the comparison to each of the other three (although they tied Gustavus Adolphus).

I think Centre is also out so I’m going to be different. They only played four matches against regional ranked teams and, although they finished 2-2, I think this won’t be enough under the old system.

I was leaning towards Gustavus Adolphus simply because it’s close with Babson and we didn’t have a team from Region IX (which would have been an argument in the committee). The problem with them is that they don’t do well against Calvin or Eastern.

Calvin ties Babson and probably wins on secondary criteria. Same story with Eastern and Babson. Ugh.

So, I loop back to Centre as they don’t have a Calvin and Eastern issue (they win both comparisons). Their 2-2 Ranked Record can really play two ways. It hurts when you are going against teams with double-digit matches that did okay but maybe helps when going against teams that played more ranked opponents but really didn’t do well.

My pick is Centre and I match exactly what NPI had this year. If you took Calvin or Eastern then you have some comparison issues. If you took Babson then you also have some comparison issues. I took Centre who has fewer comparison issues. All would have been in the running with the old system.

For the loyal readers out there, you know that I’m very critical of NPI and I don’t think it’s a good system. I will admit, however, that it was good in 2024 for the at-large team selection. A blind squirrel found the nut. Congratulations? Remember that there is still the issue of Whitworth and Gallaudet. Two teams who had too high of an NPI but are forgotten because they won their conferences. Still, I think this exercise was necessary to see how the old criteria would have done. Would the selection committee make my same choices? I don’t think so. I don’t think they would have chosen seven Region VIII teams despite them being deserving. Maybe you see a Babson in over one of those teams and we all are scratching our heads. Maybe another ‘S’ team like Swarthmore from Region V makes it in because they always did from that region.

Below is the summary for the three methods used.