"Let's say on the one hand you guess a 50% getting it if the headbands don't matter. Then this situation has a chance of 0.5^10 of coming up, since each one was a coinspin anyway."

"But if you instead thought 60% with headbands and 40% without, then for the headbanded people it'd be 0.6^3*0.4^2, with the winners being 0.6s and the losers being 0.4s, and then for the non-headbanded people you'd have 0.6^3*0.4^2 too, for the opposite reasons. So that's a chance of 0.6^6*0.4^4. Then we can multiply everything by 10^10 to make the math easier, and the idea of 60/40 and 40/60 increased in probability vs pure 50/50, by, umm..."

"(6^6)*(4^4)/(5^10)"

"Mm, not the Science!tific way of thinking about it, exactly, but not at all bad, either.  You even simplified the math some."

"Though, saying they got artifact-headbanded is not what the original experimental spec was.  We were specifying natural all-14s which will probably give us noticeably different results.  One does want to learn to think precisely about that sort of thing."

"To compute that number there, let's observe it'll be the square of 6^3*4^2/5^5, or 216*16/3125, 2160+1296 is 2160-4+1300=3456, divided by 3125... about 1.1, since 3456 is around 3125+312, squared is about 1.2."

"So the results you saw are 1.2 times more likely if the all-10s have 40% pass rates, and the all-14s have 60% pass rates, compared to everyone having 50% pass rates."

"Now, I realize this ought not to go in a 'published-experimental-report', but what do you happen to believe, seeing that?  What would you believe if you saw that in real life?"

"Oh, yeah, sorry about the headbands I was distracted by the cool math careless."

"But if I saw it in real life, I wouldn't believe very much of anything? It was only five in each group, and two of them randomly going the other way could've made it all look backwards. But times 1.2 isn't a lot of a chance boost either, compared to really knowing stuff about how the world works. So the math seems to look about right? It wasn't too convincing, and the math isn't too convinced?"

"I would go from believing that natural all-14s are going to be at a disadvantage relative to natural all-10s, to believing that having stronger mental stats and having had the experiences that go with them at least isn't actively disadvantageous, and probably does help."

(Oh gods why did she say that that was a terrible idea he's going to make her do math in baseline right now and then everyone will know how stupid she is about this.)

Keltham nods at Korva, then addresses Willa.  "Seems a bit sus that the hypothesis you're comparing to the random-coinspin is one where the supposed natural frequency of thirty-minute guessers, at those two Intelligence levels, exactly matched the data."

"Suppose we'd tested a thousand 'subjects', instead of ten.  245 out of 500 'subjects' with all-10 stats guess within thirty minutes.  256 out of 500 subjects with all-14 stats guess within thirty minutes."

"Then, clearly, to ask how we should react to this data according to your methodology, we should compute the likelihood ratio over the fair coinspin, of the hypothesis that among all-10s, 245/500 guess within thirty minutes, and among all-14s, 256/500 guess:"

(245/500)^245 * (255/500)^255 * (256/500)^256 * (244/500)^244 / (250/500)^1000

"This simplifies to... let's see... a likelihood ratio of about 1.276."

Keltham will now pause dramatically.

"I'll bite, since you're obviously dying for somebody to ask.  How'd you simplify that?  I'm frankly not seeing it."

"Well, if I've managed to remember all the numbers correctly, it's from a version of this problem I worked a while ago, back in Civilization where you could just toss 'expressions' like that into 'computers'*."

(*) This is clearly the same word as appeared in 'computer-science'.

"I see."

"And if you're wondering where the numbers 245 and 256 came from, in that problem, they just happened to be the results of generating 500 random 'bits', twice, and counting 'YES' answers."

"But clearly, the 'computer' could not have been truly randomizing!  After all, it is 1.276 times more likely that these numbers would appear if they had truly been generated by a 'computer' that first picked 'YESes' with 49% probability, and then picked 'YESes' with 51.2% probability.  The Conspiracy has been at work again!"

"Would you agree with this stance, Willa?"

This is more than a little mortifying but at least a bunch of her peers didn't get this right first, she can keep her composure, really she can.

"This is something to do with making the guess about what happened after seeing the data, isn't it? Instead of guessing the ratios first for stats-matter and stats-don't-matter, we did the experiment, and then made up the exact shapes of our guesses after, which is a naughty thing we weren't supposed to do."

"And there's some deep mathematical thing about doing that, looking at data you already have and matching that way, that tends to produce these relative probabilities around 1.25 or so, isn't there?"

"Yup.  Though embarrassingly I don't recall the exact constant.  Back when I was memorizing those numbers in order to impress the younger kids I was teaching, by pretending to calculate it all on the spot, they were too young to ask that question so I didn't memorize the answer."

"But as you say, Willa, you did indeed try to do something naughty, by calculating the likelihoods for 'latent' frequencies that exactly matched the observed frequencies.  Maybe if you'd run a pilot experiment and found 40% YESes for all-10s and 60% YESes for all-14s, you would have written down the numbers 40% and 60% in advance, in the 'preregistered' version of your 'published-experimental-report' that you'd put out and gotten... signed and time-stamped... by larger Civilization, before you'd collected any data.  Then nobody would call you naughty if you ran the likelihoods for the 40%-and-60% hypothesis, and compared that exact hypothesis to the fair-coinspin hypothesis."

"So hopefully it's starting to become clear that all of the pieces of Science! hang together, even if I didn't explain them all that well?"

"Anyways, let's say you didn't run a 'pilot-experiment' that produced 40%-and-60% as a hypothesis distinguished in advance, or 49% and 51.2%.  Then what might you say in your 'preregistered' 'published-experimental-report' that doesn't have the results filled in yet, about how you plan to analyze the data?  People besides Willa are allowed to answer too."

"What are we trying to figure out? We want people who can solve all the problems in 30 minutes? Then we probably care about whether all-14s are enough better than all-10s to make up for how much rarer they are, so we should just check, how likely does it look that all-14s are a hundred times better at this, or even five times better at this?"

"Seems slightly shaded towards 'How do I make more spellsilver?' instead of 'How do I understand what's going on inside of spellsilver refining?'  But leaving that aside, suppose we accept that 'goal-framing'.  What math do we do to it?"

" - huh, so, my complaint is that doing math on how much we should change our minds from a starting assumption there's no difference feels fake. Unless we're doing this study out of idle curiosity it's only a worthwhile study if Civilization was starting with the belief not just that there's a difference but that there's a large one. If 14s aren't wildly better at this then there's no way you'd use them. So the math is easier if you assert that we thought there'd be no difference, but I can't really imagine thinking that. But on the other hand, there's no math you can do on the thing I just said - you would get wildly different answers if you started out thinking 14s are 5 times better versus if you think they're 100 times better - and I thought maybe there was some way to specify 'started out thinking 14s are at least 5 times better' but I'm actually not seeing it. I don 't know if that means there's no math directly for answering the question that would actually motivate my Civilization to do this experiment, or if there is but it's far more complicated than the starting math and we need to get that nailed down first."

"Asking 'what if there's no difference' and 'what if tri-14s are at least five times more likely to guess than tri-10s' are both questions that are semantically well-formed - it's easy to visualize ways reality can be that makes those propositions true or false - but the second question is hard to answer directly using the statistics that 'published-experimental-reports' are supposed to calculate."

"To be clear, you could see a report saying that they tested 1000 subjects of each type, and 103 tri-10s and 622 tri-14s guessed in time.  You could look at that report and say, 'Yep, sure looks to me like tri-14s are at least five times as likely as tri-10s to guess in time.'  But in stating that, you'd be arriving at a 'posterior', not stating the kind of evidence that a 'published-experimental-report' summarizes.  You'd be doing something that intrinsically revolves around 'priors' and not just 'likelihoods'."

"Do any of the candidates want to take a stab at saying why Carissa's question of ultimate interest to her Civilization - 'Are tri-14s at least five times as likely as tri-10s to guess in thirty minutes?' - is something that's ill-formed for an experimental summary to directly summarize an 'evidential-update' about?  Why it can't just report on that the way it could report on a preregistered hypothesis that tri-10s were 40% likely to guess, and tri-14s 60% likely?"

Asmodia, after querying Security somewhat plaintively, sighs and gives the green light to the only new candidate who is reckless enough to risk directly criticizing the ideas of the Chosen of Asmodeus at this stage in their acquaintance.

Can this person please at least look around hesitantly before she tentatively raises her hand?

She very hesitantly justifies this particular incident of recklessness in her mind, both to herself and anyone listening. She just wants to learn things. But she's also very happy she can help the project right now. She does do her best to also look very hesitant, as is in character for alter Willa and also generally appropriate.

"'Five times better at this' seems like a tricky statement to use with this math. When you say a category has a specific chance of its own, you can go through and multiply all the probabilities in the category independently, and compare it to other categories that have been multiplied up independently. But with 'five times better', the two categories sort of depend on each other."

"You could guess that 10s have a 10% chance, so 14s would need a 50% or better chance. Or 10s could have a 15% chance and 14s would need to be better than 75%. Or 10s could have a 25% chance and it would be impossible for 14s to be five times better. And somehow 'are 14s five times better than 10s' needs to take account of all of those at once and everything in between. Or maybe you mean something more complicated than that by 'five times better', but I think that would make this more of an issue, not less of one."

"And it's even worse, because there's an implied 'at LEAST' five times better there. So we need to account for all the imagined frameworks where the multiplier is at least five, all the way up to infinity."

She smiles pleasantly! There are no power dynamics here, they're all just speculating about math! "I agree that all of those make it terribly complicated, and yet - I look at these experimental results and I think 'ah, so 14s, it looks like, aren't better at this by a factor of five.' Clearly in my head I am calculating the odds of observing these results given my theory, and judging them as low; I'm just not doing it precisely. And so what's bothering me is - is it impossible to do precisely, so the thing it feels like I'm doing in my head is an illusion? Or is it possible but hard enough we shouldn't bother without the 'computers' of Keltham's world? Or is there elegant math that makes all those numbers actually possible to calculate?

The reason this is bothering me is - the closest thing I've seen to science is merchant houses, going, does this supplier provide higher quality by enough of a margin to justify their higher prices? They do tests like this all the time. But I don't see how to use this math to answer their question."

"There's obviously ways to do Lawfully anything you can do with your brain by seeing the results yourself.  Sometimes it takes more Law than you have.  Sometimes it involves calculations that wouldn't make sense to put directly into a 'published-experimental-report'.  Still, Willa has put her 'publication-priority-timestamp' squarely on the key issue."

"By way of approaching the complicated proper math by a simpler 'hacky' route, suppose the 'preregistered' analysis said that they were going to compare hypotheses for each 'subject-group' having seven possible 'latent propensities' to solve the puzzle in time, of 0, 1/10, 1/5, 2/5, 1/2, 3/5, 4/5, 9/10, and 1."

"Since some subjects in both groups solved the puzzle, and some didn't, we can cross off the 0 and 1 'propensities' from both cases; they've been falsified outright, the 'likelihood' is zero 'conditional' on those propensities being the true 'latent' state of reality."

"Now, let's go ahead and compute the 'likelihoods' for each surviving 'propensity', in each group..."

All-10s (2 YES, 3 NO): Propensity:     Likelihood: ---------------     -------------- 1/10                (1/10)^2 * (9/10)^3 =   729/100000 2/10                (2/10)^2 * (8/10)^3 = 1024/100000 4/10                (4/10)^2 * (6/10)^3 = 3456/100000 5/10                                               = 3125/100000 6/10                (6/10)^2 * (4/10)^3 = 2304/100000 8/10                                               =   512/100000 9/10                                               =     81/100000 All-14s (3 YES, 2 NO):                                                       = same but table flipped

"The point which Willa is gesturing at, is that it's not possible in the same way to speak of the combined 'likelihood' of all our observations, if all-14s guess more often than all-10s, or if all-14s guess at least five times as often as all-10s."

"At least, not without dragging in some additional assumptions."

Korva doesn't at all see why you can't just ask whether all-14's get it within thirty minutes five times more often than all-10's, do the experiment, and then report whether your records of it confirm or deny that this happened, but Willa and the Chosen of Asmodeus are obviously much better at this than her, so if they're stuck on it then whatever Korva is doing is going to end up being humiliatingly wrong.

"Keltham, advisory, the new candidates have spent their previous educational lives in unimaginably-to-you-awful Golarian schools designed by Intelligence 14 teachers for Intelligence 10 students, where asking stupid questions means that you lose face in front of the other students and the teacher rebukes you.  Right now, most of them are staying quiet and not asking any questions.  This is not the good sign that it would be in dath ilan.  Here, it means that they have made a completely reasonable decision to let other candidates ask stupid questions first, and see whether we - the current researchers - were being honest with them about whether that's safe here."

"I'm not quite sure whether they're internally wondering 'What's wrong with just testing a lot of people and reporting whether all-14s get it five times as often?' or more like 'Why would anybody do this sort of experiment in the first place?' or maybe 'But what's an "experiment" actually?' because the Baseline word translated in my head but not in theirs, but..."

"Thank you, Ione."

"Can I possibly solve this problem by offering anybody here one silver to ask whatever stupid questions they weren't asking?  At least in dath ilan, offering to pay for something is taken as a credible sign that you want it."

"I wouldn't bet on it.  Maybe if you told them to Message me with the questions, let me ask it for them, and then you pay me and I pay them after class."

"What if I literally truthspelled myself about -"