"Same problem as in the cooperation-defection dilemma if you're a Chaotic Neutral outsider.  They wouldn't be certain you would know correctly whether they'd lose face in front of you, and it wouldn't solve the problem of losing face in front of the other students."

"You know, I've read about the concept of 'low-trust environments' but never really understood it before."

"Ione, you okay with being tapped by truthspell about how, if somebody else asks a question through you, you won't reveal who it was?"

"Most of the current researchers will see who, Ione.  We've made our soul arrangements and our arcane sight will detect the Message spell."

"Right."

"Well, I can still re-ask a question if anybody Messages it to me, and at least Keltham and the other candidates won't know who."

"And if that doesn't work, Ione can introduce a delay between being Messaged and when she speaks, and I'll start paying a copper to anybody who pretends to message Ione but doesn't actually subvocalize anything to her, so as to create plausible deniability about who was actually asking questions."

"If that doesn't work, I'll stop and actually think about the problem, though that's something of an extreme measure."

"Seems like a good time to mention that I had that question about 'What's wrong with just testing a bunch of people and reporting whether or not there were at least five times as many all-14s who solved it?'  And didn't ask because I am flawed and imperfect, and forgot the rules about what truly serves Asmodeus here."

"I'm actually not sure how Keltham would answer that one?  It's not thinking in probabilities, though.  And what do you do if you test a few thousand people and 100 all-10s solve it and 499 all-14s solve it, report that the theory failed?"

Korva is sitting on about ten different stupid questions at this point. She would just ask them out loud, at this point, if there was only one of them, but with ten it's hard to pick.

The most recent question is that she thinks she can see what Keltham's doing now - it's something about converting the results of the experiment into probabilities for different guesses about how the world is being the right guess, or something - but she has no idea why the calculations he did were the right ones in order to get those probabilities out the other side.

Ione holds up a hand and seems to listen.  "Okay, Message from somebody who says that their issue is actually that they've got no idea why the calculations you did on the whiteboard are how you convert the results of the experiment into probabilities for how those are the right guesses for how the world is."

Because alterCheliax students would not be this cautious and Asmodia is sick of it and Security is running Detect Thoughts anyways.  This is Cheliax and they have ways of obtaining questions from reluctant askers.

"...huh.  Asmodia, you may have needed to do more basic Probability problems with your students, not just run them through all of the abstract high-level stuff, though I appreciate that you had sharply bounded time."

"Yeah, understood.  I'll see if I can fix my fault there without taking up your own time."

Korva didn't message, so instead of being nervous she's assuming that someone behind her also had one of her questions (not surprising, if Pilar also generated a different one of her questions independently), which makes her feel a little less bad about herself. Maybe she'll at least have another day to go over it and figure out what just happened.

Amusing.  Now the Security wants to see who works it out before they get their silvers after class, and the expressions on their faces if they don't.

Meanwhile, the ever-oblivious Keltham is turning to gesture at his Prestidigitated whiteboard:

All-10s (2 YES, 3 NO): Propensity:     Likelihood: ---------------     -------------- 1/10                (1/10)^2 * (9/10)^3 =   729/100000 2/10                (2/10)^2 * (8/10)^3 = 1024/100000 4/10                (4/10)^2 * (6/10)^3 = 3456/100000 5/10                                               = 3125/100000 6/10                (6/10)^2 * (4/10)^3 = 2304/100000 8/10                                               =   512/100000 9/10                                               =     81/100000 All-14s (3 YES, 2 NO):                                                       = same but table flipped

"Right, so, in the first row, we've got the hypothesis, what if all-10s have a 0.1 'propensity' towards YES."

"Where, in this context, what that means, is:"

"Suppose the world is such a way that a random person from our 'survey-pool', about whom we know only that they have 10s in all mental stats of Intelligence, Wisdom, and Charisma, has a 10% 'propensity' to guess the correct answer on the 2-4-6 challenge within 30 minutes.  Where 'propensity' is sort of a metaphysically fraught concept.  But for our purposes in the kid's version of the lesson, 'suppose 10% YES propensity in all-10s' means, suppose the world itself is such that, from our perspective, it works out to 10% of the all-10s guessing within 30 minutes.  Nethys, maybe, would know that some people in that pool were 99% likely to get it and some were 0.1% likely to get it, so it's not that the 10% chance of YES is an objective property of every single person.  But from our perspective, the world is such a place that Nethys would say that around 10% of the people in that pool were 99% likely to get it, and we don't know which people."

"That's one hypothesis, the one in the first row.  The second row's hypothesis is that all-10s in our survey-pool have a 20% propensity towards YES when we'll test them using our current experimental setup."

"Suppose we test five all-10 subjects - one after another, rather than in a simultaneous session - and see these exact results in order:  NO, YES, YES, NO, NO."

"After getting a NO result with the first all-10 subject, we've seen something that was 90% likely in the world where all-10s have a 10% propensity to YES.  But 80% likely in the world where all-10s have a 20% propensity to produce YES results."

"After seeing the YES from the second subject, we've now seen an additional fact that was 10% likely in the world where all-10s have a 10% YES propensity, and 20% likely in the world with 20% propensity."

"The two facts combined, that we've seen so far, are then 0.9 * 0.1 = 0.09 likely in the 0.1-propensity world, and 0.8*0.2 = 0.16 likely in the 0.2-propensity world."

"We aren't saying anything there, one way or another, about the probability that the world is like that.  We're not saying anything about the probability that Nethys would, if we could ask him or Ione could, write CORRECT on our sheet of paper, if we wrote down the guess, 'all-10s have a 10% propensity to YES'.  To say that we'd need to know the 'prior-probability' of that hypothesis, and the 'prior' on all the other hypotheses, and have already computed the likelihood of all the other hypotheses, and 'renormalized' to get the 'posterior'."

"Which isn't a sort of thing that experimental reports try to do.  Other people could know other evidence that would be relevant to whether Nethys was likely to write CORRECT on the guess.  Stating probabilities like that are what prediction markets are for."

"What we're saying is - just flatly suppose that the world is in utter fact a place where all-10s have a 10% propensity to solve the 2-4-6 challenge within 30 minutes, relative to the experimental procedure we're using.  Then it is valid, as a matter of 'logical-deduction', to say that you're 9% likely to get a NO followed by a YES on the first two subjects tested.  It is likewise valid to say that, flatly assuming the hypothetical world where all-10s have a 20% YES propensity, you have a 16% chance of getting a NO followed by a YES."

"This is the key fact that other people need to know in order to update their beliefs based on your experimental results, so it's what the experimental report summarizes."

"It's a very local fact.  It's like how, if you suppose that X=3 and Y=4, you can calculate that X*Y=12 without worrying about whether Z=5 or Z=7."

"If somebody else already ran tests on 100 subjects with all-10s, they might have seen results that pretty strongly updated them on the chance that Nethys would write 'CORRECT' on the 20%-propensity-hypothesis.  They could've gotten 80 YES results, for example, which would make them pretty sure that hypothesis was wrong."

"But they can't have gotten any results relevant to the proposition that, in the hypothetical world where 20% of all-10s guess within 30 minutes given our experimental procedure, there's a 16% chance that we'll get a NO followed by a YES."

"You don't need to read all of the experimental reports in the world, you don't need to follow any prediction markets, to report that summary of the results you got."

"Suppose your third, fourth, and fifth results are YES, NO, NO."

"After the third result, we've seen data that's 0.9*0.1*0.1 = 0.009 likely in the 10%-propensity world, and that we'd have a 0.8*0.2*0.2 = 0.032 chance of getting in the 20%-propensity world."

"After the fourth result, a NO, we've seen things we're 0.0081 likely to get in the 10% world, and 0.0256 likely to get given 20% propensity."

"After the fifth result, a NO, it's 0.00729 or 729/100000, and... haha, whoops, 0.02048 or 2048/100000 for 20% propensity."

"This, by the way, being among the reasons not to trust your teacher even when he seems like such an uncomplicated straightforward reliable person, it is literally actually possible for him to be mistaken."

Keltham Prestidigitates the corrected table of likelihoods:

All-10s (2 YES, 3 NO): Propensity:     Likelihood: ---------------     -------------- 1/10                (1/10)^2 * (9/10)^3 =   729/100000 2/10                (2/10)^2 * (8/10)^3 = 2048/100000 4/10                (4/10)^2 * (6/10)^3 = 3456/100000 5/10                                               = 3125/100000 6/10                (6/10)^2 * (4/10)^3 = 2304/100000 8/10                                               =   512/100000 9/10                                               =     81/100000 All-14s (3 YES, 2 NO):                                                       = same but table flipped

"I wish I had some way of asking you whether that helped, anonymous question-asker, but maybe communicate again to Ione if that was still unclear?"

Security, don't relay anything further about that topic to Ione unless somebody takes the initiative to ask.  The meaning should be perfectly clear at this point, and if any newbies can't keep up and also can't ask questions, they're obviously headed for the dropout category.

"Nothing yet, which hopefully indicates it was clear.  And if not, well, they were adults and made their own decisions."*

(*)  A saying now acquired from the kidnapped rescued Taldane girls in the secondary site.  Though the same idea certainly does exist in Cheliax; indeed, there exist many Chelish variations on this saying, including several with Infernal loanwords in them and a couple straight from Hell.

Alexandre raises his hand. "I have a question. If I have two hypotheses, say 40% and 60%, I know how to update them based on this data. But at all times -" Keltham has said, Alexandre obviously does not have it at all times "- you have a hypothesis that is - vague, not specific, 'what if all my guesses are wrong'. How do you weight the answer, 'anything but my hypotheses'?"

"Good question!  I'll ask you to hold that thought pending subtopic #3."  Keltham gestures back to his previous list of subtopics he shouldn't forget to talk about.  "Or actually I should maybe just write that one down..."

#1 - 'Published-experimental-reports' usually don't assign 'priors' or calculate 'posteriors', they just report all cheap details of the raw data, and maybe calculate some 'likelihoods' from obvious hypotheses

#2 - Separate experiments are usually supposed to avert 'conditional-dependencies', watch out for when that isn't true

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

#4 - How to specially process the special meta-hypothesis 'all-other-hypotheses'

"Oh, and if that last Baseline word isn't translating, maybe the Taldane equivalent would be - everything we haven't thought of explicitly, all the theories we're not considering?"

"Anyways, this has hopefully ended up making #1 a little clearer."

"Going back to #2... what would make a good entrance point..."

"All right, so this isn't addressing #2 right away, just introducing an idea we'll use there, but."

"Suppose that we tried summarizing the hypotheses here into three buckets.  One bucket that the YES-propensity is 0.1 or 0.2 or 0.4, one bucket that the propensity is 0.5, one bucket that the propensity is 0.6 or 0.8 or 0.9."

"Is it then possible to describe the likelihood of our data NO YES YES NO NO, conditional on the first or third bucket?  For the middle bucket it's obviously 3125/100000 or 1/32."

...why did the only candidate who's letting herself generate responses have to be her.

Yes.  But BE FUCKING SHY OKAY.

She makes sure to look back and forth before answering, as if she's making sure nobody else is going to say anything.

"It'd have to be the sum of the likelihoods right, or the whole idea of probability wouldn't make any sense? So 0.1 or 0.2 or 0.4 would be total likelihood of (729+2048+3456)/100,000 = 6233/100,000 vs 0.5 at 3125/100,000"

"Except that only works if you thought 0.1 and 0.2 and 0.4 had the same chance to begin with, before you did the experiment. And even 0.5 too, as likely as all the others. If you thought they all had different before-chances it would mess up the weights and I'm not sure doing that would work anymore. Maybe you'd have to do them individually if you couldn't trust they started off the same?"

"Suppose I've got a coin that might have a 4/10 chance of coming up Queen, or maybe a 6/10 chance of coming up Queen, I'm not really sure, seems about equally likely to be each.  Would you agree that in this case, the coin is (4+6)/10 = 10/10 likely to come up Queen?"

Oh no something's definitely wrong, but she can fix it, she knows she can fix it it's just math...

"No, definitely not, but this is relative likelihoods and not absolute likelihoods? Maybe that doesn't help, hmm"

"Is the idea that we have to divide by the number of cases our bucket used for it to work? And maybe that the before-chance we use then is the sum of the before-chances of all the pieces of the bucket together?"