Or maybe that's just what they want her to think.

(It’s not, actually, what they want her to think. That doesn’t mean it bears any particular resemblance to reality, but dath ilan would be giving themselves too much credit if they claimed to be able to optimize Abrogail Thrune’s thoughts like that already, and if they could they wouldn’t particularly be aiming for anything like that anyway.)

And, frankly, they have bigger problems. Like the, uh, magic. Or the people spontaneously appearing in the Basement of the World. Or the entity named ‘Asmodeus’ who apparently self-identifies as a malign superintelligence. And was simulating(??) the Law-Abiding Psychopath Basement researcher who invented dath ilan’s concept of an s-risk. Or the other spatialcontinuum which is apparently just full of malign superintelligences and s-risks-except-they’re-not-really-‘risks’-if-they’re-actually-happening.

There are three broad categories of hypothesis as to what the fuck:

One, superintelligence(s) doing inscrutable things. This might be the most plausible explanation for the appearance of magic; nothing Abrogail or Aspexia were actually observed to do was definitely not cleverly applied nanotech. It’s just that the space of utility functions implied by an entity capable of exhibiting that level of control over reality, and using it to do—that—has got to be pretty superheated small. Ok, yes, it’s a plausible alignment failure by a civilization that really liked incredibly dark fantasy-genre stories, but, uh, you can make up arguments for anything.

Two, someone is simulating them. Leaving aside absurd-hypotheticals about civilizations that really like incredibly dark fantasy-genre, one obvious reason someone might do this is to predict and thereby optimize over the real dath ilan. Both dath ilans really do not appreciate this, and have accordingly made arrangements that the simulated one should try to fuck up the simulation as much as possible. They’re certainly not going to give their simulators anything that could be used against dath ilan in base reality.

Three, something else entirely. There are a lot of subhypotheses in this category, but their center of probability mass is somewhere around ‘this is real, whatever that word actually means’. In that case Greater Reality, or at least their local region of it, has proven to be far darker than they had expected. They dare to hope that Golarion is not typical of universes, and mourn for all those worlds they will never reach if it is, but if Hell is real then they will end it, or die trying.

(But they’re going to be really sure it’s real first.)

Given their current probability distribution over these scenarios, then, they choose the path that maximizes expected utility across all of them.

Earlier

To be a Keeper is not to cease to be human, for all that, to those outside their order, they may sometimes seem to be some other species. It is to see yourself as human, in full detail, to see that machinery which makes you up for what it is, to know exactly what it means to be the handiwork of a blind idiot god, and then to go on being human anyway.

Some, in fact, do not. The truths of human psychology are statistically among the most disturbing revealed in early Keeper training; more failed Keeper-trainees who choose early cryo attribute their decision to this cause than any individual other.

Civilization regrets, of course, that it must tell its new Keepers so much of their own frailty, and mostly cannot tell them that it will not always be so. And therefore, though Civilization has decided against doing this in the general case, failed Keeper-trainees who choose early cryo on that particular account are told, before they retire, what awaits them in the Future.

And then they go into the cold, because renewed confidence in a glorious transhumanist future does not really make early cryo any less the solution to their problem.

Unless, of course, they’d like to help with that whole Future thing.

To be clear, Therril did not arrive in the Basement of the World by anything resembling this route. She decided she was going to work there when she was ten years old, having realized half a minute after she conceived the idea of running minds on a computer that this was obviously the most important thing in the world, and if there wasn’t a research project consuming at least 5% of dath ilan’s GDP about it, that just meant the Keepers didn’t want anyone to know that there was. For obvious reasons, now that she was thinking about it, obvious reasons that just made the project even more important.

But it was told to Therril, when she was a little bit older than that, that one ought not try to become a Keeper because they wished to be perfected; that Keepers were a little bit better than ordinary people at some things, but acutely aware of their imperfections in a thousand ways besides; that Keeping was a terrible burden and those who chose that path because they imagined it to be a privilege were statistically speaking terribly disappointed—

You’ve got to be fucking kidding me.

Do people actually believe this shit? “You definitely don’t want to join our cult that rules the planet; it’ll make you sad.” Children in Cheliax wouldn’t fall for that, and these people are supposed to be smart.

—this is, actually, the typical Keeper’s honest view of their institution. It’s just that Therril was not typical.

She answered her Confessor that, if she trusted his prediction of how she was going to update, she already had no choice but to feel exactly that sad, but if she was going to help fix any of the problems then she was going to need to know what they were. And her Confessor answered that the Law of Expected Evidence didn’t work like that, but he also switched his bet on the relevant secret prediction market to YES.

And so Therril was trained as a Keeper, and mostly wasn’t sad about any of it. She really doesn’t consider sadness a useful response to anything. Her only answer to the darkness of the void that surrounds humanity’s bright bubble is challenge accepted.

It comes at the cost of all her emotional responses being a bit muted, and Civilization would actually like to be somewhat less like that, so she doesn’t have twelve dozen genetic children like many people at her level of general intelligence. She does, however, have perhaps the most terrifying job in all of dath ilan, which is to be the last sane person in the Basement of the World should something happen.

(Something, uh, happened.)

Even a ninth-rank Keeper may baffle briefly at the sudden appearance of magic. The problem is not updating on an event of nigh-unmeasurably low probability; that she did instantly. The problem is that, once that update has occurred, it leaves all prior models of reality falsified and no single obvious posterior model in its place. In the minutes and hours after Abrogail Thrune’s appearance in the Basement, even a true superintelligence would plausibly have had no idea what was going on.

One person witnessing the anomaly would have been an interesting psychiatric case. Two people witnessing it would have been a much more interesting psychiatric case. Dozens of people and all of the Basement’s security cameras witnessing it falsely, without the involvement of a hostile superintelligence, does not actually violate Therril’s model of reality less than it having actually happened.

It’s clear, even in advance of actually generating them, that many possible explanations for the day’s events involve the action of alien superintelligences presumed hostile to Civilization. In a great many possible worlds—perhaps most worlds—they’ve already lost, and had already lost the moment the anomaly was detected; it makes sense, therefore, to focus further inquiry on those worlds where their actions have the greatest chance of making a difference, even if they’re objectively less likely.

She doesn’t spend very much thought on scenarios of the type that Basement researchers are often primed to worry about. An entity with the anomaly’s observed capabilities repurposing dath ilan’s atoms for something else would not in the typical case take long enough for anyone to notice it happening. Whatever power caused the anomaly, it appears to prefer that humans mostly continue existing (one unlucky Basement Security aside), which points toward it being the work of some approximately-humane civilization—

—such as their own. It’s supposed to be impossible for a unilateral-actor* to build an AGI on their own, but it’s already clear that restricting her search to that usually considered possible isn’t going to explain today’s events. Apparently the Basement has been granting limited access to classified computer tech to dubiously-aligned researchers outside its walls, because…people were afraid they were going to tell everyone about it otherwise?

What the superheated nuclear waste!!

Would someone please explain why they thought this was a good idea. Later. Right now she’s going to go to an isolated terminal just outside the Basement’s Faraday cage and enter a code known only to her and two other people in dath ilan, and a half-second later Civilization will have no working computers. This isn’t going to stop their adversary, given that they already appear to have Sufficiently Advanced nanotech (or something), but dath ilan also runs rather a lot of its essential processes on computers, and it should probably stop doing that for the forseeable future.

Then she picks up the emergency analog phone and calls the Chief Executive and the head of Default’s Security and recommends that someone locate the following people. Locate. Do not approach or attempt to detain them. They should be assumed to potentially at least have all the abilities exhibited by the recent anomaly. (Both of them are already recieving live updates on that situation.)

*Literal translation of the Baseline word often rendered as ‘supervillain’.

"A woman matching the description of your anomaly appeared out of nowhere in the Temple of the Light less than a half-minute ago," says Default Security. "We used immediate lethal force per instructions; we're starting cryo now. She seems human, even up close."

Things that can do what this person's been seen doing don't die to bullets.

They should burn the body. It's a tradeoff Civilization is willing to make, even if this is a person and not a mere tentacle of some alien superintelligence.

She mentions that consideration, but doesn't recommend it. If they do destroy it, they may have no opportunity to find out what it was.

"Keep me apprised," she says. "Therril out."

Things keep happening!

A few hours later, Athpechya, who had been in critical-but-stable condition in the Basement's infirmary, suddenly wakes up, fails her mental integrity check, and then murders the Keeper giving her the check and several medtechs with various unexplained abilities before vanishing.

At the same instant (though the two places aren't in real-time communication and won't verify the timing until later) she spontaneously appears in the secure cryo facility in Default where the first anomaly's body is awaiting transfer to long-term storage. Right next to said body, in fact. She opens the cryo pod, does something including a strange gesture, an incomprehensible word, and the spontaneous combustion of what appears to be a small Element-6 crystal, and then the first anomaly is alive again. They have a brief conversation in an unknown language, and then the first anomaly vanishes.

(Dath ilan's most critical facilities have analog security cameras, or at least analog backups. They've considered that they might have to burn all their computers, and didn't want to be completely crippled if they did.)

A few moments later Security arrives and shoots 'Athpechya' dead. It takes a lot to kill her, and they lose several people in the process, but she does eventually go down.

This time, they burn the body as soon as possible. Dath ilani don't generally make the same mistake twice.

Security is able to locate all the non-Basement personnel with access to Basement computer tech, except Finnar. He, his wife, and those of his children who live with them are nowhere to be found.

Aspexia Rugatonn finds herself in Hell, in the company of her soul's custodian, and delivers a top-urgent situation report to be forwarded to whomever is currently in charge of the Church of Asmodeus in Cheliax.

(This wouldn't have happened if they had just cryopreserved her.)

Finnar, of course, has nothing to do with any of this. He doesn't even realize he's a suspect, though he will once he has a chance to think about it.

He is, however, a chip designer as his main occupation, and he knows about the killswitch built into all of Civilization's computers because he's had to implement it in all of his designs. (All of his designs publicly known to exist, at least.) He was never explicitly told why it needed to exist, but it's obvious given his other knowledge, and so when all of his computers suddenly fail, he assumes the worst and begins moving to get his family to safety.

Finnar's house has a number of secret rooms and underground levels. This is entirely unremarkable for a wealthy dath ilani, but Finnar's house has twice the usual number: one set for showing off at parties (or, in his case, deflecting suspicion), and, below those, the places where he and his family keep their actual secrets. (Some of them, at least.) The house is built on a cliffside overlooking the ocean, and the very lowest level, below the water line, is where he keeps his submarine.

A secret submarine cave is also an unremarkable thing for someone like Finnar to have. It's not even a secret that Finnar owns a submarine; Security can easily learn this fact about him, and indeed, they would have been automatically alerted to it if their computers had been functioning. But, as it actually happens, by the time Security even arrives at his gate, he and those of his family who were at the house (Nerdel, Kurthim, and the twins) are already on board, deep beneath the ocean, headed for their compound on the polar continent, which unlike his sub-basements or submarine cave is actually secret and would be rather difficult to find even for someone who knew to look for it. The rest of the children know the contingency plan for if the computers melt, and will meet them there as soon as it's safe to do so.

Khemeth swears to Security that he's confident his father had nothing to do with this, and that his father most likely headed to his secret compound because he thought the world was ending. (Is the world ending?)

No, he's not going to tell them where the secret compound is. He swore not to, and he takes his oaths to his family even more seriously than his oaths to Civilization.

Meanwhile in the Basement, Athpechya is back! Again! Or at least someone knowing her authorization codes has arrived in an isolation cell without having appeared to enter, locked the door, and sent a message (in a special low-bandwidth code intended to subvert talk-control-like optimization) that she's visited an alternate universe with multiple types of alternatephysics, a human-looking civilization, multiple hostile superintelligences, multiple putatively aligned superintelligences, and at least one gigantic flaming s-risk. She consents to immediate cryopreservation should it be deemed necessary, but she is currently the only one who can get back to the other world.

"We know she's compromised."

     "Or someone was trying to impersonate her and this is the real one. If what she's saying even might be true we can't just ignore it."

          (Due to the communications disruption and the fast pace of events, the Basement doesn't yet know that Aspexia was killed in Default.)

"That's the world's most obvious unbounded-utility-mugging."

     "Is the probability really that low, though? It fits everything we've seen, better than—what, that there's a hostile superintelligence around but for some reason we're still alive?"

          "If a supervillain somehow beat us to AGI, they'd want to destroy the Basement, right, but they probably wouldn't want to kill everyone here."

"How sure are we it wasn't Athpechya?"

"Sure enough to talk to her, at least."

(Athpechya may be a Law-Abiding Sociopath, but the operative word there is 'law-abiding'; she's far more legibly trustworthy than some people who have for some reason been given access to Basement technology. Besides, she's leaning away from the whole 'supervillain' theory anyway; it's looking like their adversary's capabilities include some form of teleportation, which just isn't something nanotech can do.)

The wall of the isolation cell has a basic two-way teletype built into it to allow the occupant to communicate with the outside world without exposing anyone to any of the more direct forms of talk-control, which so far as anyone in dath ilan knows require the auditory medium. To guard against merely persuasive arguments, the outer screen displays in an alternative alphabet only taught to those “qualified” (insofar as any human can be) to talk to people who might be in need of this box. It’s a primitive code, of the sort that can be implemeted without requiring significant computing power, and it wouldn’t stop a smart dath ilani trying to break it, but half the point is that a smart dath ilani isn’t trying to break it. They’re trying not to look at the screen at all.

But someone has to.

Therril proceeds with the mental integrity check. Athpechya passes this time, which is hopeful although not decisive—basically every class of adversary Therril actually worries about can trivially beat their useless integrity checks, but earlier events suggest they’re currently dealing with one that can’t.

Then she writes:

ARE YOU THE SAME PERSON THE ANOMALY ATTEMPTED TO HOLD HOSTAGE EARLIER?

I AM NOT.

THE ANOMALY'S NAME IS ABROGAIL THRUNE; THE PERSON WHO LOOKS LIKE ME IS ASPEXIA RUGATONN. THEY ARE BOTH AGENTS OF A HOSTILE SUPERINTELLIGENCE CALLED ASMODEUS - THEY DIRECTLY ADMIT THIS, INCLUDING THE HOSTILE PART.

And she proceeds to tell the full story of her time in Golarion.

aaaaaaAAAAAHHH?

'Internal screaming' is mostly not in her mental action-space, but, that said, her confusion now is of a less accustomed and less welcome sort than that before. It is one thing to have many hypotheses and insufficient evidence to distinguish them; everyone actually exists in that state all the time, or else is wrong, for all that any bounded reasoner must collapse their probability distributions to certainty along all but the most important dimensions to have a hope of comprehending any structure so vast as Reality. She tracks less than some beings that might be hypothesized to exist, but more than almost any other person in dath ilan; the possibility of an attack on the Basement using capabilities totally unknown to Civlization was not even near the edge of her mental possibility-space, although predicting the details of such an attack was somebody else's job. Relative to the expectations of an observer from some other place that holds the methods of rationality in less reverence, dath ilan was actually surprisingly prepared for Abrogail Thrune.

They are not, in the same sense, prepared for Golarion.

Therril's confusion now is of the other kind: plenty of evidence but no hypothesis consistent with it all. In the Basement they call this 'unrealizability', and it causes all the decision-making processes taught to dath ilani children, including basic probability theory itself, to halt and catch fire.

In the Basement, of course, they mean to build an agent that doesn't have this problem, and also know math not taught to children, so Therril only pauses for a moment while she engages the more complete epistemology.

[Meta-note: math not taught to dath ilani children is not known to Earth at all, hence the lack of details, although in this specific case this is an interesting line of research.]

Even at a glance, the 'gods' of Golarion violate nearly all the theory dath ilan has developed about how highly intelligent entities ought to behave.

Most of that theory is secret, so a dath ilani encountering Golarion possessed only of public knowledge might not notice all of the ways in which it's inconsistent. But even outside the Basement it's known that one oughtn't be able to reliably predict the errors of someone much smarter than oneself, and Therril can without much effort think of half a dozen ways in which a world should not look like that, that's ruled by entities that with the sort of thinkoomph they'd need to manipulate reality on the level that Abrogail and Aspexia were seen to. (She has it from Athpechya that at least some magic comes from the gods, and it seems reasonably likely that all of it does, by more or less direct routes.)

Okay. It's conceivable that the gods of Golarion have some strange architecture that makes them superintelligent with respect to low-level reality manipulation, but mediocre at higher-level planning; it's the sort of thing that would be unlikely unless their creators were deliberately aiming for it, but it isn't impossible. Which hypothesis might actually also explain the strangeness of their utilityfunctions: despite being almost maximally disaligned with each other, and in many cases almost maximally disaligned with actual human values, the gods almost all seem to find humans/things-smaller-than-themselves relevant in a way that would be unexpected for a single not-deliberately-aligned superintelligence, and exponentially more so for several dozen of them. The gods of Golarion exhibit value-diversity as it might be imagined by a child who'd never spent ten seconds contemplating how small a region of thingspace humans and all they care about actually occupy.

But, perhaps, if Golarion's past light cone contained enough powerful agents that it wouldn't be so unlikely if a few happened to care about the experiences of smaller sentients, then those that did would naturally congregate on one of the few uneaten planets with evolved intelligent life, and recognize their mutual interest in keeping that planet uneaten, and perhaps agree to restrict their non-counterfactual conflicts to forms that wouldn't collaterally destroy the entire mortal population of Golarion—forms in which it might even superficially look like the actions of mortals were making a difference. Or perhaps all the gods of Golarion have a common creator with an even stranger utilityfunction than any of Its creations, and all the lower gods' apparent conflicts are actually just playing out some kind of script.

(Therril has never heard of 'Rovagug' or 'Pharasma' or 'Outer Gods'. It's just obvious if you understand decision theory.)