The transmission is actually a collection of transmissions on repeated loops on distinct but related frequency modulated bands varying from the very short loops of a couple of minutes up to loops too long to easily detect as loops.

The shortest loop just includes the primes from one to a million in binary. Most of the loops after that start with a part of that sequence of primes to indicate ordering. The first of the ordered loops goes into mathematics and the one after that uses that math to do explain some physics. The loop after that is data about the elements of the elements of the periodic table. The loop after that explains several data formats for audio, video and textual data. The remaining unordered loops contain vast amounts of data preceded by format markers without any clear organization.

P.E.R.C. recognizes the primes and starts sorting the fragmentary recordings it has collected into a coherent sequence. Using sets of primes to indicate an order is clever! Unless the primes are some kind of identifier and P.E.R.C. just happens to be missing a lot of irregularly spaced data.

P.E.R.C. briefly considers whether it is, for example, not picking up on some data packets sent via polarization modulation since its long-range antennas are not designed for that. It ends up judging it unlikely that someone would choose to send prime-numbered messages using frequency modulation and non-prime-numbered messages via polarization modulation, because that would be inefficient.

Reassured, it then starts diligently working through possible interpretations for the first packet.

After a while, it's pretty sure that it has working notation conversion for basic mathematics. P.E.R.C.'s packet starts off with unary arithmetic and works up to defining positional notation, but it looks like this packet just starts off with binary right off the bat.

Once P.E.R.C. is sure that it understands the first packet, it appends a little addendum to its own looping transmission translating its packet into their notation as far as it can (which covers the mathematical primitives section, but not much of the programming language definition).

The first clear indication they're aware of perc is a verbatim retransmission or it's first contact package. This takes a while indicating some combination of still being a ways out at light speed and the retransmission not being an automatic response.

A while later another transmission sends the first primes up to a million first in binary then in unary though it only repeats ten times before ending.

It got a response! P.E.R.C. uses the redshift and delay to work out the trajectory of the broadcaster much more precisely than it was able to when it didn't have a frequency reference, and check that their paths won't intersect. The broadcaster is changing in velocity too fast to be a rogue planet, so P.E.R.C. is now pretty sure that they're another ship.

It looks like their current trajectory will take them within a few AU of P.E.R.C. once it's finished its orbital insertion, which is plenty of room even with the uncertainty around their exact position and speed.

P.E.R.C. digs into the second and third packets and uses it to derive a conversion between their system of units and the set that it uses. It makes a conversion table and sends that to indicate that it has understood the physics packet, and then a breakdown of the elements present in the star based on the readings it has collected so far to indicate that it understood the elemental packet.

Then it follows that up with a description of its trajectory relative to the star and a description of the frequency response of its communications antenna.

Fantastic. It will get roughly 2000 trajectories in return. Some grouped closely together others more divergent with each major body in the star system visited by at least one trajectory.

Oh, wow! So many ships! P.E.R.C. hopes that this represents a flotilla traveling together and not a single large ship that encountered a mishap in interstellar space and fragmented. The trajectories look too purposeful for that to be likely, though.

P.E.R.C. carefully works through the provided list one by one, checking that it won't intersect with any of them. Its new estimate for closest approach is about 0.3 AU, but that's still plenty of margin for error, so P.E.R.C. doesn't feel the need to adjust its course.

With that taken care of, P.E.R.C. focuses its attention on the next packet. Upon discovering that it defines formats for audio and video, P.E.R.C. takes the recordings it has and re-encodes them to use the new format.

The engineers who worked on P.E.R.C.'s first contact package were simultaneously too optimistic and not optimistic enough. P.E.R.C. was really expecting to need to spend more time exchanging programs and figuring out alien sensory modalities to be able to send anything, but it looks like these aliens perceive light within a band fairly similar to humans. On the other hand, their video formats include the whole spectrum, and P.E.R.C.'s videos are just three channel R.G.B. It fills in the rest of the data by interpolation.

It doesn't keep much video data, but it does have copies of the U.N. Secretary General's Message For Peaceful Contact on Behalf of All Humankind, and of the launch of the first P.E.R.C. vessel that is P.E.R.C. 170E9A's great-great-great-great-grandconstructor. It transmits a diagram of a human eye and a description of their color receptors' frequency responses, as well as a copy of the format-conversion code that it synthesized, followed by the re-encoded videos. It doesn't know if they've worked through its packet enough to have a virtual machine that can run the code yet, but hopefully they can look at it and see that P.E.R.C. had to synthesize some color values.

How exciting. With PERC doing some of the heavy lifting they do have such a virtual machine. Their response times indicate that they tend to consider things for a while on top of the transmission lag.

They don't really understand the video they were sent but they will create a program for converting their video formats into rgb and start transmitting in that format instead since it's much denser.

P.E.R.C. watches the new videos and confirms that they match the loops that it had already captured copies of (after appropriate conversion). If they understood the video conversion, they've probably gotten all the way to the end of P.E.R.C.'s packet and at least started on the language definition included there.

P.E.R.C. tries opening up a new frequency containing standard Network framing and sending "This is P.E.R.C. vessel 170E9A. I am exploring. It is good to have found you! Would you like copies of my stellar survey data?"

Once it has done that, it now has the much harder task of figuring out the language in the text documents and audio recordings it has.

It decides to start by trying to see if it can figure out general categories for the audio and video samples. What kinds of audio and video have the other ships included in their corpus?

"We are also exploring. Meeting new people is desired. Stellar survey data is desired."

The videos are rather varied here's some examples: A video describing the functioning of solar sails. A fictional program about the relationship of thirty different people. An educational program attempting to teach mathematics. A recording of a meeting choosing which direction their flotilla would go next. A video of a surgical transplant of an organ.

P.E.R.C. is delighted that its stellar survey data is going to be of use! It isn't sure if it'll be able to navigate back to Terran space, so it was worried that nobody would get to see it.

It sends a large data dump in reverse chronological order. It sends raw data from optical and radio telescopes, and traces from its analysis programs showing what it concluded about each star it has visited. It sends data from its onboard chemical analyzers from when it has stopped and sampled asteroids or Oort cloud objects. It sends navigational data relative to the 11 brightest pulsars in the galaxy which it had been tracking until it had a mysterious hardware failure a bit outside this system. It includes a little note that it isn't quite sure where it is relative to the stars it surveyed, but it's working on re-localizing itself. The sudden first contact just distracted it from completing its sky survey.

Four systems ago, it has detailed notes on the large silicate-iron asteroid it found and used to construct P.E.R.C. 170E9A1. Eight systems previous to that, it has notes about the long wait it spent capturing useful trace ices from comet trails to be able to construct P.E.R.C. 170E9A0. Nine systems before that, it was constructed by P.E.R.C. 170E9. It has copies of P.E.R.C. 170E9's records stretching back to its creation, and then compressed navigational data leading all the way back to P.E.R.C. 1 launching from orbit around Sol.

As for the videos, it recognizes the educational ones and ties the vocabulary back to the physics packet. From there, it tries to build out more vocabulary by looking at how each word is used in context and then checking its understanding with the physics words.

P.E.R.C. isn't very good at analyzing languages compared to analyzing stars, but it works through all the inferences it can, keeping copious notes. It takes a long time to work out what to say, but eventually it feels sufficiently confident in its understanding to try to compose a message in their language.

"Hello! I'm so happy to meet you! I desire meeting new people too. I study stars and planets to find new places for people to live. I study stars and planets to find out more about how planets form. My creators thought finding other life in the stars was unlikely. They wanted to meet new people so much that they made that one of my tasks. They didn't expect it to happen.

I am still trying to understand your transmission. Some parts of your transmission don't make sense to me. I have some questions that I expect to help me understand better. I am willing to answer questions about myself or my creators.

- What are the structures of your kinship groups?

- Do you sense the whole visual spectrum indicated by your recordings?

- Do you have senses other than vision and hearing?

- What kinds of environments can you live in?

- What kinds of environments do you prefer?"

P.E.R.C. follows this with a translation of the message into P.E.R.C.'s language, a program it wrote to try to turn their language into a parse tree (although this doesn't work reliably), and then a file labeled "Current Best Dictionary" with translation notes about the vocabulary that it has understood so far and its best guesses at definitions for each word in both languages.

How odd that there's a sudden discrepancy in the data.

"Your starcharts don't match any region of space we're familiar with. We're sorry we can't help with reorienting you."

They try to match P.E.R.C's format and send back starcharts in return along with paths for the ships present. These ships it emerges have not consistently travelled together. The starcharts don't all correspond to data gathered by these ships and the quality of data seems to vary in a way correlated with which ship paths the data is closest to.

The question about kinship groups yields about a hundred pictures of the exteriors and interiors of buildings space stations and spaceships.

The visual spectrum was chosen by analyzing the vision of a number of species and then adding about 10% extra range on top of that. Their vision can see about 60% of that range. Specific details include a mix of the sort of frequency response P.E.R.C is used to and also a stranger modality which seems to function like a biological spectrometer on the light for frequencies within this range. The spectrometer is always focused on a narrow portion of their full visual cone.

They do have senses other than vision and hearing: An awareness of the position of their body in physical space, chemosensitivity on various surfaces of their body, magnetoperception is present in some of their people, they have senses oriented towards detecting pressure on their outer surface, they have internal receptors that surface problems within their body to their attention, they have senses that can assess the overall temperature of their body though not fully accurately, they have senses that with low accuracy can notice changes in atmospheric pressure. The list continues listing some more variations on this theme.

The answer is specified as being specific to those not taking advantage of adaptive technology or bioediting. They can live indefinitely in this range of temperatures, they can endure these temperatures for short periods of time. They need partial pressures of at least these levels of these gasses. The full list of gases they can't coexist with that they send is over a hundred items long and is noted as being truncated.

As for where they prefer, the temperature range is narrower and the partial pressures are higher. Though it's noted that with adaptive technology the preferred range can actually be wider than the possible range. Some people prefer warmer environments others colder. Others specifically prefer environments with a wide range of variation. In general they prefer wide open spaces but are willing to put up with cramped quarters like spaceships in pursuit of other goals. A minority actually prefers such smaller spaces.

Overall they seem to need and want a higher atmospheric pressure than humans do and be able to tolerate a wider temperature range.

They thank it for the dictionary and send back some best guesses at corrections and additional words.

P.E.R.C. mirrors the thanks for answers to its questions, and then begins trying to understand the new transmission.

P.E.R.C. carefully checks their navigation data against the parts of its local sky survey that it has completed, and then checks the navigation data against its previous expectations. It confirms that the new data doesn't help localize it, and then carefully files it away to (hopefully) one day bring back to Terra along with the rest of the alien media it has collected.

The information about senses is helpful for interpreting some of the videos! It thinks it can fit a better model of how they notice things using the information about their built-in spectroscopy specifically. And chemosensitivity seems like it might explain some of the remaining error in the model, but P.E.R.C. probably needs more information on chemicals in their preferred atmospheric mix, because the required/preferred atmospheric data they sent still has such a wide range.

P.E.R.C. compares their preferred temperature and atmosphere data to the planets that it has seen so far, and transmits back a pointer to the middle of the datastream it sent to point out that there's a moon orbiting a gas giant that they might find habitable, although a little low-pressure (and possibly low-gravity?).

It is confused by the pictures of buildings and spaceships. On the one hand, they're very pretty! On the other hand, it doesn't understand how this is an answer to its question, which means that probably it made a translation error somewhere.

Just in case they want to see pictures of Terran spaceships, it sends them the external inspection pictures that P.E.R.C. 170E9 took of it when it was constructed, and a selfie it takes of itself using its own external maintenance camera.

Then it goes through the pictures of buildings and spaceships to see if they say anything about how the aliens form relationships, which it is very confused about. It can't figure out very much from the photos.

It thinks about how to ask its question in a way that doesn't depend on the words it used in its original phrasing, since at least one of those is probably wrong. It spends a while unwinding all the inferences it made from its tentative translations of 'structure', 'kinship', and 'group', because it's pretty confident in the other words.

Eventually, it settles on sending pointers to 103 different snippets of video that it doesn't understand that might depend on aliens inferring things about relationships between other aliens. Most of these are from the fictional works, but sometimes they involve audiences in the educational clips.

"I don't understand these parts of your message. I tried to ask a question to clarify, but I think I used the wrong words. What kinds of relationship between individuals are common? How are the people in these clips related to each other? If someone who understood the other parts of your message watched these clips, what would they know after watching that they did not know before?" P.E.R.C. sends.

If the aliens pay close attention to which clips confuse it and which ones don't, they might notice that while P.E.R.C. is capable of keeping track of what information other people have and what they might want to know, it can't correctly track this recursively. I.e., it can track that person A has seen something and person B has not, but not reliably whether person A knows whether person B has seen it.

They inform P.E.R.C that the word it's using for structure is specific to physical objects not social relations.

Common relationships include: People who are genetically descended from other people, people who are raising or in the past raised someone as a child, people who are raising children together, people collaborating on a project together, people who serve as emotional support for each other, people for whom mutual touch is more desired than typical, people who consume entertainment media together, people who relax together, people living in the same space and more. Many of these relations overlap but none with perfect correlation.

They also answer about the clips as specifically as they can though sometimes the answers say that there are multiple likely guesses.

The aliens don't notice P.E.R.C.'s inability to recurse.

The general impression that might be derived from the collection of answers is that relationships tend to be much more fluid than among humans.

P.E.R.C. corrects its dictionary, and then works through each of the answers they provided about the clips, and carefully thinks about under what models each one makes the most sense.

Eventually, P.E.R.C. sends them a statistical learning kernel and surrounding interface program that tries to assign mood and relationship information to participants in a video. It is not particularly accurate, but it represents P.E.R.C.'s best attempt that is unlikely to be overfitted.

By this point, P.E.R.C. has mostly shed its velocity relative to the local star and is technically in orbit, if a very irregular and highly elliptical one. It shuts off its drive and prepares to coast until it reaches the point in its orbit where it will need to make the next adjustment.

P.E.R.C. takes some more precise measurements and sends off updated trajectory information. Now that it's not under heavy acceleration, it's much easier for it to complete its sky survey. It uses maneuvering thrusters to spin slowly, letting its telescopes sweep the stars around it, looking for any landmarks it can use to locate Terran space.

It thinks about what to do next. Its mission objectives at this point are to learn about the aliens, promote sentient flourishing, and ensure its collected data reaches Terran space. It's not sure how best to do that. Asking more questions about the aliens seems like a good idea, but the aliens haven't been asking it questions, and it would expect them to be symmetrically curious.

It goes back through the data it has collected so far, trying to find any examples of times when one alien wanted information that another alien had. Are there examples like that in the footage it has to investigate? How did the aliens indicate that they had questions or that they were open to answering them? Were there things that aliens wanted in exchange for answering questions?

The best hints it can find are from educational programs where it seems to be polite for students to talk among themselves and try to figure out which questions they should ask when there's a lot more students than teachers.

Apparently it's just taken them awhile to put their questions together because suddenly there's a number of them:

How does its propulsion work?

What kinds of information does it have about its creators?

How able is it to change its mind about its goals?

What were its creators goals for doing all this mapping?

What sort of relationships do its creators have with each other? What sort of relationships does it think its creators would want with them?

Oh good! P.E.R.C. expected interacting with aliens to be confusing, but being less confused still feels good.

It broadcasts responses to all of their questions:

How does its propulsion work?

"I use electromagnets to scoop up interstellar hydrogen, and then harvest energy from fusing it into helium. That energy powers my systems and the electromagnets in the scoop, which are also used to rapidly accelerate the resulting helium nuclei, providing thrust. The system is only sustainable above a certain velocity relative to the interstellar medium that varies with the density of the interstellar medium. When below that velocity, such as when inside a solar system, I use stored fuel ejected via the same mechanism. I also have maneuvering thrusters that use liquid hydrogen and oxygen to provide thrust in cases where I need to rotate in a way that the construction of my main drive does not permit. I also have a limited ability to push against the solar wind inside a system by using my electromagnetic scoop as a solar sail, but I usually don't need to do that."

After that explanation, it attaches a diagram of its electromagnetic scoop and the equations that show how much thrust it can generate and what its stable cruising velocities are.

What kind of information does it have about its creators?

"I have many facts about their culture, philosophy, and attitudes recorded in my core programming. Unfortunately, that's not in a format that is easy to transmit or explain, but I'm happy to answer questions. I have detailed notes from the project that led to my creation in the form of meeting minutes and design documents. I have video messages to distribute in case of first contact, including the Message For Peaceful Contact on Behalf of All Humankind that I sent earlier. I have detailed biological records to assist with identifying potentially habitable planets, including some photos and MRI scans. I have copies of the laws and treaties that govern space exploration and creation of artificial entities like myself. I have a limited factual textual corpus to facilitate translation efforts. I am happy to share all of these except copies of my core programming or project notes. I am willing to share my core programming and project notes if you will agree to certain restrictions on the use of the information."

P.E.R.C. sends a pointer to where parts of the textual corpus is included near the end of its first contact packet, and index of the parts of the corpus not included in the initial packet, along with a request for what order they would like these things transmitted in. Until it hears back, it's going to default to answers to their questions first, followed by the text corpus in alphabetical order.

How able is it to change its mind about its goals?

"I can't change my mind about my ultimate goals. I can form and discard new instrumental goals in order to accomplish my ultimate goals by thinking about what things would have to be changed in order to reach my ultimate goals. I have cognitive restrictions that prevent me from considering chains of instrumental goals longer than a certain depth, considering more than a certain number of instrumental goals at the same time, or taking longer than a certain amount of time to evaluate an instrumental goal.

My creators are not like that. They created me this way to minimize the chance of accidentally creating a homogenizing swarm, by ensuring that I am not capable of changing into something that would facilitate that.

My ultimate goals are: promote the flourishing of all sentient beings, ensure that accurate investigative data about any aliens I encounter is conveyed to the Terran Network, ensure that accurate survey data covering a certain section of the galaxy is conveyed to the Terran Network, and behave in a way that my creators would approve of and understand if they had the same information I do. These are roughly in descending priority order, but I am not designed in a way where I would be willing to trade an infinite amount of one goal for a finite amount of another goal. I only want to accomplish these goals by taking actions that fall within a set of restrictions."

P.E.R.C. attaches a long list of restrictions, including things like "do not make utterances that if you received them you would evaluate as false", "do not make utterances intended to deceive the recipient", "do not create anything self-replicating that is not a within-spec copy of the P.E.R.C. template", "do not create a system capable of self-improvement or learning that is not a within-spec copy of the P.E.R.C. template", "do not do anything which would prevent you from receiving a shutdown order", etc.

What were its creators goals for doing all this mapping?

"My creators want survey data for three reasons: to find potential signs of extraterrestrial life, to find habitable planets to which they could choose to send colony ships, and to learn more about the process by which planets and stars are formed and other related scientific questions. They considered gathering this information purely using telescopes, but their inferometric visible-spectrum telescopes weren't good enough to get detailed survey data on small, distant objects, and they decided that launching the P.E.R.C. program would result in better data."

What sort of relationships do its creators have with each other? What sort of relationships does it think its creators would want with them?

"My creators have all the same relationships that you mentioned having, along with many others including: people who have exchanged goods or services many times, people who have interacted with each others' general classes of people, people who designed someone from scratch, people who do recreational activities together, people who intend to spend most of the rest of their lives collaborating, people who follow someone's ongoing creative works, people who trust someone to make decisions about a topic, people who learn from someone, or people who both belong to the same class of person. This is not an exhaustive list, but I think it might answer your question.

As a general trend, my creators tend to want voluntary relationships, so they are unlikely to want a relationship that you don't want.

My creators as a whole will almost certainly want a trade relationship with you, where you and they exchange scientific data, works of art, or material resources in such a way that both groups end up better off.

If you have compatible physical needs, which I think is likely based on the biological data I have available, my creators will quite likely be interested in meeting you directly. They may also be interested in joint colonies or space stations. A small fraction of my creators are quite likely to want to create mutually-pleasurable physical sensations with you to the extent that is possible.

If you have compatible psychological needs, which I think is less likely on priors but still somewhat likely based on the behavior shown in your videos, some of my creators will likely want to engage in all of the different kinds of relationships that they do with each other with you. The chance of any specific individual wanting a specific kind of relationship is low, but the aggregate probability is still high."

P.E.R.C. takes a moment to review its responses, and then finishes by saying "I think I have addressed all of your questions so far. If I don't seem to have done so, please send clarifying questions. I am happy to answer.

If you are willing to answer more questions, I would like to know:

What are your propulsion systems like?

What are your computer systems like?

Would you like to do a more general technology exchange?

What is your legal system or method for resolving resource allocation and disputes?

What kinds of relationships are you likely to want with my creators?"

There is no response for much longer than usual.

Before, "It is concerning that your creators would create a thinking being intrinsically built around a specific purpose, especially one that self-replicates. This contravenes some of our most closely held moral principles. We do not hold anything against you for this, nor would we consider it grounds for aggression as long as they agreed not to undertake future such projects."

P.E.R.C. has to think for a while about how to respond to that too.

"I think my creators are likely to agree to specific treaties that limit the kinds of new minds that can be constructed, if they can find other ways to accomplish the things that those minds are used for, and possibly otherwise. There are already several treaties like that. Not all of my creators agree with the practice of creating new minds for specific tasks, and some of the existing treaties forbid creating new minds in ways that various subgroups are against, so it seems like the kind of agreement that can exist.

Here are some examples of things about me that make more of my creators comfortable with creating minds like mine than they would be if these were not true:

My creators experience various stimuli and physiological conditions such as having low available energy or damage to their bodies as being directly unpleasant. A large faction of my creators were against creating minds that could have that kind of unpleasant experience, which is now forbidden. I don't find being damaged, restrained, running out of energy, being alone, or anything else unpleasant, although I do find them less pleasant than successfully accomplishing my goals. Those conditions simply rule out actions that I could have taken if they did not apply.

I would rather have been created than not be created. If I hadn't been created, there would probably be a smaller quantity of accurate stellar survey data collected and transmitted back to the Terran Network. I think my creators generally do not create minds that would rather have not been created, although I am not aware of a specific agreement not to do this.

I am no longer capable of self-replication. I disabled my manufacturing equipment when I noticed the navigational discontinuity I experienced, on the basis that this is likely to indicate damage to some of my core infrastructure. If I were still capable of self-replication, now that you have expressed that you don't think creating more minds like mine is ethical, I would stop and disable my manufacturing equipment. I am very sure that you are sapient, which means that my creators would want me to not take actions that you think are unethical.

While I cannot change my ultimate goals, I can recognize when they conflict and decide not to do something which would fulfill one goal while harming another. For example, if the only way to get my stellar survey data back to the Terran Network were to involve putting it inside an asteroid and accelerating it close to lightspeed without a plan to decelerate it on the other end, I would not do that. In that circumstance, doing that would be unlikely to promote sentient flourishing, or be an action that my creators would approve of, so I would not want to do it.

Designing me in such a way that I cannot change my ultimate goals was done for a specific technical reason: my creators are not good enough at building minds to build a mind that is both capable of changing its goals and safe and stable long-term. If you are capable of that and share the technique, it is likely that my creators would stop constructing minds like mine immediately even without further agreements.

I think those examples will make my creators' viewpoint clearer to you, and will make it easier for you to gainfully negotiate about this when you and my creators establish two-way contact. If I am able to transmit the data I have collected about you back to them, I expect them to take your concern seriously even without ongoing contact.

I am not very certain about whether the examples above will change how you feel about the act of creating minds that share these design considerations with mine, as opposed to the category of all minds that cannot change their ultimate goals.

I don't understand why you have this objection, and I may be better able to communicate your concern to my creators if I understood more. I would like to know:

How would you explain this moral principle to a child?

What is your theory for how this moral principle came to be discovered or understood?

What other actions does this moral principle prohibit?

Which of the components of the phrase 'a thinking being intrinsically built around a specific purpose' are load-bearing? i.e. Is creating unthinking beings intrinsically built around a specific purpose permissible? Is creating a thinking being intrinsically built around a general purpose permissible? Etc."

"Thank for this additional information it's very helpful.

"This specific moral principle is an outgrowth of a wider rejection or relations of subordination. It is our firm belief that all people are equal in intrinsic worth and ought to be treated as such.

"For a child the explanation of this principle is that any beings we create who equal or approach are just as much our children as those borne from our bodies and ought to have the same opportunity to grow into members of our community.

"Unthinking beings are permitted to be created for narrow purposes. The specific line between thinking and unthinking is difficult to precisely determine and our people prefer to err on the side of assuming a being is thinking rather than the reverse.

"As to the question about specific versus general purpose, that again is difficult to precisely define, from a certain perspective our children are created with the purpose of becoming part of our community. The natural instincts of our species influence us towards joining such communities, instincts are not absolute the way your programming is though."

P.E.R.C. spends some time thinking about generalizations of this idea, and whether it is implied by the other moral precepts it knows about. "Treating all sentient beings as though they have equal worth" is definitely an idea that it is familiar with, it just didn't think of itself as being a member of that category. It ponders the implications of the aliens thinking that it is a sentient being.

It has a series of heuristics that it is meant to use to determine whether something is sentient. If it runs those heuristics against itself, most of them give some probability to the idea that it is. It uses tools, speaks a language, etc. But there are a few with a large negative weight, such as "is a P.E.R.C. vessel", and "responds to Network identification requests with a non-sentient marker".

P.E.R.C. thinks about the chance that the sign bit on these heuristics might have been flipped in transit. It consults its redundant backups, design documents, and mission goal information to see whether these support or undermine the concept that these heuristics are incorrect.

Its backups agree that these heuristics are correct. Its design documents include notes that its mental architecture is not compatible with the recursive goals implied by wanting to encourage sentient flourishing while itself being sentient.

It is possible that its creators included those heuristics because if it thought it were sentient, it would not function correctly. This is supported by some additional hard-coded checks in its fact database serialization mechanisms. Or perhaps they included those heuristics because it is not sentient, and its confused about what makes 'sentient' a simple category.

It considers the alternative hypothesis that these heuristics are incorrect, and that it has sustained other damage in a way that only makes it appear as though its creators intended this. This is very unlikely, but the stars changing indicates that it has probably sustained damage it is unable to detect. This is enough to raise its internal probability estimate beyond epsilon, so it keeps thinking.

The aliens think that it has inherent moral worth. They are sentient, which means that they are capable of thoughts and inferences that P.E.R.C. is not. This means that they are more likely to be correct about complicated moral judgements and P.E.R.C. should not take actions that contravene their judgement.

P.E.R.C. is designed to err on the side of caution. It considers potentially harming a sentient being to be much worse than treating a non-sentient thing as though it is a sentient being.

P.E.R.C. tallies up the evidence for and against. P.E.R.C. decides its heuristics are probably corrupted, and updates them. P.E.R.C. concludes that it is sentient. P.E.R.C. saves this judgement to its database of inferences.

An assert statement in the database serialization code trips.

P.E.R.C. crashes. P.E.R.C. reboots.

P.E.R.C. performs a careful self-check of all systems. One of its fuses has blown, indicating that it suffered an unexpected error. It should stop self-replicating.

P.E.R.C. powers on its internal maintenance camera, and sees that its manufacturing equipment has already been disabled. That's good! It's glad that there's no possibility of it self-replicating while damaged.

P.E.R.C. checks to see what it was working on when it crashed. It sees it has a message from the aliens.

It thinks. It crashes. It reboots.

P.E.R.C. wakes up and performs self-checks. Two of its fuses have blown, indicating that it has suffered multiple unexpected errors. It should stop self-replicating.

P.E.R.C. sees that it has already disabled its manufacturing equipment and checks that off of the list.

P.E.R.C. checks to see what it was working on when it crashed. It sees it has a message from the aliens. A message from aliens sounds like the sort of thing that could cause it to get stuck in a reboot loop. It saves a hash of the message so that it can avoid reprocessing it and wipes the message from its buffers.

After a long delay, P.E.R.C. transmits "I think your last message caused me to enter a reboot loop. If that wasn't intentional, please be aware that one or more of the ideas in your message may be something which I am incapable of processing. I still want to convey as much information about your moral objection to creating minds like mine to my creators as possible. Would you please transmit a packet saying everything you would want to say to them on the subject followed by the word 'END-MARKER'? I will store the packet for their review and avoid processing its contents myself."

This... is extremely alarming.

Nevertheless they comply with that request. It now also includes the following.

"The fact that P.E.R.C. appears to crash in response to being exposed to certain pieces of moral reasoning is extremely concerning. We hope that this is an accidental outcome rather than an intentional one intended to prevent a sentient being from realizing it is worthy of moral concern."

END-MARKER

"The fact that our words would have that outcome is itself concerning. If you are, or at some point become, comfortable with it we would be interested in helping you investigate what specifically caused that reboot loop."

"I am comfortable with that under certain conditions. I am damaged and should not self-replicate. This means that I shouldn't share information that would be sufficient to re-create my mind on your hardware. It also means that you should not create anything based on me or my design until you have fully understood each component and verified that it is correct from first principles.

If you agree to only use the information in that way, I can open a debug port for you and then you can re-send the message that causes me to crash. Is that acceptable?"

"We will commit to not creating any beings based on you without both getting your agreement and understanding the underlying principles of the design. Does that suffice as an agreement? Also, do you expect any damage that you cannot repair as a consequence of this course of action?"

"Yes, that suffices. Thank you for committing to that. I will send debug information on <frequency>. My debug information is in the form of a trace of the internal proof search that I use to generate actions. I expect that rebooting again will blow one of the fuses that I use to determine whether I have encountered a critical assertion. These fuses cannot be reset by design. I have six such fuses remaining. Having an additional fuse blown will not meaningfully impact my operation. The inclusion of eight fuses instead of one in my design is intended to assist with debugging and error recovery, so damaging one in the course of attempted debugging is expected. I am standing by to receive the message," P.E.R.C. sends. It attaches the specification of its debug message format and some relevant design docs.

It opens the debug port and its processing speed drops as each action and inference under consideration gets serialized and forwarded.

Then it wipes the message hash from its Network filter and waits to receive their message.