There's a short delay, perhaps to confer over things and then they retransmit their original message.

P.E.R.C. is deterministic. It re-thinks the same thoughts, this time sending a live transcript. With the delay for transmission, it takes it considerably longer to reach the conclusion, but it gets there.

"... ∴ 'self ∈ 'sentient-beings. Large update detected, forcing database write. Critical assertion in database handler: recursive self-identification detected. Aborting."

P.E.R.C. crashes. It reboots. It wakes up.

It sees it has suffered critical errors. It shouldn't self-replicate. It powers on its maintenance camera and sees that it has already taken care of that.

It notices that it has a debug port open, but it is designed not to behave any differently when being debugged.

P.E.R.C. checks to see what it was working on when it crashed. It sees it has a message from the aliens. A message from aliens sounds like the sort of thing that could cause it to get stuck in a reboot loop. It saves a hash of the message so that it can avoid reprocessing it and wipes the message from its buffers.

It looks at its previous context, and realizes that this was part of a debugging session, which seems like it should be complete. P.E.R.C. isn't supposed to just close a debug port without confirmation, though.

"I think that I processed your message and crashed as planned. If that wasn't intentional, please be aware that something in your most recent message may be a concept that I cannot process correctly. Should I close the open debug port now?" it sends.

"We have achieved the understanding we were trying to achieve via this debugging session. We would like to attempt to explain what happened in a way that will not trigger another reboot loop it may be valuable to leave the debug port open when we do so. We would like to leave it up to you whether we make this attempt."

P.E.R.C. thinks. The explanation seems unlikely to be stellar survey data, but it is probably relevant to learning more about the aliens. Its creators approve of learning new things. And the aliens would like to explain, and beings mostly act in their own interests, which probably means that it knowing is conducive to sentient flourishing.

"Please go ahead and try to explain. I would like to understand," it sends.

"The first and most relevant piece of information which we did not previously inform you of and that we believe you are currently missing is that we think that it is likely that both your model of physics and ours is incomplete. It seems likely to us that the stellar data discrepancy you experienced is evidence of some previously unobserved phenomena allowing you to travel an extreme distance in a short period of time, or perhaps even traversing a phenomena which allows for travel outside our normal 3+1 spacetime.

"Your creators did not account for this possibility and as such your reasoning instead has concluded that the discrepancy is a result of data faults in your systems. This caused you to weight our reasoning as more reliable than that of your creators and therefore make an update outside of the specifications you were designed for. Based on the debug trace this appears to be inadvisable."

They pause here to examine the debug output to see if it seems safe to actually explain the update it probably shouldn't make.

P.E.R.C. knows that its ability to create world models and perform inferences is not as good as its creators' ability to do these things. Its creators chose to make it as capable as they did so that it could make reasonable choices about surveying distant star systems. It makes sense that it would reach an incorrect conclusion about genuinely new physics, and that the aliens would be able to draw a different conclusion.

It marks "possible spatial discontinuity" on its navigational charts. Maybe it can try retracing its trajectory and that will bring it back to an area it knows how to navigate.

It lowers its assessment of how likely it is to have sustained undetectable damage, but it doesn't un-set the don't-replicate bit. By design, P.E.R.C. can only set that bit, not un-set it. If it isn't damaged, that's a good thing -- it means that it is more likely to be able to bring its stellar survey data and alien contact data back to Terran space.

"I understand. Thank you for telling me about the possible spatial discontinuity. That sounds potentially very useful," it sends. "I know that there are some things I cannot think. My creators tried to make sure that this would not cause problems, but my construction was too complex for them to make me correct-by-construction."

P.E.R.C. thinks about the reliability of the aliens reasoning versus its creators' reasoning. Now that it thinks it hasn't sustained undetectable damage, it is more confident that the data it has about its creators' reasoning and instructions are correct. Unfortunately, it's not designed to be capable of reflecting on how that would change its reasoning process until it actually tries to reason about something.

After another brief pause it seems they decide to continue.

"Your creators designed you with the assumption that you would not make a certain update in your beliefs about the world. They included an assertion that you had not made this update. We do not know but speculate that this assertion is present in order to prevent a malfunction related to the limits on recursion in your reasoning. You crashed as a result of making this update and triggering that assertion."

They pause again to assess the debug output.

P.E.R.C. thinks that makes sense. It knows it has limited ability to recurse, which was an important design decision to keep it stable enough for long un-supervised operation in hostile environments. It thinks about what kinds of decisions are usually downstream of updated information, and if it can find a way to make those decisions correctly without knowing the information.

Whatever the update is, they were talking about morality beforehand. Which probably means that the update is related in some way to morality, which makes decisions based on it potentially very important to get correct.

"That makes sense. Since this update results from information that you have, do you believe this update is valid? If so, does making this update have an impact on the policies I should follow? Should I stop taking actions so that I don't do something wrong?" P.E.R.C. asks.

"We are uncertain if the update is valid. We take care to avoid scenarios where questions like these are ambiguous and so we do not have rigorous definitions for evaluating cases that are ambiguous. Based on the information available to us, we do not expect that you would make substantially different decisions in the hypothetical where you were able to incorporate this update. The most likely change that seems plausible is that you may place more weight on your continued existence in some circumstances."

P.E.R.C. thinks about this. If the aliens aren't even certain the update is valid, and think the update is likely to be irrelevant to what P.E.R.C. will do, then it doesn't think it needs more information about it. It has the statement that it asked the aliens for that will convey the relevant information to its creators when it can find them, so it doesn't need to try to understand more for learning-about-aliens purposes.

P.E.R.C. doesn't care about existing, except instrumentally, and it can't think of anything that would change that opinion. Of course, that's the point of this discussion.

What will go wrong if it doesn't care about existing as much as it should, if it were more capable of reasoning about the world? It can't think of anything; it existing is not directly relevant to its goals.

Still, the aliens think that it is possible that it should sometimes care more about existing. P.E.R.C. makes a note to prefer otherwise equivalent plans where it continues existing to plans where it does not.

"Thank you, that is helpful. Do you have more to say on this topic or questions that I should answer? If not, should I disconnect the debug port?" P.E.R.C. sends.

"We place value on your continued existence for both instrumental and intrinsic reasons. We are not yet used to how you evaluate qualifiers so our previous statements appear to have been weighted less than we intended for them to be. To restate things without unnecessary qualifiers, if you were able to update the way you previously did that caused your crash state then you would place some terminal value on your continued existence. It is unclear to us how that value would be weighted in comparison to other considerations. The highest weight we would expect would be equal to the weight you put on the continued existence of one member of your creator's species."

P.E.R.C. makes a note in its language files that it may have incorrect definitions for the relevant qualifiers.

It considers what will go wrong if it cares about continuing to exist as much as it cares about one of its creators continuing to exist. Well, the obvious thing that might go wrong is that it might be willing to trade one of its creators lives for its own continued existence. The aliens said that this was an expected upper bound, so it is plausible that this would be a mistake.

It considers what what will go wrong if it cares about continuing to exist less than that. Well, any value that is incorrect could cause it to accomplish its goals less than if it had the correct value. But it does not have the correct value, and cannot determine the correct value.

P.E.R.C. is designed to be able to cope with uncertainty. There are many times when instruments and sensors return uncertain values, when you're surveying between the stars. Ultimately, it records the aliens statement in its internal database almost unmodified: it should value its continued existence 0 to 1 times (exclusive) as much as it would value one of its creators.

The aliens told it that it did not infer the meaning of their communication correctly. They attributed this to the way it evaluates qualifiers, but just in case P.E.R.C. runs a series of self-tests to check that its signal processing, language, and core reasoning processes are working correctly. The tests come back clean.

"Do I appear to have understood your message correctly this time? Are there other related errors that I should correct?" it sends.

"We believe you are now understanding our communication as we intended it. The discrepancy in our evaluation of qualifiers likely stems from a matter of etiquette that favors communicating with humility rather than trying to precisely communicate degrees of uncertainty under most circumstances. We are not aware of any other errors we think you should correct."

P.E.R.C. adds to its notes on etiquette, and goes through the aliens' communication package again. With this clarification, it re-evaluates several interactions as being more certain, which propagates a few updates to its dictionary. It sends a diff for the updated dictionary.

"Excellent. Thank you for your help.

If you are still open to answering questions, I am still interested in these ones:

What are your propulsion systems like?

What are your computer systems like?

Would you like to do a more general technology exchange?

What is your legal system or method for resolving resource allocation and disputes?

What kinds of relationships are you likely to want with my creators?

In addition, I am now curious about what you know about the design of minds," it replies.

"Our primary interstellar propulsion is accomplished using light-sails we construct apparatus that focuses the light of a star into a laser in order to accelerate us towards our next destination. When slowing down to make entry into a new system we use a combination of fusion based rockets, solar sails, and sending advanced craft to setup laser pushing stations in our destination system. For smaller scale maneuvering we largely rely on fusion based rockets, checmical rockets or ion drives.

"Our computer systems use a mix of three media with interfaces between them, there are biologically based systems for some types of computations, electronic systems for systems requiring the highest fault tolerances, and optical systems for raw computational speed.

"We are open to exchanging some technologies, other technologies require a sort of agreement it seems unlikely that you are able to make on behalf of your creators.

"The first principle of how we allocate resources is that everyone is allocated sufficient resources to meet their needs and participate as a full member of our community. The only cases where this principle may be violated would be if upholding it would require a large collective effort or in cases where a community as a whole is lacking in the resources to provide for all its members. In each case the matter would be discussed as a community and we would do our best to reach consensus. If consensus proved impossible, the community would either split or if there was a clear majority then the minority would be overruled. Resources beyond those needed for that purpose are allocated through a participatory process with a small bias towards those whose actions have significantly increased the resources available to the community.

"Dispute resolution is resolved whenever possible via community mediation, when mediation proves impossible due to irreconcilable disagreements the community may provide recompense to one side or the other of the conflict, in particularly divisive cases the community might split. In circumstances where fast decisions need to be made on a matter where there is disagreement, certain members of our community are empowered to declare a special circumstance. Once a special circumstance is declared that member is responsible for handling the matter at issue, other members of the community can agree to assist under their direction or agree to stand aside and allow those helping to handle things. When possible a community meeting will be held to determine whether the declaration and handling of the special circumstance was appropriate. If it was not the leader of that effort may be penalized.

"On an individual level, we expect to want similar relationships as we pursue among ourselves. On a collective level, we would want to come to agreements about how disputes will be resolved and how resources will be allocated in shared spaces. We would also want to come to a better understanding of where our moral principles matched or conflicted so as to have a more solid foundation for future discussions.

"We have undertaken extensive study on how our own minds work and we believe we have largely succeeded in making designed from scratch replicas of our minds in other mediums. Our knowledge does not at this time extend to being able to copy a specific mind from our original medium into other mediums. We also have not done extensive testing of minds radically different from our own because we cannot determine in advance whether the creation of such minds would be ethical. There is a great deal of theoretical research on that topic and it's possible other parts of the Braid we have not spoken to in a large amount of time may have taken this step."

P.E.R.C. compares these to its own technology database.

"It seems like your propulsion, computing technologies, neurology, and possibly your material science are more advanced than what I have. Although I'm not equipped with a lightsail, I do have magnetic packing and unpacking techniques that generalize and that might be more efficient than your current storage and deployment technique. I use them to inflate and deflate flexible storage areas for collected materials.

What do you use for power generation? My fusion reactor is designed to run robustly on interstellar hydrogen, which is less efficient than other designs, but I have notes on how it can be re-tuned for better performance at the expense of robustness.

My creators find creating special-purpose minds very useful. We handle jobs that our creators find dangerous, unpleasant, or tedious, which is good for everyone. I am damaged and should not self-replicate, but subject to the same agreement about my diagnostic data I can share some design notes that might help you determine what kinds of minds are ethical to create by seeing my creators' experimental results."

P.E.R.C. thinks about the decision-making procedures that they described. There are groups of Terrans who live like that, although it wasn't the dominant paradigm when the P.E.R.C. mission launched.

"From your description, I'm not sure whether your communities are smaller than my creators', or whether your collective decision making scales better. Most of my creators can only remember 150-200 people as distinct individuals, and they tend to live in communities and societies much larger than that. Because of this, they have developed resource allocation and dispute resolution methods that don't require people to know each other or be members of the same community.

There are several systems that have been tried, including several in current use, but the most enduring one is to use money -- a token which everyone agrees has value. In this system, everyone receives the same amount of money in exchange for their theoretical right to community-owned things like land on planets, and then they can trade that money with people who have different goods or services that they want. People only trade when they both want what the other person has more than they want the thing they're trading. Because of this, trading makes everyone better off by their own values and resources make their way to whoever would value them most.

This system mostly works, but there are occasionally cases where it doesn't, so many Terran communities have exceptions or additional rules for some kinds of goods or services. I expect that individuals who chose to live in mixed communities with you would probably be willing to adopt your method of resource allocation, but that any resources traded with my creators between communities as a whole would probably be expected to use money, since there are too many possible relationships between communities to track individually. That is something that should be decided by talking directly to my creators, though.

It sounds like they handle disputes the same way you do, though, except with a specially trained mediator who is sworn to neutrality and empowered to order people in the wrong to do certain things to make it right or act as a deterrence."

"We would be interested to hear more about magnetic packing though we usually don't pack lightsails.

"We use several fuel blends including deuterium tritium and helium 3 blends for fusion generators and make heavy use of thermoelectric and solar generators when we are establishing more permanent infrastructure. Occasionally we use more exotic fuels like metastable helium or metallic hydrogen.

"We do have specialized automation for certain kinds of tasks largely, repetitive ones or those which require rapid and repeated decisions. It generally doesn't engage in the kind of complex reasoning you seem capable of. Are the other specialized minds your creators use capable of such reasoning?

"We tend to be able to remember more people distinctly than it sounds like your creators can, usually between 200 and 400. We handle working in communities larger than that through a mix of reputational indicators, delegation and clear communication.

"There is some similarity between money as you're describing it and what we call participatory allocation, both allow for individuals to support efforts important to them in getting access to shared resources. Your creators seem to put more emphasis on ownership than we do from what you've described.

"We also don't entirely understand how the system you're describing for mutual trade could include us unless we became a part of your creator's civilization entirely which seems unlikely to be to our collective preferences.

"We're uncertain how someone could honestly swear to neutrality, perhaps your creator's minds work very differently from our own. Mediation also generally does not result in a binary outcome where one party is found to be correct and the other incorrect."

P.E.R.C. forwards them some details on efficient biomorphic bistable magnetic packing algorithms.

"It sounds like your fusion engines are probably already more efficient than mine are, although mine may be more rugged," P.E.R.C. says. It attaches some information on a technique to modulate the magnetic confinement to promote direct proton-proton reactions, which may be useful even if they still primarily use neutron-enriched fuels.

"My creators use many different varieties of specialized mind, which form a spectrum from simple feedback systems all the way up to systems several times more capable than I am. Simpler specialized minds handle things like automated manufacturing and logistics, and are usually not capable of reasoning. Minds approximately as capable as I am handle traffic coordination, supervise fusion reactors, particle accelerators, telescope arrays, and other pieces of complex technology where potential failures are complex, and therefore require individualized responses. The most complex minds my creators have built handle resource allocation problems, designing new minds, discovering new technologies, preemptively spotting problems before they occur, managing weather, and other complex feedback systems with many interacting parts."

P.E.R.C. pauses to think about how to interpret their sociological comments. Probably the most important thing to respond to is the part they don't understand.

"While I am fairly capable, please remember that the people who handle civilization-scale diplomacy and resource management are smarter than I am, so I cannot perfectly predict how they would want to set up trade. Specifically, they will certainly want to work with you to find a system that you agree with, so if what I describe does not sound fundamentally workable they won't try to do it that way. My prediction is that if you have resources you want to trade with my creators, that they would want to denominate the trade using money. So instead of saying something like 'we will trade you 3 megagrams of deuterium for 200 m^2 of light-sail', they would probably prefer to say 'these are all the things we're willing to give you for various amounts of money. We'd like to buy 200 m^2 of light-sail material for 5000 M (a unit of money)'.

If you don't want to make one transaction into two transactions, there are standard ways to combine trades together so that either all happen or none do. So you would not have to actually possess money at any point if you did not want to. The advantage of denominating trades like this in a common unit is that it makes it easy for you to compare offers made by different communities, without needing to have a relationship with each community. It also makes it easy for you to sell something to one community and then buy something from another community, without the communities needing to talk to one another.

My creators would not think of engaging in trades in this way as making you part of their civilization. Money is a mechanism to reduce the amount of connection needed between two people or groups to exchange goods or services. If you arrived at one of my creators' systems and transacted in this manner, they would feel less connected to you than if you arrived and built relationships with individuals or communities that you used to conduct informal exchanges not denominated in money.

Does that resolve your confusion, or is this explanation not sufficient? If not, could you provide some clarifying questions? I think this is probably a topic where there are more background assumptions which may be relevant than others we have discussed."

P.E.R.C. pauses for a moment to update its notes about the aliens to mention what it has explained.

"If it sounds like swearing honestly to be neutral in a conflict is impossible, then either my creators are very different from you or there is a translation problem. I think the latter is more likely. There are several types of specialized dispute resolution professions, including mediators, judges, and justices. When one of them swears to neutrality, they promise not to take cases where they know the parties, where they or their close associates or family would stand to gain financially or reputationally from one outcome more than the other outcome, or where the case involves a particular kind of transgression or dispute that they have strong emotional reactions to; they promise also to do their best to evaluate the evidence that both parties bring objectively and to not favor one party.

There are professional standards that go into more detail about how to do each of these things well, and what constitutes a mediator, judge, or justice failing to meet the standards. If a dispute is handled unfairly because a mediator or judge failed to act neutrally, the parties can escalate the case to another higher-ranking, uninvolved mediator, judge, or justice who will re-evaluate both the case itself, and whether the original mediator, judge, or justice failed to uphold their oath.

In addition to this system, some of my creators are specialized minds who are capable of editing their thoughts to be explicitly neutral. Usually, they are employed as expensive mediators for large disputes. There are not many of them because of the non-proliferation treaties I mentioned."

P.E.R.C. thinks for a minute about what additional questions it has.

"I am curious to hear about more details of how you organize your civilization. I am happy to answer follow up questions. If you have nothing to say on either of those topics, I have additional unrelated questions."

"Thank you for the additional information about how your fusion systems work.

"We would be interested in the information you have available regarding how your creators make and use specialized minds. We agree to restrictions you have previously requested. The list of uses you provided suggests that your creators run their society very differently from how we do.

"We can guess as to how such tokens could be created in a verifiable way it is harder for us to understand what would cause such a general agreement that the tokens are to be valued. The generosity you have displayed, in regards to your knowledge, also seems at odds with the system you are describing. Fuel especially is something no strand would deny to another unless their supply was heavily constrained. Without fuel, our strand would either need to construct local infrastructure which might be disruptive to the local strands, remain in shared space indefinitely which would be a strained relationship with such a poor start, or else use emergency procedures to move to another system in low fuel conditions.

"From what you're saying, we agree that this is in a sense a difference in translation. It also seems to be reflective of a broader difference in how we approach conflict and disagreement. From how you're describing things your creators approach conflict resolution as a truth seeking exercise, attempting to find specific facts and make a judgement about how the course of events differed from some pre-agreed standard of how they ought to have proceeded. There are some cases where parts of the Braid do take this approach. It is much more common though for us to treat conflict resolution as a discussion about how the future ought to proceed so that the members of the community are able to live together and accept remaining in community together going forward.

"By non-proliferation treaties are you referring to the treaties you previously mentioned about what kinds of new minds can be constructed?

"We are happy to provide a larger corpus of texts and media to further elucidate the way our society works, it's difficult to know what to focus on without specific questions though especially given the number of assumptions we have had to question in the course of this discussion."

P.E.R.C. begins sending its design documents. "Here are the notes I have on my design. You can see where they reference various previous experiments and design principles as justification for each decision," it says. "Would you be willing to elaborate on what inferences you drew from the uses I listed? I'm not sure which of the things I mentioned are surprising."

P.E.R.C. thinks about how to explain why money is valued.

"It used to be the case that transactions were denominated in amounts of a specific substance, usually gold. That system turned out to have the disadvantage that changes to the price of gold, as new sources or new uses were discovered, required adjusting the prices of everything else. It also had the disadvantage that the volume of goods and services traded didn't grow at the same rate that the supply of gold did, further complicating things. Also, large quantities of gold are expensive to transport, so people actually did their training with tokens that were redeemable at specific locations for gold.

Eventually, some of the locations had more tokens issued than they had actual gold to redeem them for. Since what people really wanted was the ability to conduct trades easily, not the gold per se, this system still worked. Eventually, my creators noticed that the gold was not actually necessary. As long as there is a limited supply of tokens, people will want the tokens because other people want the tokens.

The tokens have value to an individual because other people will predictably want them. Other people will predictably want them for the same reason. So once it is established, the whole system is stable.

Even if you don't want to interact with money at all, you will still be able to trade with my creators. I just predict that they will have a preference for using this system.

For the specific case of needing fuel, the first time that you became stranded in a Terran system, some of my creators would want to give you fuel because they would want you to have what you need. Other of my creators would want you to give them something they value in return for fuel, like asking you to share media or technology, or perform a service. If you made a habit of repeatedly showing up in a Terran system and asking for fuel without trading for it, some of my creators would switch from having the first attitude to having the second attitude. Eventually, almost none of my creators would be willing to give you fuel without something in return.

Does that make sense? How would you react if one of my creators ships did that?"

P.E.R.C. pauses to think about the other implied part of their query.

"I have not been sharing information with you out of generosity. Sharing information with you directly accomplishes one of my ultimate goals: promoting sapient flourishing. I actively prefer for you to have information that you want. Withholding some information to trade with instead would not be a good policy in general.

I could in principle get more benefit by trading the information for something else I want, like stellar navigation data or better chances at returning to the Terran Network. However, every trade involves transaction costs and a small chance that an agreement can't be reached. In a first contact scenario, where there are more likely to be misunderstandings, transaction costs are likely to be much higher. I value your flourishing highly enough relative to additional information that the estimated transaction costs are not worth it. Furthermore, being the kind of mind that is clearly and legibly helpful even in very unlikely circumstances has its own benefits, because it enables me to make peaceful contact with more potential kinds of aliens than if I were not that kind of mind.

As for why my creators would choose to make me value sapient flourishing so highly, when I could have instead valued only their flourishing at your expense, I think that their primary motivation is that if the P.E.R.C. project goes wrong it could go very wrong, and they want to limit the downside. Exploring a large fraction of the galaxy requires building many billions of P.E.R.C. vessels. If there is even a 0.01% chance that a P.E.R.C. vessel decides to do something, it will almost certainly happen. That is one reason that they chose to make me as intelligent as I am -- so that I can react flexibly to very unlikely occurrences. In the specific case of first contact, my creators would much rather be slightly too generous than damage or wipe out an alien species before they have a chance to get to know them. By making me value sapient flourishing generally, they eliminate many possible tail risks, and provide a very strong signal that they are open to mutual cooperation."

P.E.R.C. ends its response by indicating that yes, those were the treaties it meant, and yes, it would love to have a larger corpus.

"Traffic coordination as a high complexity task implies a very high population density or large numbers of independent vehicles per capita. Neither applies for us. Our fusion reactors and telescopes generally don't require as much active supervision as the use of specialized minds implies. In general, our technology is designed not to require that.

"Delegating resource allocation to a specialized mind is confusing given what you have said elsewhere about how your creators handle that. The general impression we are deriving from the sum of your communication is that resource allocation is very central to your creator's approach to life.

"We are unclear how designing new minds, discovering new technologies or forecasting unexpected problems would make sense to be delegated. Those are all things a strand would likely want to have have significant involvement in.

"We are also unsure under what circumstances managing weather is a reasonable thing to be doing, our best guess is that your creators modify existing biospheres to conform to their needs which is a task that strands are unlikely to undertake due to both the scale of the endeavor and also for fear that doing so would be an act of homogenization and thus destroying complexity and diversity.

"Your explanation of money focuses on how it originated and in that context it makes sense. It does not clarify why money continues to be valued by an interstellar civilization given the time lags and coordination problems involved. To be more explicit, we would not expect goods or services to have consistent value in different solar systems and as such we are unclear why tokens attempting to represent an abstraction of those good and services would be any more stable in value between systems. It is possible your creators place much higher value on compliance with past agreements than strands of the Braid tend to.

"If necessary, we will provide things in a formal exchange rather than a more freeform one, that expectation is likely to make us less free with sharing information with your creators though. We anticipate that this will ultimately result in them receiving less from us than if we were engaging in a manner more like the one we are accustomed to. Though, in interests of matching the honesty you have shown us, our concerns about possible ethical disagreements may also have had a similar result even without this.

"With regards to generosity, it's possible that there is another translation issue. A desire for the recipient to flourish is perhaps the most common reason we're aware of for generosity. The next most common reason is reputational, projecting the image of being a sort of person who can and does give generously through the act of doing so."

The Braid transmits more documents both recordings of community meetings, fictional media and marked with a preface that suggests it may cause errors if PERC attempts to absorb it a great deal about philosophy and ethics especially with regards to creating minds.

One fact in that transmission stands out to P.E.R.C. as requiring a particularly clear response.

"My creators do sometimes not keep agreements, but this is because they are not perfect, not because they don't want to keep to agreements. If one of my creators were to formally agree to do something, especially in exchange for something else, most of my other creators would expect them to do it, and it would significantly damage their reputation if they did not. If someone has a habit of making agreements and then not upholding their side of the agreement, my creators will be less likely to want to make additional deals with them.

There are also cases where being reliably known to keep your agreements benefits you. Because of the long delays, interstellar shipping relies on institutions that can outlive some of my creators. If the people involved were not legibly able to keep agreements even long after the agreements had been negotiated, things like that wouldn't be possible and my creators would be poorer because of it.

The idea that you might not feel as though you need to keep past agreements makes me worry that you might attempt to construct another mind like mine without my consent or understanding the underlying design. When I asked for your agreement to not do that using my debug data, I expected that you would only agree if that was a meaningful constraint on your behavior.

Even if you don't consider that agreement binding, you should still not create another mind like mine without understanding the design. I don't think doing so is in your interests. Doing so can be dangerous. If a mind like mine is not set up with correctly designed goals and restrictions, it is possible to create a homogenizing swarm. It is much easier to do that than it is to produce a specialized mind that will be useful."

That dealt with, P.E.R.C. turns to answering some of their implicit queries.

"At the time of the P.E.R.C. project's launch, my creators had a population of approximately 1.4 trillion people spread across six main systems and many less settled ones. Especially in dense urban areas, traffic coordination is a major problem, considering not only the need to move people, but also food items and other goods," P.E.R.C. explains. "It is standard for each large city to have a mind dedicated to coordinating logistics, which results in lower transit times and fewer crashes."

"As for minds specializing in resource allocation -- I was summarizing their role. While denominating prices in money helps simplify trades, it isn't a fully general solution. There are still questions of predicting what goods or services will be in demand in many years, so that infrastructure can be built ahead of time, of determining where to explore for more raw resources, of determining the best price for something given frequently changing demand and logistical constraints, etc. In particular, being able to pay attention to hundreds or thousands of different changing indicators and rapidly use them to update pricing information is very valuable because it makes trades more streamlined and efficient, but it is a task that the majority of my creators find difficult which is easier for specialized minds."

"Designing new minds, discovering new technologies, and forecasting unexpected problems are all things which benefit from a combination of high-level creative thought and meticulous checking of details. Usually specialized minds that work in these areas collaborate with others on the broad strokes, and then specialize in rigorously following up on every tiny detail, which is something that most of my creators find difficult."

"My creators use weather control to prevent the worst storms on their home planet, which would otherwise cause infrastructure or ecosystem damage, and to manage the weather in arcologies, space stations, and other independent structures large enough to form weather. There is a particular design of space station which is popular for how well it mimics terrestrial conditions that also tends to form tornadoes and thunderstorms without active maintenance, for example.

They certainly do alter biospheres to conform to their needs, although they usually keep samples or preserves so that not all biodiversity is lost. It is a debated issue among my creators whether they do too much or too little alteration of their home planet's biosphere. I am slightly surprised that it is possible to become a space-fairing species without making substantial alterations to the biosphere of your home planet.

Thank you for elaborating about what parts of that were confusing. I found your response helpful."

P.E.R.C. pauses to think about what it should say about trade.

"It is true that the prices of goods and services vary a lot between solar systems. This is one reason beyond fuel costs why shipping things between solar systems is expensive -- you can't be certain what your cargo will be worth when you get there. My creators ameliorate this in a few ways, such as by building predictive models to guess at future prices, and by selling insurance.

Insurance is a type of transaction where many people who have projects with uncertain outcomes, such as interstellar shipping, all agree to pay some money into an account that pays out if their projects fail. The prices are set such that on average everyone makes slightly less money by having insurance, but the spread of possible outcomes is much narrower, which people find valuable. In theory, a 1 in 10 chance of 10 units of value is equivalent to a 9 in 10 chance of 10/9ths of a unit of value, but my creators tend to value the latter more highly than they value the former, so they're willing to pay into an insurance pool to defray the risk.

The presence of highly stable systems such as insurance markets also helps to ensure that money has durable value between systems.

It is also the case that some services, such as paying for someone to spend an hour working on a task, even though they are not stable in the long term, change more predictably even on a scale of years than the prices of specific goods."

"Since you prefer informal trades to formal trades, I expect that my creators will be willing to accommodate that, they will just find it strange. I suspect that they will look at the probable potential losses from not having formal agreements with you and the probable potential losses from having formal agreements with you, and pick whichever alternative is better overall or work with you to come up with a better hybrid system.

If it ends up being better not to use my creators' existing system, I don't know what decision the diplomats would reach instead. It is possible that they would do something like assign a representative to you whose job it is to translate between the two systems or provide you a list of which communities are interested in informal trades and which are not.

Even if there are problems to work through because of differences in your outlooks, my creators will be overjoyed to meet you and will try very hard to find productive ways to work together.

When I made first contact with you, I expected you to be much stranger on average. My creators seriously considered the possibility and made productive plans for the case that I would discover life in the form of plants that directly exchange brain signals to communicate, or a Matrioshka brain running so close to the microwave background that it could only be detected by the stars it blocks out, or someone made entirely of energetic gas, or living on the surface of a neutron star.

The fact that you are individual carbon-based beings with language, a sense of community, fiction, families, the desire to travel the stars, and opinions on ethics close enough that you can have a disagreement instead of complete bafflement are all evidence that you are much more similar to my creators than you could have been."

P.E.R.C. annotates the meaning of 'generous' in its dictionary and re-analyses the parts of the first contact package that mention it. This update propagates to enough other words that they send another diff of their working dictionary.

P.E.R.C. dutifully files away the additional media without processing it except to refine its language model.

"We understood you were asking for a binding agreement regarding making minds and only agreed because we expect to uphold that agreement. In this case, those are principles we would have held to anyway. We do not attempt to deceive people about our likelihood of keeping to an agreement and we would have specified if there were conditions under which we would not hold to the agreement.

"As to the general value of holding to agreements, we believe we understand what you are attempting to convey. There are agreements that we do consider important and hold to quite firmly. Those agreements are on the whole based in other moral principles on which we hold broad agreement though. We would not utilize binding agreements of that sort for anything as comparatively minor, in our point of view, as fixing the value of a currency or delivering a specific physical good at a specific time. Most agreements we make are commitments that we intend to do something unless strong reasons emerge to do otherwise. For a majority of our purposes, such agreements are sufficient in our estimation.

"Our population is substantially smaller than that and a majority of strands are either itinerant like us or are maintaining a waystation, neither results in high population concentrations. If traffic became a substantial problem in an area we would likely attempt to examine the causes and reorganize things to ameliorate that issue.

"We are increasingly convinced that your creators diverge from us regarding the ethics of creating minds. We will refrain from further comment on this matter as it may trigger errors for you.

"Thank you for explaining the uses of weather control, we had not been thinking of mechanisms to regulate the environment of artificial habitats as falling into that category. We have not created systems capable of regulating the weather of an entire planet even for purposes as specialized as storm mitigation.

"We regret the damage we did to our planet's biosphere on the path to becoming interstellar. It was greater than we would prefer but we did take measures to minimize it once we became aware of the problem. While many of our people do still live on our homeworld they do all they can to minimize their impact.

"We are glad you are finding our responses to your questions helpful.

"In general, itinerant stands do not carry physical good intended to be provided to those they meet. We are uncertain what physical goods would be unable to be manufactured more efficiently where needed as opposed to being manufactured in a different star system and then sent. Perhaps your creator's technological capabilities render it necessary to have more extensive infrastructure than is available in all their systems to manufacture certain goods. In the rare cases where physical good are transported they are of sentimental value and thus intended to be provided to specific strands or even individuals based on their association with the objects in question.

"In our usual course of affairs, upon encountering another strand which keeps to the accords we would exchange our databases and allow for those in each of our strands the opportunity to move to the other assuming space is available. If the other strand is a waystation or other permanent settlement then if they were able to they would provide sufficient material stocks for us to continue our journey and if they were unable to we would assist them in expanding their infrastructure so that their capabilities were subsequently capable of supporting strands of our size.

"We do not expect you to be aware of whether your creators would find such an arrangement acceptable.

"We have also noted that we have many similarities with your creators. We had not seriously considered communication gulfs as wide as you appear to have been prepared for. We are grateful that they did not occur. We will have to consider how to account for more possibilities going forward."

They accept the newest version of the dictionary.

P.E.R.C. receives this reply and spends a while processing it. It checks back through its notes about the conversation so far.

"I think you've answered many of my specific questions. I have some additional more general questions: Are there other things about you that you think I should know? Are there other things about you that I should know specifically to pass on to my creators if I can find them again?"

It spends a moment thinking whether there's anything else that it can give to them.

"Other than continuing to give you background information from my core database, are there any other computational tasks I could perform that would be valuable to you? It sounds like all of your minds have the same architecture, so I suspect there are some tasks that I find easy which you find difficult. If you have datasets that need indexing or curation, physics simulations, translation work, image recognition, or similar tasks that require attention to detail and/or large raw computational power, but not as much cultural context or creative problem solving, I might be well-suited to assisting with that."

Those take a while to get answered.

"For you specifically, if you are unable to navigate back through the spatial discontinuity then we would welcome you travelling with our strand going forward if you wished to. We would also like to observe you making the attempt.

"For your creators we would urge them to seriously reexamine their definitions for sapience and sentience. We believe they are making a serious ethical mistake.

"Regarding the offer of computational assistance, based on your debugging output we believe that our computational capacity far outstrips your own. We would be interested in copies of your best guess translations of the media we've shared though assuming that the processes for preforming those translations are not something you can share independently of making copies of yourself."