Keltham: "Perhaps.  Why?"

lintamande: "- it just seems like anything you do only in your head, like making a prediction, a god could just - have part of them that didn't have the answer do that, or more realistically probably just do that after the fact but not be influenced by knowing the answer. But humans are weak and flawed and so would err if we tried that, and predicting beforehand makes us less vulnerable to that error."

DeAnno: "I remember `3. No empirical theory can prove itself except by risking its disproof.'"

"Is part of it that if you don't make the theory before the fact of seeing the evidence, it can't have a real risk of being disproved by that evidence? Because if the evidence was already there you knew how it would interact with the theory beforehand, and so there couldn't be a real risk involved?"

Keltham: "More of an issue with mortals than with 'ideal-agents' or, I suspect, gods.  Or maybe it's just the Lawful ones, I don't know.  But an 'ideal-agent', even a 'bounded ideal-agent', if It considers the theory after seeing the evidence, won't be at all influenced by having seen the evidence, in what predictions It says the theory makes.  So while It might not even bother to invent some wrong theory, if the evidence contradicts it, it would nonetheless be the case, that if you asked It what the evidence said about that wrong theory, It would know the correct answer there."

"I nonetheless claim that if you hook up to the Keltham-environment a random input generator that picks three random numbers each independently between negative one thousand and positive one thousand, and let a god passively observe that, the god will learn less than a mortal playing the game with Keltham.  Well, maybe not with the 'timing side-channel' information added, but definitely without that timing information."

"Furthermore, I claim that this mysterious superiority of the mortal, over the god, has to do with a process which includes the mortal predicting things in advance of seeing them, nor may it be done without the mortal making those predictions."

lintamande: "Well, say the god has several theories and isn't sure which is right. If the god doesn't get to choose the numbers then they can't pick numbers that distinguish between their theories. You can figure things out a lot faster if you get to choose what tests you're doing. So being able to choose which tests to do is really valuable, but only if you have - some underlying guess about what's going on, so you can choose tests that show if you're right. If you have no idea what's going on then tests aren't any more valuable than the random numbers."

Keltham: "Yeah, that's basically it.  In terms of what you should be feeling using Probability-Sight, it's that you've got to extend out the predictions, the 'likelihood-functions', from the things that might be true, to possible observations, so you can imagine what sorts of observations might distinguish between things that might be true, and then steer into that region of reality so you get to see those observations."

"In terms of underlying Law, there's a huge amount of math here of 'computer-science', about what determines how many observations you need, in principle, to narrow down between theories in particular classes.  None of which is a practical priority for you to learn right now, because mortals are so enormously less efficient than gods, or so I expect, who in turn are enormously less efficient than 'ideal-agents', so the principled bounds hardly matter, you're going to be buying way way more evidence than a god would need to arrive at the same conclusion.  But among the things you learn from studying those impossible ideals, is that it takes much much less evidence to figure something out, when you're allowed to pick which questions to ask.  It's often the difference, not so much between 'can be done with ten copper' and 'can be done with a million gold', as 'can be done at all' versus 'can't be done at all'."

"Next I ask - is there any wise advantage in looking at a quantity, a 'scalar-quantity' like the amount of time I take to answer?  Why the amount of time, instead of, say, my facial expression and where my eyes were moving?  Not that I was letting my eyes move down the page in order, or anything that easy to catch, but still.  Why focus on the amount of time I took, which collapses down to a 'scalar-quantity', and not, say, the order in which I approached students when more than one student had their hand raised?  Wouldn't the order in which I picked students be a more complicated, structured, interesting thing to try to predict?  Is there any advantage to trying to measure 'scalar-quantities' besides that they could be measured more precisely, using a pocketwatch that I didn't catch anybody actually looking at?"

Alexandre Esquerra: Asking dumb question is better than asking no questions. And so he will let himself be scourged by failure, again. "Even without a pocketwatch, the difference between five seconds and six seconds is noticeable. You can compare questions that received five-second waits to those that received six-second waits, and notice that there are gradual increases in the wait over time... I think that latter is important. You cannot notice a gradual effect changing over time in the order you approach students - without a truly tremendous number of tests, that is. You can in the time. But I do not know how to put that into the language of probability-sight."

Keltham: "Wrong!  But positive feedback for being courageously wrong out loud instead of quietly clutching your error to yourself and leaving me, your teacher, in a silent void of zero evidence about what any of my students could possibly be thinking!  I'm also a kind of thing that requires observations to feed on in order to know things!  Lots of them, even, since I'm not a god!  Yet!"

"There's all sorts of phenomena in the ordering that you could potentially notice right away, if you knew to look for them, and that they were there.  For instance, maybe after you choose a sequence with any big number, I approach you last out of all students currently with their hand raised, and after you choose a sequence of all small numbers, I approach you first."

"But even if that had been true, and somehow important - like big numbers being harder for me to calculate and therefore unpleasant and so I'd subconsciously started avoiding people in the middle of testing those hypotheses - it would have been a harder pattern to notice in the ordering data, compared to noticing how sometimes a number is big and sometimes small.  Actually, even the example I gave of a pattern is really something that got projected down to a scalar-quantity - on a scale of beginning to end, where does my selection of you land inside the set of people with hands raised?"

"If it had been anything more subtle than that - maybe, like, my approaching a new candidate rather than current researcher next, if you showed me any big number - then while it might be very easy to verify that was going on, in terms of updating probabilities, once you noticed anything interesting at all and came up with that hypothesis, that pattern would be a lot harder to notice initially and pretheoretically, compared to noticing a number getting bigger or smaller."

"This is both a useful fact and a cautionary one.  Measuring things in scalar-quantities can give us results that are, in a certain sense, easier to work with - easier to try to notice things about - but that we find it easier to work with scalar-quantities is a fact about us, not a fact about the things.  If you're not measuring the right scalar-quantity, or if the critical measurement isn't a scalar-quantity at all, then you're just - uh, dath ilani proverb.  Somebody loses... a small precious object... a platinum piece, something worth their time to search for... while walking outside at night.  While retracing your path, it might make sense at first to go looking in any well-lighted places along which you'd walked, just in case your platinum piece happened to be there."

"But if you don't find it there, the first time, you need to go get a 'flashlight', a light-generating object, and go looking along the rest of your dark path where it isn't as convenient to look.  It's a comical character, a silly-child character, who insists that they'll go on searching the well-lighted areas because those are the easiest places to look."

"If what you need to watch is in fact my facial expression and not how long I take to answer - then you'd better not get too attached to measuring things in scalar-quantities.  But scalar-quantities sure do make sense as a thing to quickly check out first, if you have a choice of things to look at.  In fact, you might find it so much easier to notice patterns there, that your Science-mind gets out of the habit of even looking at things that aren't easily-measurable scalar-quantities!"

Keltham: "With that caution in mind - scalar-quantities are easily related to each other.  Not just that, easily related to each other by math, which makes it easier to notice all sorts of mathematical patterns in them."

"If you'd started recording my times more exactly, and found that my average delay times were growing over time for every student, but growing at different rates for different students, you could start looking for other scalar-quantities that were growing at different rates for different students.  Then one of the scalar-quantities you checked might be 'the number of previous inputs that student had already written down'.  Then when you found that my delay time was roughly and on average proportional to the number of previous inputs - in a way that was similar across all the students, even as different students had different numbers of previous inputs growing over time - you'd have a critical insight into the structure of the Keltham-environment's behavior."

"A habit of measuring all the even slightly relevant-looking things that come out in scalar-quantities, or in Baseline, 'quantifying' the 'quantifiable-observables', is something that Civilization has in fact found incredibly useful.  When we start messing around with spellsilver, we're not going to be trying a bunch of random crap to see if anything works, even as written in the language of 'chemistry' where random tries are much more likely to do interesting things.  We'll crush a lot of spellsilver ore down to sand, and mix it all thoroughly so every batch of ore-sand we use is almost exactly the same across experiments.  We'll try small perturbations to the process, most of which will be harmful, but we'll quantify exactly exactly how much less spellsilver came out the other side, and compete to come up with theories that try to call those experimental results in advance."

"Even if most things we find make the process worse, those are still facts we can use - especially in combination with other dath ilani knowledge - to build up 'testable-hypotheses' about what's going on inside the spellsilver-refining process."

lintamande: "Am I saying the same thing as you if I say 'it's good to measure lots of things, but it's especially valuable to measure things you can turn into numbers on a common scale, because those are some of the easiest for humans to notice patterns from? So when stuck, measuring your response times was likelier to be useful than noting whether you blinked or not?"

Keltham: "Yup!  Well done at simplifying back into Taldane!  Though, to be clear, it's even more easier for machines to notice patterns in numbers, but you won't have any number-noticing machines for probably a fair while."

Keltham: "But all of that, this entire previous discussion, is something you could also say to an individual trying to understand, analyze, and improve spellsilver refining."

"What distinguishes 'SCIENCE!' from merely ordinary individual Lawful experimentation with the goal of probing for unexpected patterns or distinguishing among hypotheses already formed, is that 'SCIENCE!' is a protocol for stacking the efforts of multiple individuals - the efforts, indeed, of all Civilization.  It's a designed artifact-shadow of Coordination in much the same way as Governance."

"What, do you suppose, would be the key ingredients of 'SCIENCE!' as a multi-agent practice of Civilization?"

Aevylmar: "They get the smartest people to organize it?"

lintamande: "They offer rewards for correct theories." And punishments for incorrect ones. 

Alexandre Esquerra: "Standardization of weights, terms and measures, and requiring universal and public record-keeping." Because otherwise people could pretend to lose less 2s than they, in fact, lost, by selectively reporting only the predictions they did well on.

Peranza: "They do it with prediction markets."  Because he explained those shortly before explaining this.

Carissa Sevar: "You need either a way for people to see and interpret the results of other peoples' individual experiments, like standard reporting of results, or a way to aggregate individual guesses into an overall best guess without anyone having a reason to say something other than their real best guess - if you have that, you don't need everyone reporting their experiments, just betting on reality with the knowledge their experiment gave them. Prediction markets do that, and also do another thing you need, which is a reason for people to go out and do experiments instead of staying home and making their numbers up."

Keltham: "Prediction markets actually don't work well enough at paying people to go out and secretly do experiments.  The market traders are pretty good at guessing when somebody else is just coming in with new information, and the old traders won't sell at their previous prices - the price moves sharply enough that the experimenter often can't make much of a profit off the prediction market.  Like, the current price is 80%, but that doesn't mean you can just buy a million shares at 80%, reveal your experimental results, and resell at 90%.  Once you start buying in massive quantities like that, the price starts moving upward from 80%, because market participants have guessed you have private information."

"It's more straightforward and direct and less clever than that, actually; what happens is that people will just straight-up promise 'impact-purchases' on discoveries they want to see made."

"Like most dath ilani males, I am 'gynosexual' and very 'straight', meaning that I'm only attracted to women, and mostly only if they present with fairly central feminine gendertropes.  Like most such men, I committed to the outstanding bounty for finding a reasonably-completely-safe 'medical-enhancement' intervention that would make me also be attracted to men, because, as all the 'bisexuals' keep telling the rest of us, we're missing out on half the fun.  The collective commitment is up to something like a hundred billion unskilled-labor-hours at this point... about five hundred million gold pieces in Golarion terms, maybe possibly?  That's how rich you would be in dath ilan, if you singlehandedly found a safe way to turn men bisexual, all by yourself with nobody else contributing.  It's one of the biggest, most famous unclaimed bounties."

Alexandre Esquerra: What.

(No comment, this is a class.)

Carissa Sevar: - most men, if their battalion gets deployed somewhere where there aren't any hangers-on around to keep them company, like at the Worldwound, find they're actually into men after all. Or, you know, able to get a blowjob and pretend it's a girl at absolute minimum.

Carissa is not sure this is a valuable observation.

Lady Eulàlia Avaricia de Seguer: "Is this in fact a problem you, personally, want to solve."

Keltham: "It's relatively less important to me now that I have otherwise saturated dating opportunities, but that doesn't mean I'm not curious."

"...I think it would legitimately be way way down on my list of priorities, given the current competition for my priorities list.  Why?"

Lady Eulàlia Avaricia de Seguer: "I know some categorically adequate sex therapists who'd probably take you on out of sheer curiosity. People in Cheliax have also encountered this problem, usually the other way. It does seem like a stupid use of your time but who am I to discourage you on those grounds."

Keltham: "How do those work - is there a spell for it, or a magic-item for that matter?  It's a less stupid use of my time if it's less time."

Lady Eulàlia Avaricia de Seguer: "You drink potions and have a great deal of unusual sex that expands your horizons. I haven't undergone it, if you want details I'll have to call on a friend. And then murder her, I guess."

Carissa Sevar: - would alter!Lady Eulàlia -

Lady Eulàlia Avaricia de Seguer: Yes, absolutely, Chosen of Asmodeus. If anything Taldor's nobles are worse.

Keltham: "I'm going to assume that was a joke even though I can't tell which of, like, three different possible jokes it was.  Specifically, are you supposed to be too honest to just not tell your friend why you're asking, too carefree to care about her putative discarnation, or just signaling the lengths you're willing to go for 'security-mindset'?  Not actually important, don't answer right now." 

"Anyways, I think my current sexual horizons are expanding fast enough, so I will table this issue for probably several months, unless somebody turns up who'd be an incredibly perfect match for me if only I were attracted to men."

lintamande: "You were telling us about Science," says Gregoria.

Keltham: "I was, and in particular about how Science! gets paid-for by Civilization collectively.  Something like the long-standing Quest for Safe Bisexuality Enhancement is not likely to get completed by one person; even the whole corporation that invents it will be building off the work of hundreds of other researchers.  The commitment I made doesn't get paid out just to whatever corporation invents the treatment; it goes to one of several extremely reputable philanthropic-redistribution groups that would, if Safe Bisexuality Enhancement got invented, pay out to purchase the 'impact' of whoever had turned out to contribute the most to getting it done."

"With a genuinely massive bounty like that, there's multiple 'venture-philanthropists' who are playing a multi-decade game of funding promising scientists to work on related investigations in exchange for a portion of their 'impact', and then reselling their shares of 'impact' of people who made discoveries that will plausibly be worth hundreds of thousands of gold pieces later, if Safe Bisexuality Enhancement gets invented with their work having contributed 0.1% of what got done."

"That infrastructure is too sophisticated for Golarion in its present state.  The key point is that, like everything else in life, if you want good Science! you've got to pay for it.  'What you're not willing to pay real money for, you shouldn't complain you didn't get', as the proverb goes out of dath ilan."

"What we'll do on the Project instead is as follows:  If you come up with a truly unique and brilliant idea for refining spellsilver, and, this is the part I worry may end up generating 'drama' - emotional-fraughtness - it's not a brilliant idea that I strongly expect I or anybody else would've come up with anyways given another thirty minutes, you get a larger share of the Project."

"Where the issue here is that, especially once you're speaking the correct language of 'chemistry', there are going to be useful ideas that seem brilliant and that, in fact, anybody else who competently thinks in 'chemistry' language would probably also come up with given slightly more time, and we can't afford to give away 1% shares of the Project every time somebody has one of those ideas."

"I don't have anything better to do about that than appoint myself final judge, and go under truthspell on request to say that I'm being honest and impartial about it.  Tap myself with the Fair Pricing spell, too, in case that also helps show the absence of rationalization and self-serving reasoning.  In Civilization there'd be people who spend their whole adult work-lives just specializing in that one form of judgment."

lintamande: " - this is in addition to, uh, the salary and project-shares already discussed?"

Keltham: "The shares already discussed are based on your expected contribution, as expected by me at that time.  It includes some amount of unique contributions, because everyone on the Project is expected to make some unique contributions."

"But Asmodia, for example, is angling to clearly contribute more than the other tier-1 researchers and more than I expected at the time I made my hiring offer.  If she ends up much better able than anyone else to teach Law to other researchers in my place, she can in fact pull that off."

"Similarly, if one person on the Project proves to be the only person in the first year able to master 'chemical' Prestidigitation, and ends up spending a lot of their time tediously overseeing the refinement of thousands of pounds of spellsilver, that's likewise a bigger and more unique contribution than I originally expected when I offered them 0.1% or 0.2% of the Project.  You can afford to give 2% to everyone like that."

Lady Eulàlia Avaricia de Seguer: She could really stand for this project to involve less openly talking about money. It's so gauche.

Keltham: "Going back to Science.  In answer to the suggestion about getting the smartest people to organize Science! - that follows automatically from offering huge payouts for discoveries that people are sufficiently interested in.  Smart people, being smart, will go where the money is; why would they want to be paid less per amount of effort?  Even Good smart people will usually go where the money is, because that's what Civilization is saying it cares about."

"Also to be clear, if you discover something important that nobody even thought to ask for or expect, plenty of 'utility-buyers' will show up and pay for the 'impact' on that, too; Civilization has done that many times before, because it's important to be consistent and predictable about that sort of thing.  I'll be aspiring to do the same on this Project, which is all that Golarion has of Science! for a time."

"Standardization of measures... I'm not sure what it would mean for something to be a 'measure' that wasn't 'standardized', but, this being Golarion, I'm afraid to ask.  Yes, if you measure something in dath ilan, you would report on that in the same units as everyone else in Civilization uses.  Otherwise people wouldn't build on your results, and that would decrease the amount of 'impact' you got paid for."

Korva Tallandria: Surely there are also smart people who don't particularly care about money. Or maybe she's just too stupid to understand why being smart enough to do Science necessarily entails also caring about money.

- wait, actually that makes sense, if Science is specifically about wanting to understand the universe so you can conquer it. So if you're like Korva, and you don't care about that, then you won't be any good at Science in the first place.

Ugh.

Keltham: "So I'm noticing that I'm sort of bouncing off the question of how to do Science! exactly, because it's too large, and not really the sort of thing that gets taught to dath ilani children in a single organized lesson, we take the pieces for granted -"

"One of those pieces is legit the part where contributors get paid for doing it, and people who make big contributions to Civilization that way get paid a lot.  I'll be improvising that for the Project as it goes, not ignoring it, and everyone should know that.  Obviously whatever Science! you get will be whatever kind of Science! you pay contributors for and incentivize.  Okay, that said, move on."

"Peranza suggested that Science! would use prediction markets.  That's true, but an easy guess, because every part of Civilization uses prediction markets for everything unless there's a specific reason not to.  Where exactly would Science! use prediction markets, Peranza?"

AlterPeranza: "What seems obvious is, to predict which experiments would work?"

(AlterPeranza is less terrified of being called upon by Keltham like that, of having it exposed that she was just wildly guessing based on what Keltham calls the 'meta-game'; she is able to quickly think of a plausible thing Peranza could have been thinking.)

Keltham: "What does it mean for an experiment to work?  You get results.  The results are Reality.  In what sense could Reality 'work' or 'not work'?"

AlterPeranza: "Well, in the most recent case, 'works' is getting to your first NO instead of all the YESes.  You could have a prediction market on which tries would do that successfully."

Keltham: "That's a dangerous way to start thinking - one we're explicitly warned against as kids.  You don't say that some results Reality can give you are the experiment 'working' and some are 'not working'.  Think of the timing measurements.  Willa's and Alexandre's first observations there didn't get them 'NO' answers, but those were far from failures."

"That's the same mindset you'd have to break out of to solve spellsilver refining.  Experiments on spellsilver don't have failed results, or successful results, they just have results.  Every experiment tells you something truthful about Reality, which is the world you are embedded in, it never lies.  Maybe your instruments are broken or don't do what you think they do, and then what Reality is really telling you is not what you think you are hearing.  But that's your own error.  Reality itself never lies."

Keltham: "With that said, Civilization is obviously running prediction markets all of the time on what some particular experiment will say, in its results, or what a hypothetical future experiment will say.  Prediction markets are a fundamental part of doing community experimentation, because prediction markets are a fundamental part of collective 'epistemology' - the way that a Civilization can be said to 'believe' anything apart from the beliefs of the individual people in it."

"For example.  What would it mean to say that an experiment had a 'surprising' result, if there's no prediction market assigning that result a low probability?  Maybe it surprised you personally, but then maybe you were just being stupid and not seeing what others would consider obvious.  Why should Civilization care, if just you personally were surprised?  Why should an 'impact-buyer' believe you, if you claim the results would have been a surprise to society?"

"Or let's say that you make what seems to you like a really incredible discovery, a way to refine ten pounds of spellsilver ore into twenty pounds of spellsilver.  Call the newspapers!  The newspapers are... somewhat skeptical about this incredible claim?  If there's a prediction market, they can check that prediction market to see what Civilization thinks will be the result of duplicating your experimental procedure.  If there's no prediction market, the newspaper just has no idea what to believe and it sounds weird, or so I imagine, and so your possibly incredible discovery drops into the void."

"Or maybe I'm too optimistic about how it works in Golarion, and instead the newspapers print excited stories about your spellsilver discovery, which any real spellsilver scientist knows has to almost be certainly false, but Intelligence 10 people read the newspapers and now they think that's how spellsilver mining totally works and they're confused about why nobody else is adopting this great idea.  Actually that's still too optimistic, actually they'd just stop believing things they read in newspapers, including the things that experts actually did believe.  Or maybe that's too optimistic and -"

"Anyways!  Golarion having no prediction markets, is perhaps alone an 'already-sufficient-explanation' of why Golarion has no Science! and is so far below dath ilan's level of knowledge.  Nobody knows, now, at which point in history Science! was invented - at what point pre-Civilization really started learning collectively rather than as a set of individual experimenters working in isolation or small groups or factions - but I would expect that it only happened after the simpler idea of prediction markets got invented first.  You have to be able to say what Civilization believes, before you can observe or measure how fast Civilization is learning, or figure out how to make it learn faster."

lintamande: Checking with Asmodia - in alter Cheliax like in real Cheliax are the newspapers forbidden from publishing sensationalist nonsense? 

Asmodia: Snap decision - in alterCheliax it's illegal for a broadsheet to say that the Church or Crown affirms something they don't affirm, and that cuts down on the worst of the nonsense among relatively intelligent people.  But it's not illegal to be wrong, so long as nobody gets their person or reputation injured as a result, and nothing being said is treasonous to the Crown or insulting to the Church.  Sevar, that sound right?

Carissa Sevar: They're going to have to produce broadsheets consistent with it, possibly immediately, but sounds about right in the sense that going more dath ilani than that would be uncharacteristic.

Asmodia: Oh.  Right.  They have to be able to produce the broadsheets on demand, if Keltham insists that somebody Teleport right out and get one right away, as a Conspiracy check.  Asmodia should have thought of that.  And they have to look like Taldorian broadsheets, not Chelish ones like people in Cheliax already know how to produce...

...this sounds like a nightmare for the Conspiracy, actually.  Maybe broadsheets were found to produce enough misinformation and riots that they had to be allowed only to a few Crown-supervised offices?  It doesn't make Cheliax look like Absalom, but it would explain why alterCheliax's broadsheets are relatively small and look government-produced.

The alternative would be grabbing some Taldorian broadsheets and rapidly retraining some Crown authors to be able to produce more of the same, with most of the content ready to go and print as soon as Keltham demands it for any particular day.

Ione or Meritxell, comment something that doesn't have anything to do with newspapers?

Security: Korva Tallandria has a newspaper-unrelated question ready, that she wasn't sure whether to ask, Security can prompt that one.

Korva Tallandria: - oh shit, someone is listening, okay -

"Does that mean that political unification and a lack of factionalism are also necessary prerequisites for Science?"

Keltham: "Not sure I'm following the logic behind the question?  You need a single market for predictions, or rather, you'd expect to end up with as many separate Science!s as there are separate markets.  But a single worldwide market follows automatically from 'arbitrage' - if one market is buying a proposition at 80% and one market is selling at 60%, you can buy in the 60% market and sell in the 80% market and make a guaranteed profit and do that until the prices equalize."

Korva Tallandria: Why did people tell her to ask it if it was a dumb question!!

"Well, uh, I guess I was thinking that if different countries are doing their Science secretly, then that's sort of like what you were contrasting it with, the experimentation of a small group or a faction, and I was wondering whether a worldwide effort was a necessary part of the concept. I guess you might not need political unification for a worldwide effort like that, at least on some things, but I don't think it happens automatically - the markets would have to know about each other, both in terms of not being secret and in terms of having the ability to reliably trade information even about things that aren't secret."

Keltham: "Oh, sure, there's all kinds of secret research in Civilization.  Startups working on something proprietary, probably some stuff Governance does that would be socially-infohazardous like scalable-weapons research, and, one assumes, almost everything to do with the Keepers."

"That's just, like, mini-Science! where you pretend to be your own little Civilization.  Obviously if you could have one big Civilization under the Law of Coordination you could also have several smaller ones being governed by the same rules.  The only difference I can see is that any smaller mini-Science! like that would be 'shoulder-standing' the bigger Science! project of the rest of Civilization of which it's part."

Asmodia: Relayed to Korva:  We needed a question for Keltham that wasn't about 'newspapers' while we tried to figure out how broadsheets worked in alterCheliax.  Your sacrifice and obedience is acknowledged.

Korva Tallandria: Oh that makes sense. She is glad to have been of service to the project, and surprised and grateful for the context. She's going to nod and shut up now.

Keltham: "Okay, what else is part of Science!... replication and reproducibility, right, you'd think that would be obvious and emergent-from the rest of the setup, but it got taught to me as a distinct principle so somebody thought it needed separate focus and emphasis."

"If in Civilization you report that some procedure works for refining ten pounds of spellsilver ore into twenty pounds of spellsilver, the prediction markets on the result of a general or generic experiment like that one, don't all immediately go to 100 and pay out.  In this case, even if you took 'video recordings' of your experiment so people knew you weren't just lying - that you had, at least, put in a lot of effort to fake the video - market traders would still be pretty skeptical.  The relevant prediction market would be on what would happen if known, competent, careful, previously validated, third-party professional 'replicators' tried the same experiment."

"And I doubt that prediction market would go very far above 0%, if the claim was that you could refine ten pounds of ore into twenty pounds of spellsilver, assuming spellsilver was otherwise known to be mostly pure metal and not have components that could be drawn from the air or other refining materials."

"Now, if you're the original discoverer, you probably find that pretty annoying, right?  You published your experimental results, and those stinky prediction markets still don't believe you?  So after double-checking your own work - at least, you double-check it first if you're at all sane or smart -"

"Well, mostly, you discover that enormous blatant error you made, where the scales you were using to measure the ore weight were broken."

"But let's say that doesn't happen.  Then you buy up all those prediction-market shares that are trading at under 1% that your experiment replicates.  The market price starts to go up, obviously, but as it starts to get above 2%, more traders start to come in, checking to see if they think they can make a quick 2% profit off you.  Your prediction sounds completely nuts, they figure this is just a prank or somebody making a weird point, so they start to sell at 2%."

"You buy more.  Some of the less confident counter-traders drop out, when they see you're willing to spend that much."

"Sometimes the way the story goes from here is that the probability gets as high as 20%.  If the proposition at stake is an important one, 20% is high enough that an outside 'venture philanthropist' will come in and fund a 'replication' of the research done by professional 'replicators', because the 'impact' of 'replicating' the original maniacal-experiment, if the results 'reproduce', will sell for more than five times the cost of doing the 'replication'."

"Let's say that doesn't happen, because the idea you can refine ten pounds of ore to twenty pounds of spellsilver is such a ridiculous one that your counterparties are still confident.  They keep buying the prediction's price down even as you keep trying to buy it up.  It never gets above 4%."

"Once you've spent a large portion of your savings on amassing a huge position like that, you spend a pretty large chunk of whatever's left, on the actually quite expensive project of paying respected 'replicators' with the 'certs' to actually resolve the prediction market, to test your method.  Which they will do very very carefully, taking videos of everything, that you inspect to say beforehand rather than afterwards if you think they're doing anything wrong along the way."

"Usually the way this story ends is that somebody loses a chunk of their savings and gains a valuable life lesson about overconfidence."

"Sometimes, very rarely, it ends with a lot of shocked prediction-market traders who lost a much bigger collective chunk of their own wealth, and a story that makes all the newspapers.  Your name becomes one of the glorious Science!-heroic stories that inspires the next generation to not just believe what everybody else believes.  World-class professional sex workers will compete to seduce you just to be able to advertise to their other clients that they won that competition."

"Also you can sell your prediction market shares at a 25-fold profit.  But that's not why you did it, really.  The money is just how - how other people can know that it was all real, because real money changed hands."

Alexandre Esquerra: ... "Where does the typical researcher get the funds to pay for all these 'prediction-market shares' and 'replications'?" It's not like researchers are wizard with spells to sell, right; dath ilan doesn't have wizards.

Keltham: "Well, if you are ten years old, and your parents aren't very wealthy and indulgent, and your theory isn't so persuasive that you could convince other 'venture scientists' to go in with you on it, then you are not in fact going to pull this off."

"That's considered a 'feature' of Civilization, not a 'bug'.  It prevents Civilization from being snowed under with futile replication attempts for silly ideas.  You can be poor and persuasive and still get an expensive experiment performed, or rich and unpersuasive, but you've got to be at least one of rich and persuasive."

"That said, let's say that the prestigious replicators earn four times what you do per workday, and it'll take five of them working for ten days to replicate your experiment, plus some relatively lower costs to rent equipment.  Then their cost is on the order of two hundred days of your salary.  If you save a third of what you earn for five years, you can pay for one experiment like that and still have two-thirds of your savings left over."

Keltham: "Also to be clear, if an experimental report is being put forward by anyone who's not a ten-year-old, there will always be some prediction market about whether a 'replication' would 'reproduce' those results.  You're expected to state your initial price and subsidize that market at least a little, which is how you say to the world that you actually believe your own results at all.  But Civilization knowing more stuff, is to some irreducible degree a 'non-rival non-excludable public good', so the 'philanthropists' come in and subsidize a process where some experiments will be randomly selected for 'replication'.  That is, the selection is random from a probability distribution that goes according to complicated formulas that include the market odds, the 'market-volume', and any advance-obvious 'impact'."

"The point being, almost any experiment has some probability of being actually 'replicated', in which case the conditional prediction market on the outcome of the 'replication', will pay out.  That incentivizes accurate prediction markets and careful trading, even on relatively minor experiments that wouldn't usually pay to get 'replicated' by one of the big expensive high-trust professional 'replication' firms."

"Or as my teachers emphasized to me a lot - if an experiment didn't have any prediction market price on 'reproducing', how would you have any idea what Civilization thought of it?  Would there really be a sense in which Civilization collectively knew anything about the result?  There being a prediction market, with some chance of the replication actually being performed in order to incentivize careful trading, is a key condition for an experiment really being part of Science! at all.  Without that, maybe somebody knows something about that result, but Civilization doesn't know anything about it."

"Probably the people who actually think about Science! in tremendous detail are betting that, if we stop making prediction markets for everything, Science! will start to fall apart.  So that part gets emphasized to children, to prevent that scenario?  I don't know the details, unfortunately, I'm just guessing based on how much different stuff got emphasized to me."

Carissa Sevar: "Are there - other known Science failure modes, ways all of Civilization stays wrong about something?"

Keltham: "So another precaution they emphasized was 'preregistration', you describe the experiment before you perform it and open a prediction market on the results before they're in.  That forces Civilization to make its own prediction in advance.  It lets anybody object beforehand instead of afterwards, if they have an issue with your experimental procedure.  It makes them look less persuasive if they wait until you get a result they don't like, to claim something was wrong with your methodology.  On a Civilizational level, we were told that 'preregistration' guards against a scenario where people think that boring or expected results are 'failures' and don't bother to report on those, until somebody by random luck gets a misleading exciting result and reports on that."

"There's a whole dramatic debate about whether or not, in principle, we ought to pay any attention to results that aren't 'preregistered', if somebody reports on them anyways.  Because of how, if you say we should throw the results like that away, it means we're virtuously refusing to update on what is in fact evidence.  Conversely, if we pay attention to results like that, we're creating bad systemic incentives."

"Obviously, that scenario basically never comes up in real life.  To the extent it ever did come up in real life, I figure people would just toss the original result and pay a 'replicator' to do it over again, if it was anything important."

"But, uh, I'm realizing as I say it, that none of this stuff is what we need to think about on a mini-Science! project the size of Project Lawful.  We're small enough that we can just know about all the experiments everyone is doing, and say all our individual probabilities on them out loud in advance.  We don't have professional replicators to appeal to, all we can do basically is run any important experiments twice with a different researcher in charge."

"I'm sort of flailing here, going through things in random order, as you can probably tell.  In dath ilan we all sort of grow up knowing how Science! is supposed to work, if only vaguely to start with, so at no point does anybody try to teach you the whole thing all at once.  There's just little pieces here and there where you learn in more detail how some bit of Science! works."

Keltham: "What else is there to doing Science!... I suppose there's integrating the results from multiple experiments, reporting them in a unified way so that multiple lines of evidence can be collected?  Somebody mentioned that part already?  That part is sort of obvious if you know any Law of Probability, there's only one obviously correct way to do that...  Well, no, there was that one kid in class who invented his own wacky way of doing it, so it's not that obvious.  Probably I need to get out of the mindset of believing that saying obvious things will bore everyone here.  Golarion isn't doing it yet, so it's not that obvious."

"Though collecting results across multiple experiments is also something you'd do just as an individual investigating something on your own, meaning it's not really Science! as such... but you know, whatever, it'll be actually useful to the Project.  I'm just going to talk about accumulating data across different people's experiments, whether or not you call that by the term 'Science!'"

"Or no, let's go ahead and maniacally experiment to determine whether it actually is obvious or I just grew up that way.  Asmodia, how would you collect together the results from three experiments meant to distinguish among three hypotheses?"

Asmodia: "Keltham, you literally ran that exact problem on me earlier, when you were 'trolling' me, remember?  Luckily for your real question, I'd worked it out on my own before then, while wearing the artifact headband, and I can confirm that it's obvious."

"But yeah, say you've got three hypotheses, like, 'Ordinary Asmodia', 'Conspiracy Asmodia', and 'Time-Traveling Asmodia'.  They all start out with some probability-weights attached to them - Keltham didn't actually say what his prior weights were, but let's say they were 0.90 Ordinary, 0.06 Conspiracy, 0.03 Time-Travel, and 0.01 for Something Else but we're not going to consider that part -"

Keltham: "Asmodia, I was supposed to be dead forever and found myself in another dimension with 'alternatephysics' that is both 'economicmagic' and 'conceptualmagic' like the story isn't even trying to be respectable about considering their consequences separately, was immediately allocated a research harem containing masochists, then the god of mind-altering substances cursed one of my researchers to give out cookies and Governance forgot to tell me about that for two days.  I am WELL above 1% on the None-Of-The-Above Hypothesis."

"People in dath ilan are above 1% on the What If Everything About Reality Is Actually Completely Different From How It Looks Hypothesis.  It's just that, usually, there's not much practical you can do about that."

Asmodia: "Uh, 80% Ordinary Asmodia, 10% Conspiracy Asmodia, 3% Time-Traveling Asmodia?  Actually I'm just going to run with that.  Speaking as either Ordinary Asmodia or somebody pretending to be her in great detail, Ordinary Asmodia doesn't want to help Conspiracy Asmodia by asking Keltham to reveal what his probabilities on Conspiracy were at the time."

Keltham: "Now you're getting it, or realistically pretending to have just now gotten it."

Asmodia: Security, tell some of these idiots to look incredulous or ask 'What?' or something.  AlterCheliax did not have me using this exact case as an Important Lesson in Probability-Law and Also Conspiracy Maintenance and Also Keltham; their alter-personas should be confused about now.

lintamande: "Um, what?"

Asmodia: "That was probably a little high-context, wasn't it."

Keltham: "Really?  D'you think?"

Asmodia: "All right, different example.  Suppose there's a murder case with three possible suspects -"

Keltham: "Asmodia, you can't just not explain now and leave them wondering, that would be mean."

Asmodia: "Keltham is tracking on an ongoing basis the possibility that his new world of Golarion is constructing an elaborate lie around him, including, for example, there not really being any such country as Cheliax.  Do not worry about this or try to do anything proactive about it, Keltham is a dath ilani and will not automatically conclude that everything is a giant lie even if he is looking for possible evidence of that.  We just need to be ourselves, and have faith in his ability to discern reality as reality."

"Keltham was also, I think briefly but I haven't actually asked, tracking the possibility that my overnight personality change and sudden mastery of the Law of Probability was due to my mind traveling a few months backward in time, rather than Manohar dropping an artifact headband on me for two hours."

"Keltham then observed me do two things and fail to do a third.  I forget what his exact probabilities were, at this point," she'll probably still remember them with awful clarity a hundred years later, "but the point is, he - guessed? calculated? I don't understand this part very well - some probabilities for how likely I was to do thing #1, if I was Ordinary or Conspiracy or Time-Traveler.  Then probabilities for my doing thing #2, and then probabilities for my doing #3.  Except I didn't do thing #3, so negate those probabilities - I mean, subtract them from 1..."

Keltham: "Specifically, I'd just offered Asmodia a lower salary than all of my other researchers, she'd just written something down on a piece of paper, and what I was predicting was, one, the chance that what she'd written was a Prediction about me trolling her, two, if she'd made that Prediction, the chance that she'd assign a probability to her statement, three, if the first two things happened, the chance that she would write down that I would predict her Prediction."

Asmodia: Asmodia goes up to the board and writes the example:

                       Ordinary      Conspiracy    Time-Traveler                       --------      ----------    --------------Prior:                   .80            .10            .03'Writes' (Y)             .50            .20            .80'Quantifies' (Y)         .60            .70            .90'Prediction' (N)        1 - .40        1 - .20        1 - .90

"And then - Keltham didn't actually say this part, because it was obvious given what we already learned - he multiplied his starting probabilities by all the likelihoods for each of the events, and ended up with... give me a second..."

Product:                 .14            .011           .002

"Not that it was actually that, because those weren't actually his probabilities, but that's the obvious way to combine multiple pieces of evidence.  Uh, though I'm leaving out the part where what really changes, is the ratio between the chances, not really the chances themselves.  At the end it's like... 13 times as much Ordinary as Conspiracy, and 5 times as much Conspiracy as Time-Traveler, where previously it was 8 times as much Ordinary as Conspiracy and 3 times as much Conspiracy as Time-Traveler."

"If we didn't already have the Law and were looking for Law-fragments, I'd observe that it doesn't matter what order you consider the data in, because it doesn't matter what order you multiply numbers.  Or that you could take the first results and multiply by those and call the result your new 'prior', and then multiply by the second and third results to get the new 'posterior', and that'd also be the same.  Those seem like the kind of 'coherence-constraints' the Law would need to obey, if we didn't know the Law already, or if it wasn't so easy to find a simple Law that fits together like that."

Keltham: "That's basically correct, except that I didn't actually calculate any prior odds or multiply them?  When you're dealing with weird hypotheses on the order of 'my new world is actually a Conspiracy at a low level of sophistication distinguishable to me from reality', where your own mind is doing a lot of thrashing, it can make more sense for a non-Keeper like myself to make up the 'likelihoods', calculate the combined likelihood, and then just... sort of let my intuitive mind keep track of the intuitive update that it feels after I stare at those likelihoods a bit?  If I was keeping formal track of the chance of Conspiracy, it would probably swing all over the place, because of the degree to which I'm making all of the likelihoods up.  The thing to notice is if it starts to feel to me like there's a trend, after I keep on making up likelihoods."

"But, yes, that's the Law for combining the results from multiple experiments."

Korva Tallandria: Korva doesn't think that she could do that stuff on the board unassisted right now, so she's feeling a little bitter about Asmodia (who is, after all, the only other person in the room qualified to teach classes about alien math concepts right now) declaring it obvious.

It doesn't actually look that complicated, so hopefully Korva will be able to confirm its obviousness and apply it to future things later on, when she's gotten a transcript of this class and her brain isn't dribbling out of her ears. Right now she's busy panicking about all of the rapid-fire baseline words in this lecture that haven't been explained and don't seem to have direct Taldane equivalents, and also trying to figure out what exactly an experiment is. It seems like that might be kind of important to figure out, given that she's apparently not allowed to do any of them without publicly announcing it and giving her predictions in advance.

She's not going to ask this, because a bunch of other people seem to know it already, and if she's in the slower half of the class they're going to kick her out and ensure that her soul is permanently worthless.

Keltham: "Some additional points, when applying this Law inside of Science!, that might not be obvious... let me actually 'whiteboard' the three points, so I don't forget the later two while talking about the first one."

#1 - 'Published-experimental-reports' usually don't assign 'priors' or calculate 'posteriors', they just report all cheap details of the raw data, and maybe calculate some 'likelihoods' from obvious hypotheses

#2 - Separate experiments are usually supposed to avert 'conditional-dependencies', watch out for when that isn't true

#3 - If every obvious hypothesis has unexpectedly low 'likelihood' over all the combined data, it means the true theory wasn't in your starting set, often that different experiments had different hidden conditions

Keltham: "Point #1, a 'published-experimental-report' wouldn't make up 'priors' or calculate a 'posterior' the way Asmodia did in her example.   That would be a weird kind of thing to claim as the knowledge of Civilization!  Somebody looking at your report might've seen other evidence.  They could be working on a secret Governance project, and know about evidence you can't access at all.  They might be considering hypotheses you haven't thought of at all.  Or, just, if you don't have formal ways of assigning 'priors', they might have more life experience in guessing those."

"Making claims about what Civilization should believe in the light of an experiment is a prediction market's job, not the job of one experimenter who publishes a report."

"If there's very standard or very obvious particular hypotheses to consider, you might make people's lives easier by precomputing the likelihood of the data on each of those hypotheses.  But one of the big cautions is to consider that other people might have hypotheses you don't, which means you should try to report more details of the data than you already know to be necessary."

"Taking the 2-4-6 example.  Suppose that people just hadn't thought of the possibility that there was 'temporal-dependence' in the results, that they hadn't broken out of the mindset where they were expecting the Keltham-environment to be a timeless 'function' that always produced identical output measurements for identical inputs.  They did run across one duplicate input, but didn't notice it was a duplicate input when they recorded the 'NO' - maybe because two different researchers were querying the environment separately."

"Now think of how inconvenienced you'd be, if that data was reported with input-ordering information missing.  Somebody reading through later, couldn't see which inputs had happened before each other, because you'd implicitly believed that the Keltham-environment was 'time-invariant' and the inputs were 'exchangeable' in the ordering."

"And think of how helpful it would be, if the person had instead happened to include 'timestamps' on each input, even if they didn't know it was important, so you could notice the Keltham-environment taking longer to answer on some inputs, even if they'd missed that themselves."

"If the experiment was on people, the full report includes pseudonyms you can use to contact any of the experimental subjects, just in case somebody comes along later and wants to test subjects to see if they've got a particular 'gene' in their heredity, say."

"Obviously, newspapers can't report all the data of an experiment in the middle of the text.  Even most people who read the 'published-experimental-report' directly won't review all the raw data themselves.  Summaries matter!  So yeah, the report will say, briefly up front and in more detail at the end, 'Here's the theoretical likelihoods of the data for the major hypotheses under consideration.'  The newspaper story will say that, plus, 'Here's how prediction markets on related observables shifted immediately after the results came out.'"

"But the real report, above all, is the data itself - what Reality answered back to you when you asked it a question."

Keltham: "Or, how did the 'Watcher-over-children' put this... some more of this is coming back to me now..."

"It's not the job of the 'experimentalist' who writes the 'published-experimental-report' to say what Civilization believes, to tell everybody else what to believe."

"It's not the experimentalist's role in society to say what's true."

"That's the job of prediction markets to say."

"Only prediction-market traders get to tell everybody what they should believe.  Because they're the ones who'll lose money if they're wrong.  And if you think you know better, step up and bet yourself, because anybody can."

"So I'd guess there was a failure mode they were worried about, where 'experimentalists' start to take on a social role of saying what's true, and don't lose a lot of money when they're wrong?  Or something like that, anyways.  Though that failure mode itself sounds a bit ill-defined to me, what happens when two experimentalists point in different directions?"

"But yeah, the whole Science! system is put together as carefully as Governance.  Unfortunately I did not pay as much attention to exactly why everything had to be exactly that way, because I wasn't skeptical about the Science! system the same way I was skeptical of Governance as a kid."

Korva Tallandria: Most people, in most parts of the world, probably don't have enough money that it would matter that much, even to them, if they lost it. Not that she's going to point this out.

Alexandre Esquerra: Keltham's view of a world dominated by prediction-market-traders telling people what to believe is honestly adorable. Would he reconsider, after Asmodeus just was smarter than he is, and won all the bets, and took all his money? Which, after all, would just happen.

lintamande: "Did this start with - a government doing it, and then doing so well they conquered everywhere else? I don't see how you'd know it was the best way if there aren't other ways it beat out."

Keltham: "Would you accept 'the prediction markets say the results won't be as good if we don't use prediction markets' as an answer?"

lintamande: “No.”

Keltham: "Me neither."

"One answer is that there's always cities inside Civilization trying something different, because, if there is anything dath ilan lacks, it is not people who think they might have Better Ideas.  So far, none of the cities trying something other than prediction markets and traditional Science! have produced vast quantities of new knowledge and technology enabling them to impress the rest of dath ilan into stunned agreement."

"If you mean historically... we don't know a lot of things about our own history that you might otherwise expect us to know, for reasons I'm not getting into at this point.  I'm guessing that dath ilan - had a different trajectory of Intelligence, that something catastrophic happened in Golarion to drive the average Intelligence level down to 10 after writing had been invented at average Intelligence 12 and spellcraft had been invented at average Intelligence 14, or whenever the leading geniuses would be smart enough to do that.  Or writing got carried over from a world that is cousin or common ancestor to dath ilan.  And then something happened to lower the average Intelligence here..."

"My point is, I'm guessing that, by the time prediction markets were around, people generally were smart enough and had enough Law that they wouldn't go to war and conquest, that they'd do something else which was not that, and just, you know, form Civilization, because why not."

"I'm also guessing none of you are going to believe that, and there are obviously other options."

"Maybe prediction markets came along, and Science! came along, and there were factions that refused to adopt that, that didn't know Law, didn't want to be taught Law, didn't want their children to be taught Law.  And then the way of Civilization or what became Civilization, if they thought anything like Civilization thinks now, would be to say - you can refuse to learn the Law yourself, and that's fine, but you cannot choose for your children that they'll have no chance to learn it.  And everybody in your faction is going to have their head 'cryopreserved' upon their death, because to refuse that, is the one mistake that people cannot learn from in time; and you do not pass the competence test to credibly claim to Civilization that you know all the reasons not to do that, and you are choosing to do it anyways."

"And the people in those factions would tell pre-Civilization to go die in a fire.  After which pre-Civilization would say sorry, and come in with superior technology that shrugged off whatever they had in the way of primitive pre-scientific explosives, and teach their children, and save everyone's heads when they died, and plan to apologize about that a thousand years later."

"I - am not sure a world of Kelthams would do exactly that.  I'm not sure we wouldn't, either.  Children are not their parents' stuff, children don't have imaginary ownership-tags pointing to their parents.  I think in the Kelthamverse we'd probably - be less inclined to storm in and do things anyways - if it was about people telling us they didn't want their heads cryopreserved.  Because their heads, their souls, are their own stuff and not ours."

"The children?  Are not anybody's stuff.  If the parents in that dissident faction were, like, not letting their kids own stuff?  The kids there aren't allowed to buy books with Law in them?  I think the Kelthamians probably invade them over that.  All the adults in the Kelthamverse used to be children themselves."

Korva Tallandria: Some parts of those stories are plausible. Some parts are not at all plausible. There are a lot of things she'd like to object to, in that, now that she's thinking about what things are true, but you never object to claims about history, and you super extra double don't object to claims about history with the threat of The Wall hanging over everyone.

...even so, Korva makes a note to ask the project leadership whether they are permitted access to the known information about ancient Azlant, and about the more recent trajectory of standardized wizarding education over the course of the last hundred years, and maybe also about the origins of wizardry itself, if anyone remembers them, and how they relate to other kinds of arcane magic. If that seems like a safe thing to ask for. Obviously.

Asmodia: Asmodia's tempted to just kidnap Korva Tallandria off the Project and put her to work on the Wall, frankly.

Alexandre Esquerra: This might be a slightly dangerous question, but - "Why is average intelligence important for inventions? For societies, yes -" he cannot imagine the average person in his village functioning in dath ilan "- but how does it contribute to inventing writing, except as it produces inventors of genius?" He has the idea - you could have a broader distribution with more failures and more geniuses or a narrower with more nobodies and what matters for this is the geniuses not the average - he's just not sure how to express it -

Keltham: "You need both?  There's a lot of work to be done in Science! that doesn't need to be done by geniuses but does take Intelligence 18.  That'd be one of my primary guesses for why Golarion is missing so many dath ilani ideas despite having +6 headbands and artifact headbands.  My other primary guess is that even the artifact headbands are only enhancing some of the key aspects of mentation that go into Science!.  At some later point, we may experiment to see whether Fox's Cunning, Owl's Wisdom, and Eagle's Splendour all together actually help naive subjects on the 2-4-6 challenge, at all.  Though that would be only one of many tests."

Willa Shilira: Willa's been musing over something in her head, but then she gets a mental approval from Asmodia to just ask about it out loud!

"If your pre-Civilization mostly paid attention to prediction markets when deciding what to do and what to believe, couldn't the factions around it just convince it not to attack them by buying into and distorting their markets? Or to do lots of other things that might not be in its interest? It sounds like a huge vulnerability."

Keltham: "Do you know what we call it in Civilization when somebody comes in and spends a lot of money to try to shift a prediction market away from the Lawful value?"

"In Baseline:  'Free money!'"

"Prediction markets don't generate money out of nothing.  You can only make as much money off a prediction market by being right, as there is wrong money to bet against you, for you to take.  A ton of wrong money coming into a prediction market is a subsidy that can pay smart people the cost of their time to come in, stare at that market, figure out what the correct price ought to be, and buy up shares trading significantly away from that price."

"From the perspectives of smart traders, it doesn't matter whether the person is foolishly trying to distort the market, or just foolish.  It's a subsidy to them either way.  The more stupid money goes in to be taken, the more smart people crowd around competing to take it at exactly the right price.  People who try to distort prediction markets just end up making them more accurate.  You'd need to be able to literally outbid the rest of Civilization combined, to still be standing there buying shares at a bad price after everyone else in Civilization who wanted free money had run out of money they could use to take your money."

"That's why Civilization trusts prediction markets so much.  They're a form of speech where lying costs you, and then if you try to lie anyways no matter how much that costs you, it just makes the truth stronger."

Keltham: "To be clear, there is a fundamental vulnerability of this sort, but you can't exploit it just by being rich.  The Hypothetical Corrupted Keepers could prevent Civilization from doing anything they didn't want Civilization to do, by bidding in the prediction markets, openly as the Keepers, to say that dreadful bad results will happen if Civilization does that.  Relatively few people would want to bet against the Keepers - not because it's illegal, to be clear, but because you'd probably lose all your money unless you're Literally Nemamel."

"Then Civilization doesn't do that thing, and the market on that policy never resolves, and the Keepers don't lose any money for lying that way."

"This is a central problem in all of decision theory - that we never get to observe the results of the actions we don't take.  We never get to see our expected utility estimates tested for everything we thought had less than optimal utility.  There's a lot of places where Civilization solves that problem by randomizing slightly, if doing slightly the wrong thing won't be catastrophic.  But the really important predictions - you don't do that there.  And that does open you up to assault by the Hypothetical Corrupted Keepers, or other actors inside Civilization who can afford to put up a lot of money and come up with persuasive arguments and have enough of a reputation to make other traders nervous."

"But it requires acting openly, to make the other traders nervous to bet against your reputation.  There's a limit to how much you can corrupt Civilization's decisions and markets that way, because if you do it a lot, people will be suspicious.  And not just anyone can do it, either."

"...that said, yes, the Keepers have bid against things in inscrutable ways, now and then.  It wouldn't be an especially helpful act to compile a public list of all the times they've done that, but they've done that even in markets I've been tracking.  To this day I have absolutely no idea why the Keepers fear long-term consequences specified to the rest of us only as 'people will later vote that was a bad idea', if Civilization makes a harder push on teaching average kids more 'computer-science' once my generation's kids are slightly smarter.  I mean, it's very credible that 'computer-science' reshapes some people's thoughts in some internally-damaging direction, which the Keepers would rather not point out explicitly for obvious reasons.  It doesn't obviously fit into any plan of corrupt world domination.  But... yeah, what the Keepers bid against, largely doesn't get done, and if they were Hypothetically Corrupted, they could in fact be steering Civilization that way."

Alexandre Esquerra: Alexandre is now curious what computer-science is. Anything the Keepers don't want him to know has to be powerful.

Asmodia: Asmodia is now curious what computer-science is and if it's going to DESTROY ALL OF GOLARION unless SOMEBODY STOPS IT where SOMEBODY is probably going to be ASMODIA because the rest of Project Lawful is composed of SUICIDAL DISASTER MONKEYS.

Otolmens: The mortal is thinking CORRECTLY again.

It's UNFORTUNATE that this mortal already has an ownership-tag pointing somewhere in Hell, preventing Her from clericing it.  Otolmens has never really understood before what some gods seem to see in their clerics.

Now, however, Otolmens sees.  This mortal would not just be a USEFUL TOOL.

This mortal would be a useful tool requiring MINIMAL ONGOING MAINTENANCE.

Keltham: "Anyways, I'm going to move onto #2 now, in hopes of eventually traversing this conversational 'data-structure'."

"Point #2, since I was dealing with the same Asmodia over time, I had to consider the 'conditional-dependencies' in my 'evidence' and couldn't treat them as 'approximately-independent'.  I had to ask, assuming that Asmodia was writing down a prediction that I was trolling her, having updated my model of Asmodia off that fact, what chance would she then have of putting a probability on that prediction?  That's not my baseline probability for Asmodia putting probabilities on things, it's the Asmodia who was already predicting well enough to know I was trolling her, which is an Asmodia more tuned in on the rhythm-that-is-Keltham than the Asmodia who didn't do that."

"Then I had to consider that question separately for each 'possible-world' of Ordinary, Conspiracy, Timetravel.  The facts inside a single world can interact among themselves, the facts between worlds don't."

"Uh, not in this example, anyways.  Different possible worlds can start interacting if you get into multi-agent logical decision theory.  Even in this example, Conspiracy Asmodia would be trying to guess what Keltham would think Ordinary Asmodia would do.  But those complications shouldn't come up every day, and in most of Science! we can ignore them."

"In Science!, usually, when the 'replicators' set up a new experiment at a new location, the results there shouldn't be 'causally-entangled' 'to-any-significant-degree' with the results of the previous experiment.  If you find yourself thinking that what you expect given a particular hypothesis, on the new experiment, is being influenced by what happened in the previous experiment, it means you're not narrowing down your hypotheses enough.  Which will hugely complicate any efforts to combine your data, if you've got to consider how all your pieces of data are interacting with each other."

"If I was really making a serious run at updating on Ordinary versus Conspiracy, I'd need to consider multiple possible Conspiracies, drawn finely enough that what I saw in one piece of evidence didn't much affect my likelihoods on other pieces of evidence..."

"Actually, this would probably be a lot clearer with some math."

"So let's do that, considering the hypothetical case of us experimenting to see whether enhancing all three Golarion-known mental stats by +4 would help 'previously-unexposed-subjects' on the 2-4-6 challenge."

Keltham: "Simplifying things down way too far, for the purposes of keeping the math simple on its very first presentation, we'll say that the only variable we measure is whether or not somebody solves the challenge during their first five minutes."

"Now, how would you do Science! to that, to see whether +4 to all mental stats helped?  I'm going to let researchers and candidates from any tier answer, because I expect this one to be pretty hard just starting out.  But if I'm wrong about that and Asmodia or Carissa are seeing it immediately, do call Prediction rather than just tossing out the complete answer."

Korva Tallandria: She still doesn't know what Science is or what it means to do it to something!!! But - there's an obvious thing to look at, isn't there -

"Well - test whether people who read 14 in every mental stat get it within five minutes more often than people who test 10 in each?"

Alexandre Esquerra: "With what resources? Strategies will vary depending on what we can afford."

Keltham: "Yeah, you're going to have some trouble finding people with exactly 14 in all mental stats, I predict, given Cheliax's current state of gathering statistics on its population.  Even if you can find them by checking with all the wizard academies where they measure that, paying to Teleport people in sounds quite expensive."  (The back of Keltham's mind also notes that this proposal is something a not-too-competent Conspiracy Korva might float, if their world secretly had much better stat-gathering and/or cheaper Teleports, and they hadn't fully updated on the fake world not having that fact true; and Conspiracy Alexandre might be trying for a hasty backpedal there.)

"But let's leave that aside for now.  Let's also leave aside that people with all 14s may have differences that aren't just about the mental stats, like, maybe they came from richer families and ate better as children and got better 'nutrition' and that's part of why they're smarter now.  Maybe they have smarter parents and their parents owned more books, or better books."

"Leave all that aside.  We test 5 people with all 10s, and 5 people with all 14s.  At those intelligence levels, five minutes won't cut it, given our previous experience here.  Let's give them thirty.  2 out of 5 people with all 10s get it in thirty minutes, 3 out of 5 people with all 14s get it in thirty minutes."

"What have you learned?  What do you now believe?"

Willa Shilira: "Let's say on the one hand you guess a 50% getting it if the headbands don't matter. Then this situation has a chance of 0.5^10 of coming up, since each one was a coinspin anyway."

"But if you instead thought 60% with headbands and 40% without, then for the headbanded people it'd be 0.6^3*0.4^2, with the winners being 0.6s and the losers being 0.4s, and then for the non-headbanded people you'd have 0.6^3*0.4^2 too, for the opposite reasons. So that's a chance of 0.6^6*0.4^4. Then we can multiply everything by 10^10 to make the math easier, and the idea of 60/40 and 40/60 increased in probability vs pure 50/50, by, umm..."

"(6^6)*(4^4)/(5^10)"

Keltham: "Mm, not the Science!tific way of thinking about it, exactly, but not at all bad, either.  You even simplified the math some."

"Though, saying they got artifact-headbanded is not what the original experimental spec was.  We were specifying natural all-14s which will probably give us noticeably different results.  One does want to learn to think precisely about that sort of thing."

"To compute that number there, let's observe it'll be the square of 6^3*4^2/5^5, or 216*16/3125, 2160+1296 is 2160-4+1300=3456, divided by 3125... about 1.1, since 3456 is around 3125+312, squared is about 1.2."

"So the results you saw are 1.2 times more likely if the all-10s have 40% pass rates, and the all-14s have 60% pass rates, compared to everyone having 50% pass rates."

"Now, I realize this ought not to go in a 'published-experimental-report', but what do you happen to believe, seeing that?  What would you believe if you saw that in real life?"

Willa Shilira: "Oh, yeah, sorry about the headbands I was distracted by the cool math careless."

"But if I saw it in real life, I wouldn't believe very much of anything? It was only five in each group, and two of them randomly going the other way could've made it all look backwards. But times 1.2 isn't a lot of a chance boost either, compared to really knowing stuff about how the world works. So the math seems to look about right? It wasn't too convincing, and the math isn't too convinced?"

Korva Tallandria: "I would go from believing that natural all-14s are going to be at a disadvantage relative to natural all-10s, to believing that having stronger mental stats and having had the experiences that go with them at least isn't actively disadvantageous, and probably does help."

Korva Tallandria: (Oh gods why did she say that that was a terrible idea he's going to make her do math in baseline right now and then everyone will know how stupid she is about this.)

Keltham: Keltham nods at Korva, then addresses Willa.  "Seems a bit sus that the hypothesis you're comparing to the random-coinspin is one where the supposed natural frequency of thirty-minute guessers, at those two Intelligence levels, exactly matched the data."

"Suppose we'd tested a thousand 'subjects', instead of ten.  245 out of 500 'subjects' with all-10 stats guess within thirty minutes.  256 out of 500 subjects with all-14 stats guess within thirty minutes."

"Then, clearly, to ask how we should react to this data according to your methodology, we should compute the likelihood ratio over the fair coinspin, of the hypothesis that among all-10s, 245/500 guess within thirty minutes, and among all-14s, 256/500 guess:"

(245/500)^245 * (255/500)^255 * (256/500)^256 * (244/500)^244 / (250/500)^1000

"This simplifies to... let's see... a likelihood ratio of about 1.276."

Keltham will now pause dramatically.

Asmodia: "I'll bite, since you're obviously dying for somebody to ask.  How'd you simplify that?  I'm frankly not seeing it."

Keltham: "Well, if I've managed to remember all the numbers correctly, it's from a version of this problem I worked a while ago, back in Civilization where you could just toss 'expressions' like that into 'computers'*."

(*) This is clearly the same word as appeared in 'computer-science'.

Asmodia: "I see."

Keltham: "And if you're wondering where the numbers 245 and 256 came from, in that problem, they just happened to be the results of generating 500 random 'bits', twice, and counting 'YES' answers."

"But clearly, the 'computer' could not have been truly randomizing!  After all, it is 1.276 times more likely that these numbers would appear if they had truly been generated by a 'computer' that first picked 'YESes' with 49% probability, and then picked 'YESes' with 51.2% probability.  The Conspiracy has been at work again!"

"Would you agree with this stance, Willa?"

Willa Shilira: This is more than a little mortifying but at least a bunch of her peers didn't get this right first, she can keep her composure, really she can.

"This is something to do with making the guess about what happened after seeing the data, isn't it? Instead of guessing the ratios first for stats-matter and stats-don't-matter, we did the experiment, and then made up the exact shapes of our guesses after, which is a naughty thing we weren't supposed to do."

"And there's some deep mathematical thing about doing that, looking at data you already have and matching that way, that tends to produce these relative probabilities around 1.25 or so, isn't there?"

Keltham: "Yup.  Though embarrassingly I don't recall the exact constant.  Back when I was memorizing those numbers in order to impress the younger kids I was teaching, by pretending to calculate it all on the spot, they were too young to ask that question so I didn't memorize the answer."

"But as you say, Willa, you did indeed try to do something naughty, by calculating the likelihoods for 'latent' frequencies that exactly matched the observed frequencies.  Maybe if you'd run a pilot experiment and found 40% YESes for all-10s and 60% YESes for all-14s, you would have written down the numbers 40% and 60% in advance, in the 'preregistered' version of your 'published-experimental-report' that you'd put out and gotten... signed and time-stamped... by larger Civilization, before you'd collected any data.  Then nobody would call you naughty if you ran the likelihoods for the 40%-and-60% hypothesis, and compared that exact hypothesis to the fair-coinspin hypothesis."

"So hopefully it's starting to become clear that all of the pieces of Science! hang together, even if I didn't explain them all that well?"

"Anyways, let's say you didn't run a 'pilot-experiment' that produced 40%-and-60% as a hypothesis distinguished in advance, or 49% and 51.2%.  Then what might you say in your 'preregistered' 'published-experimental-report' that doesn't have the results filled in yet, about how you plan to analyze the data?  People besides Willa are allowed to answer too."

Carissa Sevar: "What are we trying to figure out? We want people who can solve all the problems in 30 minutes? Then we probably care about whether all-14s are enough better than all-10s to make up for how much rarer they are, so we should just check, how likely does it look that all-14s are a hundred times better at this, or even five times better at this?"

Keltham: "Seems slightly shaded towards 'How do I make more spellsilver?' instead of 'How do I understand what's going on inside of spellsilver refining?'  But leaving that aside, suppose we accept that 'goal-framing'.  What math do we do to it?"

Carissa Sevar: " - huh, so, my complaint is that doing math on how much we should change our minds from a starting assumption there's no difference feels fake. Unless we're doing this study out of idle curiosity it's only a worthwhile study if Civilization was starting with the belief not just that there's a difference but that there's a large one. If 14s aren't wildly better at this then there's no way you'd use them. So the math is easier if you assert that we thought there'd be no difference, but I can't really imagine thinking that. But on the other hand, there's no math you can do on the thing I just said - you would get wildly different answers if you started out thinking 14s are 5 times better versus if you think they're 100 times better - and I thought maybe there was some way to specify 'started out thinking 14s are at least 5 times better' but I'm actually not seeing it. I don 't know if that means there's no math directly for answering the question that would actually motivate my Civilization to do this experiment, or if there is but it's far more complicated than the starting math and we need to get that nailed down first."

Keltham: "Asking 'what if there's no difference' and 'what if tri-14s are at least five times more likely to guess than tri-10s' are both questions that are semantically well-formed - it's easy to visualize ways reality can be that makes those propositions true or false - but the second question is hard to answer directly using the statistics that 'published-experimental-reports' are supposed to calculate."

"To be clear, you could see a report saying that they tested 1000 subjects of each type, and 103 tri-10s and 622 tri-14s guessed in time.  You could look at that report and say, 'Yep, sure looks to me like tri-14s are at least five times as likely as tri-10s to guess in time.'  But in stating that, you'd be arriving at a 'posterior', not stating the kind of evidence that a 'published-experimental-report' summarizes.  You'd be doing something that intrinsically revolves around 'priors' and not just 'likelihoods'."

"Do any of the candidates want to take a stab at saying why Carissa's question of ultimate interest to her Civilization - 'Are tri-14s at least five times as likely as tri-10s to guess in thirty minutes?' - is something that's ill-formed for an experimental summary to directly summarize an 'evidential-update' about?  Why it can't just report on that the way it could report on a preregistered hypothesis that tri-10s were 40% likely to guess, and tri-14s 60% likely?"

Asmodia: Asmodia, after querying Security somewhat plaintively, sighs and gives the green light to the only new candidate who is reckless enough to risk directly criticizing the ideas of the Chosen of Asmodeus at this stage in their acquaintance.

Can this person please at least look around hesitantly before she tentatively raises her hand?

Willa Shilira: She very hesitantly justifies this particular incident of recklessness in her mind, both to herself and anyone listening. She just wants to learn things. But she's also very happy she can help the project right now. She does do her best to also look very hesitant, as is in character for alter Willa and also generally appropriate.

"'Five times better at this' seems like a tricky statement to use with this math. When you say a category has a specific chance of its own, you can go through and multiply all the probabilities in the category independently, and compare it to other categories that have been multiplied up independently. But with 'five times better', the two categories sort of depend on each other."

"You could guess that 10s have a 10% chance, so 14s would need a 50% or better chance. Or 10s could have a 15% chance and 14s would need to be better than 75%. Or 10s could have a 25% chance and it would be impossible for 14s to be five times better. And somehow 'are 14s five times better than 10s' needs to take account of all of those at once and everything in between. Or maybe you mean something more complicated than that by 'five times better', but I think that would make this more of an issue, not less of one."

"And it's even worse, because there's an implied 'at LEAST' five times better there. So we need to account for all the imagined frameworks where the multiplier is at least five, all the way up to infinity."

Carissa Sevar: She smiles pleasantly! There are no power dynamics here, they're all just speculating about math! "I agree that all of those make it terribly complicated, and yet - I look at these experimental results and I think 'ah, so 14s, it looks like, aren't better at this by a factor of five.' Clearly in my head I am calculating the odds of observing these results given my theory, and judging them as low; I'm just not doing it precisely. And so what's bothering me is - is it impossible to do precisely, so the thing it feels like I'm doing in my head is an illusion? Or is it possible but hard enough we shouldn't bother without the 'computers' of Keltham's world? Or is there elegant math that makes all those numbers actually possible to calculate?

The reason this is bothering me is - the closest thing I've seen to science is merchant houses, going, does this supplier provide higher quality by enough of a margin to justify their higher prices? They do tests like this all the time. But I don't see how to use this math to answer their question."

Keltham: "There's obviously ways to do Lawfully anything you can do with your brain by seeing the results yourself.  Sometimes it takes more Law than you have.  Sometimes it involves calculations that wouldn't make sense to put directly into a 'published-experimental-report'.  Still, Willa has put her 'publication-priority-timestamp' squarely on the key issue."

"By way of approaching the complicated proper math by a simpler 'hacky' route, suppose the 'preregistered' analysis said that they were going to compare hypotheses for each 'subject-group' having seven possible 'latent propensities' to solve the puzzle in time, of 0, 1/10, 1/5, 2/5, 1/2, 3/5, 4/5, 9/10, and 1."

"Since some subjects in both groups solved the puzzle, and some didn't, we can cross off the 0 and 1 'propensities' from both cases; they've been falsified outright, the 'likelihood' is zero 'conditional' on those propensities being the true 'latent' state of reality."

"Now, let's go ahead and compute the 'likelihoods' for each surviving 'propensity', in each group..."

All-10s (2 YES, 3 NO): Propensity:     Likelihood:---------------     --------------1/10                (1/10)^2 * (9/10)^3 =   729/1000002/10                (2/10)^2 * (8/10)^3 = 1024/1000004/10                (4/10)^2 * (6/10)^3 = 3456/1000005/10                                               = 3125/1000006/10                (6/10)^2 * (4/10)^3 = 2304/1000008/10                                               =   512/1000009/10                                               =     81/100000 All-14s (3 YES, 2 NO):                                                       = same but table flipped

"The point which Willa is gesturing at, is that it's not possible in the same way to speak of the combined 'likelihood' of all our observations, if all-14s guess more often than all-10s, or if all-14s guess at least five times as often as all-10s."

"At least, not without dragging in some additional assumptions."

Korva Tallandria: Korva doesn't at all see why you can't just ask whether all-14's get it within thirty minutes five times more often than all-10's, do the experiment, and then report whether your records of it confirm or deny that this happened, but Willa and the Chosen of Asmodeus are obviously much better at this than her, so if they're stuck on it then whatever Korva is doing is going to end up being humiliatingly wrong.

Ione Sala: "Keltham, advisory, the new candidates have spent their previous educational lives in unimaginably-to-you-awful Golarian schools designed by Intelligence 14 teachers for Intelligence 10 students, where asking stupid questions means that you lose face in front of the other students and the teacher rebukes you.  Right now, most of them are staying quiet and not asking any questions.  This is not the good sign that it would be in dath ilan.  Here, it means that they have made a completely reasonable decision to let other candidates ask stupid questions first, and see whether we - the current researchers - were being honest with them about whether that's safe here."

"I'm not quite sure whether they're internally wondering 'What's wrong with just testing a lot of people and reporting whether all-14s get it five times as often?' or more like 'Why would anybody do this sort of experiment in the first place?' or maybe 'But what's an "experiment" actually?' because the Baseline word translated in my head but not in theirs, but..."

Keltham: "Thank you, Ione."

"Can I possibly solve this problem by offering anybody here one silver to ask whatever stupid questions they weren't asking?  At least in dath ilan, offering to pay for something is taken as a credible sign that you want it."

Ione Sala: "I wouldn't bet on it.  Maybe if you told them to Message me with the questions, let me ask it for them, and then you pay me and I pay them after class."

Keltham: "What if I literally truthspelled myself about -"

Ione Sala: "Same problem as in the cooperation-defection dilemma if you're a Chaotic Neutral outsider.  They wouldn't be certain you would know correctly whether they'd lose face in front of you, and it wouldn't solve the problem of losing face in front of the other students."

Keltham: "You know, I've read about the concept of 'low-trust environments' but never really understood it before."

"Ione, you okay with being tapped by truthspell about how, if somebody else asks a question through you, you won't reveal who it was?"

Asmodia: "Most of the current researchers will see who, Ione.  We've made our soul arrangements and our arcane sight will detect the Message spell."

Ione Sala: "Right."

"Well, I can still re-ask a question if anybody Messages it to me, and at least Keltham and the other candidates won't know who."

Keltham: "And if that doesn't work, Ione can introduce a delay between being Messaged and when she speaks, and I'll start paying a copper to anybody who pretends to message Ione but doesn't actually subvocalize anything to her, so as to create plausible deniability about who was actually asking questions."

"If that doesn't work, I'll stop and actually think about the problem, though that's something of an extreme measure."