Strategies for Question-Answering

Our goal is to design a system that can answer vague questions. The considerations in the previous section suggest that dialogs may be an effective tool for this purpose. This doesn’t constrain the space of possible systems enough for practical work. So, let’s think about what strategies an effective system for question-answering could employ.

Table of contents

1. Break down big questions into small ones.
2. Provide answers given by others to turn production tasks into judgment tasks.
3. Externalize thoughts to remove memory constraints
4. Externalize thoughts to enable incremental progress within individuals
5. Share thoughts to enable incremental progress across individuals
6. Communicate thoughts to make them more coherent
7. Let machines do some of the work
8. Focus attention on topics with high expected payoff

Break down big questions into small ones.

Based on the argument in Section I.3, we should expect that decomposition results in better answers when the question cannot be answered correctly based on mostly stored computations. For example, suppose you are thinking about some negative state of things x. “What could you do to make it less likely that x?” is a big question. If you haven’t thought much about the causes and components of x, and if you haven’t received high-quality external information on how to reduce the probability of x, then it will be difficult for you to produce a good answer. This could be easier if we first ask about the causes and components of x.

Provide answers given by others to turn production tasks into judgment tasks.

In various circumstances, it is easier to recognize good answers than to produce them. For three very different examples, see (1) Ira Glass on The Gap between having taste and producing work that is good according to one’s taste, (2) Justice Potter Stewart’s famous “I know it when I see it”, and (3) the strong belief that P!=NP in computational complexity.

In circumstances like this, providing answers that other people have given to a question can make it much easier for you to answer the same question. In cases where it is difficult to break down a complex question, it may still be the case that some people manage to answer the complex question, and that others can recognize a plausibly good answer when they see it, even if they couldn’t have produced it themselves.

There is a virtuous interaction between decomposing questions and sharing answers: the more we decompose a question, the more we should be able to reuse answers to subquestions.

Externalize thoughts to remove memory constraints

Imagine that you are planning how to drive to a new place at the other end of the city without a map. Even if you know all the roads, it will be difficult to find the best route, just because it is difficult to keep all the information in your working memory at once. You can think about all of the individual roads, one by one, but you will probably not choose an ideal route because you can’t quite keep in mind their connections.

Similarly, when you think about your own thoughts and behaviors, you may be able to analyze any given thought or behavior, but locally optimizing one of them may lead to problems with others. It may be easier to see connections, and to prioritize across possible thoughts, if thoughts are represented externally. Free-form writing (as in diaries) and mind maps are two existing tools that serve this function.

Externalize thoughts to enable incremental progress within individuals

Thoughts are fleeting. When we don’t write down our thoughts, our recollection will frequently be spotty. This makes progress difficult: whenever we think about some topic, we have to first reconstruct our previous state of investigation. If our recollection is sufficiently lossy, we may not make any progress at all; recreating our previous thoughts may take too long for us to bother. By writing thoughts down, or otherwise giving them more persistent form, we make this process easier, and we can more easily make incremental progress.

Share thoughts to enable incremental progress across individuals

Externalizing thoughts also enables incremental progress across people. Compare how we think about our personal lives to how science works: in science, there is a continuously growing global store of knowledge that individual scientists build on and contribute to. It would be a terrible idea for a scientist to discard all of the knowledge that their field has acquired, or to only acquire this knowledge from pop science articles and very occasional conversations with other scientists.

On the other hand, this is the default state for our personal lives: we read advice here and there, and we have occasional conversations with friends and family, but in the end, our choices are quite independent. Each of us contributes very little to humanity’s knowledge of how to live good lives, and even those who try don’t systematically build on what others have discovered before.

We do have a few systems that go part of the way: Online communities such as Reddit, Quora, and various forums discuss personal matters and allow in principle for progress to be made, but they suffer from a combination of low intellectual standards, fragmented discussions, misaligned incentives (posts should be funny, surprising, impressive), and limited access to personal details. Parts of the quantified-self movement have higher intellectual standards, and the movement is more personalized by nature, but it limits itself to what can easily be measured. The field of positive psychology, and possibly some non-academic publications on self-help, have arguably made some progress over the years, but much of the thinking is on a very coarse level (“Does marriage make people happier?”) and there is a clear separation between naive consumers and expert producers, which limits the number of producers.

Communicate thoughts to make them more coherent

Putting thoughts in words has benefits beyond resolving memory constraints and sharing acquired knowledge: when I think about what words will reconstruct an idea in the reader’s mind, I often find that I do not understand the idea very well myself.

I don’t understand this process very well—why does writing down ideas for the purpose of communication cause me to notice their weaknesses? Here are two guesses: First, the external representation needs to be sufficient to reconstruct the idea when combined with the reader’s background knowledge, which forces me to look at all parts of the idea, even ones that I assumed to be clear without much explicit consideration. Second, social considerations create an adversarial setting: when I know that my ideas will be examined by others, I am more motivated to find their flaws so that I can fix them before they do.

Let machines do some of the work

At this point in time, we don’t have good ways to represent and reason about arbitrary human knowledge in machines, so most reasoning will have to happen in human brains. However, we can strive to automate some relatively easy inferences, and extend this over time as the science of knowledge representation advances.

For example, if x is known to be good, and y is known to cause x, then we can infer (heuristically) that y instrumentally inherits some of the goodness of x, and we may not need to ask about the goodness of x explicitly.

More generally, whenever you notice that your thinking follows a pattern, consider automation. For example, you may notice that, when you are brainstorming, you ask a relatively fixed sequence of questions such as: (1) Write down anything that comes to mind. (2) Write down reasons why it is hard to come up with ideas. (3) Make an exhaustive taxonomy of things you could write. (4) Randomly constrain your problem (5) Etc. Outsourcing the question-asking process to machines can make it more reliable, and it makes it easier to share and incrementally improve the process itself.

Another place where machine learning can be applied is in learning which questions to ask: Record for each question whether people answer it, or whether they press “skip”. An answer is interpreted as a positive signal, “skip” as a negative signal. Each question is associated with a number of features (e.g., words used, the question type, context signals). We can use supervised machine learning techniques such as logistic regression to preferentially ask questions that are most likely to get answered.

Focus attention on topics with high expected payoff

There are many topics one could reflect on, and for any given topic, there are many questions one could ask. Without systematic efforts, we have little reason to believe that naive attempts at reflection will pick the most promising topics, and little reason to believe that we will ask the best questions for any given topic. It is likely that both choices will instead be driven by what easily comes to mind. This availability heuristic can be problematic in cases where we would like to look at our lives with fresh eyes, and where we would like to prioritize based on where we can make the most progress. Indeed, some of the questions we should ask may be the questions we least want to ask, e.g. because they have inherited some of the aversiveness of their subject matter. If I hate doing my taxes, then at some point the very thought may become aversive enough that I flinch away from it every time my thoughts get close. We can probably do better by following a more systematic process for choosing topics and questions, in particular if we can learn from what other people found helpful.

In the remainder of this document, I will lay out a proposal that uses the techniques above and that is intended to produce high-quality answers for vague questions.