My take on this: Q&A sites used to fill two distinct roles, but only one of these (the more boring one that doesn't matter anyway) is usurped by LLMs. The interesting one is not yet in danger of being supplanted.

There's always been two types of questions on StackOverflow:

• Trivial, highly specific questions that would be directly answered by documentation etc. but the asker either doesn't know there is a manual, is too lazy to read it, tried reading it but it was too confusing, or the manual is missing in parts or in whole. These tend to solve the one instance of a problem each. Even very similar other problems are often hard to recognize as such and people end up asking many duplicate questions about the same thing.
• Complex, deep questions where there's not really "one definitive answer", and it's not so much about "answering" the question but learning something from the discussion that is valuable. These usually deal with central concepts in an area that, once properly understood, can be used to independently resolve a whole class of problems.

I came to SO for the first kind, but stayed for the second. I am now here, because over the years SO moderators began to crack down on the second category. IMO the second category is the most interesting, while the first is quite boring. As an expert, I would not really want to be answering the first kind too much, the second would be the interesting one. This can be seen in how many eminent scholars choose to write comprehensive textbooks on their subject rather than practice tests.

I think LLMs currently excel at answering the first kind, because no manual is too boring for them, so they effectively "read them all". Even then, the models are generative, often hallucinate, and it's never clear what's generated and what's fact - so they tend to be unreliable with it as well. Although I expect that the reliability will improve over the next few years.

For the second kind of question, LLMs are no good at all. For one, it requires the ability to reason at a high level and synthesize laterally related information which the LLM cannot do. It can generate content if deep treatises of similar topics happened to be in its training data, but there are not always readily available analyses on every conceivable subject - not many people can write one, and it's work to do. So I think this is the real niche of a Q&A site - the ability to support thoughtful discussion of key concepts. This is probably out of reach for AI until we see a true breakthrough in the field.

posted over 1 year ago

CC BY-SA 4.0

matthewsnyder‭

2209 reputation 52 59 259 92