10 replies
If in some field with much pseudoscience and misleading science, how do you figure out whether the "information" you get from studying a neuron is 20% accurate, 50% or 80% accurate. And if no information is ever 100% accurate which I suspect because the brain is too complicated, what are the best solutions to this problem for researchers?
On the weather forecast, when it says there's an 80% chance of rain tomorrow, what does that mean? Unless they're taking bets on it, I'm not sure it means anything.
You sound like I sound like Tom Dwan.
You sound like I sound like Tom Dwan.
My understanding with the Met office was this means the expected amount of time it would raine over the period was 80%. Which is different to 80% chance of rain. (I may well be wrong about this, I'm too lazy to Google it).
I didn't understand the question. In some scientific subject you should study a single neuron? I think a single neuron either fires or doesn't fire and learns to do so in certain circumstances. This creates a classification. Or in other words can categorize something. This is what AI is based on by programming this principle. Maybe you are asking about how to verify a "truth" in general where nothing can be proven? Fundamentally you would have to rely on observation of laws, recognizing which are relatively more fundamental and Occam's razor.
Logic puzzles like this are a perfect benchmark for testing AI reasoning capabilities. For those curious about how different engines handle such teasers, using a multi-model hub like https://cabina.ai/ allows you to compare answers from GPT-4o, Claude 3.5, and Gemini simultaneously. It’s a great way to spot logical hallucinations and find the most accurate explanation for complex riddles in one place.
Unless the solution is independently established, comparing three models on one riddle doesn't show which answer is accurate; it can just produce three versions of the same wrong guess. A useful benchmark needs a larger set of puzzles with fixed answers and scoring criteria, including slightly altered wording to catch pattern matching.
fixed answers only help if the puzzle itself is unambiguous, imo. would you separate genuinely ambiguous riddles from wrong model answers in the scoring, or exclude them before testing?
Ambiguous riddles should probably be a separate bucket rather than simply excluded, since recognizing that a prompt has multiple valid readings is part of reasoning too. The scoring could require the model to state the assumptions behind its answer, then compare whether those assumptions match the intended reading.
Score ambiguity detection separately from matching the author’s intended answer. If two readings are defensible, a model giving conditional answers shouldn’t lose just because it missed the setter’s preferred interpretation; the benchmark needs human adjudication for that bucket.