Citing “lived experience” as evidence something works is literally the tactic of homeopaths and antivaxxers when confronted by large-scale or structured studies disproving their claims. That this now seems to be the go-to argument of the pro-“AI” crowd is a huge red flag.
@baldur the corporate software world always seemed to have trouble with defining things like quality or productivity. They have tried tools like coverage, cyclomatic complexity, velocity or lines of code but they're all not really adequate. That means it's no wonder that they have to benchmark LLMs based on feels. And that is what the tech firms sell. The outputs are ever more verbose and mimic thought processes in their form. That feels smart even though it isn't. The feeling of productivity.
@chris @baldur Unfortunately lack of valid metrics isn't just confined to corporate software, it's baked into software development culture. While the anti-metrics stance in software culture stems from corporate abuses, it really hurts us in the long run in our ability to choose the best software technology for our application needs.
