I feel vindicated on the number of times I’ve had to tell managers you don’t need optimizely, you can run a t-test super simply, and you can manage your type I error via simulations also easily.
For sequential testing like these online systems do there are a lot of options, but I usually direct them to Bayesian methods as they can show the uncertainty in a clearer way.
But basically, bootstrapping plus Bayes equals pretty good interpretable metrics.
Like, before LLMs I could spin up a tool for this in a day or two.
If you want to learn more read Computer Age Statustical Inference by Effron and Haste. You won’t get anything better than Effron’s writing on the topic.
ETA: it was like many many years before Optimizely actually hired statisticians to fix their awful statistical methods, and even then, they published some papers on stopping time that I think were mostly overcomplicated and for marketing reasons.
The culture to me from the outside seems like it’s long been about metrics to make product managers fork over cash instead of the making optimal decisions.
Hey I recognise that name. The article “how not to sort by average rating” is probably the most helpful programming article I’ve ever read. And nobody uses it 😂
I feel vindicated on the number of times I’ve had to tell managers you don’t need optimizely, you can run a t-test super simply, and you can manage your type I error via simulations also easily.
For sequential testing like these online systems do there are a lot of options, but I usually direct them to Bayesian methods as they can show the uncertainty in a clearer way.
But basically, bootstrapping plus Bayes equals pretty good interpretable metrics.
Like, before LLMs I could spin up a tool for this in a day or two.
See also https://www.evanmiller.org/index.html for some good content on this, so you can convince your companies to not use optimizely.
If you want to learn more read Computer Age Statustical Inference by Effron and Haste. You won’t get anything better than Effron’s writing on the topic.
ETA: it was like many many years before Optimizely actually hired statisticians to fix their awful statistical methods, and even then, they published some papers on stopping time that I think were mostly overcomplicated and for marketing reasons.
The culture to me from the outside seems like it’s long been about metrics to make product managers fork over cash instead of the making optimal decisions.
Hey I recognise that name. The article “how not to sort by average rating” is probably the most helpful programming article I’ve ever read. And nobody uses it 😂
I’ve used it a ton, it’s a useful method.
Hey I know some of those words!
implying managers know what bayes is.
They don’t. That’s fine. That’s why they pay me.
Hold up.
Am I being advertised to on Lemmy? 😆
No, and for 3 easy payments of $9.99 I’ll explain why not!
Godamit, I’m in!