Using Fable, pretty much every request hit some gate they had for no discernible reason. These provider-level rejections should be incorporated into benchmarks as 0s on the tasks since that's the experience you'll actually get using the model.
I have heard this from a bunch of folks, but that was not my experience. For the couple days I was able to use it, I didn't hit a single gate, and I was using it pretty extensively (but not for anything security-related).
Comments
Using Fable, pretty much every request hit some gate they had for no discernible reason. These provider-level rejections should be incorporated into benchmarks as 0s on the tasks since that's the experience you'll actually get using the model.
I have heard this from a bunch of folks, but that was not my experience. For the couple days I was able to use it, I didn't hit a single gate, and I was using it pretty extensively (but not for anything security-related).
Never had rejections in the short time Fable was available
It used Opus for every biology related question I asked it.
Even opus refuse to discuss micro biology for more than a around 15 turns in my experience.