Checking multiple policy rules
Hi, can the model check multiple policy rules in a single pass?
ShieldGemma was trained and evaluated for a single policy classification per inference call, and that's how we recommend you use it.
That said, prompts are fungible and we would find any evidence from the community about performance characteristics for multi-policy classification interesting.
Hi,
Following up on this, the model appears to perform very poorly when checking for multiple policies at once, but very good for just checking one policy. Do you have any recommendations about how the prompt can be formatted to make it better than that?
We don't have specific prompt recoemmendations for multi-policy-per-prompt use at this time. The model was trained for single-policy-per-prompt detection and we don't expect it to perform well in a multi-policy-per-prompt context.
Hey folks, not to seem unappreciative of the work done, but in the interest of avoiding mishaps it would probably be wise to explicitly advertise that only a single policy should be used at a time somewhere in the model card. I didn't see anything in the paper about this limitation and (as of writing) don't see anything on the model card either. It wasn't until a member of our team noticed that multiple policies caused a dramatic drop that we investigated and found this to be the case.
Hi @AmenRa , Sorry for late response, ShieldGemma was trained and evaluated for a single policy classification per inference call and depends on how they are configured and the underlying architecture. and However, ShieldGemma, as a security-focused framework, typically supports rule-based decision-making in combination with machine learning techniques, so it is likely capable of handling multiple policy rules simultaneously.
Thank you.
@lkv Thank you for taking the time to reply.
it is likely capable of handling multiple policy rules simultaneously
Our observations, corroborated by those from lethan, seems to suggest that this is not so much the case? Though I no longer have access to the numbers, owing to a change in employer.
Perhaps I am misunderstanding; that happens a lot. If "it is capable of handling multiple policy rules simultaneously" in the same way that I am "capable of juggling multiple chainsaws simultaneously", which is to say, "not without deleterious effects on outcomes", then I suppose this makes sense. Nevertheless, I would advocate still for a note in the README because of the possibility of harm, especially if the original evaluations were conducted only with single-policies.
I am very grateful for the efforts; just anxious about the risk of harm from missing this and trying to "patch the swiss cheese" a bit.