Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4096. Given: 11588 `inputs` tokens and 2400 `max_new_tokens`
I'm unable to process this image when calling my hosted endpoint. I get the error...
openai.APIError: Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4096. Given: 11588 `inputs` tokens and 2400 `max_new_tokens`
This is the image https://m.media-amazon.com/images/I/81xwfM+g1VL._AC_SL1500_.jpg
I've tested at https://olmocr.allenai.org/ and it works amazingly well.
I assume the token count comes from the image getting converted to base64 behind the scenes?
How do I remove the limitation? Thanks!
I had similar issues with other OCR Models, lmk if you find a solution. I think it's something related to the training params of the model (Not Sure!!)
Can you share your code please? Basically, you likely have a document with a very long "document-anchoring" prompt (see tech report). What we do in the web demo and pipeline is we automatically shrink that down if it gets too long:
ex. anchor_text = get_anchor_text("./paper.pdf", 1, pdf_engine="pdfreport", target_length=4000)
set a lower value for target_length, try 1000 characters.