Splitting text into paragraphs based on shifts of meaning

#47
by drmeir - opened

I would like to use bert-base-uncased to split text into paragraphs based on shifts in meaning. I do not want to install heavy packages and download the model, but rather use requests to submit a post request to the model's endpoint.

I guess it should be something of the kind:

hf_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
headers = {"Authorization": f"Bearer {hf_token}"}
url = "https://api-inference.huggingface.co/models/bert-base-uncased"

data = {
    "inputs": "Some text without paragraph breaks. And here is a new thought.",
}
response = requests.post(url, json=data, headers=headers)
print(response)

I get 400 response. Since I have used the authorization details for other models, I know that those are correct. So I guess that something is wrong with the request. Unfortunately, I don't see the expected request format documented. So, how do I use this model to split text into paragraphs? (If this model is not suited for this task, please correct me and make a suggestion for another model.)

Sign up or log in to comment