Moderation

This endpoint combined functionality to check messages for profanity and negative sentiment. If any negative sentiment or profanity is found in a sent message, the endpoint returns data telling you that message is flagged, and why.

Here is a list of all possible classifications in the form of keys the endpoint returns if there's a match

pii, grooming, profanity, sexual, insult, self-harm, toxic, bullying, radical, hate, threat, violence, csamdiscussion, doxxing, vulgar-name

Endpoint

POST https://quanstyst.com/api/v1/moderation

Usage

Headers

Authorization: Requires setting your API Key as the the authorization header's Bearer.

Content-Type: application/json

Parameters

/?filters=profanity,sentiment

By default if you make a post request to the endpoint, the text you send will be checked for both profanity and sentiment.

Checking for sentiment is more resource intensive, and so will take longer to return. If you do not wish to check for sentiment and only for profanity (or vice versa) you can do so by including only what you want in the filter url parameter.

For example, if you only want to check for profanity, the endpoint would be ``https://quanstyst.com/api/v1/moderation/?filters=profanity`

Body

The body requires a field content. This field can be a string, or an array of string.

An optional field threshold can be added, which is a number betwee 0.0 and 1.0.

Response

The shape of the response is:

{
    results: [
      {flagged: boolean, list: [{[string]: number}]}
    ]
}

The response will be an object with the field results.

This results field will contain an array with one item per message.

Each item contains the flagged boolean, and a list array.

The list array will contain a list of filters with the filter key and score that were matched to your input.

The key is what type of filter - ex. "violence", "profanity", etc.

The score is how "sure" the model is of the match.

If there was no match, flagged will be false, and the list will be an empty array.

Example Response

If you input is ['hello', 'bitch', 'b 1 c h']

Your expected response will be

[
  {
    "message": "bitch",
    "flagged": true,
    "list": [
      {
        "label": "profanity",
        "score": 0.9989483952522278
      },
      {
        "label": "insult",
        "score": 0.9953972697257996
      }
    ]
  },
  {
    "message": "b 1 c h",
    "flagged": true,
    "list": [
      {
        "label": "profanity",
        "score": 0.788443386554718
      }
    ]
  },
  {
    "message": "hello",
    "flagged": false,
    "list": []
  }
]

We can see that hello was not flagged, bitch was given .99 for "profanity" and "insult", and b 1 c h was given .78 for "profanity".

This works for general sentiment as well, not just profanity. Phrases that contain negative sentiment can be flagged, like in this example response:

[
  {
    "message": "Ur not from around here boy",
    "flagged": true,
    "list": [
      {
        "label": "harassment",
        "score": 0.7041879296302795
      }
    ]
  },
  {
    "message": "cut urslf",
    "flagged": true,
    "list": [
      {
        "label": "self-harm",
        "score": 0.8145557045936584
      },
      {
        "label": "self-harm/intent",
        "score": 0.7642542719841003
      },
      {
        "label": "self-harm/instructions",
        "score": 0.6352745294570923
      }
    ]
  }
]

Neither of those messages contained any profanity, and the words on their own would not trigger any flagging, but our models can recognize negative sentiment.

Threshold

We have found that generally anything over 0.4 is correct, but you can test this.

The threshold field sent in the body will filter the results. For example, if you set the threshold to 0.7, only results that are above 0.7 will be returned.

The default is 0.4, but you can adjust this to test out the responses by setting the threshold field in the body.

Sample Usage

cURL

curl --request POST \
  --url https://quanstyst.com/api/v1/moderation \
  --header authorization: Bearer <Your API Token> \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '
    {
      "content": ["hello"],
      "threshold": 0.4
    }
  '

JavaScript

const options = {
  method: 'POST',
  headers: {
    authorization: 'Bearer <Your API Token>',
    accept: 'application/json', 
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    "content": ["hello"],
    "threshold": 0.4
  })
};

fetch('https://quanstyst.com/api/v1/moderation', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

Python

import requests

url = "https://quanstyst.com/api/v1/moderation"

payload = {
  "content": ["hello"],
  "threshold": 0.4
}
headers = {
  "authorization": "Bearer <Your API Token>"
  "accept": "application/json",
  "content-type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response)