Splitting Input Text
On a per-policy basis, Philter can split input text to process each split individually. This can improve performance and allows for handling long input text. Splitting is disabled by default.
An example split configuration in a policy is shown below
In this example policy, splitting is enabled for inputs greater than equal to 10,000 characters in length.
The method of splitting the text will be the
newline method. This method will cause Philter to split the text based on the locations of new line characters in the input text. Additional methods of text splitting may be added in future versions.
Because the newline method splits text based on the locations of new line characters in the text, the text contained in the reassembled filter responses may not be an exact match of the input text. This is due to white space and other characters that may reside near the new line characters that get omitted during processing.
Text Splitting Policy Properties
Whether or not input texts are split. Whether or not input texts are split. When
When to split the input text. Set to
Any integer value.
How to split the text.
Alternative to Philter Splitting Text
In some cases it may be best to split your input text client side prior to sending the text to Philter. This gives you full control over how the text will be split and provides more predictable responses from Philter because you know how the text is split.
An example of splitting text into chunks prior to sending the text to Philter is given in the commands below: