The place that you would interface a custom script is on the "Custom Script Tool" tab page of the Modify Format option:
After you configure your tool, use the Presets menu and choose Save As to save it. You can even set a custom hotkey from the "Preset Hotkey Trigger" tab before saving.
The key to understanding how to use custom tools/scripts with CHS is to see from that dialog that CHS will first save the clipboard text to a file, and then pass that file as an argument to the script, and then expect the script to either save changes to that same file, or to a different file or output it to stdout.
So what you need to do to use REXX or any other scripting tool, is test calling your tool from a dos commandline. Some scripting languages will let you call them directly like "myscript.py %infile%" but others may require you to call them with the full path to the interpreter like "C:\Python.exe myscript.py %infile%".
Then just test your script from the commandline until it does what you want.
There are some sample tools in the Tools\ directory of CHS to help you see how a tool should behave.