Hi everyone, I am trying to preprocess a string a...
# flyte-support
p
Hi everyone, I am trying to preprocess a string and execute ast.literal_eval() to convert it into dictionary. I am getting errors such as
SyntaxError: unexpected character after line continuation character (<unknown>, line 1)
when I run this code as a flyte task. However, the same input works if I run in a normal jupyter notebook. I would like to know if there is any encoding issue that I should check for?
Copy code
import unicodedata

@task
def preprocess(row: Dict[str, Any]):
    row_text = row['text']
    row_text = unicodedata.normalize('NFKD', row_text)
    # Remove any remaining non-ASCII characters
    row_text = row_text.encode('ASCII', 'ignore').decode('ASCII')
    pattern = r'[^\w\s\-\.\,\:\;\'\"\{\}\\]'
    row_text = re.sub(pattern, '', row_text).strip('"')
    row_text_dic = ast.literal_eval(row_text) #Throws exception here
t
could you give us an example of what row[text] looks like?