polite-fish-93728
09/07/2024, 1:43 AMSyntaxError: unexpected character after line continuation character (<unknown>, line 1)
when I run this code as a flyte task. However, the same input works if I run in a normal jupyter notebook. I would like to know if there is any encoding issue that I should check for?
import unicodedata
@task
def preprocess(row: Dict[str, Any]):
row_text = row['text']
row_text = unicodedata.normalize('NFKD', row_text)
# Remove any remaining non-ASCII characters
row_text = row_text.encode('ASCII', 'ignore').decode('ASCII')
pattern = r'[^\w\s\-\.\,\:\;\'\"\{\}\\]'
row_text = re.sub(pattern, '', row_text).strip('"')
row_text_dic = ast.literal_eval(row_text) #Throws exception here
thankful-minister-83577