Hi All, I am trying to upload a JOSN file as a FlyteFile to remote S3, but getting error -

TypeError: Object of type FlyteFile is not JSON serializable

I am getting the same error even if I create a JSON file locally first and then pass S3 URL as remote_path. Any help would be appreciated. Thank you

Hmm this seems like a different error can you share a code snippet

@freezing-airport-6809 here is the code snippet -

def create_flyte_file(data, file_name):
    full_s3_path = f"{s3_prefix}{s3_bucket}/{s3_path}/{file_name}"
    local_path = os.path.join(flytekit.current_context().working_directory, file_name)

    with open(local_path, 'w') as w:
        json.dump(data, w)

    return FlyteFile(path=local_path, remote_path=full_s3_path)

it works for me

def create_flyte_file(data, file_name):
    full_s3_path = f"/tmp/123"
    local_path = os.path.join(flytekit.current_context().working_directory, file_name)

    with open(local_path, 'w') as w:
        json.dump(data, w)

    return FlyteFile(path=local_path, remote_path=full_s3_path)


@task
def t1() -> FlyteFile:
    return create_flyte_file({"a": 1, "b": 2}, "test.json")


if __name__ == '__main__':
    t1()

Thank you @glamorous-carpet-83516 So This code is running fine for me as well. The issue is, I am returning a FlyteFile in a separate function written in utils. And looks like if we use

FlyteFile

outside of a Flyte task, such as in a regular Python function, Flyte's automatic data management features are not triggered.

yup, we only trigger it when serialize the flytefile to protobuf.

@glamorous-carpet-83516, I am trying to use from_json and to_json methods of FlyteFile to work. I am getting an error :

TypeError: FlyteFile.__init__() missing 1 required positional argument: 'path'

Code -

ff_uploaded = FlyteFile(full_s3_path).from_json(json_data)

Can you help me to understand the use of these methods. I appreciate your help in advance