Terence Kent09/01/2023, 12:18 AM
file, and write it to object storage in a different cloud account. I can accomplish this with custom tasks that connect to the source and target object storage systems, but that seems like a lot of boilerplate. It sure seems like Flyte is not designed for these types of ETL efforts and a different tool should be used for those. Am I reading the situation right?
Terence Kent09/01/2023, 2:56 AM
file to a local file (maybe the Flyte data 3. Uploads the data to the s3 bucket All this would work fine, of course, but that's the boilerplate I'm referring too. Since neither the source nor destination would be the FlyteObject store configured for the environment, I think we'd need to use some 3rd party SDKs to interact with the object stores.
Terence Kent09/01/2023, 3:03 AM
are good fits for that type of activity - that's great
Terence Kent09/01/2023, 3:12 AM
Kevin Su09/01/2023, 3:13 AM
from flytekit.sensor.file_sensor import FileSensor from flytekit import task, workflow sensor = FileSensor(name="test_sensor") @task() def t1(): print("flyte") @workflow def wf(): sensor(tmp_file.name) >> t1()
Terence Kent09/01/2023, 3:36 AM
to interact with external object stores).