Brandon Segal
02/25/2022, 2:54 AM@dataclass_json
@dataclass
class Output:
manifest: FlyteFile
catalog: typing.Optional[FlyteFile]
run_results: typing.Optional[FlyteFile]
sources: typing.Optional[FlyteFile]
My first task makes this object and passes it to the next task but the next task errors with an error that there is no file where the manifest file is. Would love to know if maybe I'm doing something I shouldn't be doing or if there is something I need to do to get this to work.Ketan (kumare3)
Brandon Segal
02/25/2022, 3:32 PM@task(cache_version="1", cache=True)
def create_outputs_task(input_dir:FlyteDirectory)->Output:
manifest_path = os.path.join(input_dir,"manifest.json")
output = Output(
manifest=FlyteFile(manifest_path),
run_results=None,
catalog=None,
sources=None
)
open(manifest_path,"a").close()
return output
@task(cache_version="1", cache=True)
def consume_outputs_task(outputs:Output)->Artifact:
with open(output.manifest,"r") as f:
raw_json = json.load(f)
artifact = parse_artifact(raw_json)
return artifact
@workflow
def the_workflow(input_dir:FlyteDirectory)->Artifact
return consume_outputs_task(
outputs=create_outputs_task(
input_dir=input_dir
)
)
Kevin Su
02/25/2022, 3:53 PMBrandon Segal
02/25/2022, 3:57 PMTraceback (most recent call last):
File "/root/.venv/lib/python3.8/site-packages/flytekit/common/exceptions/scopes.py", line 203, in user_entry_point
return wrapped(*args, **kwargs)
File "/root/.venv/lib/python3.8/site-packages/spotify_dbt_flytekit/tasks/parse_dbt_artifacts.py", line 22, in parse_dbt_artifacts_task
return parse_dbt_artifacts(outputs)
File "/root/.venv/lib/python3.8/site-packages/spotify_dbt_flytekit/tasks/parse_dbt_artifacts.py", line 10, in parse_dbt_artifacts
manifest = parse_manifest(outputs)
File "/root/.venv/lib/python3.8/site-packages/spotify_dbt_flytekit/tasks/parse_manifest.py", line 12, in parse_manifest
with open(outputs.manifest, "r") as f:
Message:
[Errno 2] No such file or directory: '/84/f95f818cb52bf410d9d6-n0-0/ae6a7be9d55ff1a7fe3014fea510a66b/manifest.json'
User error.
Kevin Su
02/25/2022, 3:57 PMKetan (kumare3)
Kevin Su
02/25/2022, 5:32 PMKetan (kumare3)