Quentin Chenevier
09/06/2023, 9:45 AMFranco Bocci
09/06/2023, 10:56 AMmy-own-bucket
, and then you can upload data from your Flyte workflow to such bucketKetan (kumare3)
Franco Bocci
09/06/2023, 1:45 PMKetan (kumare3)
Quentin Chenevier
09/06/2023, 8:12 PMKetan (kumare3)
Quentin Chenevier
09/06/2023, 8:44 PMKetan (kumare3)
Quentin Chenevier
09/06/2023, 9:16 PM# %%
import pandas as pd
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config
# %%
flyteremote = FlyteRemote(config=Config.for_sandbox())
client = flyteremote.client
# %%
data = []
projects = client.list_projects().projects
for project in projects:
for domain in project.domains:
executions, _ = client.list_executions_paginated(
project=project.id, domain=domain.id
)
for execution in executions:
node_executions, _ = client.list_node_executions(workflow_execution_identifier=execution.id) # type: ignore
for node_execution in node_executions:
node_execution_data = client.get_node_execution_data(node_execution.id)
for k, v in node_execution_data.full_outputs.literals.items():
data.append(
dict(
project=execution.id.project,
domain=execution.id.domain,
execution_name=execution.id.name,
node_id=node_execution.id.node_id,
param_type="output",
param_name=k,
param_value=v.scalar.value,
)
)
df = pd.DataFrame(data)
df
On a side note we are working on a dataset / artifact service. You will see It sooon.Do you know what is the expected release date of this service ? Weeks, months or years ? I'm curious 😉
Ketan (kumare3)
Quentin Chenevier
09/08/2023, 7:11 AMKetan (kumare3)