Ferdinand von den Eichen
07/10/2023, 9:39 AMKevin Su
07/10/2023, 1:50 PMJan Fiedler
07/10/2023, 1:57 PMKevin Su
07/10/2023, 1:58 PMJan Fiedler
07/10/2023, 2:04 PMfrom flytekit import workflow, task
import pandas as pd
@task
def load_sample_data(environment: str) -> pd.DataFrame:
print(f"Loading Sample Data from {environment}")
sample_data_path = f"<s3://flyte-demo>-{environment}-sample-data-bucket"
sample_data = pd.DataFrame(
{
"Name": ["Broder", "Ferdi", "Jan", "Korbi", "Robert"],
"Job": ["MLOPS", "MLOPS", "MLOPS", "ML", "SALES"],
}
)
return sample_data
@task
def filter_for_job(sample_data: pd.DataFrame, job_filter: str) -> pd.DataFrame:
sample_data_preprocessed = sample_data[sample_data["Job"] == job_filter]
return sample_data_preprocessed
@task
def output_result(sample_data: pd.DataFrame):
print(sample_data.head())
@workflow
def wf(environment: str = "dev"):
job_filter = "MLOPS"
data = load_sample_data(environment=environment)
filtered_data = filter_for_job(sample_data=data, job_filter=job_filter)
output_result(sample_data=filtered_data)
Kevin Su
07/10/2023, 2:07 PMWe are using the multi cluster setup btw. dont know if this causes problemsdid you see any error in the flyteconsole pod?
Ferdinand von den Eichen
07/10/2023, 2:27 PM::ffff:122.175.45.58 - - [10/Jul/2023:14:25:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
During this time frame we frequently clicked on the respective UI section…What role is the flyteconsole service using?Kevin Su
07/10/2023, 2:32 PMJan Fiedler
07/10/2023, 2:48 PM{
"json": {
"exec_id": "al4ncm8fx74ss8pjk4bf",
"node": "n6"
},
"level": "error",
"msg": "Failed to read from the raw store [<s3://supply-fc-prod-data-lake-cluster-bucket/metadata/propeller/energy-supply-forecast-production-al4ncm8fx74ss8pjk4bf/n6/data/inputs.pb>] Error: Conf container:customer-flyte-central-cluster-bucket != Passed Container:supply-fc-prod-data-lake-cluster-bucket. Dynamic loading is disabled: not found",
"ts": "2023-07-10T14:37:06Z"
}
Looks like flyteadmin is expecting some files in his own cluster bucket not in the data plane bucketKevin Su
07/10/2023, 3:00 PMJan Fiedler
07/10/2023, 9:54 PMKevin Su
07/11/2023, 1:07 AMJan Fiedler
07/11/2023, 7:00 AM