Our inputs and outputs are always empty, why is th...
# ask-the-community
f
Our inputs and outputs are always empty, why is this? The tasks themselves definitely have both:
k
Seems like a UI bug. are you able to share the code? which version of flyteconsole are you using?
j
What code do you mean. We are using flyteconsole version: v1.4.1
k
workflow code
j
It happens to all our workflow code. Even with the simplest one:
Copy code
from flytekit import workflow, task
import pandas as pd


@task
def load_sample_data(environment: str) -> pd.DataFrame:
    print(f"Loading Sample Data from {environment}")
    sample_data_path = f"<s3://flyte-demo>-{environment}-sample-data-bucket"

    sample_data = pd.DataFrame(
        {
            "Name": ["Broder", "Ferdi", "Jan", "Korbi", "Robert"],
            "Job": ["MLOPS", "MLOPS", "MLOPS", "ML", "SALES"],
        }
    )
    return sample_data


@task
def filter_for_job(sample_data: pd.DataFrame, job_filter: str) -> pd.DataFrame:
    sample_data_preprocessed = sample_data[sample_data["Job"] == job_filter]
    return sample_data_preprocessed


@task
def output_result(sample_data: pd.DataFrame):
    print(sample_data.head())


@workflow
def wf(environment: str = "dev"):
    job_filter = "MLOPS"
    data = load_sample_data(environment=environment)
    filtered_data = filter_for_job(sample_data=data, job_filter=job_filter)
    output_result(sample_data=filtered_data)
We are using the multi cluster setup btw. dont know if this causes problems
k
thanks, looking
it works for me.
We are using the multi cluster setup btw. dont know if this causes problems
did you see any error in the flyteconsole pod?
feel like flyteconsole doesn’t have access to download input / output data
f
These are the logs for the console service
Copy code
::ffff:122.175.45.58 - - [10/Jul/2023:14:25:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:34 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:25:49 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:26:04 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.88.7 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.45.58 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
::ffff:122.175.23.11 - - [10/Jul/2023:14:26:19 +0000] "GET / HTTP/1.1" 200 1240 "-" "ELB-HealthChecker/2.0"
During this time frame we frequently clicked on the respective UI section…What role is the flyteconsole service using?
k
depends on your cluster setup, but it should have access to s3 bucket
j
We found the correct error logs for our problem:
Copy code
{
  "json": {
    "exec_id": "al4ncm8fx74ss8pjk4bf",
    "node": "n6"
  },
  "level": "error",
  "msg": "Failed to read from the raw store [<s3://supply-fc-prod-data-lake-cluster-bucket/metadata/propeller/energy-supply-forecast-production-al4ncm8fx74ss8pjk4bf/n6/data/inputs.pb>] Error: Conf container:customer-flyte-central-cluster-bucket != Passed Container:supply-fc-prod-data-lake-cluster-bucket. Dynamic loading is disabled: not found",
  "ts": "2023-07-10T14:37:06Z"
}
Looks like flyteadmin is expecting some files in his own cluster bucket not in the data plane bucket
k
j
Is there a way to control this variable via helm or how would I enable this?
k
update flyteadmin configmap
j
We got this working now 🙂 Thank you so much Kevin!