• illarion Disabled

    illarion Disabled

    5 months ago
    Hi all! Could you please help me - can i use PythonPickledFile between raw containers? Right now i am trying to use this data type with method described here https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html and getting
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
    when trying to read them by pickle.load(file): inside
    with open(f'{input_dir}/dump_file') as file:
            raw_data = pickle.load(file)
    it is saved without errors by:
    with open(f'{output_dir}/dump_file', 'wb') as handle:
            pickle.dump(raw_data, handle)
    by previous stage raw container
    stage1 = ContainerTask(
        name="stage1",
        output_data_dir="/var/outputs",
        outputs=kwtypes(generation=PythonPickledFile),
        image="<http://registry.project.com/flyte/stage1:0.0.1|registry.project.com/flyte/stage1:0.0.1>",
        command=[
            "python",
            "stage1.py",
            "/var/outputs",
        ],
    )
    
    stage2 = ContainerTask(
        name="stage2",
        input_data_dir="/var/inputs",
        output_data_dir="/var/outputs",
        inputs=kwtypes(dump_file=PythonPickledFile),
        outputs=kwtypes(stage2_out=str),
        image="<http://registry.project.com/flyte/stage2:0.0.1|registry.project.com/flyte/stage2:0.0.1>",
        command=[
            "python",
            "stage2.py",
            "/var/inputs",
            "/var/outputs",
        ],
    )
    
    @workflow
    def my_wf() -> str:
        generation = stage1()
        return stage2(dump_file=generation)
    Can i pass PythonPickledFile like that?
    illarion Disabled
    Kevin Su
    2 replies
    Copy to Clipboard
  • Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Hi, I am getting the following warning in a task Pod when I run a workflow, and I am not able to see the logs from the pods in the console, so I was wondering if that could be related?
    {
      "asctime": "2022-04-05 13:58:10,552",
      "name": "flytekit",
      "levelname": "WARNING",
      "message": "No config file provided or invalid flyte config_file_path flytekit.config specified."
    }
    Hampus Rosvall
    Samhita Alla
    3 replies
    Copy to Clipboard
  • Hampus Rosvall

    Hampus Rosvall

    5 months ago
    I am setting up authentication using OpenID Connect using Azure AD IdP. I have set up an app in the active directory and followed the OpenID Connect steps. I get stuck on the callback from the IdP seeing these logs in the
    FlyteAdmin
    pod. Could it be related to
    oidc_client_secret
    in the
    flyte-admin-secrets
    , or do anyone have an idea on what could cause this?
    flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
    flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T16:53:17Z"}
    flyteadmin-7498449bc8-pl6rw flyteadmin {"json":{"src":"token.go:104"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-04-05T16:53:17Z"}
  • Hampus Rosvall

    Hampus Rosvall

    5 months ago
    Hey, I am setting up OIDC Authentication using Azure AD IdP. I followed the steps here. Right now when I hit the console I am prompted to authenticate against our IdP, then I get sent back to the callback URI,
    https://<host>/callback
    and get stuck in some redirect loop where the
    FlyteAdmin
    Pod keeps logging like below. Does anyone know what could cause this? Thanks 🙂
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_idt] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie_manager.go:71"},"level":"info","msg":"Refresh token doesn't exist or failed to read it. Ignoring this error. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_rt], caused by: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_user_info] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:227"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:84"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"handlers.go:237"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-2m4zv flyteadmin {"json":{"src":"token.go:64"},"level":"debug","msg":"JWT parsing with claims failed failed to verify signature: failed to verify id token signature","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:80"},"level":"debug","msg":"Existing [flyte_at] cookie found","ts":"2022-04-05T17:46:04Z"}
    flyteadmin-6d648c5c7b-x9w54 flyteadmin {"json":{"src":"cookie.go:71"},"level":"info","msg":"Could not detect existing cookie [flyte_rt]. Error: http: named cookie not present","ts":"2022-04-05T17:46:04Z"}
    Hampus Rosvall
    p
    +3
    33 replies
    Copy to Clipboard
  • Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    Attention Airflow users, @Samhita Alla recently created a FlyteOperator in Airflow, and would love if some of you can upvote this operator - https://github.com/apache/airflow/pull/22646
    Ketan (kumare3)
    Samhita Alla
    +2
    8 replies
    Copy to Clipboard
  • Alex Bain

    Alex Bain

    5 months ago
    We're updating our k8s cluster to use OIDC-enabled service accounts as much as possible, but we noticed theflyte-copilotdownloader init container seems to only be able to get IAM creds from the EC2 metadata service at the moment: https://github.com/flyteorg/flytecopilot/blob/master/cmd/root.go#L87. Is that right?
    Alex Bain
    Ketan (kumare3)
    12 replies
    Copy to Clipboard
  • illarion Disabled

    illarion Disabled

    5 months ago
    Can we use NFS instead of S3 in k8s cluster with Flyte?
    illarion Disabled
    Ketan (kumare3)
    14 replies
    Copy to Clipboard
  • Samhita Alla

    Samhita Alla

    5 months ago
    set the channel topic: Next OSS Community Sync is on April 19th, 2021. Meeting Calendar: https://www.addevent.com/calendar/kE355955. Biweekly Minutes: https://bit.ly/37jvXgM. Email List: https://groups.google.com/a/flyte.org/g/users. Getting Started: https://flyte.org.
  • Sandra Youssef

    Sandra Youssef

    5 months ago
    Hi Flyers, And here it is.. Strivework's @Jake Neyer presents *

    Orchestrating MLOps With Flyte

    *: Learn about their Chariot platform, how Flyte fits in, and why they chose Flyte. Thank you for contributing!Meeting Minutes See you all on April 19th! Flyte Team
  • satyanarayana m

    satyanarayana m

    5 months ago
    Hi #general can we have multiple workflows inside one single workflow. Py file. If yes. Can you please share me the reference?
    satyanarayana m
    1 replies
    Copy to Clipboard