https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • a

    Adedeji Ayinde

    10/28/2022, 9:20 PM
    I am trying to register and run a workflow on a remote flyte cluster using an image in ecr, however I keep getting error message. Does anyone how to get around this
    img = ImageConfig.from_images(
        "6***********.<http://dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest|dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest>"
    )
    
    flyte_workflow = remote.register_workflow(
        entity=wf_bb,
        serialization_settings=SerializationSettings(image_config=img),
        version="v1.0",
    )
    Error
    [1/1] currentAttempt done. Last Error: USER::containers with unready status: [ax9m22wmc8xwpmhqmllt-n0-0]|Back-off pulling image "6*********.<http://dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest|dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest>"
    k
    • 2
    • 1
  • m

    Mehtab Mehdi

    10/30/2022, 9:22 PM
    Hi Team, i am trying to integrate the databricks with flyte.. Data is coming and UI is coming when i am using task and plugin code in single file. while i am creating ECR image and trying to integrate than it is connecting on local system while i am running pyflyte --remote --image<ECR image name> myfly.py wf it is throwing the following Error
    [1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution
    Can anyone help me to solve this issue.
    s
    e
    • 3
    • 11
  • s

    SeungTaeKim

    10/31/2022, 4:25 AM
    Hi, flyte team! I would like to ask
    Promise object
    of task's return values.
    from flytekit import workflow, task, dynamic, conditional
    from flytekit.core.promise import Promise
    
    @task
    def t1() -> bool:
        return True
    
    
    @task
    def t2() -> bool:
        return False
    
    
    @workflow
    def wf() -> bool:
        test = t1()
        print(test)
        return t2()
    
    
    if __name__ == "__main__":
        print(wf())
    when I run this sample codes through
    pyflyte
    , a task returns
    Promise object
    , and literal or objects I expect are wrapped in
    Promise
    . So, here is my question, how I unwrap this promise object? I would like to use the expected value from the function. Thank you!
    s
    k
    • 3
    • 9
  • t

    Tarmily Wen

    10/31/2022, 3:48 PM
    Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example
    @task(
        retries=2,
        cache=True,
        cache_version="1.0",
        requests=Resources(gpu=gpu, mem=mem, storage=storage),
        limits=Resources(gpu=gpu, mem=mem, storage=storage),
        secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
    )
    def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
        secrets = current_context().secrets
        wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
    And I would like to call a remote execution and inject the secret at the same time like this:
    current_config = Config.auto()
    
    remote = FlyteRemote(config=current_config)
    flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
    workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
    But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?
    t
    s
    +2
    • 5
    • 15
  • l

    Laura Lin

    10/31/2022, 4:11 PM
    Is there support for tasks with inputs that use Unions?
    Union[str, flytekit.types.directory.FlyteDirectory]
    . Trying to create a task that could take in a str or a flytedirectory but I get
    raise ValueError(f"Expected a directory. {source_path} is not a directory")
    ValueError: Expected a directory. X is not a directory
    
    AssertionError: Failed to Bind variable input for function
    k
    k
    • 3
    • 13
  • h

    Hadi

    11/01/2022, 10:31 AM
    Is it possible to select the gpu or machine type when creating a task?
    s
    • 2
    • 1
  • y

    Yash Panchwatkar

    11/01/2022, 11:38 AM
    Hello community, how to use image from AWS ECR while running pyflyte register -i command .. by default it's pointing to the public docker hub .. is there any configuration given for the same.. i have tried imagePullSecret by attaching to the default service account still it's not working so...
    s
    y
    k
    • 4
    • 15
  • z

    Zhiyi Li

    11/01/2022, 12:57 PM
    Hey folks, I am trying to set up kafka to gather platform events for external consuming. I am wondering whether all the IDL messages would be sent out via the same topic as it writes https://github.com/flyteorg/flyteadmin/blob/46598814b2bcb277762f96a666b2c4649cbdbe8a/pkg/async/cloudevent/factory.go#L67 also I’m wondering what’s the purpose of this? (is this used for notification or external event?) https://github.com/flyteorg/flyteadmin/blob/bca622830ff563984bd3270708ec8b6a5ff83945/pkg/async/notifications/factory.go#L183
    k
    k
    • 3
    • 2
  • l

    Laura Lin

    11/01/2022, 6:51 PM
    How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I would like to change it to always for all of my potential docker images/containers. Do I have to use https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages?
    k
    k
    d
    • 4
    • 23
  • y

    Yash Panchwatkar

    11/02/2022, 11:40 AM
    Hi @Kevin Su when i try to run my workflow I get the following error. I am using private ecr image .
    [1/1] currentAttempt done. Last Error: USER::containers with unready status: [apv828cglh4qs76ftwgf-n0-0]|Back-off pulling image "<http://x.y.z.dkr.ecr.us-west-2.amazonaws.com/flyte_test:adiwala1|x.y.z.dkr.ecr.us-west-2.amazonaws.com/flyte_test:adiwala1>"
    To make sure that imagePullSecret are present in service account I ran the following commands
    kubectl get pod nginx -o=jsonpath='{.spec.imagePullSecrets[0].name}{"\n"}'
    The output was
    reg-ecr-cred
    Can you please help me out here what am I missing?
    k
    • 2
    • 4
  • s

    Sampath Vaddadi

    11/02/2022, 1:37 PM
    Hi All, can anyone please tell me how can I control the type of instance that I can get as part of spark cluster when running a spark job through flyte? Is there such a provision? I see that we can specify spark_conf, does that take care of what type of instances are brought into kubernetes cluster for spark job? I have my kubernetes cluster on aws In databricks, I can control what type of ec2 instances by master and worker nodes are. I'm looking for similar feature in flyte
    k
    r
    • 3
    • 3
  • s

    seunggs

    11/02/2022, 6:00 PM
    Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?
    r
    k
    +2
    • 5
    • 14
  • l

    Laura Lin

    11/02/2022, 9:36 PM
    Are help strings for workflow inputs in the console UI still supported? I know it used to be possible (I think maybe through imperative workflows or something) but can't seem to find it in the docs.
    k
    s
    k
    • 4
    • 8
  • d

    Dennis O'Brien

    11/02/2022, 10:09 PM
    I'm just getting to adding a schedule to my first Flyte workflow and I have a few questions. My workflow takes two arguments, a datetime and an int.
    @workflow
    def cohort_ltv_inference(inference_date: datetime, n_days_horizon: int) -> bool:
        ...
    I had created a schedule in my launch plan but failed to provide
    kickoff_time_input_arg
    or
    default_inputs
    .
    launch_plan = LaunchPlan.get_or_create(
        name="pir_cohort_ltv_inference",
        workflow=cohort_ltv_inference,
        schedule=CronSchedule(
            schedule="45 17 * * *",  # At 17:45 daily.
        ),
    )
    In the workflow web ui, I see the schedule displayed:
    Schedules
    At 05:45 PM
    But when I noticed that no scheduled run was actually happening, I looked through the scheduler logs and saw messages like this:
    rpc error: code = InvalidArgument desc = expected_inputs inference_date missing
    Ideally the end user would be alerted about this earlier. I saw a related issue on github here, and a PR here. I haven't worked in Go, but I think the function
    validateLaunchSpec
    doesn't handle the case of no
    default_inputs
    and no
    fixed_inputs
    passed to the launch plan
    get_or_create
    . In general I'm missing some of the familiar scheduler UI I am used to from my experience with Airflow. I know these are very different projects, but for almost all my use cases, I'll eventually be running the ML pipeline on a schedule.
    s
    p
    • 3
    • 5
  • k

    Kamakshi Muthukrishnan

    11/03/2022, 5:18 AM
    Hi Team, I am getting module 'xgboost' has no attribute 'rabit' while using modin. Any specific version of xgboost/modin/ package will resolve this issues???
    s
    • 2
    • 1
  • k

    Kamakshi Muthukrishnan

    11/03/2022, 5:18 AM
    ERROR worker.py:399 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::ModinXGBoostActor.train() (pid=29796, ip=172.16.54.211, repr=<modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor object at 0x7fbef05837f0>)
      File "/home/ec2-user/anaconda3/envs/python3/lib/python3.8/site-packages/modin/experimental/xgboost/xgboost_ray.py", line 166, in train
        with RabitContext(self._rank, rabit_args):
      File "/home/ec2-user/anaconda3/envs/python3/lib/python3.8/site-packages/modin/experimental/xgboost/utils.py", line 101, in __enter__
        xgb.rabit.init(self.args)
    AttributeError: module 'xgboost' has no attribute 'rabit'
  • k

    Kamakshi Muthukrishnan

    11/03/2022, 6:15 AM
    Yes.. I just added rabit as a seperate utils and fixed the above error
  • w

    Willem Gillis

    11/03/2022, 12:23 PM
    Hi, our team is trying to use fast registration using
    pyflyte register
    but hitting a wall. The registration passes and it's worklows and launchplans even shows up in Flyte Console however when executing it we get error:
    ModuleNotFoundError: No module named 'flyte'
    When looking at the resulted generated file on the S3 minio we find that it is completely empty. Our full command is:
    pyflyte --config ~/.flyte/cloud-config.yaml register flyte/workflows -p {{.PROJECT}} --domain {{.DOMAIN}} -v {{add .DOCKER_VERSION 1}} -i "{{.CONTAINER_REGISTRY}}/{{.PROJECT}}:{{.DOCKER_VERSION}}"
    Folder structure see image Note that using the
    pyflyte package
    and
    flytectl register files
    in the same project does work as expected. Are we missing something?
    s
    k
    e
    • 4
    • 12
  • f

    Felix Ruess

    11/03/2022, 2:37 PM
    Is there a way to create ContainerTask with a string input and use that input in the command without writing cumbersome wrapper scripts which need to be part of the image? https://docs.flyte.org/projects/cookbook/en/stable/auto/core/containerization/raw_container.html#container-tasks
    k
    e
    • 3
    • 26
  • e

    Edgar Trujillo

    11/03/2022, 3:37 PM
    Is anyone else seeing
    FlytePropeller &  user defined pod executions
    logs having for the most part static timestamps? We're running EKS with FluentBit deployed that pushes logs to Cloudwatch, but seeing the following
    s
    k
    • 3
    • 8
  • f

    Felix Ruess

    11/03/2022, 5:26 PM
    Is there an alternative way to specify
    FLYTE_AWS_SECRET_ACCESS_KEY
    not as env var so that this secret is not visible to anyone who can see get the pod details?
    r
    • 2
    • 1
  • v

    varsha Parthasarathy

    11/03/2022, 7:10 PM
    Hi team, Is this expected? or are there workarounds?
    flytekit.core.type_engine.RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently
    k
    n
    +2
    • 5
    • 21
  • y

    yujinlee

    11/04/2022, 4:15 AM
    Hello, can I get some advice regarding to contribution in flyte? I want to setup timezone as Asia/Seoul, so I'm considering to add feature to set timezone. I was thinking to set it up in
    flyte_config.yaml
    as follows:
    admin:
      # For GRPC endpoints you might want to use dns:///flyte.myexample.com
      endpoint:
      authType: Pkce
      insecure: false
    timezone: ??
    Any suggestion regarding with this feature? I also wonder which git repository to start in FlyteOrg. Thank You!
    s
    s
    k
    • 4
    • 6
  • s

    Sanjay Chouhan

    11/04/2022, 7:22 AM
    Can not run helm upgrade. Getting error 'Error: failed to download "flyteorg/flyte-core"'
    (base) sanjaychouhan@prod-ml-jenkins2:~$ helm upgrade -n flyte -f values-eks.yaml --create-namespace flyte flyteorg/flyte-core
    Error: failed to download "flyteorg/flyte-core"
    (base) sanjaychouhan@prod-ml-jenkins2:~$
    (base) sanjaychouhan@prod-ml-jenkins2:~$ helm repo add flyteorg <https://flyteorg.github.io/flyte>
    Error: repository name (flyteorg) already exists, please specify a different name
    k
    • 2
    • 2
  • f

    Felix Ruess

    11/04/2022, 11:49 AM
    I want to inject secrets as env vars into my raw container and thought I can do it via secret_requests: https://github.com/flyteorg/flytekit/pull/1307 But I was surprised that the injected env var is named prefix+group+key, the prefix I can set to empty, the group obviously not. This essentially forces me to create multiple k8s secrets (secret name becomes group) for each env var that I need injected that has a different prefix. So it seems it was not intended to be used this way (that the key becomes the env var name)? Point in case I need to inject
    AWS_ACCESS_KEY_ID
    and
    AWS_SECRET_ACCESS_KEY
    and thought I can get the correct ones by specifying the group (k8s secret name). But how it currently works I would need to use
    AWS
    as group name and hence can have only one k8s secret with that name....
    • 1
    • 1
  • p

    Padma Priya M

    11/04/2022, 1:10 PM
    Hi, I am getting error while executing ray with optuna in flyte.
    s
    e
    k
    • 4
    • 8
  • f

    Felix Ruess

    11/04/2022, 3:51 PM
    Hi, I ran into the default memory limit:
    Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
    And I already increased the task resource limits but I still get the same error. Any pointers?
    l
    k
    y
    • 4
    • 49
  • l

    Louis DiNatale

    11/04/2022, 4:20 PM
    Hey Flyte team, happy Friday. When it comes to the maxParallelism config where should I be looking for errors as i start to increase this number?
    s
    k
    +3
    • 6
    • 15
  • j

    Jay Ganbat

    11/04/2022, 9:22 PM
    Hi flyte folks, was there any significant performance change in Flyte console? we recently swtiched to flyte 1.2 milestone version and its correspodning flyte backend. Unfortunately now its taking a lot of time display the workflow tasks as opposed to what previously and from the browser log `
    vendor-a9fbc36b.js:2          GET https://.../fizlxcwy-n4-0-dn1-0/n1?limit=10000 net::ERR_INSUFFICIENT_RESOURCES
    it might have been showing previously as well but never seem to be noticable. it takes solid min or 2 to even show what tasks were executed. previous version was flyteconsole
    1.1.6
    , propeller
    1.1.15
    and admin on
    1.1.29
    any idea on what have changed
    j
    k
    e
    • 4
    • 29
  • f

    Felix Ruess

    11/05/2022, 5:19 PM
    What endpoints do I need to scrape to get metrics? And what ingress rules might I need to set up for this? It doesn't say anything in the docs: https://docs.flyte.org/en/latest/deployment/cluster_config/monitoring.html
    s
    k
    • 3
    • 4
Powered by Linen
Title
f

Felix Ruess

11/05/2022, 5:19 PM
What endpoints do I need to scrape to get metrics? And what ingress rules might I need to set up for this? It doesn't say anything in the docs: https://docs.flyte.org/en/latest/deployment/cluster_config/monitoring.html
s

Samhita Alla

11/07/2022, 4:27 AM
Have you checked https://github.com/flyteorg/flyte/tree/master/stats?
f

Felix Ruess

11/07/2022, 8:16 AM
yeah, I found the dashboards there, but doesn't say how I can scrape them
s

Samhita Alla

11/07/2022, 10:33 AM
cc: @Ketan (kumare3)
k

Ketan (kumare3)

11/07/2022, 3:29 PM
Ohh that’s a miss. All Flyte services exposes port 10254 /metric’s Prometheus compatible endpoint
View count: 3