Thread
#ask-the-community
    k

    KS Tarun

    1 week ago
    Hi All, I have a script with a few tasks and the workflow defined. Certain information is stored in a config file which I am importing into this script. But when I run the workflow, it is giving a module not found error. What should I be doing to import other scripts ?
    Kevin Su

    Kevin Su

    1 week ago
    were you using pyflyte run?
    If so, pyflyte run doesn’t work for this case because it only upload workflow script to s3. We’re trying to improve it. Try to use pyflyte register instead
    k

    KS Tarun

    1 week ago
    Yes, I was using pyflyte run
    Should I now run it as "pyflyte register --remote ..... " ?
    Kevin Su

    Kevin Su

    1 week ago
    yes, and you have to launch the workflow by yourself on flyteconsole
    sorry, no remote flag. something like
    pyflyte --config ~/.flyte/config-remote.yaml register example_union1.py (or a folder example/)
    k

    KS Tarun

    1 week ago
    In this directory : ~/.flyte/ ; I currently have only config-sandbox.yaml.
    Should I create the config-remote.yaml file ?
    Kevin Su

    Kevin Su

    1 week ago
    that’s fine, you can use sandbox.yaml
    k

    KS Tarun

    1 week ago
    Ok, I'll try it
    Thanks Kevin !
    Any thoughts on this ? Since the same workflow is running well when I use pyflyte run --remote.
    Kevin Su

    Kevin Su

    1 week ago
    No need “main”
    pyflyte register only registers your workflow, and won’t execute it. therefore, you have to go to http://localhost:30081/console/projects/flytesnacks/workflows, and launch the wf
    you have to overwrite the aws credentials in the pods. By default, the task output will be written to minio
    $ kubectl edit cm flyte-propeller-config
      k8s.yaml: |
        plugins:
          k8s:
            default-cpus: 600m
            default-env-vars:
            - FLYTE_AWS_ENDPOINT: <http://minio.flyte.svc.cluster.local:9000>
            - FLYTE_AWS_ACCESS_KEY_ID: minio
            - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
            default-memory: 600Mi
    k

    KS Tarun

    1 week ago
    Got it
    Kevin Su

    Kevin Su

    1 week ago
    hmm, do you see any error in command line after you register the workflow?
    k

    KS Tarun

    1 week ago
    No, I don't see any error. But I see the registered workflows in the console now
    There's no issue regarding it, now.
    Kevin Su

    Kevin Su

    1 week ago
    great
    k

    KS Tarun

    1 week ago
    Thanks
    If I modify a script after registering the workflow, how should I ensure that when I re-launch the wf, the code changes are in effect. Currently I don't see that happening unless I register everytime I make a change.
    Kevin Su

    Kevin Su

    1 week ago
    yes, you need to re-register a new workflow if you modify the script. This make every version of the workflow reproducible
    k

    KS Tarun

    1 week ago
    Ok 👍
    The changes are not getting reflected even when I am re-registering as a different version in the same workflow. It's only working when I register it as a new workflow everytime. Any reason why it might be happening ?
    Samhita Alla

    Samhita Alla

    1 week ago
    Can you share with us the command you’re using to register the workflow?
    k

    KS Tarun

    1 week ago
    pyflyte --config ~/.flyte/config-sandbox.yaml register --image fib:1.0 Test1.py
    Samhita Alla

    Samhita Alla

    1 week ago
    Could you share the error, please?
    k

    KS Tarun

    1 week ago
    There is no error. The problem I'm facing is: After I register a workflow & launch it (version 1.0); And when I make some minor changes to the script, I re register the workflow and make this version 1.1. So now when I launch the workflow with the 1.1 version, the changes made in the script are not getting reflected in the output. I'm seeing the same output which was available from the previous run (basically everything is fetched from the cache).
    I want to know the reason behind this .
    Samhita Alla

    Samhita Alla

    1 week ago
    Could you send us the code you’re trying to change? A redacted version of your code should suffice.
    k

    KS Tarun

    1 week ago
    The code is structured to query certain data from a mysql db. A config.py is imported which would have the info about the table to query from and the start and end dates. The first task will query certain records between 2 mentioned dates ( taken from the config file ). Another task will do certain operations and return the output as a list. Now after registering the version 1.0, I go ahead and make some changes in the config file and change the start and end date and re-register it as version 1.1. So theoretically this new version should give a different output compared to the first one.
    Kevin Su

    Kevin Su

    1 week ago
    got you. good catch, thanks. It seems like a bug. I can reproduce this issue 1, register the workflow 2. run the workflow 3. change output to 3 4. re-register the workflow 5. The output of t1 is still 2 (fetch from cache)
    from flytekit import task, workflow
    
    
    @task(cache=True, cache_version="1.0")
    def t1() -> int:
        return 2
    
    
    @workflow
    def wf() -> int:
        return t1()
    
    
    if __name__ == "__main__":
        wf()
    @KS Tarun ^^^ is it the issue you run into?
    k

    KS Tarun

    1 week ago
    Yes, operation wise it's similar. Just that I'm making the changes in a different script (config.py in my case) and importing that into this.
    from flytekit import Resources,task
    from flytekit import workflow
    
    
    @task(cache_version="1.0", cache=True, limits=Resources(mem="500Mi"))
    def add(a:int,b:int) -> int:
        return a+b
    
    @task(cache_version="1.0", cache=True, limits=Resources(mem="500Mi"))
    def multiply(x:int,y:int) -> int:
        return x*y
    
    @workflow
    def output(l:int,m:int,n:int) -> int:
        return multiply(x=n,y=add(a=l,b=m))
    @Samhita Alla I'm registering this workflow. In this code, for version 1.0, I'm multiplying x & y. And later in the next version I'm adding x & y. The output when I run the 1.1 version is same as 1.0.
    Samhita Alla

    Samhita Alla

    1 week ago
    @Kevin Su, do we have an issue for this?
    Kevin Su

    Kevin Su

    1 week ago
    No, we don’t
    Samhita Alla

    Samhita Alla

    1 week ago
    I feel this is a high priority issue. WDYT, Kevin?
    Also, this isn’t working when run locally, right?
    Kevin Su

    Kevin Su

    1 week ago
    yes, will work on it today.
    Also, this isn’t working when run locally, right?
    yes, and running remotely doesn’t work as well
    Yee

    Yee

    6 days ago
    @Eduardo Apolinario (eapolinario)
    oh nm
    with dan on this one, i think this is the intended behavior
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    6 days ago
    we talked about deriving the version of tasks from the code itself in the past, but that's not currently implemented.
    Something like https://github.com/flyteorg/flyte/issues/770, but for all tasks