Jay Phan02/08/2023, 7:17 AM
Any idea how to fix this?
Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point return wrapped(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 499, in dispatch_execute raise e File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 496, in dispatch_execute native_outputs = self.execute(**native_inputs) File "/usr/local/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 215, in execute subprocess.check_call(gen_script, shell=True) File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) Message: Command ' mkdir -p output query=data/kitasatospora/k_sp_CB01950_penicillin.fasta db=data/kitasatospora/kitasatospora_proteins.faa blastout=output/AMK19_00175_blastx_kitasatospora.tab blastx -out $blastout -outfmt 6 -query $query -db $db >> stdout.txt 2>&1 ' returned non-zero exit status 127. SYSTEM ERROR! Contact platform administrators.
Eli Bixby02/08/2023, 12:44 PM
If i go to launch
@dataclass_json @dataclass class MyStruct foo: str bar: str = 'baz' def workflow(s: MyStruct): pass qux = LaunchPlan(workflow, default_inputs=dict(s=MyStruct(foo='baz', bar='qux')))
launch plan it instead takes the default values from the struct definition. So
will be unset and
will be set by default as
Not sure where the error is here, though. Has anyone else seen this?
Kevin Schaich02/08/2023, 5:13 PM
consistently fails to discover where config lives (
). the only way i've gotten it to work is by supplying
arg manually – but I thought that was the point of having a default location:
just me or has anyone else encountered? did I forget to set an env variable somewhere? I'm on an M1 Mac – Ventura
> flytectl config init INFO  Couldn't find a config file . Relying on env vars and pflags. Init flytectl config file at [/Users/kevinschaich/.flyte/config.yaml] > flytectl config discover INFO  Couldn't find a config file . Relying on env vars and pflags.
Ena Škopelja02/08/2023, 5:46 PM
that includes a dynamic workflow. The dynamic workflow starts a launchplan defined in
(see attached sketch). This is my code structure:
What I'm doing right now: • add
|- project_1 / | |- pyproject.toml | |- project_1 / | | |- __init__.py | | |- workflows.py | | |- tasks.py | | |- models.py | | |- ... |- project_2 / | |- pyproject.toml | |- project_2 / | | |- __init__.py | | |- workflows.py | | |- tasks.py | | |- models.py | | |- ... |- Dockerfile |- docker_build.sh
as a dependency to project one • build a docker image with
installed as a dependency to project 1 • package + register • run This approach requires rebuilding the docker image with every execution because fast register will not realise that the launchplan defined in
should have a different version than the rest of the workflow and it fails while trying to fetch the workflow. Another option would be to do something like:
but that would require my pod to be able to reach flyteadmin because it's inside a dynamic workflow, Is there anything I can do to avoid rebuilding the docker image for every run? Am I doing something wrong? Any advice is much appreciated. Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn't provide any link to the newly spawned workflows so I can't reliably track the intermediate states or see why something failed (if it did). (see screenshot, the purple task is dynamic). It also leads to all sorts of mixed status reports (see other screenshot).
remote = FlyteRemote(config=Config.auto()) launchplan = remote.fetch_launch_plan(...)
Igor Konforti02/08/2023, 6:02 PM
Derek Yu02/08/2023, 9:26 PM
varsha Parthasarathy02/08/2023, 9:29 PM
Aleksander Lempinen02/09/2023, 9:07 AM
My spark config in values.yaml is:
An error occurred while calling o125.parquet. : java.nio.file.AccessDeniedException: s3://<bucket>/<path>: getFileStatus on s3://<bucket>/<path>: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;
I can specify the AWS credentials manually and use "spark.hadoop.fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" in the spark config, but I was wondering if you can use IRSA to avoid using credentials.
spark-config-default: # We override the default credentials chain provider for Hadoop so that # it can use the serviceAccount based IAM role or ec2 metadata based. # This is more in line with how AWS works - spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain" - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2" - spark.kubernetes.allocation.batch.size: "50" - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl" - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A" - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A" - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A" - spark.hadoop.fs.s3a.multipart.threshold: "536870912" - spark.blacklist.enabled: "true" - spark.blacklist.timeout: "5m" - spark.task.maxfailures: "8"
Kamakshi Muthukrishnan02/09/2023, 1:10 PM
Théo LACOUR02/09/2023, 2:26 PM
, and then use
to package the code and then upload and run the new version of the workflow, we notice that the behavior
pyflyte --pkgs path.to.code package --image my_image --force --fast
is the same as before. As I understand it now, this could be because the Spark executors still have the previous code. Does this work as intended ? TL;DR : can we use
to avoid building / pushing Docker images for Spark tasks ?
Ferdinand von den Eichen02/09/2023, 2:34 PM
Sujith Samuel02/09/2023, 4:28 PM
Joseph Yi02/09/2023, 7:34 PM
Buvaneswari Ramanan02/09/2023, 8:08 PM
Mohd Shahid Khan Afridi02/10/2023, 7:02 AM
. Pods are unable to come up and are crashing since then with below error: Error from
helm upgrade -n flyte flyte -f values-override.yaml ./flyte/charts/flyte-core
Can anyone help me with this?
2023/02/10 06:54:59 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/database.go:114 [error] failed to initialize database, got error failed to connect to `host=postgres user=postgres database=flyteadmin`: hostname resolving error (lookup postgres on <ip address>:53: no such host) Error: failed to connect to `host=postgres user=postgres database=flyteadmin`: hostname resolving error (lookup postgres on <ip address>:53: no such host)
Mohd Shahid Khan Afridi02/10/2023, 12:11 PM
message: >- failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [tensorflow]: found no current condition. Conditions: 
Taylor Stout02/10/2023, 3:39 PM
Original exception: HTTPConnectionPool(host='signals-minio', port=9000): Max retries exceeded with url: /flyte/flytesnacks/development/YK2ZGX4HTNQPDYSEHLOIX7K2UM%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio_access_key%2F20230210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230210T152959Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11d5884c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
Eduardo Matus02/10/2023, 4:55 PM
Frank Shen02/10/2023, 5:12 PM
Frank Shen02/10/2023, 5:26 PM
Björn02/10/2023, 7:25 PM
Dennis O'Brien02/11/2023, 8:36 AM
that indicates how far back in the past the time based view should go. All past workflow executions will be scheduled when this is registered, perhaps sequentially. ◦ Defaults to a value based on the time the schedule was activated. That is, by default there is no backfill done. ◦ This determines the number of backfill "slots" to be executed. • Problem: The workflow UI shows the list of workflow executions based on the time they were run. But for scheduled launch plans, I want to see a view based on the scheduled times. How do I know if a particular failed schedule workflow execution ever had a subsequent success? ◦ Potential approach: Have a "schedule time" based view for each schedule. ◦ Visually display the status of each execution slot in the backfill by the status of the last execution. E.g., success, failure, running, etc. ◦ Click on an execution slot to relaunch a failed or successful task or launch a task that has not yet run. • Problem: Running backfills for a scheduled launch plan is tedious. Is there a better way than just a lot of clicking in the UI? ◦ Potential approach: Provide backfill capabilities in the UI for scheduled launch plans. For example, select a launch plan with a schedule, select a start datetime and end datetime, then run. ◦ Provide a few concurrency and backfill settings. (Borrowed from Airflow) ◦
sets the maximum number of executions running concurrently. Setting to 1 would backfill sequentially, one at a time. • Problem: In the workflow UI, in the section "All Executions in the Workflow", there is no easy way to distinguish between scheduled runs and manually triggered runs. And the inputs to the execution are not readily available in this view. ◦ Potential approach: Enable a filtered view of only those executions that were initiated by a schedule (rather than by the user launching via the UI). ◦ Filter by the name of the launch plan used to trigger the workflow execution. ◦ Include columns for the value of the time based argument passed to the workflow. And or include the name of the launch plan used to trigger the execution. I'd be happy to file these as feature requests, but I'd welcome any feedback before I get started.
Tim Hebbeler02/12/2023, 4:11 PM
Endre Karlson02/12/2023, 6:46 PM
should it not server a 30081?
flyte-cli sandbox start
Sujith Samuel02/13/2023, 8:52 AM
Volker Lorrmann02/13/2023, 4:03 PM
) on my VPS?
flyte demo start
Rezwan Abir02/13/2023, 10:01 PM
Rezwan Abir02/13/2023, 10:02 PM
"failed to validate that project [studio] and domain [development] are registered, err: [project [studio] not found]\"}\"\n>>"}
Franco Bocci02/14/2023, 8:17 AM
Adedeji Ayinde02/14/2023, 2:27 PM