GitHub
04/28/2023, 5:07 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/8f8d6869e1c2e064b2138d1b0e3f5745feb47849|8f8d6869>
- chore: [tlm] comprehensive node execution query (#749)
flyteorg/flyteconsoleGitHub
04/28/2023, 7:37 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/0025cbea7084089597691d88fb698bd8d7ae45f0|0025cbea>
- Added support for aborting task nodes reported as failures (#541)
flyteorg/flytepropellerGitHub
04/28/2023, 8:21 PMGitHub
04/28/2023, 11:02 PMScreenshot 2023-04-28 at 4 01 20 PM▾
input name: {
type: float
value: 0.5
}
Propose: Link/Inline OR Additional context
Implement the function here: https://github.com/flyteorg/flyteconsole/blob/8f8d6869e1c2e064b2138d1b0e3f5745feb47849/packages/console/src/components/Literals/helpers.ts#L145
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/29/2023, 2:40 AMerror <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o42.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
• https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to.
• https://discuss.flyte.org/t/39240/U03CZJFCT7T-you-should-be-able-to-find-pods-in-the-flytesnac
Expected behavior
We would expect this to work out of the box by providing sensible defaults.
I did find a solution. The task works when I include configure_s3
.
Additional context to reproduce
import pandas as pd
from flytekit import task, workflow, StructuredDataset
from flytekitplugins.spark import Spark
import flytekit
def configure_s3(spark):
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
hadoop_conf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.WebIdentityTokenCredentialsProvider")
@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:k8s_spark-latest|ghcr.io/flyteorg/flytecookbook:k8s_spark-latest>", task_config=Spark(
spark_conf={
"spark.driver.memory": "1000M",
"spark.executor.memory": "1000M",
"spark.executor.instances": "3",
"spark.driver.cores": "1",
"spark.executor.cores": "1",
}),
)
def t1() -> StructuredDataset:
spark = flytekit.current_context().spark_session
# The task fails without these settings.
# configure_s3(spark)
df = pd.DataFrame({"a": [1,2,3]})
df = spark.createDataFrame(df)
return StructuredDataset(dataframe=df)
@workflow
def wf() -> StructuredDataset:
sd = t1()
return sd
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/30/2023, 9:26 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by kumare3
<https://github.com/flyteorg/flytekit/commit/4ad64d3037cc6227019c489b13e29c3aef21bb74|4ad64d30>
- Add is_inside func to ImageSpec (#1601)
flyteorg/flytekitGitHub
05/01/2023, 4:00 PMGitHub
05/01/2023, 6:07 PMGitHub
05/01/2023, 8:47 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyteidl/commit/edc3faa12d0ce683bde19c15295ec81c34639e41|edc3faa1>
- Retract 1.4.x (#397)
flyteorg/flyteidlGitHub
05/01/2023, 8:48 PMGitHub
05/01/2023, 9:35 PMbt = ContainerTask(
name="basic-test",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(indir=FlyteDirectory),
outputs=kwtypes(),
image="<http://ghcr.io/flyteorg/rawcontainers-shell:v2|ghcr.io/flyteorg/rawcontainers-shell:v2>",
command=[
"ls",
"-la",
"/var/inputs",
],
)
@task
def get_dir(dirpath: str) -> FlyteDirectory:
fd = FlyteDirectory(path=dirpath)
return fd
@workflow
def wf():
fd = get_dir(dirpath='<s3://my-s3-bucket/cv-in>')
bt(indir=fd)
This workflow returns an empty directory at /var/inputs
according to the k8s logs despite there definitely being objects at the specified path.
Expected behavior
I would expect the above workflow to list either /var/inputs/indir
with objects copied to indir or perhaps copy them directly i.e. /var/inputs/obj1
, /var/inputs/obj2
as they are named in the object store.
Additional context to reproduce
This was run on a local sandbox via pyflyte run --remote basic_test.py wf
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/01/2023, 11:38 PMGitHub
05/01/2023, 11:40 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/e5b05110325a71ca229440da283e989bf1b228c6|e5b05110>
- Measuring the time taken for each component when executing a task (#1581)
flyteorg/flytekitGitHub
05/01/2023, 11:54 PMgpu_enabled
was set to True, fixed by adding an additional conditional to dataset.
flyteorg/flytekit-python-template
✅ All checks have passed
2/2 successful checksGitHub
05/02/2023, 12:31 AM<https://github.com/flyteorg/flyteidl/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyteidl/commit/513ff8221c68e3faccbdbd2039cee4b4d4e6ccf0|513ff822>
- Data addresses #minor (#391)
flyteorg/flyteidlGitHub
05/02/2023, 12:32 AMGitHub
05/02/2023, 12:40 AMGitHub
05/02/2023, 1:17 AM<https://github.com/flyteorg/flyteidl/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyteidl/commit/9189ff290804ae25775daf144a7c0e6511d27e15|9189ff29>
- #minor - bump to 1.5.x to avoid 1.4.0 and 1.4.2 issues (#399)
flyteorg/flyteidlGitHub
05/02/2023, 1:18 AMGitHub
05/02/2023, 2:09 AMflyte://
when getting data
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
Related to: flyteorg/flyteidl#391
flyteorg/flyteadmin
GitHub Actions: Unit Tests / Run Unit Test
GitHub Actions: Docker Build Images / Build Docker Image
GitHub Actions: Lint / Run Lint
GitHub Actions: Check Go Generate / Go Generate
✅ 2 other checks have passed
2/6 successful checksGitHub
05/02/2023, 10:16 AM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/4e8c868796a3a848ea2a42ec07715d831de068a0|4e8c8687>
- Update privileges, scope and nomination processes (#5)
flyteorg/communityGitHub
05/02/2023, 11:39 AM@asynchronous_task
decorator. External services can pull pending tasks, and complete them through an API.
Describe alternatives you've considered
When discussed on Slack, Ketan recommended:
There is no plugin that waits for manual input today, though conceptually we can definitely add one.
But this can be easily achieved in Flyte without any more code. This is how you could do it
Lets says there is a workflow with task1, task2, task3, task4, task5. In this workflow task3 is actually needing a human input
Workflow1 with task1 & task2
Workflow2 with task4 & B task5
and you can now wait for Workflow1 to finish, do your logic in your own service/component etc and then call flyteAdmin api to trigger workflow2
Though this works, it relies on the external service to make the appropriate callback. That external service may not be concerned about what happens next, and baking that logic may make the whole workflow brittle.
Flyte component
☑︎ Overall
☐ Flyte Setup and Installation scripts
☑︎ Flyte Documentation
☐ Flyte communication (slack/email etc)
☐ FlytePropeller
☐ FlyteIDL (Flyte specification language)
☑︎ #2631
☐ FlyteAdmin (Control Plane service)
☐ FlytePlugins
☐ DataCatalog
☐ FlyteStdlib (common libraries)
☐ FlyteConsole (UI)
☐ Other
[Optional] Propose: Link/Inline
Still thinking about this as I'm quite new to Flyte.
Additional context
I'm borrowing the language from quite a few other orchestrators / workflow engines with a similar feature / concept:
• Uber's Cadence Workflow: https://cadenceworkflow.io/docs/07_goclient/12_activity_async_completion
• Camunda's Zeebe: https://zeebe.io/blog/2018/11/message-correlation-working-with-messages-in-zeebe/
• Netflix's Conductor: https://netflix.github.io/conductor/configuration/systask/#wait
• Orchestra: https://github.com/b12io/orchestra (human-in-the-loop orchestration)
Is this a blocker for you to adopt Flyte
Not really. The alternative provided by Ketan may work, but again, it would be better to have the visibility, and control in one place. Otherwise, it'd defeat the purpose of having a workflow engine / orchestrator managing the control flow while services can focus purely on the business logic.
flyteorg/flyteGitHub
05/02/2023, 4:51 PMpromise.get()
or something like that, we should let them know what the legal operations are of promises (e.g. passing to other tasks/workflows, conditionals, etc.)
Goal: What should the final outcome look like, ideally?
TBD
Describe alternatives you've considered
NA
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/02/2023, 5:18 PMGitHub
05/02/2023, 5:52 PMGitHub
05/02/2023, 5:59 PMNone
. We need to raise a warning if the user specifies any other value so that this feature isn't misused.
Goal: What should the final outcome look like, ideally?
When a user does something like:
@task
def t1(
arg: Optional[str] = "",
)
Flytekit should raise a warning, something like:
Only None can be used as a default argument in flyte tasks
Describe alternatives you've considered
NA
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/02/2023, 7:01 PMSeedProjects
key to the Admin configuration struct in cmd/single
. This allows users to specify which projects should be created at start up. This allows management of the flyte installation to be more declarative via the helm chart installation path.
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Note to reviewers
The original ticket suggests that configuration.admin.seedProjects
should be settable; however, that key doesn't exist in the current setup. Instead this works in the same way one can set task_resources, via `configuration.inline`:
# values.yaml
flyte-core-components:
admin:
seedProjects:
- flytesnacks
- flytemeals
In my local sandbox: POD_NAMESPACE=flyte go run -tags console cmd/main.go start --config ~/.flyte/local-dev-config.yaml
with a ~/.flyte/local-dev-config.yaml
containing:
flyte:
admin:
seedProjects:
- flytesnacks
- flytemeals
Results in:
Screenshot 2023-05-02 at 1 48 25 PM▾
GitHub
05/02/2023, 8:45 PMflyte://
uri.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Other PRs:
flyteorg/flyteadmin#546
flyteorg/flyteidl#391
Tracking Issue
No issues, this is just a use-ability enhancement we found
flyteorg/flytekit
Codecov: 47.88% of diff hit (target 70.16%)
Codecov: 69.93% (-0.23%) compared to e5b0511
GitHub Actions: docs
GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.11, flytekit-snowflake)
GitHub Actions: build-plugins (3.11, flytekit-polars)
GitHub Actions: build-plugins (3.11, flytekit-papermill)
GitHub Actions: build-plugins (3.11, flytekit-pandera)
GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.11, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.11, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.11, flytekit-hive)
GitHub Actions: build-plugins (3.11, flytekit-envd)
GitHub Actions: build-plugins (3.11, flytekit-duckdb)
GitHub Actions: build-plugins (3.11, flytekit-dolt)
GitHub Actions: build-plugins (3.11, flytekit-deck-standard)
GitHub Actions: build-plugins (3.11, flytekit-dbt)
GitHub Actions: build-plugins (3.11, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.11, flytekit-dask)
GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.10, flytekit-vaex)
✅ 8 other checks have passed
8/30 successful checksGitHub
05/02/2023, 9:24 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/418628c31fc112bbd92fe808c1ac505ff03cc8d7|418628c3>
- Add adopters meetup notes placeholder
flyteorg/communityGitHub
05/02/2023, 9:29 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/9075a4bde832d48e52d01e05f07898d4832f311c|9075a4bd>
- May 2nd meeting notes
flyteorg/community