GitHub
02/20/2023, 12:54 PMif (!isAndroid()) {
staticLoggerBinderPathSet = findPossibleStaticLoggerBinderPathSet();
reportMultipleBindingAmbiguity(staticLoggerBinderPathSet);
}
// the next line does the binding
StaticLoggerBinder.getSingleton();
findPossibleStaticLoggerBinderPathSet
does the resource load so even we prevent delegation to parent at that point, StaticLoggerBinder.getSingleton()
will still delegate to parent, without this PR.
Tracking Issue
_Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_
fixes flyteorg/flyte#3147
Follow-up issue
NA
flyteorg/flytekit-java
✅ All checks have passed
3/3 successful checksGitHub
02/20/2023, 4:23 PM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by narape
<https://github.com/flyteorg/flytekit-java/commit/32af350a1950469f82082e722837a3003cf0294c|32af350a>
- Only load certain classes from child (#196)
flyteorg/flytekit-javaGitHub
02/20/2023, 5:00 PMGitHub
02/20/2023, 5:00 PMGitHub
02/20/2023, 5:58 PMGitHub
02/21/2023, 12:03 AMGitHub
02/21/2023, 5:56 AMGitHub
02/21/2023, 6:32 AMScreenshot 2023-02-21 at 2 19 20 PM▾
Failed to launch job, resource quota exceeded. err: [BackOffError] The operation was attempted but failed, caused by: pods "fec52e914a8c0b0b7000-n3-0" is forbidden: exceeded quota: project-quota, requested: limits.memory=64Gi, used: limits.memory=192Gi, limited: limits.memory=200Gi
Goal: What should the final outcome look like, ideally?
• Users should be able to identify quickly if their workflow execution is affected by project quota.
• Transition to "RUNNING" state should be accurate
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/21/2023, 9:04 AMGitHub
02/21/2023, 5:16 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/860e356718e3a6a9cfb07460baf2eccf152ec75f|860e3567>
- Add support for ApprovedCondition for GateNodes (#688)
flyteorg/flyteconsoleGitHub
02/21/2023, 7:27 PMflytectl get executions -p flytesnacks -d development f12ce5ce3aae045378c1 --details
TABLE format is not supported on detailed view and defaults to tree view. Choose either json/yaml
└── start-node - SUCCEEDED - 2023-02-21 19:20:26.545951047 +0000 UTC - 2023-02-21 19:20:26.545951047 +0000 UTC
│ ├── Outputs :
│ └── a: 2
└── n0 - SUCCEEDED - 2023-02-21 19:20:26.777756742 +0000 UTC - 2023-02-21 19:20:41.033027942 +0000 UTC
│ ├── Attempt :0
│ │ ├── Task - SUCCEEDED - 2023-02-21 19:20:27.069960835 +0000 UTC - 2023-02-21 19:20:40.827311935 +0000 UTC
│ │ ├── Task Type - python-task
│ │ ├── Reason - [ContainersNotReady|ContainerCreating]: containers with unready status: [f12ce5ce3aae045378c1-n0-0]|
│ │ ├── Metadata
│ │ │ ├── Generated Name : f12ce5ce3aae045378c1-n0-0
│ │ │ ├── Plugin Identifier : container
│ │ │ ├── External Resources
│ │ │ ├── Resource Pool Info
│ │ ├── Logs :
│ │ └── Name :Cloudwatch Logs (User)
│ │ └── URI :<https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252Fopta-oc-canary$252Fapplication$3FlogStreamNameFilter$3Dvar.log.containers.f12ce5ce3aae045378c1-n0-0_flytesnacks-development_f12ce5ce3aae045378c1-n0-0>
│ ├── Outputs :
│ └── o0: <s3://union-oc-canary-playground/7y/f12ce5ce3aae045378c1-n0-0/2a1d85ef4a87410f5a04ce889e876776>
└── n1 - SUCCEEDED - 2023-02-21 19:20:41.420976535 +0000 UTC - 2023-02-21 19:20:57.090747824 +0000 UTC
│ ├── Attempt :0
│ │ ├── Task - SUCCEEDED - 2023-02-21 19:20:41.720309792 +0000 UTC - 2023-02-21 19:20:56.914747315 +0000 UTC
│ │ ├── Task Type - python-task
│ │ ├── Reason - [ContainersNotReady|ContainerCreating]: containers with unready status: [f12ce5ce3aae045378c1-n1-0]|
│ │ ├── Metadata
│ │ │ ├── Generated Name : f12ce5ce3aae045378c1-n1-0
│ │ │ ├── Plugin Identifier : container
│ │ │ ├── External Resources
│ │ │ ├── Resource Pool Info
│ │ ├── Logs :
│ │ └── Name :Cloudwatch Logs (User)
│ │ └── URI :<https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252Fopta-oc-canary$252Fapplication$3FlogStreamNameFilter$3Dvar.log.containers.f12ce5ce3aae045378c1-n1-0_flytesnacks-development_f12ce5ce3aae045378c1-n1-0>
│ ├── Outputs :
│ └── o0: <s3://union-oc-canary-playground/g9/f12ce5ce3aae045378c1-n1-0/e340ecf4db7d64e3a539665cb7648047>
└── n2 - SUCCEEDED - 2023-02-21 19:20:57.318835153 +0000 UTC - 2023-02-21 19:21:11.021544907 +0000 UTC
│ ├── Attempt :0
│ │ ├── Task - SUCCEEDED - 2023-02-21 19:20:57.56183156 +0000 UTC - 2023-02-21 19:21:10.828786539 +0000 UTC
│ │ ├── Task Type - python-task
│ │ ├── Reason - [ContainersNotReady|ContainerCreating]: containers with unready status: [f12ce5ce3aae045378c1-n2-0]|
│ │ ├── Metadata
│ │ │ ├── Generated Name : f12ce5ce3aae045378c1-n2-0
│ │ │ ├── Plugin Identifier : container
│ │ │ ├── External Resources
│ │ │ ├── Resource Pool Info
│ │ ├── Logs :
│ │ └── Name :Cloudwatch Logs (User)
│ │ └── URI :<https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252Fopta-oc-canary$252Fapplication$3FlogStreamNameFilter$3Dvar.log.containers.f12ce5ce3aae045378c1-n2-0_flytesnacks-development_f12ce5ce3aae045378c1-n2-0>
│ ├── Outputs :
│ └── o0: <s3://union-oc-canary-playground/o4/f12ce5ce3aae045378c1-n2-0/0d1195b0dbded5310076263af1f96315>
└── end-node - SUCCEEDED - 2023-02-21 19:21:11.401510279 +0000 UTC - 2023-02-21 19:21:11.541818271 +0000 UTC
Type
☐ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
fixes flyteorg/flyte#2757
Follow-up issue
flyteorg/flyteidl
✅ All checks have passed
13/13 successful checksGitHub
02/21/2023, 7:37 PMGitHub
02/21/2023, 11:58 PM{"json":{"exec_id":"atfkcwwv5cfr7wzhqq94","node":"n1","ns":"flytesnacks-
development","res_ver":"5501760","routine":"worker-3","tasktype":"python-
task","wf":"flytesnacks:development:<http://example_test.wf|example_test.wf>"},"level":"error","msg":"DataCatalog failed to get outputs from
artifact 45bd1d68-a013-43b1-a56b-b7597b559125, err: unexpected artifactData: [o0] type:
[structured_dataset_type:\u003c\u003e ] does not match any task output type:
[structured_dataset_type:\u003cformat:\"parquet\" \u003e ]","ts":"2022-09-12T06:56:41Z"}
When the cache is enabled, we'll retrieve artifacts from datacatalog and check if the structured dataset's schema and format match the expected type.
However, the default format of the structured dataset in the expected type is always Parquet
, but the format of the output structured dataset is ""
.
@task(cache=True, cache_version="1.0")
def t2() -> StructuredDataset: # The default format of structured dataset is Parquet here
df = pd.DataFrame({"len": [len(sd.open(pd.DataFrame).all())]})
return StructuredDataset(df, uri=bq_uri) # The format of structured dataset is ""
Two ways to fix it.
1. Change these lines to
if len(structuredDatasetType.Format) != 0 && !strings.EqualFold(structuredDatasetType.Format, t.literalType.GetStructuredDatasetType().Format) {
return false
}
2. Change the default format of the expected type to "" in flytekit, and change these lines to the below. However, it will break existing users. If users upgrade flytekit, they have to upgrade the propeller as well.
if len(t.literalType.GetStructuredDatasetType().Format) != 0 && !strings.EqualFold(structuredDatasetType.Format, t.literalType.GetStructuredDatasetType().Format) {
return false
}
structuredDatasetType
is input type
t.literalType.GetStructuredDatasetType()
is expected type
Expected behavior
BQ task should run successfully even if the cache is enabled
Additional context to reproduce
import uuid
import pandas as pd
from typing_extensions import Annotated
from flytekit import task, workflow, StructuredDataset, kwtypes
@task(cache=True, cache_version="2.0")
def t1() -> StructuredDataset:
df = pd.DataFrame({
"name": ["dylan", "steve"],
"age": [33, 32]
})
return StructuredDataset(df)
@task(cache=True, cache_version="2.0")
def t2(sd: StructuredDataset) -> StructuredDataset:
df = pd.DataFrame({"len": [len(sd.open(pd.DataFrame).all())]})
table_id = str(uuid.uuid4())
bq_uri = f"<bq://flyte-test-340607.dataset>.{table_id}"
return StructuredDataset(df, uri=bq_uri)
@workflow
def wf() -> StructuredDataset:
return t2(sd=t1())
if __name__ == "__main__":
wf()
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/22/2023, 2:58 AMstdout
.
So the ability to pipe the stdout
or the stderr
to a host:port
would be the best way to go about it.
Goal: What should the final outcome look like, ideally?
Being able to use a syntax like:
def stdout_pipe_callback(data: str):
# send data to "<https://myfavoritehost:1234>"
...
def stderr_pipe_callback(data: str):
# send data to "<https://myotherfavoritehost:1234>"
...
@stream_task(stdout_pipe=stdout_pipe_callback, stderr_pipe=stderr_pipe_callback)
def do_some_work(input:InputType) -> OutputType:
.....
return results
This would allow the user, to define custom callback methods to stream back the stdout.
Describe alternatives you've considered
I've considered using the shell task, but that won't allow me to "stream" the outputs until the end of the process. But I need help defining python code to use.
Another alternative I'm considering is to figure out a way to modify the docker container to monitor all the python stdout. I haven't had any luck with that.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/22/2023, 3:16 AM<https://github.com/flyteorg/flyteidl/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flyteidl/commit/e69fd54aa9a68403361926a674e272aa85060943|e69fd54a>
- Adding support for structured dataset (#369)
flyteorg/flyteidlGitHub
02/22/2023, 3:16 AM--details
flag to flytectl command, it throws a StructuredDataset error as follows:
flytectl get execution -p flytesnacks -d development a47w6dwnr454vzzgqhxj -o yaml --details
Error:
Error: unsupported literal scalar type *core.Scalar_StructuredDataset
Expected behavior
The command should work.
Additional context to reproduce
Run flytectl get execution -p flytesnacks -d development <execution-id> -o yaml --details
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/22/2023, 3:17 AMGitHub
02/22/2023, 8:58 AMdataclass_json
Goal: What should the final outcome look like, ideally?
Add support for custom encoders and decoders to dataclass
types, as described here.
https://lidatong.github.io/dataclasses-json/#extending
Describe alternatives you've considered
Currently the ugly workaround is decoding in __post_init__
and encoding via a @classmethod
Propose: Link/Inline OR Additional context
Should be easy to add in https://github.com/flyteorg/flytekit/blob/245c4143b4ed20487d837850df4a140c7b230986/flytekit/core/type_engine.py
Simply test for encoder and decoder when serializing/deserializing.
Would be good to add support for mm_field
as well, to allow for custom serializers/deserializers that use non-string representations.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/22/2023, 9:02 AMflyteplugins
from 1.0.34
to 1.0.37
to pull in the change from flyteorg/flyteplugins#315.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
flyteorg/flytepropeller
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
02/22/2023, 11:03 AMn
retries are allowed, n-1
of them are on spot instances and for the last attempt, Flyte resorts to a regular instance.
I tested the behaviour using this workflow ...
from flytekit import task, workflow, Resources
@task(interruptible=True, retries=5, requests=Resources(gpu="1"))
def train():
import time
time.sleep(600)
@workflow
def wf():
train()
... and by simulating a preemption with gcloud compute instances simulate-maintenance-event <node name> --zone <zone>
.
Result:
• Pods with FLYTE_ATTEMPT_NUMBER
from 0-4 had the spot/preemptible node selector.
• The pod with FLYTE_ATTEMPT_NUMBER=5
did not have the spot/preemptible node selector anymore.
flyteorg/flytesnacks
GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dolt)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dbt_example)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/external_services/databricks)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_training)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_pytorch)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/batch)
GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/athena)
GitHub Actions: Build & Push to GHCR (cookbook/core)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/spark_horovod)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/pima_diabetes)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/nlp_processing)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/mnist_classifier)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/house_price_prediction)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/feast_integration)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/eda)
GitHub Actions: Build & Push to GHCR (cookbook/case_studies/bioinformatics/blast)
GitHub Actions: Docs Warnings
DCO: DCO
✅ 2 other checks have passed
2/20 successful checksGitHub
02/22/2023, 11:11 AM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/752f55e9c7f7e357de707aa3eaae6e3af03b186f|752f55e9>
- Update flyteplugins to 1.0.37 (#527)
flyteorg/flytepropellerGitHub
02/22/2023, 11:33 AMRequest URL: <https://flyte.example.com/me>
Request Method: GET
Status Code: 501
Status Code: 501
Expected behavior
I look for all issue about this error and I can't find it.
Additional context to reproduce
No response
Screenshots
Status Error 501
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/22/2023, 11:59 AMGitHub
02/22/2023, 12:02 PMGitHub
02/22/2023, 12:06 PMGitHub
02/22/2023, 1:20 PMGitHub
02/22/2023, 3:45 PMDraggableFileUpload
component to support uploading files with drag and drop functionality
Tracking Issue
#3317
Follow-up issue
flyteorg/flyte#3317
flyteorg/flyteconsole
GitHub Actions: Build & Push FlyteConsole Image
GitHub Actions: Get Release Tag
GitHub Actions: Generate Release
GitHub Actions: Build Flyteconsole Image / Push to Github Registry
GitHub Actions: unit_tests_with_coverage
✅ 4 other checks have passed
4/9 successful checksGitHub
02/22/2023, 5:41 PM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/4183cf27159a974dfecdd5f73086fa9751022169|4183cf27>
- [maven-release-plugin] prepare release 0.4.4
flyteorg/flytekit-javaGitHub
02/22/2023, 5:41 PM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/b88e987f8973a11c5719dcb9b8a27502b8712b2d|b88e987f>
- [maven-release-plugin] prepare for next development iteration
flyteorg/flytekit-javaGitHub
02/22/2023, 5:58 PMimport Task, Workflow, Tqdm from flytekit
import torch as tf
@Task
def some_func_with_mapping(...) -> ...:
model = tf.load(...)
output = []
tqdm(for row in ...):
output.append(model.forward(row))
return output
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte