GitHub
02/13/2023, 8:51 PMGitHub
02/13/2023, 9:24 PMCERTIFICATE_VERIFY_FAILED
on auth even if user has specified the certificate path or has set insecureSkipVerify
to False
. This change adds the verify
to requests library according to user configured settings.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (windows-latest, 3.7, -spark2)
GitHub Actions: build (windows-latest, 3.7)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.7, -spark2)
GitHub Actions: build (ubuntu-latest, 3.7)
GitHub Actions: docs
GitHub Actions: lint
GitHub Actions: build-plugins (3.10, flytekit-vaex)
GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.10, flytekit-spark)
GitHub Actions: build-plugins (3.10, flytekit-snowflake)
GitHub Actions: build-plugins (3.10, flytekit-ray)
GitHub Actions: build-plugins (3.10, flytekit-polars)
GitHub Actions: build-plugins (3.10, flytekit-papermill)
GitHub Actions: build-plugins (3.10, flytekit-pandera)
GitHub Actions: build-plugins (3.10, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.10, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.10, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.10, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.10, flytekit-hive)
GitHub Actions: build-plugins (3.10, flytekit-dolt)
GitHub Actions: build-plugins (3.10, flytekit-deck-standard)
GitHub Actions: build-plugins (3.10, flytekit-dbt)
GitHub Actions: build-plugins (3.10, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.10, flytekit-dask)
GitHub Actions: build-plugins (3.10, flytekit-bigquery)
✅ No checks have passed
0/30 successful checksGitHub
02/14/2023, 12:10 AMGitHub
02/14/2023, 1:14 AM<https://github.com/flyteorg/flytectl/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytectl/commit/5aefa07cbab99770cf3f642f6cd18e63006bb89a|5aefa07c>
- refactor: remove redundant YAML packages (#387)
flyteorg/flytectlGitHub
02/14/2023, 1:23 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/606e0c4e024c62151d760f8aa7fb06af7986c98e|606e0c4e>
- Use non-root user in default flytekit image (#1417)
flyteorg/flytekitGitHub
02/14/2023, 1:48 AMGitHub
02/14/2023, 2:03 AM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/c523cc38aa06be1ef300f0383be44a4041460682|c523cc38>
- Brew formula update for flytectl version v0.6.30
flyteorg/homebrew-tapGitHub
02/14/2023, 3:12 AM{{ principal }}
in the template.
Then, an example scenario would be as such, given this Flyte Admin configuration:
namespace_mapping:
template: "{{ principal }}"
When the user hankfanchiu
triggers a task execution, the Kubernetes workload would run in the hankfanchiu
namespace.
Describe alternatives you've considered
None
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 3:15 AMx-stripe-*
request header containing an access token;
2. Updating every Flyte client to pass this header.
We would like to use Flyte Admin's `auth.httpAuthorizationHeader` and `auth.grpcAuthorizationHeader` configurations to customize the request header as some x-stripe-*
.
However, currently, these configurations are only evaluated when the `security.useAuth` configuration is enabled, as seen in the following:
• `server.newHTTPServer()` → `auth.GetHTTPRequestCookieToMetadataHandler()`
• `server.newGRPCServer()` → `auth.GetAuthenticationCustomMetadataInterceptor()`
Goal: What should the final outcome look like, ideally?
Introduce a new Flyte Admin configuration that allows the auth.httpAuthorizationHeader
and auth.grpcAuthorizationHeader
configurations to be used, without actually enabling the security.useAuth
configuration.
This new configuration could perhaps be named security.forceUseAuthHeaders
. The default value would be false
. We could optionally consider that enabling both security.useAuth
and this new configuration is invalid, causing a panic.
Describe alternatives you've considered
None
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 6:54 AMScreenshot 2023-02-14 at 2 51 25 PM▾
/api/v1/executions/
API response returning an execution object that has a huge closure.error.message
payload.
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 8:04 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by andresgomezfrr
<https://github.com/flyteorg/flytekit-java/commit/305495e32dcd458b5bc0a8d15dfc82212dffa26b|305495e3>
- Refactor SdkBindindData factory methods (#193)
flyteorg/flytekit-javaGitHub
02/14/2023, 8:07 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/b9694501e3a4929c955a9de289ff7fb38cd06492|b9694501>
- [maven-release-plugin] prepare release 0.4.3
flyteorg/flytekit-javaGitHub
02/14/2023, 8:07 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/b217cd66d56fc44e921f12574798e7243263a191|b217cd66>
- [maven-release-plugin] prepare for next development iteration
flyteorg/flytekit-javaGitHub
02/14/2023, 9:39 AMGitHub
02/14/2023, 11:05 AMmap
function. The Spark Executors can only access the code from the image it pulled.
Expected behavior
The Spark Executors should use the same code as the Spark Driver - which is the code that was pushed using Fast Registration.
Additional context to reproduce
1. Following this exemple : use the code from https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html , building a Docker image like this one https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/index.html#step-2-environment-setup
2. pyflyte --pkgs my_code --image my_image --force
3. flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.1
4. Go to the Flyte Console, run the workflow : it works as expected, the output of my_spark
is close to 3.14
5. Then, change the code from the function f
in step 1: now, f
always returns 1.
6. I updated my code but I don't need to update my environnement, so I can use Fast Registration :
pyflyte --pkgs my_code --image my_image --force --fast
7. flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.2
8. Go to the Flyte Console, run the workflow's new version : the output of my_spark
is close to 3.14, but I was expecting it to be 4.
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 2:28 PMread_session = client.create_read_session(parent=parent, read_session=requested_session)
stream = read_session.streams[0]
reader = client.read_rows(stream.name)
frames = []
for message in reader.rows().pages:
frames.append(message.to_dataframe())
return pd.concat(frames)
Expected behavior
The BQ decoder should be able to read all data for bigger datasets as well.
This decoder code needs to be changed to either set the max stream to 1 in the read session or read from all streams as below snippet:
frames = []
for stream in read_session.streams:
reader = client.read_rows(stream.name)
for message in reader.rows().pages:
frames.append(message.to_dataframe())
if len(frames) > 0:
df = pd.concat(frames)
else:
schema = pyarrow.ipc.read_schema(
pyarrow.py_buffer(read_session.arrow_schema.serialized_schema)
)
df = schema.empty_table().to_pandas()
return df
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 4:25 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/d60c9af85a59ebb4c2265f76cb082b992078a309|d60c9af8>
- Update Flyte components (#3336)
flyteorg/flyteGitHub
02/14/2023, 5:03 PMflytectl demo
cluster, though the commands should transfer over to most any other Flyte cluster.
Here are example topics that would fit in this section:
• Scoped down to flytectl demo
cluster
• Getting error logs from failing pods
• Getting cluster configuration values (e.g. project, domain, task-level resources)
• Building and pushing a custom docker image to a flytectl demo
cluster
• Move the contents from the Troubleshooting guide here
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 5:15 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/fb33975ff3df43ba6cce954b25bebc15c9795a49|fb33975f>
- Deprecate FlyteSchema (#942)
flyteorg/flytesnacksGitHub
02/14/2023, 5:24 PMExecutionNodeViews
page
• Moved childGroups fetch to onToggle
in Table and Timeline components - requests trigger when user clicked on expander
• Moved map tasks fetch to details page to back fill one-off nodeExecutions data, and to ReactFlowGraphComponent
to batch them for all known nodeExecutions, so the graph is rendered properly (this will slow down the graph tab, but based on user's feedback, it's ok, if graph tab takes some time to load).
Known issues
☐ some unit tests are broken
☑︎ add childGroups fetch in graph on node click
☑︎ fix rerender of graph on childGroups fetch
☑︎ fix compiledWorkflowClosure
to check for is_dynamic
field (available in node execution's metadata) instead of pulling multiple requests on repeat
☐ test filters in Table tab
☑︎ display parent node as nested
in graph to indicate it has children
☐ child task type stays in Loading state in the Details Sidebar
Follow-up issue
NA
Test workflows
• normal core.control_flow.subworkflows.parent_wf
• dynamic <http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>
• nested parent core.control_flow.subworkflows.nested_parent_wf
• map task <http://flyte.workflows.example_map_task.wf|flyte.workflows.example_map_task.wf>
• small fan-out dynamic <http://workflows.branching_dynamic_example.wf|workflows.branching_dynamic_example.wf>
flyteorg/flyteconsole
DCO: DCO
✅ 1 other check has passed
1/2 successful checksGitHub
02/14/2023, 5:52 PMGitHub
02/14/2023, 6:20 PMfrom typing import List
from flytekit import task, workflow
@task
def my_task_1() -> int:
return 1
@task
def my_task_2(integers: List[int]) -> int:
return sum(integers)
@workflow
def subworkflow(integers: List[int]) -> int:
return my_task_2(integers=integers)
@workflow
def my_workflow() -> int:
result = my_task_1()
return subworkflow(integers=[result])
if __name__ == "__main__":
print(f"Running {__file__} main...")
print(my_workflow())
with:
Traceback (most recent call last):
File "/Users/bstadlbauer/workspace/pachama/protocol-baseline/flyte/baseline/demo.py", line 29, in <module>
print(my_workflow())
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 258, in __call__
return flyte_entity_call_handler(self, *args, **input_kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/promise.py", line 1050, in flyte_entity_call_handler
result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 273, in local_execute
function_outputs = self.execute(**kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 708, in execute
return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
return wrapped(*args, **kwargs)
File "/Users/bstadlbauer/workspace/pachama/protocol-baseline/flyte/baseline/demo.py", line 24, in my_workflow
return subworkflow(integers=[result])
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 258, in __call__
return flyte_entity_call_handler(self, *args, **input_kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/promise.py", line 1042, in flyte_entity_call_handler
return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/workflow.py", line 269, in local_execute
kwargs[k] = Promise(var=k, val=TypeEngine.to_literal(ctx, v, t, self.interface.inputs[k].type))
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 767, in to_literal
lv = transformer.to_literal(ctx, python_val, python_type, expected)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 924, in to_literal
lit_list = [TypeEngine.to_literal(ctx, x, t, expected.collection_type) for x in python_val] # type: ignore
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 924, in <listcomp>
lit_list = [TypeEngine.to_literal(ctx, x, t, expected.collection_type) for x in python_val] # type: ignore
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 751, in to_literal
transformer.assert_type(python_type, python_val)
File "/opt/homebrew/Caskroom/miniforge/base/envs/baseline-dev/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 90, in assert_type
raise TypeTransformerFailedError(f"Type of Val '{v}' is not an instance of {t}")
flytekit.core.type_engine.TypeTransformerFailedError: Type of Val 'Resolved(o0=scalar {
primitive {
integer: 1
}
}
)' is not an instance of <class 'int'>
Expected behavior
The workflow should succeed locally.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 7:10 PM<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: ip
Check all the applicable boxes
☐ I updated the documentation accordingly. Not sure if needed, seems to be referenced directly via eks-values.yaml mostly.
☐ All new and existing tests passed.
☑︎ All commits are signed-off.
Screenshots
Note to reviewers
flyteorg/flyte
✅ All checks have passed
9/9 successful checksGitHub
02/14/2023, 7:38 PMGitHub
02/14/2023, 7:39 PMGitHub
02/14/2023, 7:43 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteadmin/commit/32c8dbdc3d8936c3fed9d783129bbe1047a88cc4|32c8dbdc>
- Use ResourceManager to aid in validating task resources (#517)
flyteorg/flyteadminGitHub
02/14/2023, 7:43 PMtask_resources:
defaults:
cpu: 500m
memory: 500Mi
storage: 500Mi
limits:
cpu: 8
gpu: 1
memory: 10Gi
storage: 1Gi
2. Set a higher limit for flytesnacks/development, tra.yml
defaults:
cpu: "1"
memory: 1Gi
limits:
cpu: "12"
memory: 25Gi
project: flytesnacks
domain: development
3. verify: flytectl get task-resource-attribute -p flytesnacks -d development
{
"project": "flytesnacks",
"domain": "development",
"defaults": {
"cpu": "1",
"memory": "2Gi"
},
"limits": {
"cpu": "12",
"memory": "25Gi"
}
}
4. Try to run task with e.g. 20Gi memory requested:
Requested MEMORY default [20Gi] is greater than current limit set in the platform configuration [10Gi]. Please contact Flyte Admins to change these limits or consult the configuration
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
02/14/2023, 8:24 PMGitHub
02/14/2023, 9:56 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/44a2c483ce593566de357ae658bdb108299028b4|44a2c483>
- Remove deploying workflows page (#952)
flyteorg/flytesnacksGitHub
02/14/2023, 10:13 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/9ceea42e4b24fb31a7b5cf4a4a6f421fe53d0c8f|9ceea42e>
- Governance v1
flyteorg/community