GitHub
01/20/2023, 10:10 PMGitHub
01/20/2023, 11:28 PMflyte-binary
chart for extra configuration to be passed in via an existing configmap (inlineConfigMap
). If specified, this will be merged with the base configuration via a projected
volume.
• Leverages the above option in sandbox-bundled, and creates an empty configmap that can be updated by users to inject additional configuration and cluster resource templates for the flyte-sandbox.
• Changed some values so that resources names are more concise in the sandbox. Updated the local single-binary flyte config example to reflect the new values.
• Add secrets/webhook test to the make test
target for sandbox-bundled
.
flyteorg/flyte
✅ All checks have passed
11/11 successful checksGitHub
01/20/2023, 11:28 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/ec92a2c46e53954b1283a46f44e1826591cc246f|ec92a2c4>
- [charts/flyte-binary] Add provision for specifying configmaps for extra configs (#3248)
flyteorg/flyteGitHub
01/20/2023, 11:59 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flytekit/commit/b6605bc95bd6f03434aeb5ca8f69eba161924ef8|b6605bc9>
- [Docs] SynchronousFlyteClient API reference #3095 (#1416)
flyteorg/flytekitGitHub
01/20/2023, 11:59 PMSynchronousFlyteClient
to Flytekit API reference guide.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/22/2023, 3:40 AMGitHub
01/22/2023, 8:49 PMGitHub
01/23/2023, 5:58 AMGitHub
01/23/2023, 10:00 AMGitHub
01/23/2023, 10:23 AMimage▾
GitHub
01/23/2023, 10:39 AM2.17.0
and autovalue to 1.10.1
Type
☑︎ Bump dependencies
☐ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Folds in the changes added by:
• #174
• #160
Snyck doesn't sign its commits and also format the pom and incompatible way
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit-java
✅ All checks have passed
3/3 successful checksGitHub
01/23/2023, 1:34 PM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by narape
<https://github.com/flyteorg/flytekit-java/commit/f38be8009e0bcf0dea0867e7a05198cff67e5534|f38be800>
- Bump errorprone and autovalue libs (#176)
flyteorg/flytekit-javaGitHub
01/23/2023, 1:40 PMGitHub
01/23/2023, 4:39 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by bstadlbauer
<https://github.com/flyteorg/flytesnacks/commit/2aefe89e3973f831412e476b1d6b4c422ad8768b|2aefe89e>
- Dask plugin docs (#929)
flyteorg/flytesnacksGitHub
01/23/2023, 4:39 PMdask
integration developed in flyteorg/flyte#427
Todo before merging (after flyteorg/flyteplugins#275 and flyteorg/flytekit#1366 are in):
☑︎ Revisit all links to make sure they work
☑︎ Add and lock cookbook/integrations/kubernetes/k8s_dask/requirements.in
(on Linux)
flyteorg/flytesnacks
GitHub Actions: Mark github pre-release as Release
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
✅ 26 other checks have passed
26/30 successful checksGitHub
01/23/2023, 4:58 PMDuckDBQuery
task plugin that runs queries using DuckDB as the DBMS.
Type
☐ Bug Fix
☐ Feature
☑︎ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Capturing the crucial assumptions I made while building the task plugin:
• The DuckDBQuery
task parameter that a user needs to send argument to includes query
and can contemplate adding includes inputs
.
• query
can include a set of queries that'll be run sequentially. The last query needs to be a SELECT query.
• inputs
can include structured dataset or a list of parameters to be sent to the queries.
• The output
is a pyarrow table. Can be converted to any structured dataset compatible type.
• The connection mode is set to :memory
, i.e., the data is always stored in an in-memory, non-persistent database. It can be set to a file, but it's difficult to make the file accessible to different DuckDBQuery
pods, which otherwise wouldn't make sense because file is persistent, and it needs to be leveraged.
Example:
duckdb_query = DuckDBQuery(
name="read_parquet",
query=[
"INSTALL httpfs",
"LOAD httpfs",
"""SELECT hour(lpep_pickup_datetime) AS hour, count(*) AS count FROM READ_PARQUET(?) GROUP BY hour""",
],
inputs=kwtypes(params=list[str]),
)
@workflow
def parquet_wf(parquet_file: str) -> pd.DataFrame:
return duckdb_query(params=[parquet_file])
assert isinstance(
parquet_wf(parquet_file="<https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-02.parquet>"),
pd.DataFrame,
)
Tracking Issue
Fixes flyteorg/flyte#3246
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
GitHub Actions: build-plugins (3.10, flytekit-vaex)
GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.10, flytekit-spark)
GitHub Actions: build-plugins (3.10, flytekit-snowflake)
GitHub Actions: build-plugins (3.10, flytekit-ray)
GitHub Actions: build-plugins (3.10, flytekit-polars)
GitHub Actions: build-plugins (3.10, flytekit-papermill)
GitHub Actions: build-plugins (3.10, flytekit-pandera)
GitHub Actions: build-plugins (3.10, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.10, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.10, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.10, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.10, flytekit-hive)
GitHub Actions: build-plugins (3.10, flytekit-duckdb)
GitHub Actions: build-plugins (3.10, flytekit-dolt)
GitHub Actions: build-plugins (3.10, flytekit-deck-standard)
GitHub Actions: build-plugins (3.10, flytekit-dbt)
GitHub Actions: build-plugins (3.10, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.10, flytekit-dask)
GitHub Actions: build-plugins (3.10, flytekit-bigquery)
GitHub Actions: build-plugins (3.10, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.10, flytekit-aws-batch)
GitHub Actions: build-plugins (3.10, flytekit-aws-athena)
GitHub Actions: build-plugins (3.9, flytekit-whylogs)
GitHub Actions: build-plugins (3.9, flytekit-vaex)
GitHub Actions: build-plugins (3.9, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.9, flytekit-spark)
GitHub Actions: build-plugins (3.9, flytekit-snowflake)
GitHub Actions: build-plugins (3.9, flytekit-ray)
✅ 1 other check has passed
1/30 successful checksGitHub
01/23/2023, 5:00 PMGitHub
01/23/2023, 6:01 PMpyflyte backfill
and equivalent remote.launch_backfill
that can create a backfiller workflow for a given launchplan automatically and register it statically
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☑︎ Code documentation added
☑︎ Any pending items have an associated Issue
Issue
fixes flyteorg/flyte#3212
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
01/23/2023, 11:02 PMGitHub
01/23/2023, 11:27 PMSQLAlchemyTask
where multi-line strings were not valid inputs.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Like the SQLite3Task
, modify SQLAlchemyTask
to sanitize the input string.
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
01/23/2023, 11:35 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flyteplugins/commit/4634a81403e501882e3b1f39bcbd229b78768a4e|4634a814>
- Add env FLYTE_FAIL_ON_ERROR to aws batch job (#306)
flyteorg/flytepluginsGitHub
01/23/2023, 11:41 PMGitHub
01/24/2023, 12:05 AMflyte
repo. This PR moves that to the flytesnacks repo, where flyteorg/furo#22 needs to be merged in order for the theme updates to take effect.
The following subsections have been added:
• Flyte Fundamentals: basically this is a gateway into the User Guide section, introducing newcomers to Flyte's main concepts and development lifecycle
• Core Use Cases: this is a gateway to the Tutorial section, introducing data, ML, and analytics use cases.
flyteorg/flytesnacks
GitHub Actions: Mark github pre-release as Release
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
✅ 26 other checks have passed
26/30 successful checksGitHub
01/24/2023, 12:22 AMGitHub
01/24/2023, 1:47 AMGitHub
01/24/2023, 3:19 AM// CreateDownloadLocationRequest specified request for the CreateDownloadLocation API.
message CreateDownloadLocationRequest {
// NativeUrl specifies the url in the format of the configured storage provider (e.g. <s3://my-bucket/randomstring/suffix.tar>)
string native_url = 1;
// ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this
// exceeds the platform allowed max.
// +optional. The default value comes from a global config.
google.protobuf.Duration expires_in = 2;
}
to something like:
message CreateDownloadLinkRequest {
oneof source {
// WorkflowId is the unique identifier for the workflow execution.
core.WorkflowExecutionIdentifier execution_id = 1;
// NodeId is the unique identifier for the node execution. For a task node, this will retrieve the output of the
// most recent attempt of the task.
core.NodeExecutionIdentifier node_id = 2;
// TaskId is the unique task execution identifier that points to a specific attempt of the task.
core.TaskExecutionIdentifier task_id = 3;
}
ArtifactType artifact_type = 4;
// ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this
// exceeds the platform allowed max.
// +optional. The default value comes from a global config.
google.protobuf.Duration expires_in = 5;
}
Describe alternatives you've considered
Admin parsing the request URL and mapping to an execution: puts a lot of assumptions on the schema of the URL.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/24/2023, 8:10 AMMap<String, SdkBindingData<?>>
. This feature will be used as a part of the typed input implementation, which will be implemented in other PR.
Goal: What should the final outcome look like, ideally?
The goal is to have a method that allows you to transform an input/output class in a Map<String, SdkBindingData>` where the key is the attribute names and the values are the SdkBindingData value of the inputs/outputs.
Describe alternatives you've considered
no alternatives
Propose: Link/Inline OR Additional context
flyteorg/flytekit-java#177
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/24/2023, 8:11 AMMap<String, SdkBindingData<?>>
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Add a new method into the SdkType interface to allow to transform an input/output class in a Map<String, SdkBindingData<?>>
This PR also adds their implementations (JacksonSdkType, SdkScalaType)
Tracking Issue
• flyteorg/flyte#3258
Follow-up issue
NA
flyteorg/flytekit-java
✅ All checks have passed
3/3 successful checksGitHub
01/24/2023, 8:27 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by andresgomezfrr
<https://github.com/flyteorg/flytekit-java/commit/57ad4540348bc2ef487cfd4be8d78beb5d9503fb|57ad4540>
- Add toSdkBindingDataMap() to SdkType (#177)
flyteorg/flytekit-javaGitHub
01/24/2023, 11:54 AM