StructuredDataset

FlyteDirectory

FlyteFile

@task 
def t(ff: StructuredDataset):
    # we have a pyspark DataFrame

    df = StructuredDataset(pyspark.sql.DataFrame).all()
    ff = FlyteFile.new_remote_file()
    df.write.parquet(ff.path, partitionBy=["country"])
   return ff

# filter hive partitioned parquet with pandas to avoid loading it all into memory
def t_pandas():
   df = sd.open(pd.DataFrame).all(filters=[("year", ">=",  2021)])

# write a hive partitioned table
def t_pandas()
    ...
    return StructuredDataset(dataframe=df, partition_cols=['year', 'month', 'day'])

# append to a table instead of overwrite it
def t_write_bq():
    ...
    return StructuredDataset(df, uri="bq:/", append=True)

# support polars streaming
def t_polars_stream():
    ...
    return StructuredDataset(dataframe=df, streaming=True)

# read with low memory setting
def t_polars_low_mem(sd: StructuredDataset):
    df = sd.open(pl.DataFrame).all(low_mem=True)

<https://github.com/flyteorg/flyteadmin/tree/master|master>

<https://github.com/flyteorg/flyteadmin/commit/ed4d771f311f2633d1bc4f61aaefdf7830b8b1d8|ed4d771f>

maxArrayJobSize

DetermineDiscoverability

ExternalResources

n0-0-n0-n1-0-dn0

TYPE

<https://github.com/flyteorg/flyteconsole/tree/master|master>

<https://github.com/flyteorg/flyteconsole/commit/356b2af44fa3f522af4fe5f08b89a04a2bfb3314|356b2af4>

<https://github.com/flyteorg/flyteadmin/tree/master|master>

<https://github.com/flyteorg/flyteadmin/commit/7915ae843e24683e3ad7dab90030218323019827|7915ae84>

<https://github.com/flyteorg/flyte/tree/master|master>

<https://github.com/flyteorg/flyte/commit/5ab8cb887a5a7051070cb93fca603ed1f22f2f74|5ab8cb88>

v2.31.0

2.31.0 (2023-05-22)

Security

• Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to potential forwarding of

Proxy-Authorization

headers to destination servers when following HTTPS redirects.

When proxies are defined with user info (https://user:pass@proxy:8080), Requests will construct a

Proxy-Authorization

header that is attached to the request to authenticate with the proxy.

In cases where Requests receives a redirect response, it previously reattached the

Proxy-Authorization

header incorrectly, resulting in the value being sent through the tunneled connection to the destination server. Users who rely on defining their proxy credentials in the URL are strongly encouraged to upgrade to Requests 2.31.0+ to prevent unintentional leakage and rotate their proxy credentials once the change has been fully deployed.

Users who do not use a proxy or do not supply their proxy credentials through the user information portion of their proxy URL are not subject to this vulnerability.

Full details can be read in our Github Security Advisory and CVE-2023-32681.

v2.30.0

2.30.0 (2023-05-03)

Dependencies

• ⚠️ Added support for urllib3 2.0. ⚠️

This may contain minor breaking changes so we advise careful testing and reviewing https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html prior to upgrading.

Users who wish to stay on urllib3 1.x can pin to

urllib3<2

.

v2.29.0

2.29.0 (2023-04-26)

Improvements

• Requests now defers chunked requests to the urllib3 implementation to improve standardization. (#6226)

• Requests relaxes header component requirements to support bytes/str subclasses. (#6356)

2.31.0 (2023-05-22)

Security

• Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to potential forwarding of

Proxy-Authorization

headers to destination servers when following HTTPS redirects.

When proxies are defined with user info (https://user:pass@proxy:8080), Requests will construct a

Proxy-Authorization

header that is attached to the request to authenticate with the proxy.

In cases where Requests receives a redirect response, it previously reattached the

Proxy-Authorization

header incorrectly, resulting in the value being sent through the tunneled connection to the destination server. Users who rely on defining their proxy credentials in the URL are strongly encouraged to upgrade to Requests 2.31.0+ to prevent unintentional leakage and rotate their proxy credentials once the change has been fully deployed.

Users who do not use a proxy or do not supply their proxy credentials through the user information portion of their proxy URL are not subject to this vulnerability.

Full details can be read in our Github Security Advisory and CVE-2023-32681.

2.30.0 (2023-05-03)

Dependencies

• ⚠️ Added support for urllib3 2.0. ⚠️

This may contain minor breaking changes so we advise careful testing and reviewing https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html prior to upgrading.

Users who wish to stay on urllib3 1.x can pin to

urllib3<2

.

2.29.0 (2023-04-26)

Improvements

• Requests now defers chunked requests to the urllib3 implementation to improve standardization. (#6226)

• Requests relaxes header component requirements to support bytes/str subclasses. (#6356)

2.28.2 (2023-01-12)

@dependabot rebase

@dependabot rebase

@dependabot recreate

@dependabot merge

@dependabot squash and merge

@dependabot cancel merge

@dependabot reopen

@dependabot close

@dependabot ignore this major version

@dependabot ignore this minor version

@dependabot ignore this dependency

Traceback (most recent call last):
  File "debug_remote.py", line 45, in <module>
    flyte_workflow_execution = remote.fetch_execution(project=project, domain=domain, name=execution)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/remote/remote.py", line 362, in fetch_execution
    self.client.get_execution(
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/remote/remote.py", line 192, in client
    self._client = SynchronousFlyteClient(self.config.platform, **self._kwargs)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/clients/raw.py", line 44, in __init__
    self._channel = wrap_exceptions_channel(cfg, upgrade_channel_to_authenticated(cfg, get_channel(cfg)))
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/clients/auth_helper.py", line 165, in get_channel
    credentials = grpc.ssl_channel_credentials(load_cert(cfg.ca_cert_file_path))
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/__init__.py", line 1611, in ssl_channel_credentials
    _cygrpc.SSLChannelCredentials(root_certificates, private_key,
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 143, in grpc._cython.cygrpc.SSLChannelCredentials.__cinit__
TypeError: expected certificate to be bytes, got <class 'OpenSSL.crypto.X509'>

credentials = grpc.ssl_channel_credentials(load_cert(cfg.ca_cert_file_path))

load_cert

grpc.ssl_channel_credentials

credentials = grpc.ssl_channel_credentials(crypto.dump_certificate(crypto.FILETYPE_PEM, load_cert(cfg.ca_cert_file_path)))

load_cert

output = foo_task(
              param=value
            ).with_overrides(
                requests=Resources(mem="32Gi", cpu="4", ephemeral_storage="4Gi")
            )

requests

@task(
    requests=Resources(mem="16Gi", cpu="4", ephemeral_storage="4Gi"),
)

<https://github.com/flyteorg/flyteconsole/tree/master|master>

<https://github.com/flyteorg/flyteconsole/commit/b15640ad3674a737c54fe3ff77b3abeede4c8dd5|b15640ad>

pip

virtualenv 20.23.0 has requirement importlib-metadata>=6.4.1; python_version < "3.8", but you have importlib-metadata 6.0.0.
virtualenv 20.23.0 has requirement platformdirs<4,>=3.2, but you have platformdirs 2.6.2.
virtualenv 20.23.0 has requirement filelock<4,>=3.11, but you have filelock 3.9.0.
sphinxext-remoteliteralinclude 0.4.0 has requirement MarkupSafe<=2.0.1; python_version < "3.8", but you have MarkupSafe 2.1.2.
Sphinx 4.5.0 requires sphinxcontrib-htmlhelp, which is not installed.
Sphinx 4.5.0 requires sphinxcontrib-applehelp, which is not installed.

<https://github.com/flyteorg/flyteadmin/tree/master|master>

<https://github.com/flyteorg/flyteadmin/commit/15d5c7bb568b570890b1c696f0d01466ea6bef2d|15d5c7bb>

pyflyte