@ancient-wolf-19325, How flyte creates namespace in kubernetes ?? sometime it is taking time to create it ??

Flyte creates Kubernetes namespaces for each project-domain combination using the ClusterResourceManager (also called clusterresourcesync), which applies namespace templates defined in the deployment configuration. When a new project is created, the controller generates a namespace (e.g., project-domain) by rendering a template like:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ namespace }}

This process can be delayed if the clusterresourcesync job or cronjob is not running, or if there are issues with Kubernetes API responsiveness. Manually running the sync job can force namespace creation if it's stuck. See namespace template example and discussion on delays. Would you like more detail on troubleshooting or optimizing namespace creation? Sources: - GitHub Issue: Creating workflows in sandbox projects return 500 - namespace template example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 is it syncresources service does this ??

Yes, the syncresources service (also known as ClusterResourceManager or clusterresourcesync) is responsible for creating Kubernetes namespaces in Flyte. It periodically applies resource templates (including namespaces) for each project-domain using the configuration provided, ensuring required namespaces and other resources exist before workflow execution. If syncresources is not running or misconfigured, namespace creation may be delayed or fail. See clusterresourcesync deployment and related discussion. Would you like more detail on how syncresources works or how to troubleshoot it? Sources: - clusterresourcesync deployment - GitHub Issue: Creating workflows in sandbox projects return 500 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, i am seeing logs like this

time="2025-11-20T10:28:30Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml]"
I1120 10:28:32.056930       1 request.go:697] Waited for 1.005357527s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/spv.no/v2beta1?timeout=30s>
I1120 10:28:42.848880       1 request.go:697] Waited for 1.005900167s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/networking.istio.io/v1beta1?timeout=30s>
I1120 10:28:53.664114       1 request.go:697] Waited for 1.006154385s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/external.metrics.k8s.io/v1beta1?timeout=30s>
I1120 10:29:04.505501       1 request.go:697] Waited for 1.007857689s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/api/v1?timeout=30s>
I1120 10:29:15.299616       1 request.go:697] Waited for 1.007806427s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/serving.knative.dev/v1beta1?timeout=30s>
I1120 10:29:26.101972       1 request.go:697] Waited for 1.007481108s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/elbv2.k8s.aws/v1alpha1?timeout=30s>
I1120 10:29:36.890503       1 request.go:697] Waited for 1.007090247s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/external-secrets.io/v1beta1?timeout=30s>
I1120 10:29:47.722655       1 request.go:697] Waited for 1.006806129s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/secrets-store.csi.x-k8s.io/v1?timeout=30s>
I1120 10:29:58.546108       1 request.go:697] Waited for 1.006969524s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/reports.kyverno.io/v1?timeout=30s>
I1120 10:30:09.351203       1 request.go:697] Waited for 1.006792175s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/external-secrets.io/v1beta1?timeout=30s>
I1120 10:30:20.135857       1 request.go:697] Waited for 1.007484358s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/source.toolkit.fluxcd.io/v1?timeout=30s>
I1120 10:30:30.960252       1 request.go:697] Waited for 1.00675405s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/serving.kserve.io/v1beta1?timeout=30s>
I1120 10:30:41.913228       1 request.go:697] Waited for 1.006577323s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/elbv2.k8s.aws/v1beta1?timeout=30s>
I1120 10:30:52.726863       1 request.go:697] Waited for 1.005899978s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/generators.external-secrets.io/v1alpha1?timeout=30s>
I1120 10:31:03.536732       1 request.go:697] Waited for 1.007651619s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/secrets-store.csi.x-k8s.io/v1alpha1?timeout=30s>
I1120 10:31:14.328961       1 request.go:697] Waited for 1.00731417s due to client-side throttling, not priority and fairness, request: GET:<https://172.20.0.1:443/apis/spv.no/v1alpha1?timeout=30s>

It seems likely that these logs indicate the syncresources (ClusterResourceManager) service is being throttled by the Kubernetes API server due to exceeding client-side rate limits. The message Waited for ... due to client-side throttling means the Kubernetes client is pausing requests to avoid overwhelming the API server, which can slow down resource creation like namespaces or quotas. This is controlled by the QPS (queries per second) and burst settings in the kube-client config for syncresources. You may want to increase the QPS and burst values in the syncresources kube-client config to reduce throttling, but be cautious as setting them too high can cause server-side throttling or instability. For more, see the discussion on Kube API server throttling and tuning and syncresources deployment config. Would you like more detail on how to adjust these settings or monitor API server health? Sources: - Kube API server throttling and tuning 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, how to adjust

To adjust the client-side throttling for syncresources (ClusterResourceManager), you need to increase the kube-client QPS (queries per second) and burst values in its configuration. In Flyte, this is typically set in the

kubeClientConfig

section of the syncresources deployment YAML or configmap. For example:

kube-client-config:
  qps: 100
  burst: 25
  timeout: 30s

Edit the relevant config (often in your Helm values or configmap for clusterresourcesync), then restart the syncresources deployment for changes to take effect. For more, see example config and performance tuning discussion. Would you like step-by-step instructions for your specific deployment method? Sources: - example config - performance tuning discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325

kube-client-config

i dont see this in syncresource deployment. is it in flyte admin config ? or some where else in config map ?

The kube-client-config section is not present by default in the syncresources (clusterresourcesync) deployment or its configmap. It is typically found in the FlytePropeller config, but not in syncresources. For syncresources, Kubernetes client rate limits are not directly exposed in the standard Helm chart or config files, so you cannot adjust QPS/burst for syncresources via a documented config section. If you need to tune these values for syncresources, you would have to modify the source code or the deployment to add support for kube-client-config, as is done for FlytePropeller. This is not currently documented or supported out-of-the-box in the official Flyte Helm charts or deployment manifests. See flyte_helm_dataplane_generated.yaml for Propeller's config, but note that syncresources does not have this section. Would you like more detail on possible workarounds or how to request this feature? Sources: - flyte_helm_dataplane_generated.yaml 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, i see this in flytepropeller config

kube-client-config:
    burst: 25
    qps: 100
    timeout: 30s

can u explain what burst 25 , qps 100 and timeout 30s means

manager:
  pod-application: flytepropeller
  pod-template-container-name: flytepropeller
  pod-template-name: flytepropeller-template
propeller:
  downstream-eval-duration: 30s
  enable-admin-launcher: true
  gc-interval: 12h
  kube-client-config:
    burst: 25
    qps: 100
    timeout: 30s
  leader-election:
    enabled: true
    lease-duration: 15s
    lock-config-map:
      name: propeller-leader
      namespace: infra
    renew-deadline: 10s
    retry-period: 2s
  limit-namespace: all
  max-workflow-retries: 50
  metadata-prefix: metadata/propeller
  metrics-prefix: flyte
  prof-port: 10254
  queue:
    batch-size: -1
    batching-interval: 2s
    queue:
      base-delay: 5s
      capacity: 1000
      max-delay: 120s
      rate: 100
      type: maxof
    sub-queue:
      capacity: 1000
      rate: 100
      type: bucket
    type: batch
  rawoutput-prefix: <s3://uniphore-flyte-staging-us/>
  workers: 40
  workflow-reeval-duration: 30s
webhook:
  certDir: /etc/webhook/certs
  serviceName: flyte-pod-webhook

In the FlytePropeller config,

qps: 100

sets the maximum number of Kubernetes API requests per second, and

burst: 25

allows up to 25 requests in a short burst above the QPS rate before throttling occurs.

timeout: 30s

specifies the maximum time to wait for a single API request before giving up. These settings control how aggressively FlytePropeller interacts with the Kubernetes API and help prevent client-side throttling or overloading the API server. For more, see the FlytePropeller config reference and performance tuning discussion. Would you like more detail on how these values affect performance or how to tune them? Sources: - performance tuning discussion - FlytePropeller config reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, How much i can increase it ??

There is no strict upper limit for

qps

and

burst

, but increasing them too much can cause the Kubernetes API server to throttle requests or become unstable, especially under high load. Typical safe values are

qps: 100-500

and

burst: 25-200

, but the optimal setting depends on your cluster size, API server capacity, and workload. You should increase these values gradually and monitor for signs of API server throttling or degraded performance. See the warning and tuning advice in the performance doc update. Would you like more detail on how to monitor for throttling or best practices for tuning? Sources: - performance doc update 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, github link where i can see the source code for syncresources service

The source code for the syncresources service (ClusterResourceManager) is located in the Flyte repository under flyteadmin, specifically in the flyteadmin/pkg/clusterresource directory. Would you like more detail on the main files or logic within this component? Sources: - https://github.com/flyteorg/flyte/blob/master/flyteadmin/pkg/clusterresource/controller.go - https://github.com/flyteorg/flyte/issues/2927 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, Print debug logs in flyte admin and syncresources service is by adding LOG_LEVEL env to DEBUG ??

It seems likely that setting debug log level in FlyteAdmin and syncresources is done by configuring the logger level in their config files, not just by setting a LOG_LEVEL environment variable. For FlyteAdmin, you adjust the

logger.level

field in the logger config (e.g.,

logger.yaml

) to a higher verbosity (such as

3

or

DEBUG

). For syncresources, if it uses the same logging library, you would also set the logger config accordingly. See logger.yaml example and logger config code. Would you like more detail on the exact config values or how to apply them? Sources: - https://github.com/flyteorg/flyte/issues/250 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes

Certainly! Here’s how you can enable debug logs for FlyteAdmin and syncresources (ClusterResourceManager): --- 1. FlyteAdmin FlyteAdmin uses a logger configuration file (commonly

logger.yaml

or as part of your Helm values/configmap). To set the log level to DEBUG, set the

level

field to a lower value (lower numbers are more verbose):

logger:
  show-source: true
  level: 3  # 5=FATAL, 4=ERROR, 3=WARNING, 2=INFO, 1=DEBUG, 0=TRACE

-

level: 1

is DEBUG,

level: 0

is TRACE (most verbose). - You can also set this via an environment variable if your deployment supports it, e.g.:

env:
    - name: FLYTE_LOG_LEVEL
      value: "1"

Reference: logger.yaml example --- 2. syncresources (ClusterResourceManager) syncresources uses the same logging configuration approach as FlyteAdmin. If you deploy via Helm, you can set the logger config in your values file or as a configmap:

logger:
  show-source: true
  level: 1  # DEBUG

Or, if you want to set it via an environment variable (if supported by your deployment):

env:
  - name: FLYTE_LOG_LEVEL
    value: "1"

--- 3. Applying the Configuration - Update your configmap or values file with the new logger settings. - Restart the FlyteAdmin and syncresources pods for the changes to take effect. --- 4. Additional Notes - The numeric log levels are defined in flytestdlib/logger/config.go. - If you use Helm, you can set these values in your

values.yaml

under the relevant component (e.g.,

flyteadmin.logger.level

). --- Summary Table: | Log Level | Value | |-----------|-------| | TRACE | 0 | | DEBUG | 1 | | INFO | 2 | | WARNING | 3 | | ERROR | 4 | | FATAL | 5 | --- References: - Flyte logger config example - Logger config code --- If you need step-by-step instructions for your specific deployment method (Helm, Kustomize, etc.), let me know! 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 if is use otherthan 5, flyte admin pod is up but grcp server is not starting only print these logs and nothing is working

time="2025-12-07T13:14:24Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
time="2025-12-07T13:14:25Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
time="2025-12-07T13:14:26Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
time="2025-12-07T13:14:26Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
time="2025-12-07T13:14:27Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"

It seems likely that setting the logger level to anything other than 5 (FATAL) in FlyteAdmin is causing the pod to hang and not start the gRPC server, with repeated "Using config file" logs. This behavior is not expected and may indicate a bug or misconfiguration in the logging setup. There are reports of similar issues where verbose logging or incorrect logger config can prevent FlyteAdmin from starting properly, especially if the logger config is malformed or incompatible with the Flyte version in use. See GitHub Issue: Flyte logger overflooding and logger.yaml example. Would you like more detail on troubleshooting logger config or known workarounds for this issue? Sources: - Flyte logger overflooding 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes

To troubleshoot this issue: 1. Ensure your logger config is valid YAML and matches the expected structure. For FlyteAdmin, it should look like:

logger:
     show-source: true
     level: 2  # or 1 for DEBUG, 0 for TRACE

2. If using a numeric level, confirm your Flyte version supports it. Some older versions may expect string values like "INFO" or "DEBUG" instead of numbers. 3. If FlyteAdmin still hangs, try reverting to the default config (

level: 5

) and incrementally lower the level to see which value causes the issue. 4. Check for any additional logs or errors in the pod (use

kubectl logs

) that might indicate a parsing or initialization error. 5. Review the logger config code and logger.yaml example for reference. Would you like step-by-step instructions for debugging the pod startup or for using string log levels as a workaround? Sources: - https://github.com/flyteorg/flyte/issues/6353 - https://github.com/flyteorg/flyte/issues/5578 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 flyte-admin-base-config

@RunLLM,

apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-admin-base-config
  namespace: infra
  uid: b9396bc0-0c55-45ea-b097-4e9a543cfa4a
  resourceVersion: '13657669688'
  creationTimestamp: '2024-09-11T07:49:00Z'
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyteadmin
    <http://helm.sh/chart|helm.sh/chart>: flyte-core-v1.12.2
    <http://helm.toolkit.fluxcd.io/name|helm.toolkit.fluxcd.io/name>: flyte
    <http://helm.toolkit.fluxcd.io/namespace|helm.toolkit.fluxcd.io/namespace>: infra
    k8slens-edit-resource-version: v1
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: infra
  selfLink: /api/v1/namespaces/infra/configmaps/flyte-admin-base-config
spec: {}
data:
  cluster_resources.yaml: |
    cluster_resources:
      customData:
      - production:
        - projectQuotaCpu:
            value: "150"
        - projectQuotaMemory:
            value: 400Gi
        - defaultIamRole:
            value: arn:aws:iam::255035174406:role/uniphore-flyte-binary-us
      - staging:
        - projectQuotaCpu:
            value: "100"
        - projectQuotaMemory:
            value: 300Gi
        - defaultIamRole:
            value: arn:aws:iam::255035174406:role/uniphore-flyte-binary-us
      - development:
        - projectQuotaCpu:
            value: "100"
        - projectQuotaMemory:
            value: 300Gi
        - defaultIamRole:
            value: arn:aws:iam::255035174406:role/uniphore-flyte-binary-us
      refreshInterval: 1s
      standaloneDeployment: false
      templatePath: /etc/flyte/clusterresource/templates
  db.yaml: |
    database:
      dbname: flyteadmin
      host: '<http://flyte-us-svc.c0ybjbwzg67e.us-east-2.rds.amazonaws.com|flyte-us-svc.c0ybjbwzg67e.us-east-2.rds.amazonaws.com>'
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: 'dbadmin'
  domain.yaml: |
    domains:
    - id: development
      name: development
  remoteData.yaml: |
    remoteData:
      region: us-east-1
      scheme: local
      signedUrls:
        durationMinutes: 3
  server.yaml: |
    Logger:
      level: 5
      show-source: true
    auth:
      appAuth:
        authServerType: External
        externalAuthServer:
          AllowedAudience:
          - <api://f918e381-c854-4566-9b6c-1d99d1d7874d>
          baseUrl: <https://login.microsoftonline.com/0fa04d0d-7f60-496e-9896-91ce76dd7b9a/v2.0>
          metadataUrl: .well-known/openid-configuration
        selfAuthServer:
          staticClients:
            flyte-cli:
              grant_types:
              - refresh_token
              - authorization_code
              id: flyte-cli
              public: true
              redirect_uris:
              - <http://localhost:53593/callback>
              - <http://localhost:12345/callback>
              response_types:
              - code
              - token
              scopes:
              - all
              - offline
              - access_token
            flytectl:
              grant_types:
              - refresh_token
              - authorization_code
              id: flytectl
              public: true
              redirect_uris:
              - <http://localhost:53593/callback>
              - <http://localhost:12345/callback>
              response_types:
              - code
              - token
              scopes:
              - all
              - offline
              - access_token
            flytepropeller:
              client_secret: ''
              grant_types:
              - refresh_token
              - client_credentials
              id: flytepropeller
              public: false
              redirect_uris:
              - <http://localhost:3846/callback>
              response_types:
              - token
              scopes:
              - all
              - offline
              - access_token
        thirdPartyConfig:
          flyteClient:
            clientId: 'f918e381-c8dd-4566-9b6c-1d99d1ddd74d'
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - <api://f918e381-c8dd-4566-9b6c-1d99d1ddd74d/.default>
      authorizedUris:
      - <https://localhost:30081>
      - <http://flyteadmin:80>
      - <http://flyteadmin:81>
      - <http://flyteadmin.infra.svc.cluster.local:80>
      - <http://flyteadmin.infra.svc.cluster.local:81>
      - <https://flyte.us.cloud.uniphorestaging.com>
      - <https://flyte.us.cloud.uniphorestaging.com/console>
      userAuth:
        openId:
          baseUrl: '<https://login.microsoftonline.com/0fa0dddd-7f60-496e-9896-91ce76dd7b9a/v2.0>'
          clientId: 'f918e381-c8dd-4566-9b6c-1d99d1ddd74d'
          grant_types:
          - refresh_token
          - authorization_code
          scopes:
          - profile
          - openid
    flyteadmin:
      eventVersion: 2
      metadataStoragePrefix:
      - metadata
      - admin
      metricsScope: 'flyte:'
      profilerPort: 10254
      roleNameKey: <http://iam.amazonaws.com/role|iam.amazonaws.com/role>
      testing:
        host: <http://flyteadmin>
    server:
      grpc:
        port: 8089
      httpPort: 8088
      security:
        allowCors: true
        allowedHeaders:
        - Content-Type
        - flyte-authorization
        allowedOrigins:
        - '*'
        secure: false
        useAuth: false
  storage.yaml: |
    storage:
      type: s3
      container: "uniphore-flyte-staging-us"
      connection:
        auth-type: iam
        region: us-east-2
      enable-multicontainer: false
      limits:
        maxDownloadMBs: 1000
  task_resource_defaults.yaml: |
    task_resources:
      defaults:
        cpu: 1000m
        memory: 1000Mi
        storage: 1000Mi
      limits:
        cpu: 24
        gpu: 1
        memory: 50Gi
        storage: 2000Mi

It seems likely that your

flyte-admin-base-config

ConfigMap is valid and contains the expected configuration sections for FlyteAdmin, including cluster resources, database, domains, remote data, server, storage, and task resource defaults. The logger level is set under

server.yaml

as:

Logger:
  level: 5
  show-source: true

This means only FATAL logs will be shown. If you want more verbose logs (e.g., DEBUG or INFO), you can lower the

level

value (e.g.,

level: 2

for INFO,

level: 1

for DEBUG). However, as discussed earlier, some Flyte versions may have issues starting if the logger level is set below 5, so test carefully after changing this value. The logger config should be under the

Logger

key, not

logger

, in this YAML structure. Would you like more detail on how to safely change the log level or troubleshoot startup issues after changing it? Sources: - https://github.com/flyteorg/flyte/issues/4695 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes

It seems likely that to safely change the log level in your FlyteAdmin config, you should: 1. Edit the

Logger

section in your

server.yaml

(inside the

flyte-admin-base-config

ConfigMap) and set the desired level, for example:

Logger:
     level: 2  # INFO
     show-source: true

2. Apply the updated ConfigMap to your cluster:

kubectl apply -f <your-configmap-file>.yaml

3. Restart the FlyteAdmin pod to pick up the new configuration:

kubectl rollout restart deployment <flyteadmin-deployment-name> -n infra

4. Monitor the pod logs (

kubectl logs ...

) to ensure FlyteAdmin starts correctly and the gRPC server is running. If the pod hangs or fails to start, revert the log level to

5

and restart again. If you still encounter issues with lower log levels, it may be a version-specific bug or a config parsing problem. In that case, check for errors in the pod logs and consult Flyte logger issues for workarounds. Would you like more detail on debugging startup failures or on Flyte version compatibility with log levels? Sources: - Flyte logger overflooding issue 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, if i set to 3 then i am seeing this log

{
  "json": {
    "src": "stow_store.go:460"
  },
  "level": "warning",
  "msg": "stow configuration section missing, defaulting to legacy s3/minio connection config",
  "ts": "2025-12-07T13:58:08Z"
}

This log message means that FlyteAdmin did not find a

stow

configuration section in your

storage.yaml

, so it is defaulting to the legacy S3/MinIO connection config. This is a warning, not an error, and is expected if you are using the

connection

block (legacy style) instead of the newer

stow

block for storage configuration. See storage config code for details. You can safely ignore this warning if your storage is working as expected. If you want to remove the warning, you can migrate your storage config to use the

stow

section, but it is not required for S3/MinIO setups. Would you like more detail on how to migrate to the stow config or on the implications of this warning? Sources: - storage config code 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, having more no of projects will create problem in creating namespace ??

It seems likely that having a large number of projects in Flyte will result in a large number of Kubernetes namespaces (one per project-domain by default), which can increase the load on the clusterresourcesync (syncresources) service and the Kubernetes API server. While Flyte is designed to handle multiple projects and domains, Kubernetes clusters have practical limits on the number of namespaces and resources they can efficiently manage. If namespace creation is slow or fails, it may be due to API server throttling, resource limits, or syncresources performance, as discussed in GitHub Issue: Document how to limit Flyte deployment to use restrictive namespaces. You can mitigate this by tuning syncresources, increasing API server capacity, or configuring Flyte to use fewer namespaces (e.g., one per domain instead of per project-domain). Would you like more detail on namespace scaling strategies or best practices? Sources: - GitHub Issue: Document how to limit Flyte deployment to use restrictive namespaces 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325, from command line can i trigger flyteadmin clusterresource run or is there any flyteadmin http api to trigger clusterresource run ?

You can trigger the cluster resource sync manually from the command line by running the FlyteAdmin binary with the

clusterresource run

subcommand. For example:

flyteadmin clusterresource run

This command starts the ClusterResourceController and performs a sync immediately. There is no documented FlyteAdmin HTTP API endpoint to trigger a clusterresource run; it is only available as a CLI subcommand. See clusterresource.go source. Would you like more detail on running this in your environment or automating it? Sources: - clusterresource.go source 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.