Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

I have two questions about flyte copilot.

• Is my understanding correct that when using copilot, task inputs and outputs are not downloaded/uploaded using `fsspec` in python at all?
• How much of a “first-class-citizen” is copilot? The <https://github.com/flyteorg/flyte/tree/master/flytecopilot#raw-notes|“raw notes”> in its readme sound a bit like its WIP.
Thank you :pray:

correct, we are using stow to download/upload. <https://github.com/flyteorg/flyte/blob/5d792c09bef73c12df46c48fe197418254093f0c/flytecopilot/cmd/download.go#L80>

there is no flytekit in the raw container

the default behavior is 3
&gt; Use shared process namespace. This allows all pids in a pod to share the namespace. Thus pids can see each other

the thing is i wrote it over a weekend 4 years ago

I have one more question, will `fsspec` still be used to load/write from/to local filesystem even if copilot is used? We have a problem with `fsspec` because it sometimes just randomly deadlocks and prevents anything from being written to blob storage, causing tasks to hang. (And it’s not the “known” deadlock issue with fsspec and forked processes.)
So far we haven’t figured out what breaks fsspec but are looking for ways to not rely on fsspec for uploading task outputs.

2 things
1. There is a regression in fsspec, known use older one. I’ll put some info here
2. We are looking to move away from s3fs and gang to rust ob store

this one <https://github.com/flyteorg/flyte/issues/5936>

Thank you :pray: Unfortunately we are on an older version and still have deadlock issues.

I have another question about copilot, is this only for raw containers or can I also use this for “normal” python function tasks to avoid having to download/upload data from/to blob storage with fsspec?

```	} else if pod := taskTemplate.GetK8SPod(); pod != nil {
		dataLoadingConfig = pod.GetDataConfig()
	}

    if dataLoadingConfig != nil {
		if err := AddCoPilotToContainer(ctx, config.GetK8sPluginConfig().CoPilot,
			primaryContainer, taskTemplate.Interface, dataLoadingConfig); err != nil {
			return nil, nil, err
		}

		if err := AddCoPilotToPod(ctx, config.GetK8sPluginConfig().CoPilot, podSpec, taskTemplate.GetInterface(),
			tCtx.TaskExecutionMetadata(), tCtx.InputReader(), tCtx.OutputWriter(), dataLoadingConfig); err != nil      {
			return nil, nil, err
		}
	}```

For a normal python function task, `dataLoadingConfig` ends up being `nil` so that the copilot side car isn’t injected.

<@UNR3C6Y4T> updating to a version that has <https://github.com/fsspec/gcsfs/commit/8a83230c671b9f1c1f500791e6b2b1810f8919b9|this> commit in gcsfs fixed our hanging issue.

We were never able to reproduce it in a minimal working example, only sporadically in some actual jobs running for > 1h. Also did not occur when sleeping for >1h.