Scheduler CrashLoopBackOff After Deploying Image Built On Or After May 20th 2026 (kubernetes Python Client 36.0.0 Regression)

Summary

We have noticed an issue regarding new images pushed to Astro deployments on or after 2026-05-20, where the scheduler enters CrashLoopBackOff shortly after, and pod logs show the scheduler loop exiting with 401 responses from the kubelet. In a scenario similar to the above, the likely cause is an upstream regression in version 36.0.0 of the kubernetes Python client. Rolling back to the previous image will resolve the issue. A permanent fix is to pin kubernetes<36 or pin 35.0.0 directly with kubernetes==35.0.0 in your image's requirements.txt until a patched upstream release is available.

Symptoms

•  The deployment uses Kubernetes Executor or KPO
•  A previously healthy deployment gets stuck in DEPLOYING immediately after a new image is pushed.
•  Scheduler enters CrashLoopBackOff. It is not impossible for workers or triggers to exhibit this behavior but Scheduler seems to be the main pod affected.
•  Tasks stop being scheduled due to the scheduler being unhealthy
•  Rolling back to the previously deployed image immediately resolves the issue.
•  Pod logs show repeated 401 Unauthorized responses from the Kubernetes API server.

Cause

Version 36.0.0 of the kubernetes Python client (released 2026-05-20) shipped a regression in which config.load_incluster_config() no longer attaches an Authorization: Bearer  header to outgoing requests. Every call to the Kubernetes API from inside the pod is subsequentally rejected as unauthenticated.

Airflow's scheduler relies on this client, both directly and via the cncf.kubernetes provider, to talk to the Kubernetes API. This is frequently done to launch KubernetesPodOperator tasks or manage the Kubernetes Executor's worker pods. When authentication fails on startup, the scheduler crashes and Kubernetes restarts the pod, producing CrashLoopBackOff.

This is not an Astronomer-side change. It affects any image that pulls in kubernetes==36.0.0, which can happen even if you don't list kubernetes directly in your requirements.txt. This is a transitive dependency of apache-airflow-providers-cncf-kubernetes and several other packages, so an unpinned image rebuilt after 2026-05-20 may silently upgrade, resulting in this behavior

Resolution

Immediate mitigation: Roll back to the last known good image. This will undo all changes regarding the upgrade to the kubernetes 36.0.0.

Midterm mitigation: Pin the kubernetes package below 36.0.0 in your image. Add the following line to your requirements.txt:

kubernetes<36

Also acceptable is hard pinning the last release before 36.0.0

kubernetes==35.0.0

Longterm fix: Upgrading to awaited patched release of kubernetes package to address: https://github.com/kubernetes-client/python/issues/2591

How to confirm this is your issue

Using a local dev environment, you can either deploy your image with:

astro dev start

Then exec into the pod to run pip freeze | grep kubernetes or pip show kubernetes | grep -iE version

Alternatively, you could run the image locally without using the entire dev environment

docker run <imageTag> pip freeze | grep kubernetes or docker run <imageTag> pip show kubernetes | grep -iE version

If you are impacted, these commands should return that it is using kubernetes 36.0.0

kubernetes==36.0.0

Or

Version: 36.0.0