Skip to main content

Slow Deploy Times from GitHub Actions Managed Runners

Saketh Somaraju
Updated

 

Problem Statement

When deploying via GitHub Actions using managed (GitHub-hosted) runners, users may experience intermittent slowness or stalls during the “Pushing image to Astronomer registry” step. Logs often show repeated “Preparing” or “Waiting” messages and retries such as:

8e79fa1cca0e: Retrying in 5 seconds
8e79fa1cca0e: Retrying in 4 seconds

These issues typically resolve after retries or when the deployment is re-run from a local or self-hosted environment, indicating that the root cause lies in the GitHub client-side network path or performance rather than the Astronomer registry or cluster-side infrastructure.

 

Observed Behaviour

  • Builds and image pushes stall or slow down specifically on GitHub-hosted runners.

  • Local machines or self-hosted runners in the same AWS region (e.g., us-east-1) complete normally.

  • Investigation into the Astronomer registry, S3 bucket performance, and AWS throttling has not revealed any root cause.

  • Issues are sporadic and self-healing, often succeeding on retry without changes.

  • The behavior suggests a client-side issue within GitHub’s managed infrastructure, such as throttling, temporary regional congestion, or degraded networking performance.

Investigation Summary

We’ve confirmed that there is no throttling on the AWS S3 backing store and that the Astronomer registry, cluster network paths, and throughput are all operating normally. The issue is reproducible only on GitHub-managed runners and resolves after retries or when using self-hosted environments, indicating a transient, runner-side network or performance issue.

Recommended Troubleshooting Steps

If you encounter slow or stalled deploys from a GitHub Actions workflow, you can try the following mitigations. These are general suggestions based on observed patterns, Astronomer cannot directly diagnose or resolve GitHub Actions runner performance issues.

  1. Try Larger or Different Runners
    Consider using a higher-tier GitHub-hosted runner (for example, ubuntu-latest-xlarge, if available) or a self-hosted runner. Larger runners sometimes reduce delays when pushing large Docker images, but results may vary depending on GitHub’s infrastructure.

  2. Use a Self-Hosted Runner 
    If possible, try deploying from a self-hosted runner located in the same AWS region as your Astronomer registry (e.g., us-east-1). This can help ensure consistent network performance and remove variables related to GitHub’s shared runner environment.

    https://docs.github.com/en/actions/concepts/runners/self-hosted-runners 

  3. Contact GitHub Support
    If the issue persists, consider opening a GitHub Support ticket and include:

    • Links to the failing workflow runs and timestamps

    • Log snippets showing the docker push retries or delays

    • The runner type, region, and any recent configuration changes

    You may also mention that Astronomer’s registry and infrastructure have been verified as healthy, which can help GitHub identify whether there are broader network or regional service issues affecting managed runners.

NOTE: These steps are optional mitigations that may help reduce delays. Because the behavior appears specific to GitHub’s managed runner environments, Astronomer cannot directly troubleshoot or resolve the issue. If problems persist, we recommend reaching out to GitHub Support for further investigation.

 

Related to

Related articles

Comments

0 comments

Please sign in to leave a comment.