lupyuen · GitHub

Is your feature request related to a problem? Please describe.

  • NuttX Devs need to wait (2 hours) for the CI Build to complete across all Architectures (Arm32, Arm64, RISC-V, Xtensa), even though they're modifying a Single Architecture
  • We're using too many GitHub Runners and Build Minutes, exceeding the ASF Policy for GitHub Actions
  • Our usage of GitHub Runners is going up ($12K per month), we need to stay within the ASF Budget for GitHub Runners ($8.2K per month)
  • What if CI could build only the Modified Architecture?
  • Right now most of our CI Builds are taking 2 hours 15 mins. Can we complete the build within 1 hour, when we Create / Modify a Simple PR?

Describe the solution you'd like

  • We propose a Partial Solution, based on the Arch and Board Labels (recently added to CI)
  • We target only the Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: risc-v, Board: risc-v, Size: XS"
  • If "Arch: arm" is the only non-size label, then we build only arm-01, arm-02, ...
  • Same for "Board: arm"
  • If Arch and Board Labels are both present: They must be the same
  • Similar rules for RISC-V, Simulator, x86_64 and Xtensa
  • Simple PR + Docs is still considered a Simple PR (so devs won't be penalised for adding docs)

Fetch the Arch Labels

This is how we fetch the Arch Labels, and identify as Arm, Arm64, RISC-V, Xtensa:

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L32-L104

      # Get the Arch for the PR: arm, arm64, risc-v, xtensa, ...
      - name: Get arch
        id: get-arch
        run: |        

          # If PR is Not Created or Modified: Build all targets
          pr=${{github.event.pull_request.number}}
          if [[ "$pr" == "" ]]; then
            echo "Not a Created or Modified PR, will build all targets"
            exit
          fi

          # Ignore the Label "Area: Documentation", because it won't affect the Build Targets
          query='.labels | map(select(.name != "Area: Documentation")) | '
          select_name='.[].name'
          select_length='length'

          # Get the Labels for the PR: "Arch: risc-v \n Board: risc-v \n Size: XS"
          # If GitHub CLI Fails: Build all targets
          labels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_name || echo "")
          numlabels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_length || echo "")
          echo "numlabels=$numlabels" | tee -a $GITHUB_OUTPUT

          # Identify the Size, Arch and Board Labels
          if [[ "$labels" == *"Size: "* ]]; then
            echo 'labels_contain_size=1' | tee -a $GITHUB_OUTPUT
          fi
          if [[ "$labels" == *"Arch: "* ]]; then
            echo 'labels_contain_arch=1' | tee -a $GITHUB_OUTPUT
          fi
          if [[ "$labels" == *"Board: "* ]]; then
            echo 'labels_contain_board=1' | tee -a $GITHUB_OUTPUT
          fi

          # Get the Arch Label
          if [[ "$labels" == *"Arch: arm64"* ]]; then
            echo 'arch_contains_arm64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: arm"* ]]; then
            echo 'arch_contains_arm=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: risc-v"* ]]; then
            echo 'arch_contains_riscv=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: simulator"* ]]; then
            echo 'arch_contains_sim=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: x86_64"* ]]; then
            echo 'arch_contains_x86_64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: xtensa"* ]]; then
            echo 'arch_contains_xtensa=1' | tee -a $GITHUB_OUTPUT
          fi

          # Get the Board Label
          if [[ "$labels" == *"Board: arm64"* ]]; then
            echo 'board_contains_arm64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: arm"* ]]; then
            echo 'board_contains_arm=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: risc-v"* ]]; then
            echo 'board_contains_riscv=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: simulator"* ]]; then
            echo 'board_contains_sim=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: x86_64"* ]]; then
            echo 'board_contains_x86_64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: xtensa"* ]]; then
            echo 'board_contains_xtensa=1' | tee -a $GITHUB_OUTPUT
          fi

        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Why || echo ""? That's because if the GitHub CLI gh fails for any reason, we will build all targets. This ensures that our CI Workflow won't get disrupted due to errors in GitHub CLI.

Handle Only Simple PRs

We handle only Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: risc-v, Board: risc-v, Size: XS". If it's not a Simple PR: We build everything.

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L127-L169

          # inputs.boards is a JSON Array: ["arm-01", "risc-v-01", "xtensa-01", ...]
          # We compact and remove the newlines
          boards=$( echo '${{ inputs.boards }}' | jq --compact-output ".")
          numboards=$( echo "$boards" | jq "length" )
          # We consider only Simple PRs with:
          # Arch + Size Labels Only
          # Board + Size Labels Only
          # Arch + Board + Size Labels Only
          if [[ "$labels_contain_size" != "1" ]]; then
            echo "Size Label Missing, will build all targets"
            quit=1
          elif [[ "$numlabels" == "2" && "$labels_contain_arch" == "1" ]]; then
            echo "Arch + Size Labels Only"
          elif [[ "$numlabels" == "2" && "$labels_contain_board" == "1" ]]; then
            echo "Board + Size Labels Only"
          elif [[ "$numlabels" == "3" && "$labels_contain_arch" == "1"  && "$labels_contain_board" == "1" ]]; then
            # Arch and Board must be the same
            if [[
              "$arch_contains_arm" != "$board_contains_arm" ||
              "$arch_contains_arm64" != "$board_contains_arm64" ||
              "$arch_contains_riscv" != "$board_contains_riscv" ||
              "$arch_contains_sim" != "$board_contains_sim" ||
              "$arch_contains_x86_64" != "$board_contains_x86_64" ||
              "$arch_contains_xtensa" != "$board_contains_xtensa"
            ]]; then
              echo "Arch and Board are not the same, will build all targets"
              quit=1
            else
              echo "Arch + Board + Size Labels Only"
            fi
          else
            echo "Not a Simple PR, will build all targets"
            quit=1
          fi
          # If Not a Simple PR: Build all targets
          if [[ "$quit" == "1" ]]; then
            echo "selected_builds=$boards" | tee -a $GITHUB_OUTPUT
            exit
          fi

For Arm Arch: Identify the Non-Arm Builds

Suppose the PR says "Arch: arm" or "Board: arm". We filter out the builds that should be skipped (RISC-V, Xtensa, etc):

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L169-L234

          # For every board
          for (( i=0; i<numboards; i++ ))
          do
            # Fetch the board
            board=$( echo "$boards" | jq ".[$i]" )
            skip_build=0
            # For "Arch / Board: arm": Build arm-01, arm-02, ...
            if [[ "$arch_contains_arm" == "1" || "$board_contains_arm" == "1" ]]; then
              if [[ "$board" != *"arm"* ]]; then
                skip_build=1
              fi
            # Omitted: Arm64, RISC-V, Simulator x86_64, Xtensa
            ...
            # For Other Arch: Allow the build
            else
              echo Build by default: $board
            fi
            # Add the board to the selected builds
            if [[ "$skip_build" == "0" ]]; then
              echo Add $board to selected_builds
              if [[ "$selected_builds" == "" ]]; then
                selected_builds=$board
              else
                selected_builds=$selected_builds,$board
              fi
            fi
          done
          # Return the selected builds as JSON Array
          # If Selected Builds is empty: Skip all builds
          echo "selected_builds=[$selected_builds]" | tee -a $GITHUB_OUTPUT
          if [[ "$selected_builds" == "" ]]; then
            echo "skip_all_builds=1" | tee -a $GITHUB_OUTPUT
          fi

Skip The Non-Arm Builds

Earlier we saw the code in arch.yml (Reusable Workflow) that identifies the builds to be skipped. The code above is called by build.yml (Build Workflow) which will actually skip the builds:

https://github.com/apache/nuttx/blob/master/.github/workflows/build.yml#L119-L148

  # Select the Linux Builds based on PR Arch Label
  Linux-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        [
          "arm-01", "other", "risc-v-01", "sim-01", "xtensa-01",
          "arm-02", "risc-v-02", "sim-02", "xtensa-02",
          "arm-03", "arm-04", "arm-05", "arm-06", "arm-07", "arm-08", "arm-09", "arm-10", "arm-11", "arm-12", "arm-13"
        ]

  # Run the selected Linux Builds
  Linux:
    needs: Linux-Arch
    if: ${{ needs.Linux-Arch.outputs.skip_all_builds != '1' }}
    runs-on: ubuntu-latest
    env:
      DOCKER_BUILDKIT: 1
    strategy:
      max-parallel: 12
      matrix:
        boards: ${{ fromJSON(needs.Linux-Arch.outputs.selected_builds) }}
    steps:
      ## Omitted: Run cibuild.sh on Linux

Why needs: Fetch-Source? That's because the PR Labeler runs concurrently in the background. When we add Fetch-Source as a Job Dependency, we give the PR Labeler sufficient time to run (1 min), before we read the PR Label in arch.yml.

Same for RISC-V, Simulator, x86_64 and Xtensa Builds

We do the same for RISC-V, Simulator, x86_64 and Xtensa:

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L105-L129

            # For "Arch / Board: arm64": Build other
            elif [[ "$arch_contains_arm64" == "1" || "$board_contains_arm64" == "1" ]]; then
              if [[ "$board" != *"other"* ]]; then
                skip_build=1
              fi
            # For "Arch / Board: risc-v": Build risc-v-01, risc-v-02
            elif [[ "$arch_contains_riscv" == "1" || "$board_contains_riscv" == "1" ]]; then
              if [[ "$board" != *"risc-v"* ]]; then
                skip_build=1
              fi
            # For "Arch / Board: simulator": Build sim-01, sim-02
            elif [[ "$arch_contains_sim" == "1" || "$board_contains_sim" == "1" ]]; then
              if [[ "$board" != *"sim"* ]]; then
                skip_build=1
              fi
            # For "Arch / Board: x86_64": Build other
            elif [[ "$arch_contains_x86_64" == "1" || "$board_contains_x86_64" == "1" ]]; then
              if [[ "$board" != *"other"* ]]; then
                skip_build=1
              fi
            # For "Arch / Board: xtensa": Build xtensa-01, xtensa-02
            elif [[ "$arch_contains_xtensa" == "1" || "$board_contains_xtensa" == "1" ]]; then
              if [[ "$board" != *"xtensa"* ]]; then
                skip_build=1
              fi

Skip the macOS and Windows Builds

For these Simple PRs (One Arch Label + One Size Label), we skip the macOS and Windows builds (macos, macos/sim-*, msys2) since these builds are costly:

(macos and macos/sim-* builds will take 2 hours to complete due to the queueing for macOS Runners)

https://github.com/apache/nuttx/blob/master/.github/workflows/build.yml#L194-L281

  # Select the macOS Builds based on PR Arch Label
  macOS-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        ["macos", "sim-01", "sim-02"]

  # Run the selected macOS Builds
  macOS:
    permissions:
      contents: none
    runs-on: macos-13
    needs: macOS-Arch
    if: ${{ needs.macOS-Arch.outputs.skip_all_builds != '1' }}
    strategy:
      max-parallel: 2
      matrix:
        boards: ${{ fromJSON(needs.macOS-Arch.outputs.selected_builds) }}
    steps:
      ## Omitted: Run cibuild.sh on macOS
      ...
  # Select the msys2 Builds based on PR Arch Label
  msys2-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        ["msys2"]

  # Run the selected msys2 Builds
  msys2:
    needs: msys2-Arch
    if: ${{ needs.msys2-Arch.outputs.skip_all_builds != '1' }}
    runs-on: windows-latest
    strategy:
      fail-fast: false
      max-parallel: 1
      matrix:
        boards: ${{ fromJSON(needs.msys2-Arch.outputs.selected_builds) }}
    defaults:
      run:
        shell: msys2 {0}
    steps:
      ## Omitted: Run cibuild.sh on msys2

skip_all_builds will be set to 1 for Simple PRs on macOS and msys2.

(Except for "Arch: Simulator", which will enable the macOS Builds for sim-01 and sim-02)

Ignore the Documentation

NuttX Devs shouldn't be penalised for adding docs! That's why we ignore the label "Area: Documentation", so that Simple PR + Docs is still a Simple PR (which will skip the unnecessary builds).

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L44-L55

          # Ignore the Label "Area: Documentation", because it won't affect the Build Targets
          query='.labels | map(select(.name != "Area: Documentation")) | '
          select_name='.[].name'
          select_length='length'
          # Get the Labels for the PR: "Arch: risc-v \n Board: risc-v \n Size: XS"
          # If GitHub CLI Fails: Build all targets
          labels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_name || echo "")
          numlabels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_length || echo "")
          echo "numlabels=$numlabels" | tee -a $GITHUB_OUTPUT

Sync the CI Workflow from nuttx repo to nuttx-apps

Remember to sync build.yml and arch.yml from nuttx repo to nuttx-apps! apache/nuttx-apps#2676

build.yml refers to arch.yml (for the build rules). So when we sync build.yml from nuttx to nuttx-apps, we won't need to remove the references to arch.yml.

We could make nuttx-apps/build.yml point to the nuttx/arch.yml. But that would make the CI fragile: Changes to nuttx/arch.yml might cause nuttx-apps/build.yml to break.

Yep arch.yml is totally not needed in nuttx-apps. I have difficulty keeping nuttx/build.yml and nuttx-apps/build.yml in sync, that's why I simply copied over arch.yml as-is. In future we could extend arch.yml with Build Rules that are specific to nuttx-apps?

If we decide to remove nuttx-apps/arch.yml: This means that we need to rewrite the build.yml logic from this:

  # Select the Linux Builds based on PR Arch Label
  Linux-Arch:
    uses: apache/nuttx-apps/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      boards: |
        [
          "arm-01", "other", "risc-v-01", "sim-01", "xtensa-01", ...
        ]

  # Run the selected Linux Builds
  Linux:
    needs: Linux-Arch
    if: ${{ needs.Linux-Arch.outputs.skip_all_builds != '1' }}
    strategy:
      matrix:
        boards: ${{ fromJSON(needs.Linux-Arch.outputs.selected_builds) }}

Back to this:

  Linux:
    needs: Fetch-Source
    strategy:
      matrix:
        boards: [arm-01, arm-02, arm-03, arm-04, arm-05, arm-06, arm-07, arm-08, arm-09, arm-10, arm-11, arm-12, arm-13, other, risc-v-01, risc-v-02, sim-01, sim-02, xtensa-01, xtensa-02]

Testing

(Note: The timings here are obsolete)

When we test our updated CI Workflow, we see that the irrelevant builds are skipped in seconds. Click "Show All Jobs" to reveal the timings:

Actual Performance

We recorded the CI Build Performance based on Real-World PRs:

  • For Arm32: Simple PRs will build in 2 hours (previously also 2 hours)

  • For Arm64: Simple PRs will build in 49 mins (previously 2 hours 11 mins)

  • For RISC-V: Simple PRs will build in 1 hour 45 mins (previously also 1 hour 45 mins)

  • For Xtensa: Simple PRs will build in 1 hour 17 mins (previously 2 hours 11 mins)

  • For x86_64: Simple PRs will build in 48 mins (previously 2 hours 13 mins)

  • For Simulator: Simple PRs will build in 1 hour 32 mins (previously 2 hours 12 mins)

  • OK no big wow yet. We need to break arm-05, riscv-01 and riscv-02 into multiple smaller jobs. Then things will really zoom! (See the Build Job Details)

    Move the RP2040 jobs from arm-05 to arm-06, then add arm-14. Add jobs riscv-03 to riscv-06.

    (Update: All Done! Check the PRs below)

  • We already see a 27% Reduction in GitHub Runner Hours! From 15 Runner Hours down to 11 Runner Hours per Arm32 Build.

  • Split the Board Labels according to Arch, like "Board: arm". So "Board: arm" should build the exact same way as "Arch: arm". Same for "Board: arm, Arch: arm". Update the Build Rules to use the Board Labels

    (Update: All Done! Check the PRs below)

  • Split the others job into arm64 and x86_64

    (Update: All Done! Check the PRs below)

TODO: Reorg and rename the CI Build Jobs, for better performance and easier maintenance. But how?

  • I have a hunch that CI works better when we pack the jobs into One-Hour Time Slices
  • Kinda like packing yummy goodies into Bento Boxes, making sure they don't overflow the Time Boxes :-)
  • We should probably shift the Riskiest / Most Failure Prone builds into the First Build Job. So we can Fail Faster (in case of problems), and skip the rest of the jobs
  • Recently we see many builds for Arm32 Goldfish. Can we limit the builds to the Goldfish Boards only? To identify Goldfish PRs, we can label the PRs like this: "Arch: arm, SubArch: goldfish" and/or "Board: arm, SubBoard: goldfish"
  • How will we filter out the Build Jobs (e.g. arm-01) that should be built for a SubBoard (e.g. stm32)? Maybe like this

Hi @raiden00pl and @simbit18: Do you think this will work? And keep Our Devs and ASF happy? Thanks!

Describe alternatives you've considered

No response

Verification

  • I have verified before submitting the report.

Read the original on github.com ↗