Bash fails in Gradle task on Azure pipeline with windows-2022

eye-catch Other techs
2022.03.16

vs2017-win2016 as Agent Specification but it was removed 2022-03-15. There were some points that I needed to look into it, so I wrote down those in this article.

Sponsored links

Bash script fails for all commands in gradle

There are 3 bash.exe in the environment (Azure pipeline). I checked the path of bash.exe in Bash Script by where bash command. The result is as follows.

C:\Program Files\Git\bin\bash.exe
C:\Windows\System32\bash.exe
C:\Program Files\Git\usr\bin\bash.exe

Git-bash is what we use on our development machines too, so it looks correct. We define Gradle tasks and they are used in the pipeline. It is something like this.

task build(type: Exec) {
    description 'Builds all modules with lerna'
    dependsOn 'lerna_bootstrap'

    workingDir 'src'
    commandLine 'bash', '-c', 'lerna run build'
}

However, it fails with exit code 1. Even if the command is the simplest command “ls“, it fails. I tried the following commands.

task yuto(type: Exec) {
    description 'test'

    workingDir 'src'
    commandLine 'bash', '-c', 'ls'
}

task yuto2(type: Exec) {
    description 'test'

    workingDir 'src'
    commandLine 'C:\Program Files\Git\bin\bash.exe', '-c', 'ls'
}

task yuto3(type: Exec) {
    description 'test'

    workingDir 'src'
    commandLine 'C:\Windows\System32\bash.exe', '-c', 'ls'
}

task yuto4(type: Exec) {
    description 'test'

    workingDir 'src'
    commandLine 'C:\Program Files\Git\usr\bin\bash.exe', '-c', 'ls'
}

yuto and yuto3 tasks failed with exit code 1. It turned out that C:\Windows\System32\bash.exe was executed in Gradle even though C:\Program Files\Git\bin\bash.exe is the highest priority. It worked in vs2017-win2016 environment, so it seems something changed.

I executed where bash command in Bash Script but it might be valid only in the script itself in the new environment windows-2022.

By specifying the absolute path to the desired bash.exe, it succeeds. However, one developer uses a Virtual Machine for his dev-env and the path was different. Using the absolute path was not so nice and it doesn’t fit the developer. Therefore, I changed it to PowerShell instead.

task build(type: Exec) {
    description 'Builds all modules with lerna'
    dependsOn 'lerna_bootstrap'

    workingDir 'src'
    commandLine 'powershell', '-c', 'lerna run build'
}

It also works.

Sponsored links

How to get an Environment Variable on Powershell in gradle task

Changing bash to PowerShell works for most tasks but some of them need an adjustment because the way to get an environment variable is different. The original one is something like this.

task publish(type: Exec) {
    description 'Publish artifacts'
    dependsOn 'build', 'test'

    environment NPM_REGISTRY: project.property('NPM_REGISTRY')
    workingDir 'src'
    commandLine 'powershell', '-c', "lerna publish \"${version}-${build_timestamp}\" --registry \"\$NPM_REGISTRY\" --exact --no-git-tag-version --no-push --yes --force-publish=* --no-git-reset"

It’s not difficult. It publishes artifacts after build and test steps but it contains an environment variable $NPM_REGISTRY. It works on bash script but it doesn’t on Powershell. Normally, we can get an env variable with this.

$env:ENV_NAME

But I didn’t know how to write it in the command at first. I tried the following commands but all of them failed.

commandLine 'powershell', '-c', " echo $env:NPM_REGISTRY"
commandLine 'powershell', '-c', " echo ${env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo ${$env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo ${$env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo " + env:NPM_REGISTRY + ""
commandLine 'powershell', '-c', " echo " + $env:NPM_REGISTRY + ""

To make it work, $ mark needs to be added to the command when PowerShell is executed. It means the symbol must be escape like this.

commandLine 'powershell', '-c', " echo \$env:NPM_REGISTRY"

I applied it to the actual command.

task publish(type: Exec) {
    description 'Publish artifacts'
    dependsOn 'build', 'test'

    environment NPM_REGISTRY: project.property('NPM_REGISTRY')
    workingDir 'src'
    commandLine 'powershell', '-c', "lerna publish ${version}-${build_timestamp} --registry \$env:NPM_REGISTRY --exact --no-git-tag-version --no-push --yes --force-publish=* --no-git-reset"
}

MSBuild.exe not found error in gradle

Our project has different languages and we need to build them all in different ways. MSBuild.exe is used to build C# code but it fails because of MSBuild.exe not found error.

Azure pipeline offers MSBuild task but it was not possible to use it because MSBuild.exe is called in one of bash scripts called by Gradle task.

I looked for the MSBuild.exe file first with the following commands in the Azure pipeline.

echo "-----ls----"
ls 
echo "------Program Files---"
ls "/c/Program Files"

echo "-----vs----"
ls "/c/Program Files/Microsoft Visual Studio/"

echo "-----vs2022----"
ls "/c/Program Files/Microsoft Visual Studio/2022/"

echo "------enterprise---"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/"

echo "-----msbuild----"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/"

echo "-----Current----"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/"

echo "------bin---"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin"


echo "---X86"
ls "/c/Program Files (x86)/"

echo "---Installer"
ls "/c/Program Files (x86)/Microsoft Visual Studio/Installer"

echo "---shared"
ls "/c/Program Files (x86)/Microsoft Visual Studio/Shared"

There might be another better way to do a similar thing but I don’t have enough access right. So I investigated it in this way.

I added the desired path to MSBuild.exe to PATH variable.

PATH="/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin/":$PATH

After this, MSBuild.exe can be found by a bash script called by a Gradle task.

node-gyp does not find Visual Studio version

When we used vs2017-win2016 environment, it works without any problem but I got the following error on windows-2022.

lerna ERR! npm run build stderr:
gyp ERR! find VS 
gyp ERR! find VS msvs_version not set from command line or npm config
gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp ERR! find VS unknown version "undefined" found at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise"
gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
gyp ERR! find VS looking for Visual Studio 2015
gyp ERR! find VS - not found
gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
gyp ERR! find VS 
gyp ERR! find VS **************************************************************
gyp ERR! find VS You need to install the latest version of Visual Studio
gyp ERR! find VS including the "Desktop development with C++" workload.
gyp ERR! find VS For more information consult the documentation at:
gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
gyp ERR! find VS **************************************************************
gyp ERR! find VS 
gyp ERR! configure error 
gyp ERR! stack Error: Could not find any Visual Studio installation to use
gyp ERR! stack     at VisualStudioFinder.fail (C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:121:47)
gyp ERR! stack     at C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:74:16
gyp ERR! stack     at VisualStudioFinder.findVisualStudio2013 (C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:351:14)
gyp ERR! stack     at C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:70:14
gyp ERR! stack     at C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:372:16
gyp ERR! stack     at C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\util.js:54:7
gyp ERR! stack     at C:\hostedtoolcache\windows\node\12.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\util.js:33:16
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:310:5)
gyp ERR! stack     at ChildProcess.emit (events.js:311:20)
gyp ERR! stack     at maybeClose (internal/child_process.js:1021:16)
gyp ERR! System Windows_NT 10.0.20348
gyp ERR! command "C:\\hostedtoolcache\\windows\\node\\12.16.1\\x86\\node.exe" "C:\\hostedtoolcache\\windows\\node\\12.16.1\\x86\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\a\1\s\src\modules\module-name
gyp ERR! node -v v12.16.1
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! module-name@x.y.z build: `sh build.sh`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the module-name@x.y.z build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

node-gyp version is old, so I needed to update it first. I’ve read Updating-npm-bundled-node-gyp and applied it.

I added the following commands to update node-gyp in the right place.

cd "/c/hostedtoolcache/windows/node/12.16.1/x86/node_modules/npm/node_modules/npm-lifecycle"
npm install node-gyp@9.0.0

I don’t remember well but the path to MSBuild.exe might need to be added to the PATH env.

PATH="/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin/":$PATH

Redirect operation leads a different result

If a redirect operation is used in the command, the results can be different. See the following two commands.

commandLine 'bash', '-c', 'echo node.exe app.js > run.bat'
commandLine 'powershell', '-c', 'echo node.exe app.js > run.bat'

Both commands are the same but the execution script is different. It looks it generates exactly the same file but actually not.

The result of the bash command

node.exe app.js

The result of PowerShell command

node.exe
app.js

If using PowerShell, the line is split into two lines. For PowerShell, it’s necessary to add double quotations there.

echo "node.exe app.js" > run.bat

Then, the result becomes the same in PowerShell but it still didn’t work in the Gradle task. When I executed the run.bat, it failed with the following error.

D:\temp>.\run.bat

D:\temp>■e
'■e' is not recognized as an internal or external command,
operable program or batch file.

The characters were garbled. I found the encoding different

  • bash: Unix(LF), UTF-8
  • PowerShell: Windows (CR LF), UTF-16 LE BOM

I guess BOM causes the garbled character above. BOM stands for Byte Order Mark that is used to know in which file format the file is written.

BOM is added by default in Powershell. I somehow needed to remove it. I tried the following.

echo "node.exe app.js" | Out-File -encoding utf8 run.bat
echo "node.exe app.js" | Out-File -encoding ascii run.bat

The first one creates a file with BOM even though it doesn’t add BOM in the encoding.

The second one creates exactly what I want. It doesn’t work if the content contains characters that ASCII can’t handle but it is no problem in my case.

But it doesn’t work in Gradle script!! None of the following commands didn’t work.

commandLine 'powershell', '-c', 'echo "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'echo \"node.exe app.js\" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'echo `"node.exe app.js`" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Host "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output \"node.exe app.js\" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output `"node.exe app.js`" | Out-File -encoding ascii -FilePath run.bat'

I changed from PowerShell to cmd to make it work.

commandLine 'cmd', '/c', 'echo node.exe app.js > run.bat'

This actually works as expected but I found a better way with Gradle.

task create_run_bat() {
    description 'Generate a batch script'

    file("src/app/dist/run.bat").text = "node.exe app.js"
}

It looks a proper way for Gradle.

End

I could finally manage to adjust the Azure pipeline for the new environment. I’ve touched Azure pipeline before but it was very small things to change. I struggled with many things this time but got more familiar with Azure pipeline.

I hope this article helps other developers.

Sponsored links

Comments

Copied title and URL