tl;dr; Set a Global Environment Variable GIT_LFS_SKIP_SMUDGE=1 to fix the issue.
Recently we encountered issues with our Jenkins CI setup at Blendle with Git repositories which contain Git LFS files. The behavior of the normal workspace checkout of Jenkins has changed without any apparent reason. No plugins are updated nor did Jenkins itself receive an upgrade, we know this for sure because we have codified our CI setup and plugins using the Configuration as Code.
This is error we received
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/blendle/repo_with_lfs_files.git
> git init /home/jenkins/agent/workspace/test # timeout=10
Fetching upstream changes from https://github.com/blendle/repo_with_lfs_files.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials GitHub Username & Password
> git fetch --no-tags --force --progress -- https://github.com/blendle/repo_with_lfs_files.git +refs/heads/test:refs/remotes/origin/test
Fetching without tags
Pruning obsolete local branches
Checking out Revision 7888fec2200a056b081a077d0ec5c3023b18c70f (test)
> git config remote.origin.url https://github.com/blendle/repo_with_lfs_files.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/test:refs/remotes/origin/test # timeout=10
> git config remote.origin.url https://github.com/blendle/repo_with_lfs_files.git # timeout=10
Fetching upstream changes from https://github.com/blendle/repo_with_lfs_files.git
using GIT_ASKPASS to set credentials GitHub Username & Password
> git fetch --no-tags --force --progress --prune -- https://github.com/blendle/repo_with_lfs_files.git +refs/heads/test:refs/remotes/origin/test
> git config core.sparsecheckout # timeout=10
> git checkout -f 7888fec2200a056b081a077d0ec5c3023b18c70f
captured job exception: [class: class hudson.plugins.git.GitException] [message: Could not checkout 7888fec2200a056b081a077d0ec5c3023b18c70f]
exception has cause: [class: class hudson.plugins.git.GitException] [message: Command "git checkout -f 7888fec2200a056b081a077d0ec5c3023b18c70f" returned status code 128:
stderr: Downloading lfs_file_object (3.6 MB)
Error downloading object: lfs_file_object (46bc159): Smudge error: Error downloading lfs_file_object (46bc159dd84a8b99d14d65fcd66af5f13220b19ef8f74fd9656a32cc47a0ab3b): batch response: Bad credentials
Errors logged to /home/jenkins/agent/workspace/test.git/lfs/logs/20190930T051331.0177141.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: lfs_file_object: smudge filter lfs failed
hudson.plugins.git.GitException: Command "git checkout -f 7888fec2200a056b081a077d0ec5c3023b18c70f" returned status code 128:
stdout:
stderr: Downloading lfs_file_object (3.6 MB)
Error downloading object: lfs_file_object (46bc159): Smudge error: Error downloading lfs_file_object (46bc159dd84a8b99d14d65fcd66af5f13220b19ef8f74fd9656a32cc47a0ab3b): batch response: Bad credentials
Errors logged to /home/jenkins/agent/workspace/test.git/lfs/logs/20190930T051331.0177141.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: lfs_file_object: smudge filter lfs failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1000(CliGitAPIImpl.java:78)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2471)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.0.1.19/10.0.1.19:36878
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy116.execute(Unknown Source)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1242)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused: hudson.plugins.git.GitException: Could not checkout 7888fec2200a056b081a077d0ec5c3023b18c70f
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2495)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
On several issues, we could find on Jenkins and Git LFS, it was recommended to install git-lfs both on the Jenkins Master and the clients, so we did but without any luck. Strange part was we couldn’t duplicate the behavior if we executed the same steps in a client pod as Jenkins.
Since we aren’t interested in pulling the Git LFS files during the initial checkout, we’ve disabled it by using a global environment variable which we expose during the whole pipeline step:
Configure Global Environment Variable
After this change, our pipelines are running again without any issues, since the Git LFS files are checked out if needed in any further steps of the pipeline like it was already the case before.
