세미나
Jenkins Pipeline - Advanced
nineDeveloper
2019. 10. 22. 21:19
728x90
Jenkinsfile
- Pipeline 스크립트를 git 에서 관리할 수 있다.
Job 설정
Job 구성 > Pipeline script from SCM 선택
Jenkinsfile
properties([parameters([string(defaultValue: 'james', description: '', name: 'YourName', trim: true)])])
stage 'Compile' node() {
checkout scm
def mvnHome = tool 'apache-maven-3.6.0'
try {
sh "${mvnHome}/bin/mvn clean test"
}
catch(e) {
echo "exception"
}
stash 'working-copy'
}주의
- Properties
- UI에서도 변경 가능하나 한번만 적용되고 실행 시 Jenkinsfile 설정으로 overwrite 된다.
실습 05. - Jenkinsfile
- Jenkinsfile 생성하고 Pipeline job 만들기
- 실습 03. Repository 활용
- Job 구성 > Pipeline script from SCM
- Git 설정
- Jenkinsfile 경로 입력
Reusable Pipeline Libraries
개요
- Pipeline 스크립트를 라이브러리로 만들기
- https://jenkins.io/doc/book/pipeline/shared-libraries/
- Public libraries
Sample
- 라이브러리로 등록된
runUITest()호출
Reusable 스크립트
- Directory structure (Git Repository)
(git root)
+- src # Groovy source files
| +- org
| +- foo
| +- Bar.groovy # for org.foo.Bar class
+- vars
| +- foo.groovy # for global 'foo' variable
| +- foo.txt # help for 'foo' variable
+- resources # resource files (external libraries only)
| +- org
| +- foo
| +- bar.json # static helper data for org.foo.BarSample 1
https://github.com/hyunil-shin/pipeline-library-demo
- vars/sayHello.groovy
#!/usr/bin/env groovy
def call(String name = 'human') {
echo "Hello, ${name}."
}Jenkins에 라이브러리 추가
- Global과 Folder에 추가할 수 있음
- Jenkins > folder > Configure > Pipeline Libraries
- name
- 아무거나 (변수명과 비슷)
- 예) demo
- default version
- default 브랜치 (예) master)
- 호출 시 브랜치 지정할 수 있음
- Modern SCM (git)
- git 설정
- name
- 여러 개 등록 가능
호출
@Library('demo')_
stage('Demo') {
echo 'Hello World'
// library 함수 호출
sayHello 'Dave'
}library 브랜치 지정
@Library('demo@newBranch')_
stage('Demo') {
echo 'Hello World'
sayHello 'Dave'
}Sample 2 - src/ 사용
- src/ci/Bar.groovy
#!/usr/bin/groovy
package ci;
def runTest(param1, param2) {
node('master') {
sh "echo parameters: ${param1}, ${param2}"
sh "echo jenkins parameters: ${params.env}"
}
}
return this;runTest() 호출
@Library('demo')_
import ci.Bar
stage('Demo') {
Bar a = new Bar()
a.runTest('hello', 'world')
}실행 결과
Running on Jenkins in /data/jenkins/workspace/pipeline_training/library-demo2
[Pipeline] {
[Pipeline] sh
+ echo parameters: hello, world parameters: hello, world
[Pipeline] sh
+ echo jenkins parameters: alpha jenkins parameters: alpha
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline참고. Groovy class
helloWorld.groovy
println "Hello, World!"helloWorld.class
Compiled from "helloWorld.groovy"
public class helloWorld extends groovy.lang.Script {
public static transient boolean __$stMC;
public helloWorld();
public helloWorld(groovy.lang.Binding);
public static void main(java.lang.String...);
public java.lang.Object run();
protected groovy.lang.MetaClass $getStaticMetaClass();
}Sample 3 - 리소스 활용, shell script 호출
- resources/script.sh
#!/bin/bash
echo "this is shell script"리소스 로드
- libraryResource: return its content as a plain string
@Library('demo@shell')_
stage('Demo') {
node() {
def scriptContent = libraryResource "script.sh"
println scriptContent
writeFile file: "hello.sh", text: scriptContent
sh "chmod +x hello.sh"
sh "./hello.sh"
}
}실행 결과
[Pipeline] sh
+ chmod
+x hello.sh
[Pipeline] sh
+ ./hello.sh this is shell script
[Pipeline] }Sample 4 - Jenkinsfile
Jenkinsfile에서도 library를 사용할 수 있다.
// shell_Jenkinsfile
@Library('demo')_
import ci.Bar
node('master') {
stage('Demo') {
Bar a = new Bar()
a.runTest('hello', 'world')
}
}실습 06. - shared library
- mvn test, junit, jacoco 동작을 shared library로 만들기
- Share Library
- 신규 repository 생성
- vars/xxx.groovy 추가
- Jenkins
- Folder 설정에 shared library 추가
- Application Jenkinsfile 수정
- application code repository (maven + java)
- shared library 호출
- Jenkins
- Pipeline job 생성
Example
- shared library > vars/mvn.groovy
#!/usr/bin/env groovy
def call() {
node {
checkout scm
def testType
stage('input') {
def userInput = input message: 'Choose test type', parameters: [string(defaultValue: 'test', description: 'test or verify', name: 'testType', trim: false)], submitterParameter: 'store'
testType = userInput.testType
}
stage('build') {
withEnv(["PATH+MAVEN=${tool 'mvn-3.6.0'}/bin"]) {
sh 'mvn --version'
sh "mvn clean ${testType}"
}
}
stage('report') {
junit 'target/surefire-reports/*.xml'
jacoco execPattern: 'target/**.exec'
addShortText background: '', borderColor: '', color: '', link: '', text: testType
}
}
}- Jenkinsfile
mvn()
스크립트 모듈화 (/src 에서만 가능)
ci/Util.groovy (공통 함수)
#!/usr/bin/groovy
package ci;
def printOptions(options = []) {
// do not use echo: MissingMethodException
echo "this is module"
for(int i = 0; i < options.size(); i++) {
echo options[i]
sh "echo ${options[i]}"
}
}ci/Zoo.groovy (공통 함수 호출)
#!/usr/bin/groovy
package ci;
def abc(param1, param2) {
node('master') {
sh "echo parameters: ${param1}, ${param2}"
sh "echo jenkins parameters: ${params.env}"
def util = new Util()
util.printOptions([param1, param2])
}
}스크립트 모듈화 - load()
- https://github.com/hyunil-shin/java-maven-junit-helloworld
- Jenkinsfile_module.groovy (공통 함수)
properties([parameters([string(defaultValue: 'test', description: 'test or verify', name: 'testType', trim: false)])])
node {
checkout scm
stage('build') {
//withEnv(["PATH+MAVEN=${tool 'apache-maven-3.3.9'}/bin"]) {
withEnv(["PATH+MAVEN=${tool 'mvn-3.6.0'}/bin"]) {
sh 'mvn --version'
sh "mvn clean ${params.testType}"
}
}
stage('report') {
junit 'target/surefire-reports/*.xml'
jacoco execPattern: 'target/**.exec'
}
}- Jenkinsfile_caller (공통 함수 호출)
node('master') {
checkout scm
def m = load('Jenkinsfile_module.groovy')
m.mvnBuild("test")
}알아야 할 것들
Parallel Execution
- 병렬 실행: parallel step
def builds = [:]
// map
builds["p1"] = {
node('master') {
echo "p1"
}
}
builds["p2"] = {
node() {
echo "p2"
}
}
parallel buildsParallel Execution
def testList = ["a", "b", "c", "d"]
def branches = [:]
for (int i = 0; i < 4 ; i++) {
int index=i, branch = i+1
stage ("branch_${branch}"){
branches["branch_${branch}"] = {
node (''){
sh "echo 'node: ${NODE_NAME}, index: ${index}, i: ${i}, testListVal: " + testList[index] + "'"
}
}
}
}
parallel branches실패하는 경우
def testList = ["a", "b", "c", "d"]
def branches = [:]
for (int i = 0; i < 4 ; i++) {
int index=i, branch = i+1
stage ("branch_${branch}"){
branches["branch_${branch}"] = {
node (''){
sh "echo 'node: ${NODE_NAME}, index: ${index}, i: ${i}, testListVal: " + testList[index] + "'"
// 실패하게 만든다.
if(index == 0) {
sh "exit 10"
}
}
}
}
}
parallel branches- branch 1이 실패로 표시된다.
주의
- 변경되는 변수는 local variable로 만들어서 사용해야 한다.
- ex) index = i
def testList = ["a", "b", "c", "d"]
def branches = [:]
for (int i = 0; i < 4 ; i++) {
int index=i, branch = i+1
stage ("branch_${branch}"){
branches["branch_${branch}"] = {
node (''){
sh "echo 'node: ${NODE_NAME}, index: ${index}, i: ${i}, testListVal: " + testList[index] + "'"
echo "${i} == ${index}" // 실행 시점에 i와 index는 같지 않다.
}
}
}
}
parallel branches실행 결과
[branch_1] node: master, index: 0, i: 4, testListVal: a
[branch_2] node: master, index: 1, i: 4, testListVal: b
[branch_3] node: master, index: 2, i: 4, testListVal: c
[branch_4] node: master, index: 3, i: 4, testListVal: d실습 07. - Parallel
- maven + junit 시 여러 java 버전으로 빌드 => 병렬 실행으로 변경
- JUnit 결과 합치기
- stash, unstash
- mvn ... -Dsurefire.reportNameSuffix=java1_8
node {
stage("git") {
git 'https://github.com/hyunil-shin/java-maven-junit-helloworld.git'
}
stage('jdk8') {
def jdk = tool name: 'openjdk8', type: 'jdk'
withEnv(["JAVA_HOME=${jdk}", "PATH+JDK=${jdk}/bin", "PATH+MAVEN=${tool 'mvn-3.6.0'}/bin"]) {
sh "mvn clean test"
}
}
stage('jdk11') {
def jdk = tool name: 'openjdk9', type: 'jdk'
withEnv(["JAVA_HOME=${jdk}", "PATH+JDK=${jdk}/bin", "PATH+MAVEN=${tool 'mvn-3.6.0'}/bin"]) {
sh "mvn clean test"
}
}
stage('jdk12') {
def jdk = tool name: 'openjdk10', type: 'jdk'
withEnv(["JAVA_HOME=${jdk}", "PATH+JDK=${jdk}/bin", "PATH+MAVEN=${tool 'mvn-3.6.0'}/bin"]) {
sh "mvn clean test"
}
}
stage('report') {
junit 'target/surefire-reports/*.xml'
}
}실습 07. 예상 결과
Pipeline 스크립트 검증 방안
- commit 전에 스크립트가 올바른지 확인하는 방법
- https://jenkins.io/doc/book/pipeline/development/#pipeline-development-tools
Replay
- Jenkins UI에서 스크립트를 수정하여 실행할 수 있다.
- 빌드 > Replay
- diff
Groovy Sandbox
- 위험한 Jenkins API 호출을 방지하기 위한 기능
- 허용되지 않은 호출이 있으면 실패하고, ScriptApproval에 요청이 추가된다 (Jenkins admin이 승인해야 함).
- By default, all Jenkins pipelines run in a Groovy sandbox.
Script Approval
- Jenkins 관리자
- Jenkins 관리 > In-process Script Approval
Demo
stage("1") {
node('master') {
// Get all Causes for the current build
def causes = currentBuild.rawBuild.getCauses()
// Get a specific Cause type (in this case the user who kicked off the build),
// if present.
def specificCause = currentBuild.rawBuild.getCause(hudson.model.Cause$UserIdCause)
//println specificCause.getName()
println specificCause.getUserName()
}
}실행하면 에러가 발생한다.
[Pipeline] // stage Scripts not permitted to use method hudson.model.Run getCauses. Administrators can decide whether to approve or reject this signature.
[Pipeline] End of Pipeline
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method hudson.model.Run getCauses at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectMethod(StaticWhitelist.java:175)스크립트 승인 필요
Groovy Sandbox unchecked
- Jenkins admin 경우
- script approval 없이 실행된다.
- 일반 users
- 스크립트 전체를 admin이 리뷰하여 승인 또는 거부한다.
- users may also disable the Groovy Sandbox entirely. Disabling the Groovy Sandbox requires that the entire script be reviewed and manually approved by an administrator
불편한 점
- 허용되지 않은 호출이 한번에 하나씩 발견된다.
- 실행 -> 에러 확인 -> 승인 -> 실행 -> 에러 확인 -> 승인 -> ...
CPS (Continuation Passing Style)
A Jenkins Pipeline has been made so robust that it can resume in various failure scenarios, like a master or agent going down.
To be able to do this, every intermediate step of the Pipeline is stored, so the Pipeline know where to resume in case of a failure.
To store the Pipeline steps, they must be Serializable.
In your case
def theJobs = Jenkins.instance.getAllItems(Job); is not serializable since an entry in the list is a FreeStyleProject and that does not implements Serializable.
A way to workaround this is to move all code that uses non-serializable steps into a method and annotate that method with @NonCPS.
That signals to the Pipeline to not try and store the result of this method, so it needs not serialize it.
Should a failure occur and and the result from this method be needed in a yet to execute step, then this method will be executed again.
에러 발생
- Error Message
- Caused: java.io.NotSerializableException: hudson.model.ParametersAction
def call(String env, String tests, String extraParams) { ...
// 동적으로 추가된 빌드 매개변수에 접근하기 위함
def myparams = currentBuild.rawBuild.getAction(ParametersAction)
def pMap = [:]
for( p in myparams ) {
pMap[p.name.toString()] = p.value.toString()
}
...
}스크립트 수정 - @NonCPS 사용
- 에러 있는 스크립트를 별도 함수로 만들고
@NonCPS를 추가한다.
@NonCPS
def getGradleProperties(List extraParams) {
// 동적으로 추가된 빌드 매개변수에 접근하기 위함
def myparams = currentBuild.rawBuild.getAction(ParametersAction)
def pMap = [:]
for( p in myparams ) {
pMap[p.name.toString()] = p.value.toString()
}
// gradle -Dxxxx=yyyy
def gradle_properties = ""
for(i = 0; i < extraParams.size(); i++) {
echo extraParams[i].name
echo pMap[extraParams[i].name]
gradle_properties = "-D${extraParams[i].name}=${pMap[extraParams[i].name]} "
}
return gradle_properties
}CPS 참고
- https://github.com/cloudbees/groovy-cps/blob/master/doc/cps-basics.md
- https://stackoverflow.com/questions/44417906/jenkins-pipeline-exception-java-io-notserializableexception-hudson-model-freest
- https://stackoverflow.com/questions/42295921/what-is-the-effect-of-noncps-in-a-jenkins-pipeline-script
- https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/
- Use
@NonCPS-annotated functions for slightly more complex work
- Use
More Examples
Docker
node {
docker.image('node:7-alpine').inside {
stage('Test') {
sh 'node --version'
}
}
}- Caching data
node {
docker.image('maven:3-alpine').inside('-v $HOME/.m2:/root/.m2') {
stage('Build') {
sh 'mvn -B'
}
}
}github 연동
- git push 시 jenkins job 빌드
- commit 정보에 jenkins 빌드 결과 추가
참고
- https://dzone.com/articles/how-to-integrate-your-github-repository-to-your-je
- https://support.cloudbees.com/hc/en-us/articles/115003015691-GitHub-Webhook-Non-Multibranch-Jobs
- github ip (jenkins 서버 ip acl)
github 연동 - how to
- GitHub
- access token 발급
- Jenkins 설정
- Jenkins 관리 > 시스템 설정 > GitHub
- github 서버 추가
- 개인 access token 연결 (secret text as credential)
- github 서버 추가
- Jenkins 관리 > 시스템 설정 > GitHub
- github repository 설정
- webhook 추가
- Payload URL: [jenkins_url]/github-webhook/
- Content type: application/json
- event 선택
- webhook 추가
- Job 설정 (
- 빌드 유발
- GitHub hook trigger for GITScm polling
- GitHub hook trigger for GITScm polling
- PostBuild
- Set GitHub commit status (universal)
- Set GitHub commit status (universal)
- 빌드 유발
Pipeline Code
properties([[$class: 'GithubProjectProperty', displayName: '', projectUrlStr: 'https://github.com/hyunil-shin/github_integration/'],
pipelineTriggers([githubPush()])])
node() {
checkout scm
sh 'ls -al'
setBuildStatus("Build complete", "SUCCESS");
}
void setBuildStatus(String message, String state) {
step([
$class: "GitHubCommitStatusSetter",
reposSource: [$class: "ManuallyEnteredRepositorySource", url: "https://github.com/hyunil-shin/github_integration"],
contextSource: [$class: "ManuallyEnteredCommitContextSource", context: "jenkins build status"],
errorHandlers: [[$class: "ChangingBuildStatusErrorHandler", result: "UNSTABLE"]],
statusResultSource: [ $class: "ConditionalStatusResultSource", results: [[$class: "AnyBuildResult", message: message, state: state]] ]
]);
}Multiple Branches
- branch 설정하지 않으면 모든 브랜치에 대해 동작한다.
- UI job에서만 가능하다. Pipeline job에서는 master 브랜치만 적용된다.
Multibranch Pipeline
- 브랜치 별로 Pipeline job이 자동 생성된다.
- 브랜치 삭제하면 해당 job이 자동 비활성화된다.
- Jenkins > new Job > Multibranch Pipeline
실습 09. github 연동
- create new repository
- add Jenkinsfile
- add github webhook trigger
- add Jenkinsfile
- create jenkins job
- github 변경 때마다 job 실행
plugins
https://github.com/hyunil-shin/JenkinsPipelineTraining/tree/master/pipeline_examples/10_plugin
Groovy Postbuild
node() {
sh 'ls -al'
manager.addShortText("this is something")
}Timestamper
HTML Publisher
HTTP Request
728x90