In this post, I’d like to outline what MacLemon and I worked on for the metalab Hackathon 8. The project stems from a problem with one of the most basic rules of software development:

Let software developers develop software, don’t hog them with repeated configuration/administration stuff.

This rule is violated with large-scale Cocoa development for the iPhone that also has beta testers: If you want to allow them, you need something like HockeyApp (or TestFlight, but we didn’t use that one). Further, somebody might break the build without realizing it (that’s not specific in any way to Cocoa development). Different versions of Xcode use different compilers, which might not accept the same source code.

Apple sketched a solution for some of these problems in WWDC 2012, session 404: You can integrate Xcode into Jenkins CI, a continuous integration platform written in Java. We chose to improve on this solution, to let it do much more.

What you need:

  • A build server that runs Mac OS X, accessible by all developers and itself able to access the HockeyApp website.
  • A server runnig git. This might be the same server as the above, but doesn’t have to be. Our instructions are for the case where it’s on the same server, as this is easier to do (only “localhost” as hostname).
  • Development must happen using git & Xcode (we tested version 4.3, 4.4DP4 and 4.5DP2)
  • A very solid knowledge of the command line, since just about everything will be done there.

What this solution provides:

  • On every push, the app is checked for build errors (and the test suites can be run, if you happen to have some).
  • Working versions are automatically tagged, so you can easily correlate build logs to the git history.
  • Pushing a release to HockeyApp only requires adding a tag to a revision before pushing it to the server.

Some ideas in what way this solution can be extended very easily:

  • Push working git revisions to a different git server.
  • Email a build breaker to communicate this fact.

Concepts

The system is configured so that on every push of a new commit, Jenkins downloads the revision and builds it. When it works, Jenkins also tags it with the name jenkins_<build number>. The build number is also displayed in its web interface.

When you push, and the HEAD is also tagged with release_<major>_<minor>_<patch> (where all the <> are numbers, everything except the major number are optional), Jenkins automatically uploads the result to HockeyApp. Supply your release notes as the message for that tag (-m on the command line).

The most recent release_-tag in the commit tree is used to update the version number inside the Info.plist of the application (no matter whether it is uploaded to HockeyApp or not).

Initial Setup

Obviously, you need Xcode installed on your build server. Note the application name of all the versions you have installed (for example, “Xcode45-DP2″). Xcode comes with git, which is very handy.

First, I installed gitolite for easy repository configuration. For this, we need a git user first.

To keep things as minimalistic as possible, this is not a regular user as created by the UI, we created it using the command line (as root):

First, create the group:

dscl . create /Groups/git PrimaryGroupID 1025

The unique ID was chosen based on not having a conflict with the other groups.

Next, create the user:

dscl . create /Users/git
dscl . create /Users/git UserShell /usr/bin/false
dscl . create /Users/git RealName git
dscl . create /Users/git UniqueID 1025
dscl . create /Users/git PrimaryGroupID 1025
dscl . create /Users/git NFSHomeDirectory /Users/Shared/git
dscl . append /Groups/git GroupMembership git

The unique ID was chosen based on not having a conflict with automatically created users. Afterwards, create the user’s home directory at /Users/Shared/git, and change its owner and group to git:git.

You might want to use /bin/bash as the shell instead as long as you’re setting the whole system up, it makes things a bit simpler. Without it, you can change the active user (as root) to git with

sudo -u git -s

Which unfortunately doesn’t update the whole environment (like $HOME). If you have a proper shell, you can use -i instead of -s, which does update everything.

gitolite includes a nice installation setup in its README.txt. Just follow the (short) instructions there, then create a new repository for your project (we called ours “CIDemo”).

Jenkins is easier to do (with a caveat, see next paragraph), since there’s a fully-fledged installer for Mac OS X available. We just used that one, and it worked fine. Select using a separate jenkins user (it will be created automatically).

You need at least version 1.439 of Jenkins. Right now, this version is not released yet. Thus I had to fetch the version 1.437 from github, and cherry-pick the one fix that’s required, a811f406e785a8c8e6cb4b9246829dce9c7b97ae. Otherwise, restarting Jenkins removes the jobs due to a parsing error.

Keychain Fun

Since you want to sign your apps, you have to supply the signing keys and the provisioning profile. The profile you can just copy to ~jenkins/. For the script below, we use the file name “${HOME}/Development.mobileprovision“.

The certificate and private key are more complicated, since they have to be in a keychain. Since the jenkins user does not have UI support, you can’t just log in and use Keychain Access.

The solution I found was to launch Keychain Access from your regular user, setup the keychain there and then copy it to the jenkins user. Use “New Keychain” from the File menu (name it something like “jenkins”, use whatever password you want), then option-drag you developer certificate(s) to the new one. Then, delete the keychain (but do not remove it from the file system), and then in the shell, copy ~/Library/Keychains/jenkins.keychain to ~jenkins/Library/Keychains/login.keychain (create the directory on the destination, and also change the owner and group to jenkins). Don’t forget to set the keychain permissions to 600. Then, you can set this keychain as the default and remove the password (this is not possible from the UI):

security default-keychain -d user -s login.keychain
security set-keychain-password

The old password is the one you entered when you created it, just press return for the new one.

Jenkins Setup

In Jenkins’ web interface (port 8080 on the machine), add the following plugins:

  • Jenkins GIT plugin
  • Post Build Task

In the Jenkins configuration, in the “Git” section, change the git executable path to “/Applications/{your Xcode appname}.app/Contents/Developer/usr/bin/git“. In the “Git plugin” section, set up “Global Config user.name Value” and “Global Config user.email Value”. Also setup the SMTP setup for the email notifications there.

Also, you should enable security in the configuration, with whatever system you like. Otherwise, everyone who can reach this server via HTTP can execute arbitrary shell commands as the jenkins user, which is kinda bad.

Back on the command line, create a (standard) ssh key for the jenkins user (use sudo like explained above to switch to that user). Add its public key to your project in gitolite (with RW permissions).

You also have to accept the Xcode license agreement for all users on the system, which can be done on the command line:

sudo xcodebuild -license

Next, we created a new job in the Jenkins web interface with the “Build multi-configuration project” type. Name it distinctively.

In the “Source Code Management” section, select git. The repository URL is “git@localhost:{project name}.git” (see your gitolite config). In the Advanced section there, enter “origin” as the name. In “branches to build”, you can enter what fits your build workflow. We entered “master“.

In “Build Triggers”, activate “Poll SCM”. This doesn’t really poll, but it enables automatic building after a push.

In “Configuration Matrix”, add two dimensions:

  • name=XcodeApplication, values=Xcode Xcode44-DP4 Xcode45-DP2
  • name=SDKName, values=macosx iphone iphonesimulator

Of course, you have to alter these to fit your setup (which Xcode apps and which platforms you want to build-test). The values for the SDKs are defined by Apple, so you usually don’t want to change them.

Push “Add build step”, “Execute shell”. Insert the following script:

#!/bin/bash
export DEVELOPER_DIR=/Applications/${XcodeApplication}.app/Contents/Developer
security unlock-keychain -p "" login.keychain
if [ "${SDKName}" == "macosx" ]; then
  SchemeName="CIDemo Mac"
else
  SchemeName="CIDemo iOS"
fi
xcodebuild -workspace CIDemo.xcworkspace \
  -scheme "${SchemeName}" -sdk "${SDKName}" \
  -configuration Release \
  DSTROOT=$WORKSPACE/build.dst \
  OBJROOT=$WORKSPACE/build.obj \
  SYMROOT=$WORKSPACE/build.sym \
  SHARED_PRECOMPS_DIR=$WORKSPACE/build.pch

 

Next is adding the automatic tagging of succeeding builds (or all builds, whatever you think is reasonable).

In “Post-build Actions”, add Git Publisher. There, activate “Push Only If Build Succeeds” (if you want that), in Tags, use “Tag to push:” = “jenkins_$BUILD_NUMBER“. Create new tag active, “Target remote name” = “origin“.

Next, add “E-mail Notification”, and configure it to your liking.

The post build task to add is a bit more complicated (you might want to leave this for now and add it later). It involves uploading the result to HockeyApp (but only the iOS version built on Xcode 4.3 here):

#!/bin/bash
# by Andreas Monitzer (@anlumo1) and Pepi Zawodsky (@MacLemon)
#
# This script published under WTF license
# http://en.wikipedia.org/wiki/WTFPL
# Improvements to this script are welcome though.
if [ "${SDKName}" == "iphoneos" -a "${XcodeApplication}" == "Xcode" ]; then
 	TAG=$(git describe --exact-match --match "release_*" 2> /dev/null)
  	if [ "$?" -eq "0" ]; then
 		# publish the app
 		# get tag annotation to use as the release notes
 		NOTES=$(git tag -l -n9999 $TAG|sed -e "s/^$TAG//;s/^ *//")
  		API_TOKEN="<HockeyApp API token>"
 		SIGNING_IDENTITY="iPhone Developer"
 		PROVISIONING_PROFILE="${HOME}/Development.mobileprovision"
 		PRODUCT_NAME="CIDemo"
  		DSYM="$WORKSPACE/build.sym/Release-iphoneos/${PRODUCT_NAME}.app.dSYM"
 		APP="$WORKSPACE/build.sym/Release-iphoneos/${PRODUCT_NAME}.app"
  		HOCKEY="$WORKSPACE/build.HockeyApp/"
 		mkdir "$HOCKEY" 2> /dev/null
  		echo "Creating .ipa for ${PRODUCT_NAME}..."
  		/bin/rm -f "${HOCKEY}${PRODUCT_NAME}.ipa"
 		/usr/bin/xcrun -sdk iphoneos PackageApplication \
 					   -v "${APP}" \
 					   -o "${HOCKEY}${PRODUCT_NAME}.ipa" \
 					   --sign "${SIGNING_IDENTITY}" \
 					   --embed "${PROVISIONING_PROFILE}" > ${HOCKEY}${PRODUCT_NAME}.log
  		echo "Zipping .dSYM for ${PRODUCT_NAME}..."
  		/bin/rm -f "${HOCKEY}${PRODUCT_NAME}.dSYM.zip"
 		/usr/bin/zip -r "${HOCKEY}${PRODUCT_NAME}.dSYM.zip" "${DSYM}"
  		echo "Uploading to HockeyApp..."
  		/usr/bin/curl "https://rink.hockeyapp.net/api/2/apps/<HockeyApp app id>/app_versions" \
 					  -F "status=2" \
 					  -F "notify=0" \
 					  -F ipa=@"${HOCKEY}${PRODUCT_NAME}.ipa" \
 					  -F dsym=@"${HOCKEY}${PRODUCT_NAME}.dSYM.zip" \
 					  -F notes="${NOTES}" \
 					  -H "X-HockeyAppToken: ${API_TOKEN}"
  		echo "Uploaded to HockeyApp" 	fi fi

It’s important that you actually understand what this script is doing, so you can alter it to fit your needs!

Git Setup

What’s missing here is informing Jenkins that a new commit was pushed. For this, you have to create a new file in “/Users/Shared/git/repositories/<project name>.git/hooks” called post-receive. This is the content:

#!/bin/bash
curl http://localhost:8080/git/notifyCommit?url=git@localhost:<repository name>.git

As you can see, you could easily change this to inform another server about the new push (this is a common issue with automatic build actions which have to run a script locally to trigger something). Set this script to 755 permissions.

Updating the Bundle Version

In every OS X and iOS project you need to access certain information from the Info.plist present in every .app bundle. Some of that information is automatically created by Xcode, other stuff you need to provide manually, like entering a version number for your app in Xcode. Every time you have to manually add fields or increase a version number this is not only error prone but simply annoying.

Screenshot of Xcode Target settings showing the Version and Build fields with 1.0.

Screenshot of Xcode Target settings showing the Version and Build fields with 1.0.

This is why this script takes care to add all the useful information like a build revision and git short hash to the Info.plist automatically. This not only works for automated build systems like Jenkins but also for local builds.

Version numbers are taken from a git tag in the form of release___. You can add a release version tag by running this command: git tag -a "release_2_4_8". You’ll be prompted to enter release notes for this tag and if you’re using Hockey they will be used there automatically as well. Nifty! Don’t forget to push your new tag to the build server by issuing git push --tags origin.

This script needs to be added as a “run script” build phase in Xcode.

#!/bin/bash
# by Andreas Monitzer (@anlumo1) and Pepi Zawodsky (@MacLemon)
#
# This script published under WTF license
# http://en.wikipedia.org/wiki/WTFPL
# Improvements to this script are welcome though.

# Augments the Info.plist with a lot of nice stuff.
# It's suggested to call this script from a "run script" build phase, not copy the script's contents there.

# All Values are available from Objective-C like this:
# Example
# NSDictionary *infoPList = [[NSBundle mainBundle] infoDictionary];
# NSLog(@"CFBundleShortVersionString: %@", [infoPList objectForKey:@"CFBundleShortVersionString"]);

 echo "Checking for file ${PROJECT_DIR}/.git"

# When using git SCM
if [ -e "${PROJECT_DIR}/../.git" ]
then

    # Get a svn-like revision number that keeps increasing with every commit.
    REV=$(git log --pretty=format:'' | wc -l | sed 's/[ \t]//g')
    echo "git rev: $REV"

    # Getting the current branch
    GITBRANCH=$(git branch | grep "*" | sed -e 's/^* //')
    echo "Git Branch: $GITBRANCH"
    # To prevent local builds from displaying incorrent branches we need to delete and readd.
    /usr/libexec/PlistBuddy -c "Delete :GitBranch string" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
    /usr/libexec/PlistBuddy -c "Add :GitBranch string $GITBRANCH" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"

    # full build hash
    GITHASH=$(git rev-parse HEAD)
    echo "Git Hash: $GITHASH"
    /usr/libexec/PlistBuddy -c "Delete :GitHash string" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
    /usr/libexec/PlistBuddy -c "Add :GitHash string $GITHASH" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"

    # commonly used short hash
    GITHASHSHORT=$(git rev-parse --short HEAD)
    echo "Git Hash Short: $GITHASHSHORT"
    /usr/libexec/PlistBuddy -c "Delete :GitShortHash string" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
    /usr/libexec/PlistBuddy -c "Add :GitShortHash string $GITHASHSHORT" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"

    # parsing tags to build the CFBundleVersion in the form of <MAJOR>.<MINOR>.<PATCH>.<REV>
    # Parts of the version number that are missing are substituted with zeros.
    NEAREST=$(git describe --abbrev=0 --match "release_[0-9]*")
    echo "Nearest release Tag: \"$NEAREST\""

    MAJOR="0"
    MINOR="0"
    PATCH="0"
    if [ "$NEAREST" == "" ]
    then
        echo "No release tag found!"
    else
        MAJOR=$(echo $NEAREST | cut -d "_" -f 2)
        if [ $MAJOR == "" ]
        then
            MAJOR="0"
        else
            MINOR=$(echo $NEAREST | cut -d "_" -f 3)
            if [ $MINOR == "" ]
            then
                MINOR="0"
            else
                PATCH=$(echo $NEAREST | cut -d "_" -f 4)
                if [ $PATCH == "" ]
                then
                    PATCH="0"
                fi
            fi
        fi
    fi

    echo "Version String: $MAJOR.$MINOR.$PATCH.$REV"

    # Setting in the Info.plist file
     /usr/libexec/PlistBuddy -c "Set :CFBundleVersion $MAJOR.$MINOR.$PATCH.$REV" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
     /usr/libexec/PlistBuddy -c "Set :CFBundleShortVersionString $MAJOR.$MINOR.$PATCH.$REV ($GITHASHSHORT)" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"

    # Setting the same version number in the .dSYM file needed for symbolication in the Hockeyapp.net webinterface
     /usr/libexec/PlistBuddy -c "Set :CFBundleVersion $MAJOR.$MINOR.$PATCH.$REV" "${DWARF_DSYM_FOLDER_PATH}/${DWARF_DSYM_FILE_NAME}/Contents/Info.plist"
     /usr/libexec/PlistBuddy -c "Set :CFBundleShortVersionString $MAJOR.$MINOR.$PATCH.$REV ($GITHASHSHORT)" "${DWARF_DSYM_FOLDER_PATH}/${DWARF_DSYM_FILE_NAME}/Contents/Info.plist"

elif [ -e "${PROJECT_DIR}/../.svn" ]
then
    # Support for SVN would be happy if you'd actually implement it!
    REV=$(svnversion -nc "${PROJECT_DIR}" | sed -e 's/^[^:]*://;s/[A-Za-z]//')
    echo -n "SVN rev: $REV"
fi

if [ "$BUILD_NUMBER" == "" ]
then
    /usr/libexec/PlistBuddy -c "Delete :JenkinsBuild string" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
    /usr/libexec/PlistBuddy -c "Add :JenkinsBuild string $BUILD_NUMBER" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
else
    /usr/libexec/PlistBuddy -c "Delete :JenkinsBuild string" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
    /usr/libexec/PlistBuddy -c "Add :JenkinsBuild string Local build (not via Jenkins)" "${TARGET_BUILD_DIR}/${INFOPLIST_PATH}"
fi

#EOF

Wrapping Up

So, that’s as far as we’ve come. If you have any comments/additions, feel free to send them to me! I hope that the information we’ve covered here can help you construct a continuous integration solution that fits right into your development process.

There’s a lot of ways you can go from here. Maybe uploading to iTunes Connect is also something to think about, although I’d personally like that process to be a bit manual, in order to remove unintentional results.

(Note that the code covered here is under the WTFPL, but the text is not.)

«