19 Eylül 2016 Pazartesi

Continuous Integration and Delivery with Unity3D

This is the seventh post about Unity3D so far, if you wonder what I have said before, please take a look:

All jobs should be green 
We are already using Jenkins as our Continuous Integration (CI) platform. CI is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

We are using Jenkins as CI platform at Peak Games including server side, CRM, mobile, web... So obviously we decided to use Jenkins for Unity projects also. We have seven Jenkins jobs for a single Unity game. Let me tell you about what these jobs are doing.


Common Gaming Libraries


Since we are planning to use Unity for at least 2-3 board/card games, we started creating common libraries that will be used by all Unity games. Before Unity, we have done similar work for iOS and Android native. That is why we have the knowledge to decide what functions would be common and re-usable across all games.

Right now we have 18 common libraries written in C# and 2 native plugins written in Java and Objective-C. To be able to integrate these libraries to actual game code we didn't use any dependency management framework, we just use two different custom approaches. When we are building these integrations we couldn't find a well known, mature enough dependency management framework that is why we created our own custom solutions.

For native plugins (Peak CRM API Client, Native IAP Client). We have created one project for each. Each of them has similar approach inside the project. Actually each project includes four separate projects inside. One for iOS in Objective-C, one for Android in Java, one for creating a dll in C#, one for reference sample Unity application. And of course one build script to build all native codes and create a dll file. Creating native plugins for iOS and Android is quite different, for more information please refer to these documentations (iOS and Android).

So we have two Jenkins jobs to create these plugins and generate available links to download. Once these plugins developed, they will not change a lot. So to integrate them, there is a manual operation. Actually just copy and paste after downloading from given URL.

For C# common libraries we have created one single Unity project called GamingLibrary. Unlike native plugins we don't want to create one or more dll files. Because these common libraries might change (with additions, changes, removals) a lot during the game development. We want to see the actual code of these libraries while we are developing.

So we tried to generate Unity Package from this project and import it into actual game project. It was working like a charm in Editor. You could import any package even your code has some compile issues. It would not compile because there were dependencies to our gaming lib package. But after the import it will compile and work successfully.

We were happy till to see that importing in batch mode (via command line) has a very subtle difference according to Editor. In batch mode, Unity needs to compile the project to be able to import any package before the import action. Which would not work for us because as I said our game code is not compiling before the gaming lib package import. We try to find a work around for this, but couldn't find.

Then we write a script file to copy gaming libraries into actual game code base. Basically we just use rsync to make synchronization.  So we have one Jenkins job to compile and run unit tests on each commit to gaming library.


Game Code Base


So the rest of four Jenkins jobs are there for each Unity game. I will use Gin Rummy Plus as an example.

First one is there for actual CI purposes, it is compiling and running unit tests for each commit to main branch. Before compiling, it is using the above script file to integrate gaming libraries into code base and then using Unity's batch mode mechanism to compile it. For Gin Rummy Plus it has been 2187 run so far and it takes 40 seconds one individual run.

Second one is for running integration tests, it is working twice a day and manually when required. I have already mentioned about integration tests in more detail. And for Gin Rummy Plus it has been 462 run so far and it takes 18 minutes one individual run.

Third and forth ones are there for creating binary files for iOS and Android. Actually each of them is creating two binaries one is to send HockeyApp for testing purposes, one is to send App Stores for publishing. The jobs are not just creating binaries but also sending them to HockeyApp and App Stores. These binaries have some differences, for instance the ones for HockeyApp includes some extra debug features like cheat buttons to be able to test some scenarios, or some extra information to be able to understand a/b test groups for instance.

We are running these jobs manually, when we think that a new version of the game needs to be published. After job uploads the binaries it is informing our product managers and testers automatically. After that it is up to them to publish the binary or open tickets about the issues they have discovered in this version.

To differentiate the builds inside the code base we are using Scripting Define Symbols. I mean custom compilation flags (see Unity's platform dependent compilation page for details). We have two extra flags (DEBUG and ADHOC) a long with build-in PRODUCTION flag. DEBUG is there for development purposes it has more logs, more extra functions. ADHOC is there for testing purposes, and PRODUCTION is for stores.

Let me give more details about these binary build jobs.


Binary Builds


For iOS, when you build a Unity project it creates an intermediate Xcode project, from that project you can create the actual binary ipa file (you have to run it on MacOS system). And this job is doing it twice in a single run, to create two binaries as I mentioned above. That is why iOS jobs is taking 17 minutes on average. 

After creation of intermediate Xcode project, most of the cases you have to make configuration on this intermediate project to be able configure some native iOS settings like compile flags, like info.plist updates, like url schemes updates. So we are using Unity's post process build attribute frequently to make this custom configuration programatically.

To create the binary from the Xcode project, and upload to HockeyApp and App Store (TestFlight) we are using our custom iOS Build Library which is already being used for our native iOS games. I will not give any details of native builds and all confusing provisioning profile configurations. That is another post's topic.

The last binary that we have sent to App Store is 56 MB. It is almost two times bigger than our native iOS games. It is mainly because of splash screen. Yes, you read it correctly, just because of splash screen your binary is increased at least 10 MB. I hope there will be a solution in the next versions of Unity.

For Android, it is much more easier process. Because Unity can create binary  apk file without creating an intermediate project. So it takes way too quick to create two binaries according to iOS build. It takes around 7 minutes to create two binaries and upload one to HockeyApp and other to Play Store. We are using Jenkins plugins to make these uploads.

Unlike iOS builds, to make the native configurations we don't need to use post process build attribute. We just use the Manifest files to make the configurations.

The last binary that we have sent Play Store is 36 MB. It is also higher than our native Android games, but not like iOS. It is just 5-6 MB bigger. I can give more information about how to decrease the size of binaries on iOS and Android in another post (may be).

So as a conclusion our CI system for Unity games is consist of Jenkins, Unity's command line tools and a little bit of shell scripting, that is all.

We are very close to end of this Unity3D series, just one more :) 

Yorum Gönder