Update 2: Continuous Integration & Deployment With Sitecore

A long long time ago in a galaxy far far away…. Wait that’s not right. A long long time ago in a blog post far away… Yes that’s better :). In truth this update has been a long time coming. To set expectations this update is about introducing more automation and refining the process. I think my original post was a great kick start and I’ve seen many people site it as a source of information (thank you).

Recently I had the chance to introduce a continuous integration & delivery process for a client. I wanted to introduce more automation and perhaps simplify the process better while still staying true to my original goals :). For those who may not have read the original post here’s a quick recap of those goals.

  • Create an automated pipeline from the point of committing changes into version control to having them deployed on a target environment.
  • Use as few applications or pieces as software as possible.
  • Use as little scripting as possible.

Without further a do I present a more simple and automated CI/CD process. For each step I’ve added a bit of information where things have changed from my original post.

Software

  • Git – version control. You can use any version control you like but for this process it should support tagging of commits.
  • TeamCity (9.X) – build server.
  • Octopus Deploy (2.6.5) – deployment server.
  • Team Development for Sitecore – uses Sitecore’s serialisation to create files on disk that can then be put into version control. Has an excellent plugin for Visual Studio for synchronising what is in version control with a Sitecore instance.
  • Fortis Collections – Toolcore
  • ProGet – NuGet package feed software.

What’s changed?

  • Sitecore Courier – used to Sitecore update packages based on serialised files from Sitecore. I’ve switched to using TDS for creating the packages rather than using Sitecore Courier to create difference packages.

Version Control – Git

Set up your version control however you would like to use it.

What’s changed?

Previously I’d used a tag which is moved by TeamCity to signify the last commit that’s been deployed. This is no longer required as the package deployment process is simplified.

Visual Studio Solution & Source

Configuration

Create a Development (can be named whatever you like) configuration for your Visual Studio solution. This will be the configuration used by TeamCity to build against. In my projects I use configuration transforms to output different configuration. Further along in the process we’ll be adding Octopus Deploy variables to our Development configuration transform files. This will allow us to have all of our different environment information in Octopus Deploy and abstracted away from our source code.

Team Development for Sitecore (TDS)

For this part I’m assuming you have a single TDS project with all the items in that you want to deploy.

  1. Set up TDS to deploy the site files into a folder of your choice. In my case I always deploy to a folder called _Deploy which is in the root of my Git repository. This folder should be somewhere inside your Git repository but you shouldn’t commit these files.
    TDS_build_settings
  2. Turn on generating an update package during build and make sure only the items are included (no site files). You’ll also need to tell TDS where to find the Sitecore binaries as it uses them for creating packages.
    TDS_update_package_settings

If you have more than one TDS project that you want deployed then you can do one of two things.

  1. Use TDS’ Mult-project Properties to combine several projects and produce a single update package to be deployed.
  2. Let each TDS project create its own update package and deploy them separately one at a time.

NuGet

Eventually we’re going to create two NuGet packages which will be deployed by Octopus Deploy. For this we’ll need to create two nuspec files which defines what we want in each NuGet package.

  1. Create a folder in the root of your Git repository called BuildServer. You can call and have this where you like but be sure to replace the location you use through this post.
  2. Create an application.nuspec file inside the BuildServer folder. If you’re unfamiliar with creating nuspec files you can read about them here. Have the nuspec file include all the files created by TDS in the _Deploy folder.
    <?xml version="1.0" encoding="utf-8"?>
    <package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
        <metadata>
            <id>MySite_Application</id>
            <authors>Partner</authors>
    	<version>1.0.0.0</version>
            <requireLicenseAcceptance>false</requireLicenseAcceptance>
            <description>Application package</description>
        </metadata>
    	<files>
    		<file src="..\_Deploy\**\*.*" target="" /> 
    	</files>
    </package>
    
  3. Create a sitecore_items.nuspec file inside the BuildServer folder. Have the nuspec file include all .update files, created by TDS, that you want deployed. If you’re using multiple projects with each one creating its own .update file then you’ll need to specify each one in the nuspec file.
    <?xml version="1.0" encoding="utf-8"?>
    <package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
        <metadata>
            <id>MySite_SitecoreUpdatePackages</id>
            <authors>Partner</authors>
    		<version>1.0.0.0</version>
            <requireLicenseAcceptance>false</requireLicenseAcceptance>
            <description>Sitecore update packages</description>
        </metadata>
        <files>
            <file src="..\Source\MySite.Sitecore.TDS.Master\bin\Package_Development\*.update" target="" />
        </files>
    </package>
    

Build – TeamCity/Octopus Deploy

Build Configuration Settings

The following assumes you’ve already created a build configuration and have a decent knowledge of using TeamCity.

Version Control Settings (VCS) & Triggers

  1. Attach a VCS root for your branch that is to be deployed.
  2. Under Checkout Options ensure Clean build is checked.
  3. Create a VCS Trigger in the Triggers section for your VCS root.

Build Features

  1. Add a Build Feature.
  2. Select VCS Labelling.
  3. Select the VCS root you created previously.
  4. [Optional] Change the Labelling pattern “-” (hyphen) to a “/” (forward slash). I do this because it means all the tags for build numbers will show up in a folder in Git.

    vcs_labelling_pattern_git

    Tags organised in a folder, viewed in SourceTree

Parameters

Add a system property to tell TeamCity which configuration to use when it runs MSBuild.
TeamCity_system_properties

Build Steps

  1. Add an MSBuild runner step and point it to your solution file. When TeamCity runs this step it will cause TDS to also run its MSBuild steps. This will in turn create our application and update package files. You do need to TDS installed on the server where the TeamCity agent is running.
  2. Add a NuGet Pack runner step and specify the nuspec files created previously. Also make sure the packages aren’t published to the TeamCity NuGet feed as we’re not using it. Finally make sure to include the build number in the package version otherwise you won’t be able to upload it to the NuGet feed.
    TeamCity_NuGet_pack_step
  3. Add a NuGet Publish runner step and include both NuGet packages created. As both our packages are prefixed with MySite_ we can use a wildcard to pick up all the packages. You’ll also need to enter the details for your NuGet feed. In my case I’ve actually used a configuration parameter for both the API Key and Package Source.
    TeamCity_NuGet_publish_step
  4. Add an Octopus Deploy: Create release runner step. If you haven’t set up anything in Octopus Deploy you’ll want to skip to the Octopus Deploy section in this post. Otherwise enter the relevant details for your release. In my case below I’ve also manually increased the deployment timeout.
    TeamCity_Octopus_Deploy_create_release_step

What’s changed?

There’s several big differences between what was being done before and now.

  1. No more use of Sitecore Courier and creating difference packages. I just deploy the entire update package every time. The reason for this is help us utilise promoting of releases in Octopus Deploy. The problem came about by the fact that you may have more that one environment in Octopus Deploy and they could all be running different build numbers. For example if we had a Development and QA environment where the Development is automatically deployed to and QA triggered manually irregularly. Development might be on build 501 but QA on build 489 and wouldn’t have a difference package between builds 489 and 501. You could promote every build between 489 and 501 to eventually get there but that wouldn’t fun! By having the update packages contain everything we can promote from any older build to a newer build.
  2. As I’ve dropped the need for creating difference packages I was able to also simplify the tagging process. We no longer have to check out on the agent and can just use the standard VCS Labelling build feature. It also means we can drop the second VCS root.

Octopus Deploy

Similar to the TeamCity section I’m assuming you’ve already got some knowledge of using Octopus Deploy already. I’ve also assumed you’ve hooked up your Octopus Deploy to the private ProGet NuGet feed. Also note that a newer version of Octopus Deploy is out but I believe this should all the principles will be the same still.

Process

You’ll see a lot of Octopus Deploy variables used throughout the process. Most of them should be self explanatory. I’ve also purposefully not mentioned anything about Machine roles and Environments because these can be quite custom to your own infrastructure. The one thing I do is install an Octopus Tentacle on the same machine that runs the Octopus Server. This is so that I can run executables on the Octopus Server.

  1. Create a PowerShell script step to clean-up database backup files that are created in steps further along. I only keep the previous release’s backups and the release we’re creating.
    remove-item $DatabaseBackupFolder\Archive\*.bak
    move-item $DatabaseBackupFolder\*.bak $DatabaseBackupFolder\Archive
    
  2. Create an SQL – Backup Database step to backup the Core database. This is a custom template which you can grab from here.
  3. Create an SQL – Backup Database step to backup the Master database.
  4. Create a Deploy a NuGet Package step to deploy the Sitecore Package Installer Service. I actually package up the files into a NuGet package and host them on the private ProGet NuGet feed. The reason for doing this is it allows us to update the service and not worry about manually updating the service on every Sitecore instance.
  5. Create a Deploy a NuGet Package step to deploy the Sitecore Update Package.
  6. Create a PowerShell script step to install the Sitecore Update Package. This step uses the executable provided by Toolcore – Update to install the Sitecore update package that we’ve deployed to the website. I actually have the executable on the server running Octopus Deploy (not the tentacle), hence the reason for installing a tentacle on the same server. I normally create a Machine Role called OctopusServer which allows me to run the executable.
    &"$ToolcorePath$ToolcoreUpdateExecutable" -p "$WebsiteDirectory$SitecoreUpdatePackageFolder$SitecoreUpdatePackageName" -u "$WebsiteURL"
    
  7. Create a Deploy a NuGet Package step to deploy the application files.
  8. Create a PowerShell script step to trigger a publish on the Sitecore instance. This step uses the executable provided by Toolcore – Publishing to trigger the publish. Same as the update package installation step I also have the executable on the Octopus Deploy server.
    &"$ToolcorePath$ToolcorePublishingExecutable" -s "master" -u "$WebsiteURL"
    

Promoting

As we’re now installing full Sitecore update packages we’re able to promote releases created in Octopus Deploy. For example you could have a Development environment which is automatically deployed to and then promote releases to QA and then perhaps even production.

Where next?

Improvements

  • Clean instance every build – one of the issues at present is old files aren’t cleaned up. The best approach would be to create a clean set of Sitecore files and then deploy to that. My thoughts at present would be to copy the databases, deploy a fresh Sitecore instance with all the required module files and the deploy the application on top of that. This way you would avoid any need of figuring out what needs to be removed.
  • Automate re-indexing – a simple addition like publishing but this time if we want to trigger a rebuild of the indexes.

Alternatives?

Some other worthy options you may want to consider when determining what software is right for your process.

  • Unicorn by Kam Figy – an alternative way of serialising items from Sitecore into your Git repository.
  • Sitecore Ship by Kevin Obee – an alternative way of installing Sitecore update packages via HTTP requests. Also has some other features to it.

Wrap up!

I hope this update gives some more insight into the world of continuous integration and delivery. As always this is an ever evolving area and you should always continue to look for improvements!

Credits

I’d also like to thank Kamruz Jaman for several discussion regarding this topic :).

Jason Bert