Continuous integration open discussion

Convened by @MissDotNet

##Benefits of CI##

• Confidence (I sleep better at night)
• "I didn't buy into at first (lots of adoption of new paradigms etc.) - really painful at first, but all focused on working more efficiently. Shipping well formed, testable code."
• With Team City, automatically capture unit test stats, integration api tests, verifications - all the data you can use to feed up to management - Transparency (this software passes the tests, and it passed yesterday, and last week, etc.)
• Weird to write code that just does one thing (it took time to get into TDD - but now it's fun and fast).
• CI admin wasn't sold on TDD - took the process of fixing a bug to make him get the benefits of TDD for isolating issues.
• Ultimate goal as a CI admin is to make everything automatic so that I have confidence and can push at midnight.

Some samples from people's CI setup:

Most involved some degree of Unit testing and, in at least one instance, pushing to production WAS the acceptance test. Similarly a there were a few people who were on the way to total automation, but there were still lots of manual steps - particularly around managing the database deployment. Using Cruise-Control in conjunction with Team City feels painful (Automation guy)

Another example setup involved using Mercurial, Unit tests, and Selenium for integration tests, with the results automatically pushed to production 5-6 times a day. One of the nice features of TeamCity is that, as it pushes to production, it checks how many exceptions are thrown and, if that number crosses a given threshold, TC will do an automated rollback.

One of the problems with automatically pushing to production is that you have to deal with various transformations and environmental variables, which are painful and hard to handle well without a himan in the loop. Particularly with a service-oriented architecture, there are lots of fiddly details, to the extent that several people specifically include do-not-copy instructions in their deployment PowerShell.

Evidently, several people are using Octopus, which plugs into Team City (and costs $800) - on the other hand, Jenkins is free!

Cruise Control + Team City (using NuGet packages), with integrations tests in Microsoft Test Manager, and using Octopus to deploy to API servers (no UI) - works for 16 KanBan teams. Deployment consists of staged roll-outs across web servers using a router switch (4 servers - deploy + test - then roll the next 4). Use IOC containers to control which users can be routed where (essentially an if-then block), and push code so that it's ready, but it's not live / accepting users.

If you're doing CD without ability to separate release and deployment for features, you're gonna have a bad time. People commit bad, untested code, and code can be pushed to a sub-set of servers, ready to users to pass through when it's ready to test.

What's it gonna take for .NET people to get real CI (with toggles etc.) built in? CI in Ruby is Rake (standardised) - but every single .NET project does it differently.

##How involved was your team / org in the process? Took about 3 months - total rebuild of process to move away from quarterly releases. Got to the point where one team might be doing integration testing with something that's in prod, but the rest of the teams are working on code - lots of flow and velocity.

Big hard things are getting the culture right - getting people so that they don't commit code that isn't behind a switch (if it changes something, it's gotta go behind a switch). Switches operate down to schema change / sproc levels (additions are permitted). Databases are hard to manage - have filters that switch out / change schemas. If it violates SQL Lint, you'll need to get a DBA to sign off on that. Result is lots of child tables or related tables - it doesn't hurt per se, just looks a bit messy.

When you cross service boundaries, you need to be able to serialise data and carry it with you - switches can change code in multiple locations to allow you to control features (which will typically touch multiple locations). On a related note, Team City has a nice code-coverage feature.

Actual production push is triggered manually, but that's just for the final sign-off.

##Tools and Thoughts Word doc / wiki / dev image which sets up a fresh dev box - "Morning Coffee" (just backup resharper settings to Azure!)

• Packer / Vagrant / etc.
• immutable servers (every deployment breaks and rebuilds your server, deletes all the keys etc. Once something is in production, you cannot touch it or reach it - it's totally contained - break off the key in the lock, means that the server is totally contained).
• It keeps people from putting custom crap in your build.
• If you're dumping appropriate log data etc., why should you configure it? Just redeploy.

Dealing with MS updates (Morning Coffee)

• VM that sucks up updates etc. and is the place you clone from.
• Nothing left to install once you just copy the latest VM - except random custom stuff.

Azure Deployment

• Management portal for on-premise infrastructure, lets you provision things for on-premise.
• OS update is just a 1-line attribute change.
• PaaS (plenty of weaknesses when it comes to infrastructure, but great for just getting applications out into the world and tested)
• Competitive pricing (no charge for data upload, and a very nice environment for develops to ship applications to, test in, and deliver business value really fast).

TFS

• Can pull from a whole bunch of sources (e.g. BitBucket, GitHub, Azure, whatever) into Team Foundation Service.
• Web Deploy option - just runs a web deploy script after it builds and tests.
• Don't want something fiddle or complex - want something out of the box. Database and schema is still super hard.
• Worker roles are a well-oiled machine.
• Ordering build scripts? To do that in TFS involves something like Windows Workflow Foundation -> shell out to a batch file.
• Do you write a build script, or just TFS build steps; hard to version control (use Rake for .NET)

Octopus Deploy

• Free license
• Good at handling application code, but less good at handling databases.
• Worth also investigating Roundhouse / Deployment Manager

##Testing practices within CI

• Personal branching
• Feature branching
• Version branching

###Using GitHub

• GitHub enterprise; pull requests in that space, across teams is an interesting exercise.
• Integration between GitHub and Team City is pretty good (so long as you're using team repos vs. personal repos). Feature built in a branch, which Team City sees, tests, and green-lights.
• Using Azure for our build agent (and a sleep trigger, so builds don't occur more than once every 15 minutes)
• Comparison of GitHub vs. etc. (Github is feature-rich, but PRICY) Enterprise GitHub is distributed as a VM appliance which is run in local virtual environment and updated remotely by GH. 'Spensive.
• Check out GitLab (Gitorious)
• Multiple gits to let people push from repo to repo (for staging environments)?
• 16 teams pushing to various branches (all KanBan), and staying on those branches for various amounts of time before they're ready to merge to master.