From Git Flow to Trunk Based Development

I have worked with Git Flow for a few years and it helped me a lot to structure my Git branches. However, I have encountered some problems with Git Flow, most of them coming from long-living branches. The solution to solve those problems is trunk based development! It is an amazingly simple technique which is also the base for effective continuous delivery. In this article I tell you how I have made the transition from Git Flow to trunk based development with my iOS development team at HolidayCheck. You will learn what is the most important step to get there and what benefits you will get from trunk based development, so keep on reading!

The Problems with Git Flow
Merge Conflicts

Merge ConflictsMerge conflicts are very common when you work with Git Flow. The reason is simple: If you have multiple parallel feature branches which live for a long time then it is very likely that the same part of the code base is changed in two different branches. Merge conflicts are not only frustrating for the developer who has to solve them manually. They also increase the risk of breaking something in your code because it is easy to make a mistake when you have to decide which code to use and which not. Even if you do everything right when you merge one branch into another it can happen that the combination of two features breaks your code.

Feature Separation

You cannot test the combination of two features until they are merged into one branch. When you develop features in separate branches for multiple days or even weeks then problems which arise from the interaction of two features become visible late. When you react late on those problems you will probably have to change more parts of the code you have written for the new features. That means you have wasted more time creating code you don’t need any more.

Many different feature branches can also be confusing for the manual testers of your software. They always have to know in which staging environment the new feature can be found. Different staging environments normally also mean different build jobs, different screens for monitoring the jobs and so on.

The Unpredictable Release

There is another problem with Git Flow which is a result of the two points above: It is impossible to know how much time you will need for a release if the feature branches are not merged yet. You cannot know it because you don’t know which merge conflicts will occur and you also don’t know how the different new features will influence each other. Not being able to release at any time makes you inflexible and unreliable.

The Solution: Trunk Based Development

Great news! There is a solution for all the problems above and it is called trunk based development. You only have one main branch, the trunk (also called master or mainline). There is no develop branch anymore. Also no long-living feature branches! All your commits are merged into the trunk as soon as possible, at least once a day. By merging to the trunk so quickly, merge conflicts become very rare. Using short-living branches is one of the 4 Simple Tricks to Avoid Merge Conflicts. Even if you get a merge conflict it will probably be easy to solve because the set of changes since the last merge to the trunk is not so big. Interferences between different features become visible immediately and can be tested while the features are still in progress.

Trunk based development will also encourage your team to think and work in small steps which lead to small commits that can be merged to the main branch quickly. Usually small steps reduce the number of bugs and help to create a modular design.

The big question is: How can you avoid to get an unstable master branch if you push your code into it every day, even if a feature is not finished, yet? Read on to get the answer…

How to Get from Git Flow to Trunk Based Development
Feature Toggles

When I introduced trunk based development to my team at HolidayCheck there was one first step which was absolutely necessary in order to be able to commit to the trunk quickly! Before we even started to change our branching structure we had to make sure that we can make very short-living branches that can be merged into the develop branch as quickly as possible. The solution is rather simple! We started to use feature toggles – little switches in the source code that decide whether a feature is active or not.

if ( FeatureManager.isFeatureEnabled("NewLoginForm") )
{
    openNewLoginForm()
}
else
{
    openOldLoginForm()
}

Feature ToggleAs long as a feature is not ready to be released, it is disabled. That allows us to already push it into the develop branch without breaking anything. Developers and manual testers can enable every feature in some settings which are hidden to the normal users. The develop branch is always ready to be released because the unfinished features are switched off. They will be shipped to the user but they will not be visible. As soon as a feature is finished it is turned on and available with the next release.

When we had those feature toggles in our code we realized that they are not only useful while the features are being developed! It would be great to keep the toggles in the code even when the features are finished. If we had a possibility to control the toggles remotely for all our users then we would be able to deactivate a feature quickly whenever we see that it has a bad impact on our conversion rates or other key numbers. Also if any bugs occur or the traffic load for our servers becomes too high, we are always able to switch off the feature immediately. This is especially important as we have to wait multiple days until Apple reviews and excepts a new release of our iOS app. Being able to disable features without a new release is a very powerful weapon!

Now, with all the feature toggles set up, there is another great thing we can do: A/B testing! Since every feature toggle can be controlled remotely we have the possibility to enable a feature only for a part of our user base and disable it for the others. Doing that we can see how well a feature really performs. We can test new features on small test groups and then decide whether we should enable it for everybody or remove it if we see a negative impact. We use Optimizely to control and evaluate our A/B tests but there are also other tools available.

One Branch to Rule Them All

Now that we are able to merge our feature branches into the develop branch quickly (because unfinished features are deactivated), we are able to release the develop branch at any time. The question is: Do we still need a develop and a master branch? The answer is: No. We can commit everything directly into the master instead of the develop branch. The master is always ready for production (don’t forget to run all your tests whenever something is pushed into the master ;-)). If we want to create a release version we can either do it directly from the master branch or create a release branch for that. The latest released commit is marked with a Git tag. So the second step after introducing feature toggles is to delete the develop branch! And here we are! This is already trunk based development!

Take Aways

Trunk based development has made my team much more flexible. We can release our master branch at any time and we have no big merge conflicts anymore. A master branch that is always ready to be released to production is the prerequisite for continuous delivery. The feature toggles are the prerequisite for A/B testing. They help us to find out what our users really want and make us more confident because we know that we can disable a feature at any time. The result is smaller risk and more innovation.

36 thoughts on “From Git Flow to Trunk Based Development

  • March 26, 2016 at 2:37 pm
    Permalink

    Hi Robert,

    Thanks for the nice article, really interesting thoughts! I’m a big fan of trunk based development, but the only fear I see in what you wrote is that the code toggling your feature could stay forever in your code base (and when it gets large, there will be hundrets of conditions, with the risk of having no overview what is enabled and what not). Sure, it’s a very cool way to disable something when you have problems, but I guess this is not the puropse of doing it? (Even is on iOS you are right, you can’t simply revert something within 2 mins ;-)).
    Did you already run in such problems?

    Reply
    • March 26, 2016 at 3:54 pm
      Permalink

      Hi Rob,
      thanks for your comment! You are right, feature toggles can create some confusion, especially if you have many of them. It’s important to have the discipline to remove a toggle once it is not needed anymore, e.g. when the feature is finished and can be released. In our team we remove the toggles as soon as we have a clear winner in the A/B test. Currently we have less than 20 toggles in our app but I know others that use much more toggles and it works very well for them.

      Best regards
      Robert

      Reply
  • April 2, 2016 at 10:52 am
    Permalink

    Very nice idea. Did you and your team also worked with big improvement of existing features while at the same time there are some temporary fixes on that feature?
    Next, I am trying to think how the switches can work when the new feature is based on changes in code of different files, like Frontend and controllers .

    Reply
    • April 2, 2016 at 1:54 pm
      Permalink

      Hi Bert,
      thanks for your comment! Yes, you can always release hotfixes for features you are currently working on, even if the big changes on that features are not completed. The new stuff should still be hidden by the feature toggles while the hotfix is visible for all users. I have to admit that it’s not always easy to disable parts of a feature but normally, if you have modular code, it is possible without too much effort.

      Regarding your second question: As long as it doesn’t effect the user you can always change the code in other parts of your application if it is necessary for your new feature. But if those changes in the other part would effect what becomes visible for the user then you should also hide them with a feature toggle. In this case you would use the same feature key as for your new feature.

      Reply
      • April 2, 2016 at 7:59 pm
        Permalink

        Hi Robert,
        Many thanks for quick and clear respons. Understood. Perhaps a total other question more related to the TBD approach in general is the fact I am a little scared about not having Pull Request anymore. I mean everybody is commiting to the trunk and you it seems there is no option to ask your peer to check before commit.
        I am not sure if this on topic, but perhaps you can point me to some info about the fact I am used to work with PR’s as a method to check a commit or a group of commits

        Reply
        • April 2, 2016 at 8:23 pm
          Permalink

          Of course you can still discuss code before you push it to the trunk. If you want to create pull requests, that’s no problem. TBD does not mean that you are not allowed to create branches! You can still create short-living branches and pull requests. You will probably have multiple pull requests for big features. The point is just that your code should get into the trunk as quickly as possible.

          Reply
  • Pingback: From Git Flow to Trunk Based Development | Git Workflow

  • Pingback: Code Reviews in Trunk Based Development - The Team Coder

  • December 2, 2016 at 5:23 am
    Permalink

    Great ideas, However, I have a few questions,
    1. How do you deal with applications that don’t have short time deployments cycles (i.e. 2 weeks Prd releases instead of everyday)?
    2. How do you handle sev1 issues that come up while you have already polluted your master branch with things that are not yet ready for prod? I know you wrote about feature toggling, but I see a lot of problems with huge apps that have many components with a bunch of dependencies.

    Reply
    • December 2, 2016 at 9:55 am
      Permalink

      Thanks for your feedback :)
      We release our iOS app every 2 or 3 weeks. When we press the release button on our build server it checks out the current master branch, builds the app and sends it to Apple.

      Regarding your second question: I think too many dependencies are always a problem. The more modular you build your code the easier it will be to hide parts with a feature toggle. However, sometimes there will be a little bit of refactoring necessary in order to use the toggle.

      Reply
  • Pingback: Code Reviews in Trunk Based Development

  • February 22, 2017 at 5:56 pm
    Permalink

    Hi Robert,

    nice article, although I don’t agree ;-)
    I am a huge fan of feature branches, that’s why I have some comments conerning the problems you mentioned.

    Merge conflicts:
    Before giving up feature branches I would try to slice big stories to prevent long living branches. Additionally the developers should regularly merge the develop branch into the feature branch.

    Feature Separation:
    In my opinion this is a feature of Git Flow and not a bug. I’m not affected by the develpment of other features or bugs that are checked into these branches.
    The point where I agree with you is, that the interaction of different features should be tested as soon as possible. But that problem is also tackled by regularly merging the develop branch into my feature branch. That way the feature I am working on will be tested together with the features that are already done. And the other features that are still in development, will later be tested together with the feature I had been workling on.

    The Unpredictable Release
    Let’s imagine you are working on a new feature. After some time you notice that the development is way more complicated than expected and it won’t be possible to finish the feature in time for the upcoming release. As you mentioned, a feature toggle could solve the problem, but that can be complicated if the new feature has more touch points in different parts of the code.
    In my experience it is easier to just not merge before the release, in comparison to hiding the partly developed feature.

    Cheers!
    André

    Reply
    • February 22, 2017 at 10:39 pm
      Permalink

      Hi André,
      thanks for your feedback :)

      I understand your points and I agree with you that GitFlow has it’s benefits. However, when my current team made the transition from GitFlow to TBD we realized a huge boost in speed, flexibility and fun. That’s why I wanted to share the story with others in this article.

      Have you ever tried TBD? If not I really recommend to start with the feature toggles because they can be very helpful anyway, even if you keep GitFlow! A/B testing is super valuable! When you have the toggles you can try how it works for you to push code into the develop branch when the feature is not completely finished, yet. If you don’t like it, just continue with GitFlow. Every team should do what it feels most comfortable with!

      Cheers
      Robert

      Reply
      • May 19, 2017 at 12:59 pm
        Permalink

        The biggest problem I see with feature branching is that you don’t have visibility of what other team members’ changes in the code. You get to see these changes (most probably) only at the end of the sprint. And this is too late since you’ll have big merges that you’ll have to address. It’s the end of the sprint. Clock is ticking and features need to be tested.
        I usually hear excuse like “yeah, but I merge master into my feature branch everyday”. Yeah, but what about other guys that develop their feature on their branches, when will they get to see your changes that perhaps affect them? Only when you merge your feature branch in the master. And when you do that it’s too late (like I said, end of sprint, end of release, whatever).
        As for the feature toggles, they are a technical debt. But! You can use them as a feedback about your design. What I mean by this is that if you adhere to Open Closed Principle, you’ll be having feature toggles only in composition root! since you just switch the old implementation with the new one, or append the existing with new functionality.
        Thanks for the article btw :)

        Cheers!

        Reply
        • May 20, 2017 at 6:14 pm
          Permalink

          I really like your thoughts, Dragan :)

          Reply
        • July 11, 2017 at 5:37 pm
          Permalink

          I really like reading you guys. Very interesting topic and ideas. I believe these are all great tools to use for different situations. I believe feature branches are great just as the feature switch! Where I disagree is when we say that conflicts are surprises. If your team communicates well and efficiently through daily stand-up meetings or other means, you will know:
          1. what is coming
          2. from which branch you should pull and merge frequently
          3. with who you should work closely
          I propose to use some incremental code reviews for those long lived branches to minimise the amount of code to review in one dose, identify design clash early and improve developper communication’s efficiency.

          Reply
          • July 11, 2017 at 5:52 pm
            Permalink

            I guess you are right. Perfect communication would probably solve every problem ;-)

            Reply
  • May 26, 2017 at 12:03 am
    Permalink

    Biggest problem with feature-branch *and* feature-toggles (and most other things it seems) is the difference between doing what actually worked in the first-place, versus the various sloppy, misinformed, misexecutions that neglected the most important stuff that made it good-vs-bad in the first place.

    Case in point #1: Not just feature branches but any/all LONG-lived branches. Long-lived was never supposed to mean UNsynchronized or UNintegrated. The original practice (the one that was good) did not delay subsequent merge/integration to main-trunk. It was always supposed to happen ASAP, immediately after any change. Someone somewhere along the line left that out, and used the branch as change-prison for solitary confinement. This broke *everything*!

    Case-in-point-#2: Feature toggles (especially those using conditional compilation) — the idea was use some simple “toggles” (instead of a branch) BUT THEN REMOVE THE TOGGLES after the (short-lived feature) was complete.
    Here again, some folks started neglecting to remove the toggles, creating lots of increasingly brittle hard-to-change code, and not even bothering to “refactor” them into better+simpler structures that applied SOLID principles.

    SO then you get to choose between long-lived branching structures WITHOUT frequent integration *OR* long-lived toggles without subsequent removal or refactoring.

    Better choice: apply the appropriate practice correctly (as originally intended) with a solid understanding of the context in which it applies, and of the underlying principles that must not be violated when executing it.

    Reply
    • May 26, 2017 at 9:30 am
      Permalink

      “Case in point #1: Not just feature branches but any/all LONG-lived branches. Long-lived was never supposed to mean UNsynchronized or UNintegrated. The original practice (the one that was good) did not delay subsequent merge/integration to main-trunk.”
      How would you do keep in sync all feature branches (in case there’s people are doing feature branching)?

      Reply
  • September 28, 2017 at 9:21 am
    Permalink

    is it possible to always code in Develop branch ( with feature toggle) first and then push the change to any targeted feature or release branch as needed so that there are no merge conflicts ?

    Reply
    • September 28, 2017 at 10:30 am
      Permalink

      Hey, sure it is possible but what do you think would be the benefits of it?

      Reply
  • October 2, 2017 at 3:45 pm
    Permalink

    Hi Robert,

    In big project system there are multiple team having different modules and there is individual support team who work on always on hotfix across team, so how this trunk or git flow works with feature push.

    Even we have multiple staging servers QA/ClientQA so how this process will work with master,develop,feature branch.

    And toggle will works if single developer work on single project.

    Reply
    • October 2, 2017 at 4:45 pm
      Permalink

      Hi Megha,

      I think with many people and different teams it is even more helpful to use one branch and feature toggles instead of many different feature branches. As long as a feature is not active it will most likely not influence the features that other teams are working on.

      Maybe you don’t even need so many staging systems anymore. At the project I’m currently working on we have only one staging environment but we can run it with different configuration set-ups. Testers just enable the features that they want to test.

      Regarding the hot fixes, I would not even create feature toggles for them. I guess most hot fixes should automatically be active for everybody!

      If you would like to try out TBD I would recommend you to not change everything at once. Maybe you can start it only in one team. If it works for them they can show others how they do it. Then you can also apply it on the bigger scope of the complete system. At least in my experience this is much easier than trying to change the whole organization at once :)

      Reply
  • October 9, 2017 at 10:01 pm
    Permalink

    Robert,

    Do you use Release branches along with TBD? I am curious why or why not and what you think.

    Rex

    Reply
    • October 9, 2017 at 11:00 pm
      Permalink

      Hey Rex,
      Our master branch is always ready to be released. When we want to release a new version of our iOS app to the App Store we push a Git Tag on the master branch and start our release build job. The release is then built directly from the master branch. To be honest, I don’t see a real benefit of creating a new branch for the release.

      How do you do that?

      Reply
      • October 9, 2017 at 11:10 pm
        Permalink

        We are a little more liberal on our trunk branch – it is not technically always ready for customers. We make a version branch (v1) when we are ready to release and as customers use that version we may fix bugs on that branch from time to time. I am in a regulated industry. Often we make changes but cannot release them until the application is audited. Because of this we do not keep trunk ready at a moments notice to update a customer, instead we use the version branches to do this. Sometimes this means fixing a bug twice (once in trunk and once in the version branch) but this is usually not a big deal and we do not always fix them the same way in each (for example we may use newer technologies like async/await in trunk and use a promise in the v1 branch)

        Reply
        • October 9, 2017 at 11:24 pm
          Permalink

          Sounds like the release branches are a good choice in your case! You could also use cherry picks if you don’t want to merge your changes from the release branch into the master. Then you don’t have to do the fixes twice :)

          Maybe it would be also ok for your auditors if you release new features along with hotfixes as long as the new features are hidden with feature toggles.

          I think in the end every team should try out a few things and find the mode which is working best for their specific needs.

          Reply
          • October 9, 2017 at 11:28 pm
            Permalink

            “I think in the end every team should try out a few things and find the mode which is working best for their specific needs.”

            I agree, this is always a good idea.

            Thanks for the article and conversation.

            Reply
            • October 9, 2017 at 11:32 pm
              Permalink

              Thanks for sharing your thoughts :)

              Reply
    • November 30, 2017 at 2:45 am
      Permalink

      Release-specific branches should only be necessary when you are facing the “multiple maintenance problem” where you are in the position of having to support multiple different releases that are in production usage during the same time. This is less common in SaaS and cloud-based environments where (especially where “release” and “deploy” are close together in not only time & space, but also server/site ownership).

      But the more frequently you can release, and the less cost+impact it is for each installation to patch/update/upgrade from one release to the next, the fewer the number of different releases you have to be able to support in parallel.

      If you are using trunk-based-development, you might still need to create release-branches, but you would not create them at the beginning of development for each release. You would create/branch the release off of main-trunk only at the point in time when you needed to be able to support+maintain that release in parallel with the active development work still taking place on main-trunk.

      As it turns out, creating “release-toggles” might seem like a good alternative to creating a release-branch, but only if the release-toggle was going to be very short-lived. Otherwise, toggling-code at the release-level creates more violations of clean-coding and related principles of coupling, cohesion, modularity, SOLID, SoC, and “well-factoredness” than does creating longer-lived toggles at the feature-level or service-level (and applying refactoring [to patterns] of the toggles as the code evolves).

      Reply
  • October 21, 2017 at 8:05 pm
    Permalink

    Nice article. we have been following best practices like short living feature branches created out of trunk and feature release behind flag. i think for branching, the key takeaway is – do not have multiple long lasting branches.

    Reply
    • October 22, 2017 at 12:18 pm
      Permalink

      Yeah, short-living branches for the win! :)

      Reply
  • November 16, 2017 at 5:46 pm
    Permalink

    Great article, but does a code review happen every time you merge to the trunk or only when the feature is complete?

    Reply
    • November 16, 2017 at 6:12 pm
      Permalink

      Hey Jason, thanks for your feedback :)
      In my current team we review the code before we merge it to the trunk. We don’t wait until the whole feature is finished because that would mean a much longer time until we get feedback about the code. The smaller our cycles are the more efficient we can be. I also found out that it’s not much fun to do huge code reviews and it’s more likely to miss something if you have to read too much code. If you do small review cycles it works much better.
      I have written another article about code reviews in TBD, if you are interested:
      https://team-coder.com/code-reviews-in-trunk-based-development

      Reply
  • November 30, 2017 at 2:23 am
    Permalink

    Dragan asks: “How would you do keep in sync all feature branches (in case there’s people are doing feature branching)?”

    Normal approach was as follows: Work on the feature branch was still supposed to happen in small chunks (e.g. change-tasks). Every time a small change was made, and merged+integrated to feature-branch … once it succeeded, you would then merge+integrate the result to the main-trunk. This way you keep syncing from feature-branch to main-trunk early and regularly.

    Now … if you are able to integrate changes from feature-branch to main-trunk as frequently as you do to feature-branch, then you probably didnt need the feature branch in the first place. Otherwise, if the reason for feature-branch was that otherwise too many things are trying to merge to main-trunk during the same time-period, then you integrate from feature-branch to main-trunk somewhat less frequently (e.g., hourly rather than every 15minutes, or perhaps every 2-3 hours, but at least 1-2 times per day).

    If the integration to main-trunk generally requires more time for a more complete/clean build and more complete (automated) test-coverage, then this would allow feature-level CI to happen much more rapidly/frequently then you would otherwise be able to do on main-trunk.

    And if there was no issue merging+integrating to main-trunk at the shorter frequency with sufficiently complete (clean) building and test-coverage, then there was no need for feature-branch in the first place.

    The purpose of the feature-branch was for a feature-team to be able to continuously integrate as quickly and frequently as possible when merging all changes (across all features) to main-trunk at the same frequency across all feature teams would otherwise create too much concurrent commit contention to the same integration branch to sustain a stable main-trunk.

    If there was a need to sync (i.e. rebase) feature-branches from main-trunk (instead of keep them pristine), that could be done during (or after) integrating from feature-branch to main-trunk. The moment that became more overhead then creating (and removing) feature toggles, is the moment you opt for feature-toggles instead of branches (but then you have to remove them when done, and don’t create toggles for things that are fine-grained).

    Smaller changes+fixes that were not closely coupled to a particular feature, didn’t need to be merged to any feature branch, and could merge directly to main-trunk instead.

    But when you have a single, small team (and need for multiple teams, much less feature teams) , then creating a separate feature-integration branch per feature makes no sense, because there is no need for a separate-but-nested integration cadences at the feature-level versus the whole-product.

    Reply
    • November 30, 2017 at 2:24 pm
      Permalink

      Well, as for me, it boils down to this. If your not able to merge continuously into master then you’d have to ask yourself why is that. If there are lots of merge conflicts then perhaps teams are not aligned with business capabilities (e.g. feature teams) and you have more component based teams than feature teams. Also, postponing merge conflicts (with feature branches) is not going to help you at all but is going to harm you way more than merging as soon as possible (pull the pain to the front). If the CI cycle takes a lot of time, then we should ask ourselves why is it taking too long? Perhaps we have a monolith on the rise and should be thinking in terms of aligning code with the teams and having separate repos (did someone mention microservices? :) ) and thus separate CI cycles.
      Summed up, any difficulties with continuous integration are actually a FEEDBACK. A feedback about our team structures, code structures, design and architecture. Let’s not push that valuable feedback under the carpet (e.g. by using feature branches functionality, CI build on feature branches etc.) since this way we’re actually SOLVING THE WRONG PROBLEM, but use it to improve your code/teams/design/architecture.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*