![\"Image](\"https://commons.wikimedia.org/wiki/File:Entroncamento_do_Transpraia.JPG\")
\nImage by Mecanismo, via Wikimedia Commons. Used under the CC-By-SA 3.0 license.You are a victim of branch prediction fail.
\nConsider a railroad junction:
\n\nImage by Mecanismo, via Wikimedia Commons. Used under the CC-By-SA 3.0 license.
Now for the sake of argument, suppose this is back in the 1800s - before long-distance or radio communication.
\nYou are a blind operator of a junction and you hear a train coming. You have no idea which way it is supposed to go. You stop the train to ask the driver which direction they want. And then you set the switch appropriately.
\nTrains are heavy and have a lot of inertia, so they take forever to start up and slow down.
\nIs there a better way? You guess which direction the train will go!
\nIf you guess right every time, the train will never have to stop.
\nIf you guess wrong too often, the train will spend a lot of time stopping, backing up, and restarting.
Consider an if-statement: At the processor level, it is a branch instruction:
\n![\"if(x](\"https://i.sstatic.net/pyfwC.png\")
You are a processor and you see a branch. You have no idea which way it will go. What do you do? You halt execution and wait until the previous instructions are complete. Then you continue down the correct path.
\nModern processors are complicated and have long pipelines. This means they take forever to "warm up" and "slow down".
\nIs there a better way? You guess which direction the branch will go!
\nIf you guess right every time, the execution will never have to stop.
\nIf you guess wrong too often, you spend a lot of time stalling, rolling back, and restarting.
This is branch prediction. I admit it's not the best analogy since the train could just signal the direction with a flag. But in computers, the processor doesn't know which direction a branch will go until the last moment.
\nHow would you strategically guess to minimize the number of times that the train must back up and go down the other path? You look at the past history! If the train goes left 99% of the time, then you guess left. If it alternates, then you alternate your guesses. If it goes one way every three times, you guess the same...
\nIn other words, you try to identify a pattern and follow it. This is more or less how branch predictors work.
\nMost applications have well-behaved branches. Therefore, modern branch predictors will typically achieve >90% hit rates. But when faced with unpredictable branches with no recognizable patterns, branch predictors are virtually useless.
\nFurther reading: "Branch predictor" article on Wikipedia.
\nif (data[c] >= 128)\n sum += data[c];\nNotice that the data is evenly distributed between 0 and 255. When the data is sorted, roughly the first half of the iterations will not enter the if-statement. After that, they will all enter the if-statement.
\nThis is very friendly to the branch predictor since the branch consecutively goes the same direction many times. Even a simple saturating counter will correctly predict the branch except for the few iterations after it switches direction.
\nQuick visualization:
\nT = branch taken\nN = branch not taken\n\ndata[] = 0, 1, 2, 3, 4, ... 126, 127, 128, 129, 130, ... 250, 251, 252, ...\nbranch = N N N N N ... N N T T T ... T T T ...\n\n = NNNNNNNNNNNN ... NNNNNNNTTTTTTTTT ... TTTTTTTTTT (easy to predict)\nHowever, when the data is completely random, the branch predictor is rendered useless, because it can't predict random data. Thus there will probably be around 50% misprediction (no better than random guessing).
\ndata[] = 226, 185, 125, 158, 198, 144, 217, 79, 202, 118, 14, 150, 177, 182, ...\nbranch = T, T, N, T, T, T, T, N, T, N, N, T, T, T ...\n\n = TTNTTTTNTNNTTT ... (completely random - impossible to predict)\nWhat can be done?
\nIf the compiler isn't able to optimize the branch into a conditional move, you can try some hacks if you are willing to sacrifice readability for performance.
\nReplace:
\nif (data[c] >= 128)\n sum += data[c];\nwith:
\nint t = (data[c] - 128) >> 31;\nsum += ~t & data[c];\nThis eliminates the branch and replaces it with some bitwise operations.
\n(Note that this hack is not strictly equivalent to the original if-statement. But in this case, it's valid for all the input values of data[].)
Benchmarks: Core i7 920 @ 3.5 GHz
\nC++ - Visual Studio 2010 - x64 Release
\n| Scenario | \nTime (seconds) | \n
|---|---|
| Branching - Random data | \n11.777 | \n
| Branching - Sorted data | \n2.352 | \n
| Branchless - Random data | \n2.564 | \n
| Branchless - Sorted data | \n2.587 | \n
Java - NetBeans 7.1.1 JDK 7 - x64
\n| Scenario | \nTime (seconds) | \n
|---|---|
| Branching - Random data | \n10.93293813 | \n
| Branching - Sorted data | \n5.643797077 | \n
| Branchless - Random data | \n3.113581453 | \n
| Branchless - Sorted data | \n3.186068823 | \n
Observations:
\nA general rule of thumb is to avoid data-dependent branching in critical loops (such as in this example).
\nUpdate:
\nGCC 4.6.1 with -O3 or -ftree-vectorize on x64 is able to generate a conditional move, so there is no difference between the sorted and unsorted data - both are fast. This is called "if-conversion" (to branchless) and is necessary for vectorization but also sometimes good for scalar.
(Or somewhat fast: for the already-sorted case, cmov can be slower especially if GCC puts it on the critical path instead of just add, especially on Intel before Broadwell where cmov has 2-cycle latency: gcc optimization flag -O3 makes code slower than -O2)
VC++ 2010 is unable to generate conditional moves for this branch even under /Ox.
Intel C++ Compiler (ICC) 11 does something miraculous. It interchanges the two loops, thereby hoisting the unpredictable branch to the outer loop. Not only is it immune to the mispredictions, it's also twice as fast as whatever VC++ and GCC can generate! In other words, ICC took advantage of the test-loop to defeat the benchmark...
\nIf you give the Intel compiler the branchless code, it just outright vectorizes it... and is just as fast as with the branch (with the loop interchange).
\nif() version, as will GCC 5 and later with -O3, even though it takes quite a few instructions to sign-extend to the 64-bit sum on x86 without SSE4 or AVX2. (-march=x86-64-v2 or v3). See Why is processing an unsorted array the same speed as processing a sorted array with modern x86-64 clang?This goes to show that even mature modern compilers can vary wildly in their ability to optimize code...
\n","owner":"Mysticial","score":35287,"created_at":"2012-06-27T13:56:42.000Z","is_accepted":true,"question_id":11227809,"last_activity_at":"2024-03-04T17:37:13.000Z"},{"id":11227877,"body":"Branch prediction.
\nWith a sorted array, the condition data[c] >= 128 is first false for a streak of values, then becomes true for all later values. That's easy to predict. With an unsorted array, you pay for the branching cost.
The reason why performance improves drastically when the data is sorted is that the branch prediction penalty is removed, as explained beautifully in Mysticial's answer.
\nNow, if we look at the code
\nif (data[c] >= 128)\n sum += data[c];\nwe can find that the meaning of this particular if... else... branch is to add something when a condition is satisfied. This type of branch can be easily transformed into a conditional move statement, which would be compiled into a conditional move instruction: cmovl, in an x86 system. The branch and thus the potential branch prediction penalty is removed.
In C, thus C++, the statement, which would compile directly (without any optimization) into the conditional move instruction in x86, is the ternary operator ... ? ... : .... So we rewrite the above statement into an equivalent one:
sum += data[c] >=128 ? data[c] : 0;\nWhile maintaining readability, we can check the speedup factor.
\nOn an Intel Core i7-2600K @ 3.4 GHz and Visual Studio 2010 Release Mode, the benchmark is:
\nx86
\n| Scenario | \nTime (seconds) | \n
|---|---|
| Branching - Random data | \n8.885 | \n
| Branching - Sorted data | \n1.528 | \n
| Branchless - Random data | \n3.716 | \n
| Branchless - Sorted data | \n3.71 | \n
x64
\n| Scenario | \nTime (seconds) | \n
|---|---|
| Branching - Random data | \n11.302 | \n
| Branching - Sorted data | \n1.830 | \n
| Branchless - Random data | \n2.736 | \n
| Branchless - Sorted data | \n2.737 | \n
The result is robust in multiple tests. We get a great speedup when the branch result is unpredictable, but we suffer a little bit when it is predictable. In fact, when using a conditional move, the performance is the same regardless of the data pattern.
\nNow let's look more closely by investigating the x86 assembly they generate. For simplicity, we use two functions max1 and max2.
max1 uses the conditional branch if... else ...:
int max1(int a, int b) {\n if (a > b)\n return a;\n else\n return b;\n}\nmax2 uses the ternary operator ... ? ... : ...:
int max2(int a, int b) {\n return a > b ? a : b;\n}\nOn an x86-64 machine, GCC -S generates the assembly below.
:max1\n movl %edi, -4(%rbp)\n movl %esi, -8(%rbp)\n movl -4(%rbp), %eax\n cmpl -8(%rbp), %eax\n jle .L2\n movl -4(%rbp), %eax\n movl %eax, -12(%rbp)\n jmp .L4\n.L2:\n movl -8(%rbp), %eax\n movl %eax, -12(%rbp)\n.L4:\n movl -12(%rbp), %eax\n leave\n ret\n\n:max2\n movl %edi, -4(%rbp)\n movl %esi, -8(%rbp)\n movl -4(%rbp), %eax\n cmpl %eax, -8(%rbp)\n cmovge -8(%rbp), %eax\n leave\n ret\nmax2 uses much less code due to the usage of instruction cmovge. But the real gain is that max2 does not involve branch jumps, jmp, which would have a significant performance penalty if the predicted result is not right.
So why does a conditional move perform better?
\nIn a typical x86 processor, the execution of an instruction is divided into several stages. Roughly, we have different hardware to deal with different stages. So we do not have to wait for one instruction to finish to start a new one. This is called pipelining.
In a branch case, the following instruction is determined by the preceding one, so we cannot do pipelining. We have to either wait or predict.
\nIn a conditional move case, the execution of conditional move instruction is divided into several stages, but the earlier stages like Fetch and Decode do not depend on the result of the previous instruction; only the latter stages need the result. Thus, we wait a fraction of one instruction's execution time. This is why the conditional move version is slower than the branch when the prediction is easy.
The book Computer Systems: A Programmer's Perspective, second edition explains this in detail. You can check Section 3.6.6 for Conditional Move Instructions, entire Chapter 4 for Processor Architecture, and Section 5.11.2 for special treatment for Branch Prediction and Misprediction Penalties.
\nSometimes, some modern compilers can optimize our code to assembly with better performance, and sometimes some compilers can't (the code in question is using Visual Studio's native compiler). Knowing the performance difference between a branch and a conditional move when unpredictable can help us write code with better performance when the scenario gets so complex that the compiler can not optimize them automatically.
\n","owner":"WiSaGaN","score":3860,"created_at":"2012-06-28T02:14:03.000Z","is_accepted":false,"question_id":11227809,"last_activity_at":"2022-07-03T15:12:52.000Z"},{"id":11303693,"body":"If you are curious about even more optimizations that can be done to this code, consider this:
\nStarting with the original loop:
\nfor (unsigned i = 0; i < 100000; ++i)\n{\n for (unsigned j = 0; j < arraySize; ++j)\n {\n if (data[j] >= 128)\n sum += data[j];\n }\n}\nWith loop interchange, we can safely change this loop to:
\nfor (unsigned j = 0; j < arraySize; ++j)\n{\n for (unsigned i = 0; i < 100000; ++i)\n {\n if (data[j] >= 128)\n sum += data[j];\n }\n}\nThen, you can see that the if conditional is constant throughout the execution of the i loop, so you can hoist the if out:
for (unsigned j = 0; j < arraySize; ++j)\n {\n if (data[j] >= 128)\n {\n for (unsigned i = 0; i < 100000; ++i)\n {\n sum += data[j];\n }\n }\n}\nThen, you see that the inner loop can be collapsed into one single expression, assuming the floating point model allows it (/fp:fast is thrown, for example)
for (unsigned j = 0; j < arraySize; ++j)\n{\n if (data[j] >= 128)\n {\n sum += data[j] * 100000;\n }\n}\nThat one is 100,000 times faster than before.
\n","owner":"vulcan raven","score":2669,"created_at":"2012-07-03T02:25:30.000Z","is_accepted":false,"question_id":11227809,"last_activity_at":"2024-01-02T18:32:46.000Z"},{"id":12853037,"body":"No doubt some of us would be interested in ways of identifying code that is problematic for the CPU's branch-predictor. The Valgrind tool cachegrind has a branch-predictor simulator, enabled by using the --branch-sim=yes flag. Running it over the examples in this question, with the number of outer loops reduced to 10000 and compiled with g++, gives these results:
Sorted:
\n\n==32551== Branches: 656,645,130 ( 656,609,208 cond + 35,922 ind)\n==32551== Mispredicts: 169,556 ( 169,095 cond + 461 ind)\n==32551== Mispred rate: 0.0% ( 0.0% + 1.2% )\nUnsorted:
\n\n==32555== Branches: 655,996,082 ( 655,960,160 cond + 35,922 ind)\n==32555== Mispredicts: 164,073,152 ( 164,072,692 cond + 460 ind)\n==32555== Mispred rate: 25.0% ( 25.0% + 1.2% )\nDrilling down into the line-by-line output produced by cg_annotate we see for the loop in question:
Sorted:
\n\n Bc Bcm Bi Bim\n 10,001 4 0 0 for (unsigned i = 0; i < 10000; ++i)\n . . . . {\n . . . . // primary loop\n 327,690,000 10,016 0 0 for (unsigned c = 0; c < arraySize; ++c)\n . . . . {\n 327,680,000 10,006 0 0 if (data[c] >= 128)\n 0 0 0 0 sum += data[c];\n . . . . }\n . . . . }\nUnsorted:
\n\n Bc Bcm Bi Bim\n 10,001 4 0 0 for (unsigned i = 0; i < 10000; ++i)\n . . . . {\n . . . . // primary loop\n 327,690,000 10,038 0 0 for (unsigned c = 0; c < arraySize; ++c)\n . . . . {\n 327,680,000 164,050,007 0 0 if (data[c] >= 128)\n 0 0 0 0 sum += data[c];\n . . . . }\n . . . . }\nThis lets you easily identify the problematic line - in the unsorted version the if (data[c] >= 128) line is causing 164,050,007 mispredicted conditional branches (Bcm) under cachegrind's branch-predictor model, whereas it's only causing 10,006 in the sorted version.
Alternatively, on Linux you can use the performance counters subsystem to accomplish the same task, but with native performance using CPU counters.
\n\nperf stat ./sumtest_sorted\nSorted:
\n\n Performance counter stats for './sumtest_sorted':\n\n 11808.095776 task-clock # 0.998 CPUs utilized \n 1,062 context-switches # 0.090 K/sec \n 14 CPU-migrations # 0.001 K/sec \n 337 page-faults # 0.029 K/sec \n26,487,882,764 cycles # 2.243 GHz \n41,025,654,322 instructions # 1.55 insns per cycle \n 6,558,871,379 branches # 555.455 M/sec \n 567,204 branch-misses # 0.01% of all branches \n\n 11.827228330 seconds time elapsed\nUnsorted:
\n\n Performance counter stats for './sumtest_unsorted':\n\n 28877.954344 task-clock # 0.998 CPUs utilized \n 2,584 context-switches # 0.089 K/sec \n 18 CPU-migrations # 0.001 K/sec \n 335 page-faults # 0.012 K/sec \n65,076,127,595 cycles # 2.253 GHz \n41,032,528,741 instructions # 0.63 insns per cycle \n 6,560,579,013 branches # 227.183 M/sec \n 1,646,394,749 branch-misses # 25.10% of all branches \n\n 28.935500947 seconds time elapsed\nIt can also do source code annotation with dissassembly.
\n\nperf record -e branch-misses ./sumtest_unsorted\nperf annotate -d sumtest_unsorted\n Percent | Source code & Disassembly of sumtest_unsorted\n------------------------------------------------\n...\n : sum += data[c];\n 0.00 : 400a1a: mov -0x14(%rbp),%eax\n 39.97 : 400a1d: mov %eax,%eax\n 5.31 : 400a1f: mov -0x20040(%rbp,%rax,4),%eax\n 4.60 : 400a26: cltq \n 0.00 : 400a28: add %rax,-0x30(%rbp)\n...\nSee the performance tutorial for more details.
\n","owner":"caf","score":2213,"created_at":"2012-10-12T05:53:33.000Z","is_accepted":false,"question_id":11227809,"last_activity_at":"2012-10-18T19:20:21.000Z"},{"id":16184827,"body":"I just read up on this question and its answers, and I feel an answer is missing.
\n\nA common way to eliminate branch prediction that I've found to work particularly good in managed languages is a table lookup instead of using a branch (although I haven't tested it in this case).
\n\nThis approach works in general if:
\n\nBackground and why
\n\nFrom a processor perspective, your memory is slow. To compensate for the difference in speed, a couple of caches are built into your processor (L1/L2 cache). So imagine that you're doing your nice calculations and figure out that you need a piece of memory. The processor will get its 'load' operation and loads the piece of memory into cache -- and then uses the cache to do the rest of the calculations. Because memory is relatively slow, this 'load' will slow down your program.
\n\nLike branch prediction, this was optimized in the Pentium processors: the processor predicts that it needs to load a piece of data and attempts to load that into the cache before the operation actually hits the cache. As we've already seen, branch prediction sometimes goes horribly wrong -- in the worst case scenario you need to go back and actually wait for a memory load, which will take forever (in other words: failing branch prediction is bad, a memory load after a branch prediction fail is just horrible!).
\n\nFortunately for us, if the memory access pattern is predictable, the processor will load it in its fast cache and all is well.
\n\nThe first thing we need to know is what is small? While smaller is generally better, a rule of thumb is to stick to lookup tables that are <= 4096 bytes in size. As an upper limit: if your lookup table is larger than 64K it's probably worth reconsidering.
\n\nConstructing a table
\n\nSo we've figured out that we can create a small table. Next thing to do is get a lookup function in place. Lookup functions are usually small functions that use a couple of basic integer operations (and, or, xor, shift, add, remove and perhaps multiply). You want to have your input translated by the lookup function to some kind of 'unique key' in your table, which then simply gives you the answer of all the work you wanted it to do.
\n\nIn this case: >= 128 means we can keep the value, < 128 means we get rid of it. The easiest way to do that is by using an 'AND': if we keep it, we AND it with 7FFFFFFF; if we want to get rid of it, we AND it with 0. Notice also that 128 is a power of 2 -- so we can go ahead and make a table of 32768/128 integers and fill it with one zero and a lot of 7FFFFFFFF's.
\n\nManaged languages
\n\nYou might wonder why this works well in managed languages. After all, managed languages check the boundaries of the arrays with a branch to ensure you don't mess up...
\n\nWell, not exactly... :-)
\n\nThere has been quite some work on eliminating this branch for managed languages. For example:
\n\n\n\nfor (int i = 0; i < array.Length; ++i)\n{\n // Use array[i]\n}\nIn this case, it's obvious to the compiler that the boundary condition will never be hit. At least the Microsoft JIT compiler (but I expect Java does similar things) will notice this and remove the check altogether. WOW, that means no branch. Similarly, it will deal with other obvious cases.
\n\nIf you run into trouble with lookups in managed languages -- the key is to add a & 0x[something]FFF to your lookup function to make the boundary check predictable -- and watch it going faster.
The result of this case
\n\n// Generate data\nint arraySize = 32768;\nint[] data = new int[arraySize];\n\nRandom random = new Random(0);\nfor (int c = 0; c < arraySize; ++c)\n{\n data[c] = random.Next(256);\n}\n\n/*To keep the spirit of the code intact, I'll make a separate lookup table\n(I assume we cannot modify 'data' or the number of loops)*/\n\nint[] lookup = new int[256];\n\nfor (int c = 0; c < 256; ++c)\n{\n lookup[c] = (c >= 128) ? c : 0;\n}\n\n// Test\nDateTime startTime = System.DateTime.Now;\nlong sum = 0;\n\nfor (int i = 0; i < 100000; ++i)\n{\n // Primary loop\n for (int j = 0; j < arraySize; ++j)\n {\n /* Here you basically want to use simple operations - so no\n random branches, but things like &, |, *, -, +, etc. are fine. */\n sum += lookup[data[j]];\n }\n}\n\nDateTime endTime = System.DateTime.Now;\nConsole.WriteLine(endTime - startTime);\nConsole.WriteLine(\"sum = \" + sum);\nConsole.ReadLine();\nAs data is distributed between 0 and 255 when the array is sorted, around the first half of the iterations will not enter the if-statement (the if statement is shared below).
if (data[c] >= 128)\n sum += data[c];\nThe question is: What makes the above statement not execute in certain cases as in case of sorted data? Here comes the \"branch predictor\". A branch predictor is a digital circuit that tries to guess which way a branch (e.g. an if-then-else structure) will go before this is known for sure. The purpose of the branch predictor is to improve the flow in the instruction pipeline. Branch predictors play a critical role in achieving high effective performance!
Let's do some bench marking to understand it better
\n\nThe performance of an if-statement depends on whether its condition has a predictable pattern. If the condition is always true or always false, the branch prediction logic in the processor will pick up the pattern. On the other hand, if the pattern is unpredictable, the if-statement will be much more expensive.
Let’s measure the performance of this loop with different conditions:
\n\nfor (int i = 0; i < max; i++)\n if (condition)\n sum++;\nHere are the timings of the loop with different true-false patterns:
\n\nCondition Pattern Time (ms)\n-------------------------------------------------------\n(i & 0×80000000) == 0 T repeated 322\n\n(i & 0xffffffff) == 0 F repeated 276\n\n(i & 1) == 0 TF alternating 760\n\n(i & 3) == 0 TFFFTFFF… 513\n\n(i & 2) == 0 TTFFTTFF… 1675\n\n(i & 4) == 0 TTTTFFFFTTTTFFFF… 1275\n\n(i & 8) == 0 8T 8F 8T 8F … 752\n\n(i & 16) == 0 16T 16F 16T 16F … 490\nA “bad” true-false pattern can make an if-statement up to six times slower than a “good” pattern! Of course, which pattern is good and which is bad depends on the exact instructions generated by the compiler and on the specific processor.
So there is no doubt about the impact of branch prediction on performance!
\n","owner":"Saqlain","score":1453,"created_at":"2013-02-15T07:24:16.000Z","is_accepted":false,"question_id":11227809,"last_activity_at":"2019-02-27T10:58:32.000Z"},{"id":17782979,"body":"One way to avoid branch prediction errors is to build a lookup table, and index it using the data. Stefan de Bruijn discussed that in his answer.
\nBut in this case, we know values are in the range [0, 255] and we only care about values >= 128. That means we can easily extract a single bit that will tell us whether we want a value or not: by shifting the data to the right 7 bits, we are left with a 0 bit or a 1 bit, and we only want to add the value when we have a 1 bit. Let's call this bit the "decision bit".
\nBy using the 0/1 value of the decision bit as an index into an array, we can make code that will be equally fast whether the data is sorted or not sorted. Our code will always add a value, but when the decision bit is 0, we will add the value somewhere we don't care about. Here's the code:
\n// Test\nclock_t start = clock();\nlong long a[] = {0, 0};\nlong long sum;\n\nfor (unsigned i = 0; i < 100000; ++i)\n{\n // Primary loop\n for (unsigned c = 0; c < arraySize; ++c)\n {\n int j = (data[c] >> 7);\n a[j] += data[c];\n }\n}\n\ndouble elapsedTime = static_cast<double>(clock() - start) / CLOCKS_PER_SEC;\nsum = a[1];\nThis code wastes half of the adds but never has a branch prediction failure. It's tremendously faster on random data than the version with an actual if statement.
\nBut in my testing, an explicit lookup table was slightly faster than this, probably because indexing into a lookup table was slightly faster than bit shifting. This shows how my code sets up and uses the lookup table (unimaginatively called lut for "LookUp Table" in the code). Here's the C++ code:
// Declare and then fill in the lookup table\nint lut[256];\nfor (unsigned c = 0; c < 256; ++c)\n lut[c] = (c >= 128) ? c : 0;\n\n// Use the lookup table after it is built\nfor (unsigned i = 0; i < 100000; ++i)\n{\n // Primary loop\n for (unsigned c = 0; c < arraySize; ++c)\n {\n sum += lut[data[c]];\n }\n}\nIn this case, the lookup table was only 256 bytes, so it fits nicely in a cache and all was fast. This technique wouldn't work well if the data was 24-bit values and we only wanted half of them... the lookup table would be far too big to be practical. On the other hand, we can combine the two techniques shown above: first shift the bits over, then index a lookup table. For a 24-bit value that we only want the top half value, we could potentially shift the data right by 12 bits, and be left with a 12-bit value for a table index. A 12-bit table index implies a table of 4096 values, which might be practical.
\nThe technique of indexing into an array, instead of using an if statement, can be used for deciding which pointer to use. I saw a library that implemented binary trees, and instead of having two named pointers (pLeft and pRight or whatever) had a length-2 array of pointers and used the "decision bit" technique to decide which one to follow. For example, instead of:
if (x < node->value)\n node = node->pLeft;\nelse\n node = node->pRight;\nthis library would do something like:
\ni = (x < node->value);\nnode = node->link[i];\nHere's a link to this code: Red Black Trees, Eternally Confuzzled
\n","owner":"steveha","score":1388,"created_at":"2013-07-22T08:29:30.000Z","is_accepted":false,"question_id":11227809,"last_activity_at":"2021-01-27T10:03:36.000Z"},{"id":17828251,"body":"In the sorted case, you can do better than relying on successful branch prediction or any branchless comparison trick: completely remove the branch.
\n\nIndeed, the array is partitioned in a contiguous zone with data < 128 and another with data >= 128. So you should find the partition point with a dichotomic search (using Lg(arraySize) = 15 comparisons), then do a straight accumulation from that point.
Something like (unchecked)
\n\nint i= 0, j, k= arraySize;\nwhile (i < k)\n{\n j= (i + k) >> 1;\n if (data[j] >= 128)\n k= j;\n else\n i= j;\n}\nsum= 0;\nfor (; i < arraySize; i++)\n sum+= data[i];\nor, slightly more obfuscated
\n\nint i, k, j= (i + k) >> 1;\nfor (i= 0, k= arraySize; i < k; (data[j] >= 128 ? k : i)= j)\n j= (i + k) >> 1;\nfor (sum= 0; i < arraySize; i++)\n sum+= data[i];\nA yet faster approach, that gives an approximate solution for both sorted or unsorted is: sum= 3137536; (assuming a truly uniform distribution, 16384 samples with expected value 191.5) :-)
The above behavior is happening because of Branch prediction.
\nTo understand branch prediction one must first understand an Instruction Pipeline.
\nThe the steps of running an instruction can be overlapped with the sequence of steps of running the previous and next instruction, so that different steps can be executed concurrently in parallel. This technique is known as instruction pipelining and is used to increase throughput in modern processors. To understand this better please see this example on Wikipedia.
\nGenerally, modern processors have quite long (and wide) pipelines, so many instruction can be in flight. See Modern Microprocessors\nA 90-Minute Guide! which starts by introducing basic in-order pipelining and goes from there.
\nBut for ease let's consider a simple in-order pipeline with these 4 steps only.
\n(Like a classic 5-stage RISC, but omitting a separate MEM stage.)
4-stage pipeline in general for 2 instructions.
\n![\"4-stage](\"https://i.sstatic.net/PqBBR.png\")
Moving back to the above question let's consider the following instructions:
\n A) if (data[c] >= 128)\n /\\\n / \\\n / \\\n true / \\ false\n / \\\n / \\\n / \\\n / \\\n B) sum += data[c]; C) for loop or print().\nWithout branch prediction, the following would occur:
\nTo execute instruction B or instruction C the processor will have to wait (stall) till the instruction A leaves the EX stage in the pipeline, as the decision to go to instruction B or instruction C depends on the result of instruction A. (i.e. where to fetch from next.) So the pipeline will look like this:
\nWithout prediction: when if condition is true:\n![\"enter](\"https://i.sstatic.net/0H4gP.png\")
Without prediction: When if condition is false:\n![\"enter](\"https://i.sstatic.net/APpca.png\")
As a result of waiting for the result of instruction A, the total CPU cycles spent in the above case (without branch prediction; for both true and false) is 7.
\nSo what is branch prediction?
\nBranch predictor will try to guess which way a branch (an if-then-else structure) will go before this is known for sure. It will not wait for the instruction A to reach the EX stage of the pipeline, but it will guess the decision and go to that instruction (B or C in case of our example).
\nIn case of a correct guess, the pipeline looks something like this:\n![\"enter](\"https://i.sstatic.net/ZYUbs.png\")
If it is later detected that the guess was wrong then the partially executed instructions are discarded and the pipeline starts over with the correct branch, incurring a delay.\nThe time that is wasted in case of a branch misprediction is equal to the number of stages in the pipeline from the fetch stage to the execute stage. Modern microprocessors tend to have quite long pipelines so that the misprediction delay is between 10 and 20 clock cycles. The longer the pipeline the greater the need for a good branch predictor.
\nIn the OP's code, the first time when the conditional, the branch predictor does not have any information to base up prediction, so the first time it will randomly choose the next instruction. (Or fall back to static prediction, typically forward not-taken, backward taken). Later in the for loop, it can base the prediction on the history.\nFor an array sorted in ascending order, there are three possibilities:
\nLet us assume that the predictor will always assume the true branch on the first run.
\nSo in the first case, it will always take the true branch since historically all its predictions are correct.\nIn the 2nd case, initially it will predict wrong, but after a few iterations, it will predict correctly.\nIn the 3rd case, it will initially predict correctly till the elements are less than 128. After which it will fail for some time and the correct itself when it sees branch prediction failure in history.
\nIn all these cases the failure will be too less in number and as a result, only a few times it will need to discard the partially executed instructions and start over with the correct branch, resulting in fewer CPU cycles.
\nBut in case of a random unsorted array, the prediction will need to discard the partially executed instructions and start over with the correct branch most of the time and result in more CPU cycles compared to the sorted array.
\nFurther reading:
\nif or break), backward taken (like a loop) has been used because it's better than nothing. Laying out code so the fast path / common case minimizes taken branches is good for I-cache density as well as static prediction, so compilers already do that. (That's the real effect of likely / unlikely hints in C source, not actually hinting the hardware branch prediction in most CPU, except maybe via static prediction.)