This article was pushed to this comment . Or rather, one phrase from it.

... it is not good to spend memory or processor cycles on items in billions of volumes ...It so happened that lately I had to do just that. And, although the case, which I will consider in the article, is quite private - the conclusions and the applied solutions can be useful to someone.

The iFunny application deals with a huge amount of graphic and video content, and a fuzzy search for duplicates is one of the very important tasks. This in itself is a big topic that deserves a separate article, but today I’ll just talk a little about some of the approaches to calculating very large arrays of numbers in relation to this search. Of course, everyone has a different understanding of “very large arrays,” and it would be foolish to be with the Hadron Collider, but still.:)

If the algorithm is very brief, then for each image, its digital signature (signature) is created from 968 integers, and the comparison is made by finding the “distance” between two signatures. Considering that the volume of content in the last two months alone amounted to about 10 million images, then, as an attentive reader will easily estimate in his mind, these are the very “elements in billions of volumes”. Who cares - welcome under the cat.

In the beginning there will be a boring story about saving time and memory, and at the end there is a small instructive story, about the fact that sometimes it is very useful to look at the sources. The picture to attract attention is directly related to this instructive story.

I have to admit that I was a little wicked. With a preliminary analysis of the algorithm, it was possible to reduce the number of values in each signature from 968 to 420. Already twice as good, but the order of the values remained the same.

Although, if you think about it, I’m not so deceitful, and this will be the first conclusion: should you start thinking about the task, you should think - can it be somehow simplified in advance?

The comparison algorithm is quite simple: the root is calculated from the sum of the squares of the differences of the two signatures, divided by the sum of the previously calculated values (i.e., in each iteration, the summation will still be completely removed from the constant) and compared with the threshold value. It should be noted that the elements of the signature are limited to values from -2 to +2, and, accordingly, the absolute value of the difference - values from 0 to 4.

Nothing complicated, but the number decides.

` ````
//Threshold
const val d = 0.3
//10.000.000 objects.//Let's keep this figure in mind//so as not to mention in every second sentence
val collection: MutableList & lt; Signature & gt; = mutableListOf ()
//signature - an array of 420 values of type Byte
class Signature (val signature: Array & lt; Byte & gt ;, val norma: Double)
fun getSimilar (signature: Signature) = collection
.filter {calculateDistance (it, signature) & lt; d}
fun calculateDistance (first: Signature, second: Signature): Double =
Math.sqrt (first.signature.mapIndexed {index, value - & gt;
Math.pow ((value - second.signature [index]). ToDouble (), 2.0)
} .sum ())/(first.norma + second.norma)
```

Let's calculate what we have here with the number of operations:

` 10M `

square roots of ` Math.sqrt `

` 10M `

additions and divisions `/(first.norma + second.norma) `

` 4.200M `

subtractions and squaring ` Math.pow ((value - second.signature [index]). toDouble (), 2.0) `

` 4.200M `

additions `.sum () `

What options we have:

- With such volumes, spending a whole
`Byte`

(or, God forbid, someone would guess to use`Int`

) to store three significant bits is an unforgivable waste. - Maybe somehow reduce the amount of mathematics?
- Is it possible to do any pre-filtering that is not so costly to calculate?

One

` Long `

is 64 bits, which, in our case, will allow you to store 21 values in it (and 1 bit will remain restless). ` ````
//array of 20 Long values
class Signature (val signature: Array & lt; Long & gt ;, val norma: Double)
fun calculateDistance (first: Signature, second: Signature): Double =
Math.sqrt (first.signature.mapIndexed {index, value - & gt;
calculatePartial (value, second.signature [index])
} .sum ())/(first.norma + second.norma)
fun calculatePartial (first: Long, second: Long): Double {
var sum = 0L
(0..60 step 3) .onach {
val current = (first.ushr (it) and 0x7) - (second.ushr (it) and 0x7)
sum + = current * current
}
return sum.toDouble ()
}
```

From memory it became better (

` 4.200M `

against ` 1.600M `

bytes, if roughly), but what about calculations? I think it is obvious that the number of operations remained the same and

` 8.400M `

shifts and ` 8.400M `

logical I were added. Maybe it can and can be optimized, but the trend is still not encouraging - this is not what we wanted. I sense the gut, you can use bit magic here!

Let's store the values not in three, but in four bits. So, here, in the way:

-2 | 0b1100 |

-1 | 0b0100 |

0 | 0b0000 |

1 | 0b0010 |

2 | 0b0011 |

Yes, we will lose in packing density in comparison with the previous option, but we will be able to get

` Long `

with differences (not quite) 16 elements at once by one ` XOR `

. In addition, there will be a total of 11 options for the distribution of bits in each final nibble, which will allow the use of pre-calculated squares of differences. ` ````
//array of 27 Long values
class Signature (val signature: Array & lt; Long & gt ;, val norma: Double)
//-1 for impossible options
val precomputed = arrayOf (0, 1, 1, 4, 1, -1, 4, 9, 1, -1, -1, -1, 4, -1, 9, 16)
fun calculatePartial (first: Long, second: Long): Double {
var sum = 0L
val difference = first xor second
(0..60 step 4) .onach {
sum + = precomputed [(difference.ushr (it) and 0xF) .toInt ()]
}
return sum.toDouble ()
}
```

From memory, it became

` 2.160M `

against ` 1.600M `

- unpleasant, but still better than the initial ` 4.200M `

. Calculate operations:

` 10M `

square roots, additions and divisions (have not gone anywhere) ` 270M `

` XOR `

'ov ` 4.320 `

additions, shifts, logical AND and extraction from an array. It looks more interesting, but, all the same, too many calculations. Unfortunately, it seems that those 20% of the efforts that give 80% of the result, we have already spent here and it's time to think about where else you can gain. The first thing that comes to mind is not to bring it to the calculation at all, filtering out obviously inappropriate signatures with some kind of lightweight filter.

If you slightly transform the calculation formula, then you can get such an inequality (the smaller the calculated distance, the better):

Those. now we need to figure out how, based on the information we have on the number of bits set in

` Long `

'ah, calculate the minimum possible value of the left side of the inequality. Then just discard all the signatures that do not satisfy him. Let me remind you that x

The final formula looks like this (we will not need it, but I took it for quite a long time and it would be insulting to just forget and not show it to anyone):

Where B is the number of bits set.

In fact, in one

` Long `

's only 64 bits, read 64 possible results. And they are well calculated in advance and added to the array, by analogy with the previous section. In addition, it is not necessary to calculate all 27

` Long `

's - it is enough to exceed the threshold at the next one and you can interrupt the check and return false. The same approach, by the way, can be used in the main calculation. ` ````
fun getSimilar (signature: Signature) = collection
.asSequence ()//Why do we need an extra intermediate collection !?
.filter {estimate (it, signature)}
.filter {calculateDistance (it, signature) & lt; d}
val estimateValues = arrayOf (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 22, 25, 28, 31, 34 , 37, 40, 43, 46, 49, 52, 55, 58, 61, 64, 69, 74, 79, 84, 89, 94, 99, 104, 109, 114, 119, 124, 129, 134, 139 , 144, 151, 158, 165, 172, 179, 186, 193, 200, 207, 214, 221, 228, 235, 242, 249, 256)
fun estimate (first: Signature, second: Signature): Boolean {
var bitThreshold = Math.pow (d * (first.norma + second.norma), 2.0) .toLong ()
first.signature.forEachIndexed {index, value - & gt;
bitThreshold - = estimateValues [java.lang.Long.bitCount (value xor second.signature [index])]
if (bitThreshold & lt; = 0) return false
}
return true
}
```

Here we must understand that the effectiveness of this filter (up to negative) strongly depends on the selected threshold and, a little less strongly, on the input data. Fortunately, for the threshold we need

` d = 0.3 `

, it is possible to pass a filter to a relatively small number of objects and the contribution of their calculation to the total response time is so scanty that they can be neglected. And if so, then you can still save a little. When working with large collections, the

` sequence `

is a good defense against the extremely unpleasant ` sequence `

is not only fashionable and youthful but also an iterator with ` hasNext `

associates who are not at all free. ` ````
fun getSimilar (signature: Signature) = collection
.filter {estimate (it, signature)}
.filter {calculateDistance (it, signature) & lt; d}
```

It would seem that here it is happiness, but I wanted to "make it beautiful." Here we come to the promised instructive story.

We write on Kotlin, and here some foreign

` java.lang.Long.bitCount `

! And recently, unsigned types were brought to the language. Attack! No sooner said than done. All

` Long `

are replaced with ` ULong `

, from Java sources with a root torn out ` java.lang.Long.bitCount `

and rewritten as an extension function for ` ULong `

` ````
fun ULong.bitCount (): Int {
var i = this
i = i - (i.shr (1) and 0x5555555555555555uL)
i = (i and 0x3333333333333333uL) + (i.shr (2) and 0x3333333333333333uL)
i = i + i.shr (4) and 0x0f0f0f0f0f0f0f0fuL
i = i + i.shr (8)
i = i + i.shr (16)
i = i + i.shr (32)
return i.toInt () and 0x7f
}
```

Run and ... Something is wrong. The code began to work much slower. We start the profiler and see the strange (see the headline illustration of the article): slightly less than a million

` bitCount () `

calls almost 16 million ` Kotlin.ULong.constructor-impl `

calls. WAT !? The riddle is simply explained - just look at the code of the

` ULong `

class ` ````
public inline class ULong @PublishedApi internal constructor (@PublishedApi internal val data: Long): Comparable & lt; ULong & gt; {
@ kotlin.internal.InlineOnly
public inline operator fun plus (other: ULong): ULong = ULong (this.data.plus (other.data))
@ kotlin.internal.InlineOnly
public inline operator fun minus (other: ULong): ULong = ULong (this.data.minus (other.data))
@ kotlin.internal.InlineOnly
public inline infix fun shl (bitCount: Int): ULong = ULong (data shl bitCount)
@ kotlin.internal.InlineOnly
public inline operator fun inc (): ULong = ULong (data.inc ())
etc.
}
```

No, I understand everything,

` ULong `

is now ` experimental `

, but how is that !? In general, we recognize the approach failed, which is a pity.

Well, still, can something else be improved?

In fact, you can. The original code for

` java.lang.Long.bitCount `

is not the most optimal. It gives a good result in the general case, but if we know in advance on which processors our application will work, then we can choose a more optimal way - this is a very good article about this on Habré Carefully counting single bits , I highly recommend reading. I used the "Combined Method"

` ````
fun Long.bitCount (): Int {
var n = this
n - = (n.shr (1)) and 0x5555555555555555L
n = ((n.shr (2)) and 0x3333333333333333L) + (n and 0x3333333333333333L)
n = ((((n.shr (4)) + n) and 0x0F0F0F0F0F0F0F0FL) * 0x010101010101010101) .shr (56)
return n.toInt () and 0x7F
}
```

All measurements were made haphazardly, in the course of development on a local machine and reproduced from memory, so it’s difficult to speak about any accuracy, but it’s possible to estimate the approximate contribution of each approach.

What did you do | parrots (seconds) |
---|---|

Approach the first, naive | 25 ± |

Second approach, pack | - |

Approach three, re-pack with a sub-subvert | 11-14 |

Fourth, large sieve approach | 2-3 |

Fifth approach, get rid of sequence | 1.8-2.2 |

Approach the sixth, like the best | 3-4 |

"Combined Method" for calculating the set bits | 1.5-1.7 |

- When dealing with processing a large amount of data, it is worth spending time on a preliminary analysis. Perhaps not all of this data needs to be processed.
- If you have the option of using coarse but cheap pre-filtering, this can help a lot.
- Bit magic is our everything. Well, where it applies, of course.
- Seeing the source code for standard classes and functions is sometimes very useful.

Thanks for attention!:)

And yes, to be continued.

Source text: About bit counting, unsigned types in Kotlin and about situations when saving on matches is justified