Those who ran through the hadoop cluster understood that k-means, on a single plane, ran into the basics of tb-grade data. A question was raised as to why spark was not used, when the project was in 2014 and the clusters were the hadoop family and could only be achieved on mapreduce. Get k-means up hereDismantling the two steps of the iterative cluster -- counting distance, renewing the body -- into the map and reduce phasesI don't know. The idea is not complicated, but the details of the project are quite varied, and they go hand in hand with the projects i have done。
I don't know
The map phase is very straightforward: each data point calculates the distance to the current level of the whole of the massTake the latest quality id as the outputkeyAnd the data points themselves are written when the value is written. Note that one point here produces only one output, not the k bar, so map's output equals input volume and does not swell. Reduce ends are grouped by phid, and after all points of the group are collected, the dimension coordinates are calculated and new phc coordinates are obtained. One round of mapreduce is just one k-means iterative round, with a clear logic but a large volume of hdfs ios, which is the main bottleneck。
There's an easy-to-neglect pit when you actually run: the mass k is large, each point in the map is counted at k, and cpu costs are doubled. Our project k = 256, with 500 million pieces of data, has been running for more than 40 minutes on a single round of map, where reading and writing of hdfs takes half of the time. And then..Heart coordinates broadcast through distributed cache to each maptask local memoryThe cost of reading every round of the hdfs was saved, and the effects were immediately felt by single-wheeling to 12 minutes. There are hundreds of kbs in total, and the cost of broadcasting is almost zero。
Kmeans rounds mapleduce
The number of condensed wheels depends mainly on the distribution of data and on the beginning of the quality. The data are clean, the opening values are reasonable and the three or five rounds are stable; the multi-modular distribution or the opening values are in bad position, and it is normal that more than a dozen rounds are not constricted. The mapreduce framework itself has no built-in termination conditions and has to write circular logic in driver: every round of reduce runsTake the old and the new and the old and the average root of the mass moving distance, and stop at less than the threshold, otherwise write back the new cylindrical sequence to the hdfs for the next round. The threshold has been set with experience, with 0. 01 for our project, which is basically five to seven rounds。
Another point that can easily be overlooked is the strict symbiotic dependence between the wheel and the wheel, and the qualitative heart of round n+1 depends on the output of round nreduce and cannot cross the wheel in parallel. The total length of the algorithm is that of a round by single wheel, which, in order to increase speed, can only be reduced by a single wheel or the meaningless number of integers can be cut off without any other structural shortcut。
Run down the whole project. Kmeans mapreduce's principle is clearSplit it down into multiple rounds of mapreduce, in parallel and inter-wheelI don't know. There are two things that need to be looked at in the project: how the quality is broadcast and how the threshold is set. These two points are fine-tuned and the overall performance gap can be two or three times higher, so don't underestimate the details。
