Sampling Pig has a built-in SAMPLE operator that performs Bernoulli sampling on a relation. Apache DataFu Pig provides additional sampling techniques for when Bernoulli sampling is not applicable. …

Sampling: simple random sample with/without replacement, weighted sample, sample by keys Hashing: SHA and MD5 Link Analysis: PageRank Assorted Macros: deduplication of tables, human-readable …

It takes a bag of n items and a sampling probability p as the inputs, and outputs a simple random sample of size exactly ceil (p*n) in a bag, with probability at least 99.99%. For example, the following script …

Scalable simple random sampling with replacement (ScaSRSWR). This UDF together with SimpleRandomSampleWithReplacementElect implement a scalable algorithm for simple random …

Sampling Simple random sampling with or without replacement, weighted sampling. Link Analysis Run PageRank on a graph represented by a bag of nodes and edges. More Other useful methods like …

Package datafu.pig.sampling Description Sampling UDFs, including weighted sample, reservoir sampling, sampling by key, etc.

Sampling UDFs, including weighted sample, reservoir sampling, sampling by key, etc.

The method of sampling is to convert the key to a hash, derive a double value from this, and then test this against a supplied probability. The double value derived from a key is uniformly distributed …

For example, DEFINE SRS datafu.pig.sampling.SimpleRandomSample ('0.01'); examples = LOAD ... grouped = GROUP examples BY label; sampled = FOREACH grouped GENERATE FLATTEN (SRS …

Class Summary Class Description ReservoirSample Performs a simple random sample using an in-memory reservoir to produce a uniformly random sample of a given size. ReservoirSample.Final …