Skip to content

Latest commit

 

History

History
74 lines (47 loc) · 2.78 KB

CHANGELOG.textile

File metadata and controls

74 lines (47 loc) · 2.78 KB

Wukong v1.5.4

  • EMR support now works very well

Wukong v1.5.3

  • A couple of bugfixes. Sorry about that.
  • Documentation fixes

Wukong v1.5.0

Elastic Map-Reduce

Use —run=emr to launch a job onto the Amazon Elastic MapReduce cloud.

  • copies the script to s3, as foo-mapper.rb and foo-reducer.rb (removing the need for the —map flag)
  • copies the wukong libs up as a .tar/bz2, and extracts it into the cwd
  • combines settings from commandline and yaml config, etc to configure and launch job

It’s still way shaky and I don’t think anything but the sample app will run. That sample app runs, tho.

Greatly simplified script launching.

Incompatible changes to option handling and script launching:

  • Script doesn’t use extra_options any more. You should relocate them to the initializer or to configliere.
  • there is no more default_mapper or default_reducer

Wukong v.14.12 2010-08-31

  • Improvements to the pig conversion methods
  • hdp-rm respects the -skipTrash method

Wukong v1.4.11 2010-07-30

  • added the max_(maps|reduces)_per_(node|cluster) jobconfs.
  • added jobconfs for io_job_mb and friends.
  • added a loadable module to convert output data to pig bags and tuples
  • pulled in several methods from active_support, incl. Enumerable#sum
  • Scripts to find percentile rank of elements in a dataset
  • We are starting to move wukong to a model where streaming is from a generic
    source into a generic sink. Several stores have been landed in the code, but
    many are in a half- or un-baked state. Please ignore this for the moment.

Wukong v1.4.8 2010-06-05

  • made scripts inject a helpful job name using mapred.job.name
  • Hash.compact_blank! and HashLike.compact_blank! — eliminate all key-values whoes value is blank?

Wukong v1.4.8 2010-05-17

  • Bug in passing commandline args down to map and reduce child processes

Wukong v1.4.7 2010-03-05

Lots more examples:

  • examples/stats/avg_value_frequency.rb does an Average Value Frequency histogram
  • examples/server_logs has a quite useful apache log file parser
  • Made the base streamer use each_record, opening the door for alternative record injection (eg Datamapper!)
  • wukong/streamer/counting_reducer.rb is an um reducer and it counts things.

Wukong v1.4.6 2010-01-26

  • A HELLA AWESOME working example from retail web analytics by @lenbust

Wukong v1.4.5 2010-01-18

  • In --run=local mode, you can use ‘-’ alone as a filename to indicate STDIN / STDOUT as input/output respectively.
  • Minor tweaks to contrib/jeans

Wukong v1.4.4 2010-01-15