forked from ekoontz/hadoop-common
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathCHANGES.txt
10011 lines (6864 loc) · 391 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop Change Log
Trunk (unreleased changes)
BUG FIXES
HADOOP-6730. Bug in FileContext#copy and provide base class for FileContext
tests. (Ravi Phulari via jghoman)
Release 0.21.0 - Unreleased
INCOMPATIBLE CHANGES
HADOOP-4895. Remove deprecated methods DFSClient.getHints(..) and
DFSClient.isDirectory(..). (szetszwo)
HADOOP-4941. Remove deprecated FileSystem methods: getBlockSize(Path f),
getLength(Path f) and getReplication(Path src). (szetszwo)
HADOOP-4648. Remove obsolete, deprecated InMemoryFileSystem and
ChecksumDistributedFileSystem. (cdouglas via szetszwo)
HADOOP-4940. Remove a deprecated method FileSystem.delete(Path f). (Enis
Soztutar via szetszwo)
HADOOP-4010. Change semantics for LineRecordReader to read an additional
line per split- rather than moving back one character in the stream- to
work with splittable compression codecs. (Abdul Qadeer via cdouglas)
HADOOP-5094. Show hostname and separate live/dead datanodes in DFSAdmin
report. (Jakob Homan via szetszwo)
HADOOP-4942. Remove deprecated FileSystem methods getName() and
getNamed(String name, Configuration conf). (Jakob Homan via szetszwo)
HADOOP-5486. Removes the CLASSPATH string from the command line and instead
exports it in the environment. (Amareshwari Sriramadasu via ddas)
HADOOP-2827. Remove deprecated NetUtils::getServerAddress. (cdouglas)
HADOOP-5681. Change examples RandomWriter and RandomTextWriter to
use new mapreduce API. (Amareshwari Sriramadasu via sharad)
HADOOP-5680. Change org.apache.hadoop.examples.SleepJob to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5699. Change org.apache.hadoop.examples.PiEstimator to use
new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5720. Introduces new task types - JOB_SETUP, JOB_CLEANUP
and TASK_CLEANUP. Removes the isMap methods from TaskID/TaskAttemptID
classes. (ddas)
HADOOP-5668. Change TotalOrderPartitioner to use new API. (Amareshwari
Sriramadasu via cdouglas)
HADOOP-5738. Split "waiting_tasks" JobTracker metric into waiting maps and
waiting reduces. (Sreekanth Ramakrishnan via cdouglas)
HADOOP-5679. Resolve findbugs warnings in core/streaming/pipes/examples.
(Jothi Padmanabhan via sharad)
HADOOP-4359. Support for data access authorization checking on Datanodes.
(Kan Zhang via rangadi)
HADOOP-5690. Change org.apache.hadoop.examples.DBCountPageView to use
new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5694. Change org.apache.hadoop.examples.dancing to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5696. Change org.apache.hadoop.examples.Sort to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5698. Change org.apache.hadoop.examples.MultiFileWordCount to
use new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5913. Provide ability to an administrator to stop and start
job queues. (Rahul Kumar Singh and Hemanth Yamijala via yhemanth)
MAPREDUCE-711. Removed Distributed Cache from Common, to move it
under Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6201. Change FileSystem::listStatus contract to throw
FileNotFoundException if the directory does not exist, rather than letting
this be implementation-specific. (Jakob Homan via cdouglas)
HADOOP-6230. Moved process tree and memory calculator related classes
from Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6203. FsShell rm/rmr error message indicates exceeding Trash quota
and suggests using -skpTrash, when moving to trash fails.
(Boris Shkolnik via suresh)
HADOOP-6303. Eclipse .classpath template has outdated jar files and is
missing some new ones. (cos)
HADOOP-6396. Fix uninformative exception message when unable to parse
umask. (jghoman)
HADOOP-6299. Reimplement the UserGroupInformation to use the OS
specific and Kerberos JAAS login. (omalley)
HADOOP-6686. Remove redundant exception class name from the exception
message for the exceptions thrown at RPC client. (suresh)
HADOOP-6701. Fix incorrect exit codes returned from chmod, chown and chgrp
commands from FsShell. (Ravi Phulari via suresh)
NEW FEATURES
HADOOP-4268. Change fsck to use ClientProtocol methods so that the
corresponding permission requirement for running the ClientProtocol
methods will be enforced. (szetszwo)
HADOOP-3953. Implement sticky bit for directories in HDFS. (Jakob Homan
via szetszwo)
HADOOP-4368. Implement df in FsShell to show the status of a FileSystem.
(Craig Macdonald via szetszwo)
HADOOP-3741. Add a web ui to the SecondaryNameNode for showing its status.
(szetszwo)
HADOOP-5018. Add pipelined writers to Chukwa. (Ari Rabkin via cdouglas)
HADOOP-5052. Add an example computing exact digits of pi using the
Bailey-Borwein-Plouffe algorithm. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-4927. Adds a generic wrapper around outputformat to allow creation of
output on demand (Jothi Padmanabhan via ddas)
HADOOP-5144. Add a new DFSAdmin command for changing the setting of restore
failed storage replicas in namenode. (Boris Shkolnik via szetszwo)
HADOOP-5258. Add a new DFSAdmin command to print a tree of the rack and
datanode topology as seen by the namenode. (Jakob Homan via szetszwo)
HADOOP-4756. A command line tool to access JMX properties on NameNode
and DataNode. (Boris Shkolnik via rangadi)
HADOOP-4539. Introduce backup node and checkpoint node. (shv)
HADOOP-5363. Add support for proxying connections to multiple clusters with
different versions to hdfsproxy. (Zhiyong Zhang via cdouglas)
HADOOP-5528. Add a configurable hash partitioner operating on ranges of
BinaryComparable keys. (Klaas Bosteels via shv)
HADOOP-5257. HDFS servers may start and stop external components through
a plugin interface. (Carlos Valiente via dhruba)
HADOOP-5450. Add application-specific data types to streaming's typed bytes
interface. (Klaas Bosteels via omalley)
HADOOP-5518. Add contrib/mrunit, a MapReduce unit test framework.
(Aaron Kimball via cutting)
HADOOP-5469. Add /metrics servlet to daemons, providing metrics
over HTTP as either text or JSON. (Philip Zeyliger via cutting)
HADOOP-5467. Introduce offline fsimage image viewer. (Jakob Homan via shv)
HADOOP-5752. Add a new hdfs image processor, Delimited, to oiv. (Jakob
Homan via szetszwo)
HADOOP-5266. Adds the capability to do mark/reset of the reduce values
iterator in the Context object API. (Jothi Padmanabhan via ddas)
HADOOP-5745. Allow setting the default value of maxRunningJobs for all
pools. (dhruba via matei)
HADOOP-5643. Adds a way to decommission TaskTrackers while the JobTracker
is running. (Amar Kamat via ddas)
HADOOP-4829. Allow FileSystem shutdown hook to be disabled.
(Todd Lipcon via tomwhite)
HADOOP-5815. Sqoop: A database import tool for Hadoop.
(Aaron Kimball via tomwhite)
HADOOP-4861. Add disk usage with human-readable size (-duh).
(Todd Lipcon via tomwhite)
HADOOP-5844. Use mysqldump when connecting to local mysql instance in Sqoop.
(Aaron Kimball via tomwhite)
HADOOP-5976. Add a new command, classpath, to the hadoop script. (Owen
O'Malley and Gary Murry via szetszwo)
HADOOP-6120. Add support for Avro specific and reflect data.
(sharad via cutting)
HADOOP-6226. Moves BoundedByteArrayOutputStream from the tfile package to
the io package and makes it available to other users (MAPREDUCE-318).
(Jothi Padmanabhan via ddas)
HADOOP-6105. Adds support for automatically handling deprecation of
configuration keys. (V.V.Chaitanya Krishna via yhemanth)
HADOOP-6235. Adds new method to FileSystem for clients to get server
defaults. (Kan Zhang via suresh)
HADOOP-6234. Add new option dfs.umaskmode to set umask in configuration
to use octal or symbolic instead of decimal. (Jakob Homan via suresh)
HADOOP-5073. Add annotation mechanism for interface classification.
(Jakob Homan via suresh)
HADOOP-4012. Provide splitting support for bzip2 compressed files. (Abdul
Qadeer via cdouglas)
HADOOP-6246. Add backward compatibility support to use deprecated decimal
umask from old configuration. (Jakob Homan via suresh)
HADOOP-4952. Add new improved file system interface FileContext for the
application writer (Sanjay Radia via suresh)
HADOOP-6170. Add facility to tunnel Avro RPCs through Hadoop RPCs.
This permits one to take advantage of both Avro's RPC versioning
features and Hadoop's proven RPC scalability. (cutting)
HADOOP-6267. Permit building contrib modules located in external
source trees. (Todd Lipcon via cutting)
HADOOP-6240. Add new FileContext rename operation that posix compliant
that allows overwriting existing destination. (suresh)
HADOOP-6204. Implementing aspects development and fault injeciton
framework for Hadoop (cos)
HADOOP-6313. Implement Syncable interface in FSDataOutputStream to expose
flush APIs to application users. (Hairong Kuang via suresh)
HADOOP-6284. Add a new parameter, HADOOP_JAVA_PLATFORM_OPTS, to
hadoop-config.sh so that it allows setting java command options for
JAVA_PLATFORM. (Koji Noguchi via szetszwo)
HADOOP-6337. Updates FilterInitializer class to be more visible,
and the init of the class is made to take a Configuration argument.
(Jakob Homan via ddas)
HADOOP-6108. Add support for EBS storage on EC2. (tomwhite)
Hadoop-6223. Add new file system interface AbstractFileSystem with
implementation of some file systems that delegate to old FileSystem.
(Sanjay Radia via suresh)
HADOOP-6392. Run namenode and jobtracker on separate EC2 instances.
(tomwhite)
HADOOP-6433. Introduce asychronous deletion of files via a pool of
threads. This can be used to delete files in the Distributed
Cache. (Zheng Shao via dhruba)
HADOOP-6415. Adds a common token interface for both job token and
delegation token. (Kan Zhang via ddas)
HADOOP-6466. Add a ZooKeeper service to the cloud scripts. (tomwhite)
HADOOP-6408. Add a /conf servlet to dump running configuration.
(Todd Lipcon via tomwhite)
HADOOP-6464. Write a Rackspace cloud provider. (tomwhite)
HADOOP-6520. Adds APIs to read/write Token and secret keys. Also
adds the automatic loading of tokens into UserGroupInformation
upon login. The tokens are read from a file specified in the
environment variable. (ddas)
HADOOP-6419. Adds SASL based authentication to RPC.
(Kan Zhang via ddas)
HADOOP-6510. Adds a way for superusers to impersonate other users
in a secure environment. (Jitendra Nath Pandey via ddas)
HADOOP-6421. Adds Symbolic links to FileContext, AbstractFileSystem.
It also adds a limited implementation for the local file system
(RawLocalFs) that allows local symlinks. (Eli Collins via Sanjay Radia)
HADOOP-6577. Add hidden configuration option "ipc.server.max.response.size"
to change the default 1 MB, the maximum size when large IPC handler
response buffer is reset. (suresh)
HADOOP-6568. Adds authorization for the default servlets.
(Vinod Kumar Vavilapalli via ddas)
HADOOP-6586. Log authentication and authorization failures and successes
for RPC (boryas)
HADOOP-6580. UGI should contain authentication method. (jnp via boryas)
HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545.
(Luke Lu via Steve Loughran)
HADOOP-6692. Add FileContext#listStatus that returns an iterator.
(hairong)
IMPROVEMENTS
HADOOP-4565. Added CombineFileInputFormat to use data locality information
to create splits. (dhruba via zshao)
HADOOP-4936. Improvements to TestSafeMode. (shv)
HADOOP-4985. Remove unnecessary "throw IOException" declarations in
FSDirectory related methods. (szetszwo)
HADOOP-5017. Change NameNode.namesystem declaration to private. (szetszwo)
HADOOP-4794. Add branch information from the source version control into
the version information that is compiled into Hadoop. (cdouglas via
omalley)
HADOOP-5070. Increment copyright year to 2009, remove assertions of ASF
copyright to licensed files. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-5037. Deprecate static FSNamesystem.getFSNamesystem(). (szetszwo)
HADOOP-5088. Include releaseaudit target as part of developer test-patch
target. (Giridharan Kesavan via nigel)
HADOOP-2721. Uses setsid when creating new tasks so that subprocesses of
this process will be within this new session (and this process will be
the process leader for all the subprocesses). Killing the process leader,
or the main Java task in Hadoop's case, kills the entire subtree of
processes. (Ravi Gummadi via ddas)
HADOOP-5097. Remove static variable JspHelper.fsn, a static reference to
a non-singleton FSNamesystem object. (szetszwo)
HADOOP-3327. Improves handling of READ_TIMEOUT during map output copying.
(Amareshwari Sriramadasu via ddas)
HADOOP-5124. Choose datanodes randomly instead of starting from the first
datanode for providing fairness. (hairong via szetszwo)
HADOOP-4930. Implement a Linux native executable that can be used to
launch tasks as users. (Sreekanth Ramakrishnan via yhemanth)
HADOOP-5122. Fix format of fs.default.name value in libhdfs test conf.
(Craig Macdonald via tomwhite)
HADOOP-5038. Direct daemon trace to debug log instead of stdout. (Jerome
Boulon via cdouglas)
HADOOP-5101. Improve packaging by adding 'all-jars' target building core,
tools, and example jars. Let findbugs depend on this rather than the 'tar'
target. (Giridharan Kesavan via cdouglas)
HADOOP-4868. Splits the hadoop script into three parts - bin/hadoop,
bin/mapred and bin/hdfs. (Sharad Agarwal via ddas)
HADOOP-1722. Adds support for TypedBytes and RawBytes in Streaming.
(Klaas Bosteels via ddas)
HADOOP-4220. Changes the JobTracker restart tests so that they take much
less time. (Amar Kamat via ddas)
HADOOP-4885. Try to restore failed name-node storage directories at
checkpoint time. (Boris Shkolnik via shv)
HADOOP-5209. Update year to 2009 for javadoc. (szetszwo)
HADOOP-5279. Remove unnecessary targets from test-patch.sh.
(Giridharan Kesavan via nigel)
HADOOP-5120. Remove the use of FSNamesystem.getFSNamesystem() from
UpgradeManagerNamenode and UpgradeObjectNamenode. (szetszwo)
HADOOP-5222. Add offset to datanode clienttrace. (Lei Xu via cdouglas)
HADOOP-5240. Skip re-building javadoc when it is already
up-to-date. (Aaron Kimball via cutting)
HADOOP-5042. Add a cleanup stage to log rollover in Chukwa appender.
(Jerome Boulon via cdouglas)
HADOOP-5264. Removes redundant configuration object from the TaskTracker.
(Sharad Agarwal via ddas)
HADOOP-5232. Enable patch testing to occur on more than one host.
(Giri Kesavan via nigel)
HADOOP-4546. Fix DF reporting for AIX. (Bill Habermaas via cdouglas)
HADOOP-5023. Add Tomcat support to HdfsProxy. (Zhiyong Zhang via cdouglas)
HADOOP-5317. Provide documentation for LazyOutput Feature.
(Jothi Padmanabhan via johan)
HADOOP-5455. Document rpc metrics context to the extent dfs, mapred, and
jvm contexts are documented. (Philip Zeyliger via cdouglas)
HADOOP-5358. Provide scripting functionality to the synthetic load
generator. (Jakob Homan via hairong)
HADOOP-5442. Paginate jobhistory display and added some search
capabilities. (Amar Kamat via acmurthy)
HADOOP-4842. Streaming now allows specifiying a command for the combiner.
(Amareshwari Sriramadasu via ddas)
HADOOP-5196. avoiding unnecessary byte[] allocation in
SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes.
(hong tang via mahadev)
HADOOP-4655. New method FileSystem.newInstance() that always returns
a newly allocated FileSystem object. (dhruba)
HADOOP-4788. Set Fair scheduler to assign both a map and a reduce on each
heartbeat by default. (matei)
HADOOP-5491. In contrib/index, better control memory usage.
(Ning Li via cutting)
HADOOP-5423. Include option of preserving file metadata in
SequenceFile::sort. (Michael Tamm via cdouglas)
HADOOP-5331. Add support for KFS appends. (Sriram Rao via cdouglas)
HADOOP-4365. Make Configuration::getProps protected in support of
meaningful subclassing. (Steve Loughran via cdouglas)
HADOOP-2413. Remove the static variable FSNamesystem.fsNamesystemObject.
(Konstantin Shvachko via szetszwo)
HADOOP-4584. Improve datanode block reports and associated file system
scan to avoid interefering with normal datanode operations.
(Suresh Srinivas via rangadi)
HADOOP-5502. Documentation for backup and checkpoint nodes.
(Jakob Homan via shv)
HADOOP-5485. Mask actions in the fair scheduler's servlet UI based on
value of webinterface.private.actions.
(Vinod Kumar Vavilapalli via yhemanth)
HADOOP-5581. HDFS should throw FileNotFoundException when while opening
a file that does not exist. (Brian Bockelman via rangadi)
HADOOP-5509. PendingReplicationBlocks does not start monitor in the
constructor. (shv)
HADOOP-5494. Modify sorted map output merger to lazily read values,
rather than buffering at least one record for each segment. (Devaraj Das
via cdouglas)
HADOOP-5396. Provide ability to refresh queue ACLs in the JobTracker
without having to restart the daemon.
(Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
HADOOP-4490. Provide ability to run tasks as job owners.
(Sreekanth Ramakrishnan via yhemanth)
HADOOP-5697. Change org.apache.hadoop.examples.Grep to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5625. Add operation duration to clienttrace. (Lei Xu via cdouglas)
HADOOP-5705. Improve TotalOrderPartitioner efficiency by updating the trie
construction. (Dick King via cdouglas)
HADOOP-5589. Eliminate source limit of 64 for map-side joins imposed by
TupleWritable encoding. (Jingkei Ly via cdouglas)
HADOOP-5734. Correct block placement policy description in HDFS
Design document. (Konstantin Boudnik via shv)
HADOOP-5657. Validate data in TestReduceFetch to improve merge test
coverage. (cdouglas)
HADOOP-5613. Change S3Exception to checked exception.
(Andrew Hitchcock via tomwhite)
HADOOP-5717. Create public enum class for the Framework counters in
org.apache.hadoop.mapreduce. (Amareshwari Sriramadasu via sharad)
HADOOP-5217. Split AllTestDriver for core, hdfs and mapred. (sharad)
HADOOP-5364. Add certificate expiration warning to HsftpFileSystem and HDFS
proxy. (Zhiyong Zhang via cdouglas)
HADOOP-5733. Add map/reduce slot capacity and blacklisted capacity to
JobTracker metrics. (Sreekanth Ramakrishnan via cdouglas)
HADOOP-5596. Add EnumSetWritable. (He Yongqiang via szetszwo)
HADOOP-5727. Simplify hashcode for ID types. (Shevek via cdouglas)
HADOOP-5500. In DBOutputFormat, where field names are absent permit the
number of fields to be sufficient to construct the select query. (Enis
Soztutar via cdouglas)
HADOOP-5081. Split TestCLI into HDFS, Mapred and Core tests. (sharad)
HADOOP-5015. Separate block management code from FSNamesystem. (Suresh
Srinivas via szetszwo)
HADOOP-5080. Add new test cases to TestMRCLI and TestHDFSCLI
(V.Karthikeyan via nigel)
HADOOP-5135. Splits the tests into different directories based on the
package. Four new test targets have been defined - run-test-core,
run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr.
(Sharad Agarwal via ddas)
HADOOP-5771. Implements unit tests for LinuxTaskController.
(Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
HADOOP-5419. Provide a facility to query the Queue ACLs for the
current user.
(Rahul Kumar Singh via yhemanth)
HADOOP-5780. Improve per block message prited by "-metaSave" in HDFS.
(Raghu Angadi)
HADOOP-5823. Added a new class DeprecatedUTF8 to help with removing
UTF8 related javac warnings. These warnings are removed in
FSEditLog.java as a use case. (Raghu Angadi)
HADOOP-5824. Deprecate DataTransferProtocol.OP_READ_METADATA and remove
the corresponding unused codes. (Kan Zhang via szetszwo)
HADOOP-5721. Factor out EditLogFileInputStream and EditLogFileOutputStream
into independent classes. (Luca Telloli & Flavio Junqueira via shv)
HADOOP-5838. Fix a few javac warnings in HDFS. (Raghu Angadi)
HADOOP-5854. Fix a few "Inconsistent Synchronization" warnings in HDFS.
(Raghu Angadi)
HADOOP-5369. Small tweaks to reduce MapFile index size. (Ben Maurer
via sharad)
HADOOP-5858. Eliminate UTF8 and fix warnings in test/hdfs-with-mr package.
(shv)
HADOOP-5866. Move DeprecatedUTF8 from o.a.h.io to o.a.h.hdfs since it may
not be used outside hdfs. (Raghu Angadi)
HADOOP-5857. Move normal java methods from hdfs .jsp files to .java files.
(szetszwo)
HADOOP-5873. Remove deprecated methods randomDataNode() and
getDatanodeByIndex(..) in FSNamesystem. (szetszwo)
HADOOP-5572. Improves the progress reporting for the sort phase for both
maps and reduces. (Ravi Gummadi via ddas)
HADOOP-5839. Fix EC2 scripts to allow remote job submission.
(Joydeep Sen Sarma via tomwhite)
HADOOP-5877. Fix javac warnings in TestHDFSServerPorts, TestCheckpoint,
TestNameEditsConfig, TestStartup and TestStorageRestore.
(Jakob Homan via shv)
HADOOP-5438. Provide a single FileSystem method to create or
open-for-append to a file. (He Yongqiang via dhruba)
HADOOP-5472. Change DistCp to support globbing of input paths. (Dhruba
Borthakur and Rodrigo Schmidt via szetszwo)
HADOOP-5175. Don't unpack libjars on classpath. (Todd Lipcon via tomwhite)
HADOOP-5620. Add an option to DistCp for preserving modification and access
times. (Rodrigo Schmidt via szetszwo)
HADOOP-5664. Change map serialization so a lock is obtained only where
contention is possible, rather than for each write. (cdouglas)
HADOOP-5896. Remove the dependency of GenericOptionsParser on
Option.withArgPattern. (Giridharan Kesavan and Sharad Agarwal via
sharad)
HADOOP-5784. Makes the number of heartbeats that should arrive a second
at the JobTracker configurable. (Amareshwari Sriramadasu via ddas)
HADOOP-5955. Changes TestFileOuputFormat so that is uses LOCAL_MR
instead of CLUSTER_MR. (Jothi Padmanabhan via das)
HADOOP-5948. Changes TestJavaSerialization to use LocalJobRunner
instead of MiniMR/DFS cluster. (Jothi Padmanabhan via das)
HADOOP-2838. Add mapred.child.env to pass environment variables to
tasktracker's child processes. (Amar Kamat via sharad)
HADOOP-5961. DataNode process understand generic hadoop command line
options (like -Ddfs.property=value). (Raghu Angadi)
HADOOP-5938. Change org.apache.hadoop.mapred.jobcontrol to use new
api. (Amareshwari Sriramadasu via sharad)
HADOOP-2141. Improves the speculative execution heuristic. The heuristic
is currently based on the progress-rates of tasks and the expected time
to complete. Also, statistics about trackers are collected, and speculative
tasks are not given to the ones deduced to be slow.
(Andy Konwinski and ddas)
HADOOP-5952. Change "-1 tests included" wording in test-patch.sh.
(Gary Murry via szetszwo)
HADOOP-6106. Provides an option in ShellCommandExecutor to timeout
commands that do not complete within a certain amount of time.
(Sreekanth Ramakrishnan via yhemanth)
HADOOP-5925. EC2 scripts should exit on error. (tomwhite)
HADOOP-6109. Change Text to grow its internal buffer exponentially, rather
than the max of the current length and the proposed length to improve
performance reading large values. (thushara wijeratna via cdouglas)
HADOOP-2366. Support trimmed strings in Configuration. (Michele Catasta
via szetszwo)
HADOOP-6099. The RPC module can be configured to not send period pings.
The default behaviour of sending periodic pings remain unchanged. (dhruba)
HADOOP-6142. Update documentation and use of harchives for relative paths
added in MAPREDUCE-739. (Mahadev Konar via cdouglas)
HADOOP-6148. Implement a fast, pure Java CRC32 calculator which outperforms
java.util.zip.CRC32. (Todd Lipcon and Scott Carey via szetszwo)
HADOOP-6146. Upgrade to JetS3t version 0.7.1. (tomwhite)
HADOOP-6161. Add get/setEnum methods to Configuration. (cdouglas)
HADOOP-6160. Fix releaseaudit target to run on specific directories.
(gkesavan)
HADOOP-6169. Removing deprecated method calls in TFile. (hong tang via
mahadev)
HADOOP-6176. Add a couple package private methods to AccessTokenHandler
for testing. (Kan Zhang via szetszwo)
HADOOP-6182. Fix ReleaseAudit warnings (Giridharan Kesavan and Lee Tucker
via gkesavan)
HADOOP-6173. Change src/native/packageNativeHadoop.sh to package all
native library files. (Hong Tang via szetszwo)
HADOOP-6184. Provide an API to dump Configuration in a JSON format.
(V.V.Chaitanya Krishna via yhemanth)
HADOOP-6224. Add a method to WritableUtils performing a bounded read of an
encoded String. (Jothi Padmanabhan via cdouglas)
HADOOP-6133. Add a caching layer to Configuration::getClassByName to
alleviate a performance regression introduced in a compatibility layer.
(Todd Lipcon via cdouglas)
HADOOP-6252. Provide a method to determine if a deprecated key is set in
config file. (Jakob Homan via suresh)
HADOOP-5879. Read compression level and strategy from Configuration for
gzip compression. (He Yongqiang via cdouglas)
HADOOP-6216. Support comments in host files. (Ravi Phulari and Dmytro
Molkov via szetszwo)
HADOOP-6217. Update documentation for project split. (Corinne Chandel via
omalley)
HADOOP-6268. Add ivy jar to .gitignore. (Todd Lipcon via cdouglas)
HADOOP-6270. Support deleteOnExit in FileContext. (Suresh Srinivas via
szetszwo)
HADOOP-6233. Rename configuration keys towards API standardization and
backward compatibility. (Jithendra Pandey via suresh)
HADOOP-6260. Add additional unit tests for FileContext util methods.
(Gary Murry via suresh).
HADOOP-6309. Change build.xml to run tests with java asserts. (Eli
Collins via szetszwo)
HADOOP-6326. Hundson runs should check for AspectJ warnings and report
failure if any is present (cos)
HADOOP-6329. Add build-fi directory to the ignore lists. (szetszwo)
HADOOP-5107. Use Maven ant tasks to publish the subproject jars.
(Giridharan Kesavan via omalley)
HADOOP-6343. Log unexpected throwable object caught in RPC. (Jitendra Nath
Pandey via szetszwo)
HADOOP-6367. Removes Access Token implementation from common.
(Kan Zhang via ddas)
HADOOP-6395. Upgrade some libraries to be consistent across common, hdfs,
and mapreduce. (omalley)
HADOOP-6398. Build is broken after HADOOP-6395 patch has been applied (cos)
HADOOP-6413. Move TestReflectionUtils to Common. (Todd Lipcon via tomwhite)
HADOOP-6283. Improve the exception messages thrown by
FileUtil$HardLink.getLinkCount(..). (szetszwo)
HADOOP-6279. Add Runtime::maxMemory to JVM metrics. (Todd Lipcon via
cdouglas)
HADOOP-6305. Unify build property names to facilitate cross-projects
modifications (cos)
HADOOP-6312. Remove unnecessary debug logging in Configuration constructor.
(Aaron Kimball via cdouglas)
HADOOP-6366. Reduce ivy console output to ovservable level (cos)
HADOOP-6400. Log errors getting Unix UGI. (Todd Lipcon via tomwhite)
HADOOP-6346. Add support for specifying unpack pattern regex to
RunJar.unJar. (Todd Lipcon via tomwhite)
HADOOP-6422. Make RPC backend plugable, protocol-by-protocol, to
ease evolution towards Avro. (cutting)
HADOOP-5958. Use JDK 1.6 File APIs in DF.java wherever possible.
(Aaron Kimball via tomwhite)
HADOOP-6222. Core doesn't have TestCommonCLI facility. (cos)
HADOOP-6394. Add a helper class to simplify FileContext related tests and
improve code reusability. (Jitendra Nath Pandey via suresh)
HADOOP-6426. Create ant build for running EC2 unit tests. (tomwhite)
HADOOP-4656. Add a user to groups mapping service. (boryas, acmurthy)
HADOOP-6444. Support additional security group option in hadoop-ec2 script.
(Paul Egan via tomwhite)
HADOOP-6454. Create setup.py for EC2 cloud scripts. (tomwhite)
HADOOP-6435. Make RPC.waitForProxy with timeout public. (Steve Loughran
via tomwhite)
HADOOP-6472. add tokenCache option to GenericOptionsParser for passing
file with secret keys to a map reduce job. (boryas)
HADOOP-3205. Read multiple chunks directly from FSInputChecker subclass
into user buffers. (Todd Lipcon via tomwhite)
HADOOP-6479. TestUTF8 assertions could fail with better text.
(Steve Loughran via tomwhite)
HADOOP-6155. Deprecate RecordIO anticipating Avro. (Tom White via cdouglas)
HADOOP-6492. Make some Avro serialization APIs public.
(Aaron Kimball via cutting)
HADOOP-6497. Add an adapter for Avro's SeekableInput interface, so
that Avro can read FileSystem data.
(Aaron Kimball via cutting)
HADOOP-6495. Identifier should be serialized after the password is
created In Token constructor (jnp via boryas)
HADOOP-6518. Makes the UGI honor the env var KRB5CCNAME.
(Owen O'Malley via ddas)
HADOOP-6531. Enhance FileUtil with an API to delete all contents of a
directory. (Amareshwari Sriramadasu via yhemanth)
HADOOP-6547. Move DelegationToken into Common, so that it can be used by
MapReduce also. (devaraj via omalley)
HADOOP-6552. Puts renewTGT=true and useTicketCache=true for the keytab
kerberos options. (ddas)
HADOOP-6534. Trim whitespace from directory lists initializing
LocalDirAllocator. (Todd Lipcon via cdouglas)
HADOOP-6559. Makes the RPC client automatically re-login when the SASL
connection setup fails. This is applicable only to keytab based logins.
(Devaraj Das)
HADOOP-6551. Delegation token renewing and cancelling should provide
meaningful exceptions when there are failures instead of returning
false. (omalley)
HADOOP-6583. Captures authentication and authorization metrics. (ddas)
HADOOP-6543. Allows secure clients to talk to unsecure clusters.
(Kan Zhang via ddas)
HADOOP-6579. Provide a mechanism for encoding/decoding Tokens from
a url-safe string and change the commons-code library to 1.4. (omalley)
HADOOP-6596. Add a version field to the AbstractDelegationTokenIdentifier's
serialized value. (omalley)
HADOOP-6573. Support for persistent delegation tokens.
(Jitendra Pandey via shv)
HADOOP-6594. Provide a fetchdt tool via bin/hdfs. (jhoman via acmurthy)
HADOOP-6589. Provide better error messages when RPC authentication fails.
(Kan Zhang via omalley)
HADOOP-6599 Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics.
(Suresh Srinivas via Sanjay Radia)
HADOOP-6537 Declare more detailed exceptions in FileContext and
AbstractFileSystem (Suresh Srinivas via Sanjay Radia)
HADOOP-6486. fix common classes to work with Avro 1.3 reflection.
(cutting via tomwhite)
HADOOP-6591. HarFileSystem can handle paths with the whitespace characters.
(Rodrigo Schmidt via dhruba)
HADOOP-6407. Have a way to automatically update Eclipse .classpath file
when new libs are added to the classpath through Ivy. (tomwhite)
HADOOP-3659. Patch to allow hadoop native to compile on Mac OS X.
(Colin Evans and Allen Wittenauer via tomwhite)
HADOOP-6471. StringBuffer -> StringBuilder - conversion of references
as necessary. (Kay Kay via tomwhite)
HADOOP-6646. Move HarfileSystem out of Hadoop Common. (mahadev)
HADOOP-6566. Add methods supporting, enforcing narrower permissions on
local daemon directories. (Arun Murthy and Luke Lu via cdouglas)
HADOOP-6705. Fix to work with 1.5 version of jiracli
(Giridharan Kesavan)
HADOOP-6658. Exclude Private elements from generated Javadoc. (tomwhite)
HADOOP-6635. Install/deploy source jars to Maven repo.
(Patrick Angeles via jghoman)
HADOOP-6717. Log levels in o.a.h.security.Groups too high
(Todd Lipcon via jghoman)
HADOOP-6667. RPC.waitForProxy should retry through NoRouteToHostException.
(Todd Lipcon via tomwhite)
HADOOP-6677. InterfaceAudience.LimitedPrivate should take a string not an
enum. (tomwhite)
HADOOP-678. Remove FileContext#isFile, isDirectory, and exists.
(Eli Collins via hairong)
HADOOP-6515. Make maximum number of http threads configurable.
(Scott Chen via zshao)
HADOOP-6563. Add more symlink tests to cover intermediate symlinks
in paths. (Eli Collins via suresh)
OPTIMIZATIONS
HADOOP-5595. NameNode does not need to run a replicator to choose a
random DataNode. (hairong)
HADOOP-5603. Improve NameNode's block placement performance. (hairong)
HADOOP-5638. More improvement on block placement performance. (hairong)
HADOOP-6180. NameNode slowed down when many files with same filename
were moved to Trash. (Boris Shkolnik via hairong)
HADOOP-6166. Further improve the performance of the pure-Java CRC32
implementation. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-6271. Add recursive and non recursive create and mkdir to
FileContext. (Sanjay Radia via suresh)
HADOOP-6261. Add URI based tests for FileContext.
(Ravi Pulari via suresh).
HADOOP-6307. Add a new SequenceFile.Reader constructor in order to support
reading on un-closed file. (szetszwo)
HADOOP-6467. Improve the performance on HarFileSystem.listStatus(..).
(mahadev via szetszwo)
HADOOP-6569. FsShell#cat should avoid calling unecessary getFileStatus
before opening a file to read. (hairong)
HADOOP-6689. Add directory renaming test to existing FileContext tests.
(Eli Collins via suresh)
HADOOP-6713. The RPC server Listener thread is a scalability bottleneck.
(Dmytro Molkov via hairong)
BUG FIXES
HADOOP-5379. CBZip2InputStream to throw IOException on data crc error.
(Rodrigo Schmidt via zshao)
HADOOP-5326. Fixes CBZip2OutputStream data corruption problem.
(Rodrigo Schmidt via zshao)
HADOOP-4963. Fixes a logging to do with getting the location of
map output file. (Amareshwari Sriramadasu via ddas)
HADOOP-2337. Trash should close FileSystem on exit and should not start
emtying thread if disabled. (shv)
HADOOP-5072. Fix failure in TestCodec because testSequenceFileGzipCodec
won't pass without native gzip codec. (Zheng Shao via dhruba)
HADOOP-5050. TestDFSShell.testFilePermissions should not assume umask
setting. (Jakob Homan via szetszwo)
HADOOP-4975. Set classloader for nested mapred.join configs. (Jingkei Ly
via cdouglas)
HADOOP-5078. Remove invalid AMI kernel in EC2 scripts. (tomwhite)
HADOOP-5045. FileSystem.isDirectory() should not be deprecated. (Suresh
Srinivas via szetszwo)
HADOOP-4960. Use datasource time, rather than system time, during metrics
demux. (Eric Yang via cdouglas)
HADOOP-5032. Export conf dir set in config script. (Eric Yang via cdouglas)
HADOOP-5176. Fix a typo in TestDFSIO. (Ravi Phulari via szetszwo)
HADOOP-4859. Distinguish daily rolling output dir by adding a timestamp.
(Jerome Boulon via cdouglas)
HADOOP-4959. Correct system metric collection from top on Redhat 5.1. (Eric
Yang via cdouglas)
HADOOP-5039. Fix log rolling regex to process only the relevant
subdirectories. (Jerome Boulon via cdouglas)
HADOOP-5095. Update Chukwa watchdog to accept config parameter. (Jerome
Boulon via cdouglas)
HADOOP-5147. Correct reference to agent list in Chukwa bin scripts. (Ari
Rabkin via cdouglas)
HADOOP-5148. Fix logic disabling watchdog timer in Chukwa daemon scripts.
(Ari Rabkin via cdouglas)
HADOOP-5100. Append, rather than truncate, when creating log4j metrics in
Chukwa. (Jerome Boulon via cdouglas)
HADOOP-5204. Fix broken trunk compilation on Hudson by letting
task-controller be an independent target in build.xml.
(Sreekanth Ramakrishnan via yhemanth)
HADOOP-5212. Fix the path translation problem introduced by HADOOP-4868
running on cygwin. (Sharad Agarwal via omalley)
HADOOP-5226. Add license headers to html and jsp files. (szetszwo)
HADOOP-5172. Disable misbehaving Chukwa unit test until it can be fixed.
(Jerome Boulon via nigel)
HADOOP-4933. Fixes a ConcurrentModificationException problem that shows up
when the history viewer is accessed concurrently.
(Amar Kamat via ddas)
HADOOP-5253. Remove duplicate call to cn-docs target.
(Giri Kesavan via nigel)
HADOOP-5251. Fix classpath for contrib unit tests to include clover jar.
(nigel)
HADOOP-5206. Synchronize "unprotected*" methods of FSDirectory on the root.
(Jakob Homan via shv)
HADOOP-5292. Fix NPE in KFS::getBlockLocations. (Sriram Rao via lohit)
HADOOP-5219. Adds a new property io.seqfile.local.dir for use by
SequenceFile, which earlier used mapred.local.dir. (Sharad Agarwal
via ddas)
HADOOP-5300. Fix ant javadoc-dev target and the typo in the class name
NameNodeActivtyMBean. (szetszwo)
HADOOP-5218. libhdfs unit test failed because it was unable to
start namenode/datanode. Fixed. (dhruba)
HADOOP-5273. Add license header to TestJobInProgress.java. (Jakob Homan
via szetszwo)
HADOOP-5229. Remove duplicate version variables in build files
(Stefan Groschupf via johan)
HADOOP-5383. Avoid building an unused string in NameNode's
verifyReplication(). (Raghu Angadi)
HADOOP-5347. Create a job output directory for the bbp examples. (szetszwo)
HADOOP-5341. Make hadoop-daemon scripts backwards compatible with the
changes in HADOOP-4868. (Sharad Agarwal via yhemanth)