Transfer form DVMM website.

ColumbiaDVMM · Jul 30, 2016 · d8c5bdb · d8c5bdb
commit d8c5bdb
Show file tree

Hide file tree

Showing 107 changed files with 491 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -0,0 +1,131 @@
+MOT - Moving Object Segmentation and Tracking System
+___
+####Terms of Use
+
+Copyright (c) 2003 by DVMM Laboratory
+
+Department of Electrical Engineering</br>
+Columbia University</br>
+Rm 1312 S.W. Mudd, 500 West 120th Street</br>
+New York, NY 10027</br>
+USA
+
+
+Di Zhong and Shih-Fu Chang, Columbia University
+{dzhong,sfchang}@ee.columbia.edu
+
+####1. SYSTEM OVERVIEW
+
+
+MOT is a moving object segmentation and tracking system for salient objects
+over long period in general video sources. The system utilizes innovative
+methods of combining color and edge information in improving the object motion
+estimation results.  It then uses the long-term spatio-temporal constraints to
+achieve reliable object tracking over long sequences. 
+
+The system contains two stages.  In the first stage, we apply an iterative
+motion layer detection process based on the estimation and merging of affine
+motion models. Each iteration generates one motion layer. The difference from
+existing methods is that motion models are estimated from spatially segmented
+color regions instead of just pixels or blocks.  In the second stage, temporal
+constraints are applied to detect moving objects in spatial and temporal space.
+Layers in individual frames are linked together based on characteristics of
+their underlying regions. One or more layers will be declared as motion objects
+according to specific spatio-temporal consistency rules.
+
+The input are decoded frames in either PPM, GIF or JPEG format. The system does
+not include an interface to decode MPEG video into image files. One can use the
+AMOS system to decode MPEG streams and build database of video shots. 
+
+In this binary distribution, we provide the classes for local segmentation and 
+global detection. We also provide sample Java codes to extract region information
+described in OIF files. Before using the system, the video clip should have
+been segmented into individual shots and store individual frames in PPM format.
+
+####2.SYSTEM REQUIREMENT/ INSTALLATION
+
+
+The system is developed in Java language on Sun Solaris 2.5, Windows NT 4.0,
+Windows 2000, or Windows XP.  It requires following software packages.
+
+a) JDK 1.1.6+ or 1.2 (with JIT compiler)
+
+
+The final installation will include following directory structure:
+00README.txt         ---   this file
+test.db              ---   an example database file (generate by AMOS)
+runmot.bat           ---   run object tracking for the example database file
+image                ---   general image I/O and processing functions
+feature              ---   segmentation and feature computation
+db                   ---   management of video shots and objects
+VisualNumerics       ---   numeric functions from the third party
+odet                 ---   main program for object segmentation and tracking
+test                 ---   this is the root directory of "test.db" and repository
+                	   of frames of a sample shot "1009"
+sample		     ---   sample code for extracting region information
+makesample.bat	     ---   how to compile the region information sample code
+runsample.bat	     ---   execute region information extraction
+README_for_db_file   ---   interpretation of "db" file
+
+
+
+####3. USING THE SYSTEM
+
+
+As previously discussed, it include two steps to track moving objects, which 
+are both provided with binary Java codes. The last step is to extract the
+region information, kept in *.OIF files, and is demonstrated in a sample code.
+
+#####3.1 Local Segmentation
+
+
+Usage: Java odet.ObjectDetection {v.db} {th} {min_size}
+where: v.db      ---  the database file for video shots
+       th        ---  the color merging threshold
+       min_size  ---  minimum size of tracked regions (in number of pixels)
+
+This program generates intermedia segmentation results in OIF files, including:
+- salient regions
+- foreground or background classification at individual frames
+
+The source video sequences, in PPM formats, are located in ".\test\1009\frames\" and
+OIF files are further generated in ".\test\1009\odet\".
+
+#####3.2 Global Detection
+
+Usage: java odet.GlobalDetection {v.db} {min_mv} {min_lifespan}
+where: v.db          ---  the database file for video shots
+       min_mv        ---  the minimum number of frames with motion
+       min_lifespan  ---  the minimum number of total frames
+
+This program generates final moving object. Outputs are stored to PPM files 
+under the directory ".\test\1009\odet\" for each image sequence.
+
+#####3.3 db File
+
+
+The interpretation of the "db" file is exemplified in "README_for_db_file.txt".
+
+#####3.4 Example
+
+
+[path to Java run time program] java odet.ObjectDetection test.db 20 100
+[path to Java run time program] java odet.GlobalDetection test.db 8 15
+
+Please reference "runmot.bat" for these scripts and parameters. Before 
+executing the program, pease make sure Java environment and corresponding
+PATH are well configured.
+
+####4. Region information extraction
+
+
+Each image will be segmented into regions defined by boundary boxes
+and masks and described in *.OIF file. To know how to extract region
+information, please reference "RegionInfoDoc.tex" in directory "sample".
+
+
+####5. Misc.
+
+
+Before executing the binary distribution and compiling the sample code, please
+MAKE SURE that Java runtime and development enviroment are well configured.
diff --git a/README_for_db_file.txt b/README_for_db_file.txt
@@ -0,0 +1,21 @@
+// The interpretation of the "test.db" file is as the following
+// For simplicity, we suggest using a "db" file for a single shot.
+// If there are multiple shots to process, generate *.db files in 
+// batch.
+
+
+.\test	// storing corresponding frames within shots, here shot "1009" only
+100000	// next_shotid 
+1 	// number of shots
+	// one blank row is necessary here. This cannot be omitted.
+1009 	// shot id (same as folder name under the path designated on the first line)
+-UNKNOWN- // video file name (for mpeg2dec. This is not used in MOT)
+234300	// start time (not used in MOT)
+50	// duration (not used in MOT)
+40	// start frame number
+30	// number of frames
+352	// width of the frame
+240	// height of the frame
+2	// (not used in MOT)
+6002138 // (not used in MOT)
+
diff --git a/VisualNumerics/math/BLAS.class b/VisualNumerics/math/BLAS.class
diff --git a/VisualNumerics/math/BLASA.class b/VisualNumerics/math/BLASA.class
diff --git a/VisualNumerics/math/Complex.class b/VisualNumerics/math/Complex.class
diff --git a/VisualNumerics/math/ComplexA.class b/VisualNumerics/math/ComplexA.class
diff --git a/VisualNumerics/math/ComplexCholesky.class b/VisualNumerics/math/ComplexCholesky.class
diff --git a/VisualNumerics/math/ComplexLU.class b/VisualNumerics/math/ComplexLU.class
diff --git a/VisualNumerics/math/ComplexMatrix.class b/VisualNumerics/math/ComplexMatrix.class
diff --git a/VisualNumerics/math/ComplexQR.class b/VisualNumerics/math/ComplexQR.class
diff --git a/VisualNumerics/math/ComplexSVD.class b/VisualNumerics/math/ComplexSVD.class
diff --git a/VisualNumerics/math/ComplexVector.class b/VisualNumerics/math/ComplexVector.class
diff --git a/VisualNumerics/math/DoubleCholesky.class b/VisualNumerics/math/DoubleCholesky.class
diff --git a/VisualNumerics/math/DoubleLU.class b/VisualNumerics/math/DoubleLU.class
diff --git a/VisualNumerics/math/DoubleMatrix.class b/VisualNumerics/math/DoubleMatrix.class
diff --git a/VisualNumerics/math/DoubleQR.class b/VisualNumerics/math/DoubleQR.class
diff --git a/VisualNumerics/math/DoubleSVD.class b/VisualNumerics/math/DoubleSVD.class
diff --git a/VisualNumerics/math/DoubleVector.class b/VisualNumerics/math/DoubleVector.class
diff --git a/VisualNumerics/math/MathException.class b/VisualNumerics/math/MathException.class
diff --git a/VisualNumerics/math/Sfun.class b/VisualNumerics/math/Sfun.class
diff --git a/VisualNumerics/math/Statistics.class b/VisualNumerics/math/Statistics.class
diff --git a/db/ObjectFeature.class b/db/ObjectFeature.class
diff --git a/db/RegionFeature.class b/db/RegionFeature.class
diff --git a/db/StructureFeature.class b/db/StructureFeature.class
diff --git a/db/VideoDB.class b/db/VideoDB.class
diff --git a/db/VideoObject.class b/db/VideoObject.class
diff --git a/db/VideoRegion.class b/db/VideoRegion.class
diff --git a/db/VideoShot.class b/db/VideoShot.class
diff --git a/feature/AffineModel.class b/feature/AffineModel.class
diff --git a/feature/EdgeMap.class b/feature/EdgeMap.class
diff --git a/feature/OpticalFlow.class b/feature/OpticalFlow.class
diff --git a/feature/RegionInfo.class b/feature/RegionInfo.class
diff --git a/feature/Segment.class b/feature/Segment.class
diff --git a/feature/Snake.class b/feature/Snake.class
diff --git a/image/ImageData.class b/image/ImageData.class
diff --git a/image/ImageIO.class b/image/ImageIO.class
diff --git a/image/ImageProc.class b/image/ImageProc.class
diff --git a/image/IndexSortable.class b/image/IndexSortable.class
diff --git a/image/IntVector.class b/image/IntVector.class
diff --git a/image/Mask.class b/image/Mask.class
diff --git a/image/PubFunc.class b/image/PubFunc.class
diff --git a/image/Sortable.class b/image/Sortable.class
diff --git a/image/XPolygon.class b/image/XPolygon.class
diff --git a/image/luv4096.dat b/image/luv4096.dat
diff --git a/mksample.bat b/mksample.bat
@@ -0,0 +1,2 @@
+javac -O -d . sample\SampleRegionInfo.java
+
diff --git a/odet/GlobalDetection.class b/odet/GlobalDetection.class
diff --git a/odet/ObjectDetection.class b/odet/ObjectDetection.class
diff --git a/runmot.bat b/runmot.bat
@@ -0,0 +1,2 @@
+java odet.ObjectDetection test.db 20 100
+java odet.GlobalDetection test.db 8 15
diff --git a/runsample.bat b/runsample.bat
@@ -0,0 +1 @@
+java sample.SampleRegionInfo .\test 1009
diff --git a/sample/RegionInfoDoc.txt b/sample/RegionInfoDoc.txt
@@ -0,0 +1,222 @@
+[Usage of region information]
+
+Each image will be segmented into regions defined by boundary boxes
+and masks and described in *.OIF file. To know how to extract region
+information please reference the sample code "SampleRegionInfo.java" and
+"RegionInfoDoc.txt".
+
+* Sample code
+	sample.SampleRegionInfo
+	Each image will be segmented into regions defined by boundary boxes
+	and masks and are described in *.oif files. This program shows how 
+	to extract region information from *.oif files.
+
+	This program is complied with ..\mksample.bat, and it is run with ..\runsample.bat.
+	** Please make sure the Java environment is well configured.
+
+* Explanation about RegionInfo class
+	Region information can be handled using RegionInfo class. The following 
+	is the explanation about the class.
+
+1. Members
+
+	public int id  -- region id
+	public int pixelnum -- number of pixels within the region
+	public float[] feat1 -- mean color of the region (feat1[0]:L, feat1[1]:u, feat1[2]:v )
+	public float[] feat2 -- motion parameters of the region. motion is compensated by the following equation:
+        x= feat2[0] + feat2[1] * x0 + feat2[2] * y0 + feat2[6] * x0 * x0 + feat2[7] * x0 * y0 + x0,
+        y= feat2[3] + feat2[4] * x0 + feat2[5] * y0 + feat2[6] * x0 * y0 + feat2[7] * y0 * y0 + y0
+      where (x0,y0) is a point in the current region and (x,y) is its corresponding point in the next frame. 
+	public Point ul -- the coordinate of the upper-left corner of the bounding box
+	public Point lr -- the coordinate of the lower-right corner of the bounding box 
+	public Hashtable con_regs -- List of the adjoining regions. 
+	public boolean[][] mask -- Mask information indicating the region shape. The mask size is the same as the bounding box of the region. Hence mask[0][0] indicates the mask information at the upper-left point of bounding box.
+	public int tag -- This can be used for several way. In the Object/GlobalDetection, tag is used for distinguishing
+       foreground and background regions. If tag = 1, the region indicates foreground, otherwise, background.
+
+
+2. Methods
+
+public RegionInfo(int id, float[] clr, int x, int y)
+	Arguments:
+		int id -- region ID.
+		float[] clr -- the Luv value for the pixel at (x,y).
+		int x -- x coordinate of the pixel included in the region
+		int y -- y coordinate of the pixel included in the region
+	Explanation:
+		Constructor of RegionInfo object. This makes RegionInfo object for the region including a single pixel (x,y).
+-------------------------------------------------------------------------------------------
+public RegionInfo(RegionInfo r)
+	Arguments:
+		RegionInfo r -- region information to be copied
+	Explanation:
+		Copy constructor of RegionInfo object.
+-------------------------------------------------------------------------------------------
+public static final RegionInfo[] read(String fname)
+	Arguments:
+		String fileName  -- The OIF file name from which the region information is read. 
+	Returned values:
+		Array of RegionInfo objects, each of which indicates a region in the image.
+	Explanation:
+		This method reads region information from the file designated by fname, and return the array of RegionInfo objects.
+-------------------------------------------------------------------------------------------
+public static final void write(String fname, RegionInfo[] rlist, int nextid, int fw, int fh, float[] gm_param )
+	Arguments:
+		String fname -- The OIF file name to which the region information is written.
+		RegionInfo[] rlist -- The array of region information
+		int nextid -- next region id (Currently this is not used, so any positive integer is all right)
+		int fw -- frame width (Currently this is not used, so any positive integer is all right)
+		int fh -- frame height (Currently this is not used, so any positive integer is all right)
+		float[] gm_param -- global motion parameters (Currently this is not used, but memory for these parameters should be allocated before calling. See the sample code.)
+	Returned values:
+		None
+	Explanation:
+		This method writes the array of region information rlist to the file designated by fname.
+-------------------------------------------------------------------------------------------
+public final int perimeter()
+	Arguments:
+		None
+	Returned values:
+		The perimeter of this region
+	Explanation:
+		This method finds the perimeter of this region and return the value.
+-------------------------------------------------------------------------------------------
+public final boolean imgxy( int x, int y )
+	Arguments:
+		int x -- x coordinate of the point
+		int y -- y coordinate of the point
+	Returned values:
+		True if the point is inside the region, and false otherwise.
+	Explanation:
+		This method is used for checking whether the point is inside the region or not.
+-------------------------------------------------------------------------------------------
+public final boolean isBorder(int x,int y)
+	Arguments:
+		int x -- x coordinate of the point
+		int y -- y coordinate of the point
+	Returned values:
+		True if it is a point on the border, and false otherwise.
+	Explanation:
+		This method is used for checking whether the point is on the border or not.
+-------------------------------------------------------------------------------------------
+public final boolean isNeighbor(int x, int y) 
+	Arguments:
+		int x -- x coordinate of the point
+		int y -- y coordinate of the point
+	Returned values:
+		True if it is an adjoining point, and false otherwise.
+	Explanation:
+		This method is used for checking whether the point adjoins this region or not.
+-------------------------------------------------------------------------------------------
+public static void merge(RegionInfo r0,RegionInfo r1)
+	Arguments:
+		RegionInfo r0 -- the first region information, which carries the merge result.
+		RegionInfo r1 -- the second region information.
+	Returned values:
+		None.
+	Explanation:
+		Regions r0 and r1 are merged. The merge result is included in r0.
+------------------------------------------------------------------------------------------
+public RegionInfo nearestNeighbor(float sqthresh, float[] w)
+	Arguments:
+		float sqthresh -- threshold to determine the upper-limit of mean color distance.
+		float[] w -- weight for calculating the mean color distance of the regions.
+	Returned values:
+		RegionInfo object with the closest mean color to this region.
+	Explanation:
+		This method finds the region with the closest mean color to this region among the adjoining regions, 
+		and the found region information is returned. If the difference in the mean colors are more than 
+		sqthresh for all the adjoining regions, the function returns null instead. The distance is calculated 
+		by
+     		d = w[0]*w[0]*DL*DL + w[1]*w[1]*Du*Du + w[2]*w[2]*Dv*Dv, 
+		where DL, Du and Dv denote the difference of mean L, u and v values between two regions.
+------------------------------------------------------------------------------------------
+public final void connect(RegionInfo nb)
+	Arguments:
+		RegionInfo nb -- region information to be connected to this region
+	Returned values:
+		None.
+	Explanation:
+		Connect the region nb to this region.
+------------------------------------------------------------------------------------------
+public final void disconnect()
+	Arguments:
+		None.
+	Returned values:
+		None.
+	Explanation:
+		Disconnect this region from all the adjoining regions.
+------------------------------------------------------------------------------------------
+public final int[] getConnectList()
+	Arguments:
+		None.
+	Returned values:
+		List of the IDs of the adjoining region.
+	Explanation:
+		This method returns the list of the IDs of the adjoining region. 
+------------------------------------------------------------------------------------------
+public static boolean[][] getMask(RegionInfo[] rlist, Rectangle rect)
+	Arguments:
+		RegionInfo[] rlist -- region information array.
+		Rectangle rect -- bouding box of combined mask.
+	Returned values:
+		2D boolean array indicating the combined mask information.
+	Explanation:
+		This method is used for geting combined mask of a list of regions. The mask size equals 
+		the given bounding box.
+------------------------------------------------------------------------------------------
+public static Rectangle getBoundingBox(RegionInfo[] rlist)
+	Arguments:
+		RegionInfo[] rlist -- A list of the regions.
+	Returned values:
+		Rectangle object indicating the bouding box of the regions included in rlist.
+	Explanation:
+		This method returns the bounding box for the region included in the list rlist.
+------------------------------------------------------------------------------------------
+public void printRandomColor() 
+	Arguments:
+		None.
+	Returned values:
+		None.
+	Explanation:
+		This method displays the color of the region in a region image to the stdout.
+------------------------------------------------------------------------------------------
+public final int intersectPixels(boolean[][] mask0, Point mask_ul)
+	Arguments:
+		boolean[][] mask0 -- the gievn mask information
+		Point mask_ul -- coordinate of the upper-left corner of the mask
+	Returned values:
+		The number of pixels included in intersection.
+	Explanation:
+		This methold counts the number of pixels in intersection region with mask0.
+------------------------------------------------------------------------------------------
+public final boolean[][] intersect(boolean[][] mask0, Point mask_ul)
+	Arguments:
+  		boolean[][] mask0 -- the given mask information.
+		Point mask_ul -- coordinate of upper-left corner of the given mask.
+	Returned values:
+		Mask for intersection
+	Explnaation:
+		This method computes the intersection of the region with a given mask.
+------------------------------------------------------------------------------------------
+public static boolean[][] getCloseOpenMask(RegionInfo[] rlist, Rectangle rect, float close_r, float open_r)
+	Arguments:
+		RegionInfo rlist -- Region information array
+		Rectangle rect -- Bounding box of combined mask
+		float close_r -- The radius of closing operator
+		float open_r -- The radius of opening operator
+	Returned values:
+		Resultant mask.
+	Explanation:
+		This method is used for getting morphologocal-closed mask of a list of regions. The mask size equals the given bounding box.
+------------------------------------------------------------------------------------------
+public static boolean[][] getOpenCloseMask(RegionInfo[] rlist, Rectangle rect, float open_r, float close_r) 
+	Arguments:
+		RegionInfo rlist -- Region information array
+		Rectangle rect -- Bounding box of combined mask
+		float open_r -- The radius of opening operator
+		float close_r -- The radius of closing operator
+	Returned values:
+		Resultant mask.
+	Explanation:
+		This method is used for getting morphological-opened then closed mask of a list of regions. The mask size equals the given bounding box.
diff --git a/sample/SampleRegionInfo.class b/sample/SampleRegionInfo.class
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		java odet.ObjectDetection test.db 20 100
		java odet.GlobalDetection test.db 8 15