Trying Out Feeding in League Again With the Famvideomp4

Final updated on July 8, 2021.

Today's tutorial kicks off a new series of blog posts on object tracking, arguably one of the most requested topics here on PyImageSearch.

Object tracking is the procedure of:

Taking an initial prepare of object detections (such as an input set up of bounding box coordinates)
Creating a unique ID for each of the initial detections
And then tracking each of the objects equally they move around frames in a video, maintaining the consignment of unique IDs

Furthermore, object tracking allows us to apply a unique ID to each tracked object, making it possible for usa to count unique objects in a video. Object tracking is paramount to edifice a person counter (which we'll practise later in this serial).

An ideal object tracking algorithm volition:

Only crave the object detection stage once (i.due east., when the object is initially detected)
Will be extremely fast — much faster than running the actual object detector itself
Be able to handle when the tracked object "disappears" or moves exterior the boundaries of the video frame
Exist robust to occlusion
Be able to pick up objects it has "lost" in between frames

This is a tall guild for any computer vision or image processing algorithm and at that place are a diverseness of tricks we can play to help improve our object trackers.

But before we can build such a robust method nosotros start need to study the fundamentals of object tracking.

In today's blog mail, y'all will acquire how to implement centroid tracking with OpenCV, an easy to understand, yet highly effective tracking algorithm.

In future posts in this object tracking series, I'll showtime going into more advanced kernel-based and correlation-based tracking algorithms.

To acquire how to go started building your get-go object tracking with OpenCV, just continue reading!

Update July 2021: Added section on alternative object trackers, including object trackers built straight into the OpenCV library.

Looking for the source code to this post?

Jump Right To The Downloads Section

Simple object tracking with OpenCV

In the remainder of this post, we'll exist implementing a simple object tracking algorithm using the OpenCV library.

This object tracking algorithm is called centroid tracking as it relies on the Euclidean distance between (ane) existing object centroids (i.e., objects the centroid tracker has already seen before) and (2) new object centroids between subsequent frames in a video.

We'll review the centroid algorithm in more depth in the post-obit section. From in that location we'll implement a Python class to contain our centroid tracking algorithm and so create a Python script to actually run the object tracker and apply information technology to input videos.

Finally, we'll run our object tracker and examine the results, noting both the positives and the drawbacks of the algorithm.

The centroid tracking algorithm

The centroid tracking algorithm is a multi-step process. We will review each of the tracking steps in this section.

Step #ane: Accept bounding box coordinates and compute centroids

**Figure 1:** To build a simple object tracking algorithm using centroid tracking, the commencement step is to accept bounding box coordinates from an object detector and employ them to compute centroids.

The centroid tracking algorithm assumes that we are passing in a set up of bounding box (10, y)-coordinates for each detected object in every unmarried frame .

These bounding boxes tin can be produced past any blazon of object detector you would like (color thresholding + contour extraction, Haar cascades, HOG + Linear SVM, SSDs, Faster R-CNNs, etc.), provided that they are computed for every frame in the video.

Once we have the bounding box coordinates we must compute the "centroid", or more just, the center (x, y)-coordinates of the bounding box. Effigy one in a higher place demonstrates accepting a ready of bounding box coordinates and calculating the centroid.

Since these are the starting time initial gear up of bounding boxes presented to our algorithm we will assign them unique IDs.

Step #2: Compute Euclidean distance between new bounding boxes and existing objects

**Effigy 2:** Iii objects are present in this image for simple object tracking with Python and OpenCV. We demand to compute the Euclidean distances between each pair of original centroids (*cerise*) and new centroids (*greenish*).

For every subsequent frame in our video stream nosotros apply Step #i of computing object centroids; however, instead of assigning a new unique ID to each detected object (which would defeat the purpose of object tracking), we start demand to make up one's mind if nosotros can acquaintance the new object centroids (xanthous) with the old object centroids (purple). To achieve this procedure, nosotros compute the Euclidean distance (highlighted with green arrows) betwixt each pair of existing object centroids and input object centroids.

From Figure 2 you lot can come across that we have this time detected three objects in our image. The ii pairs that are close together are two existing objects.

We so compute the Euclidean distances between each pair of original centroids (yellow) and new centroids (majestic). But how practise we employ the Euclidean distances between these points to really lucifer them and associate them?

The answer is in Footstep #three.

Stride #iii: Update (ten, y)-coordinates of existing objects

**Figure 3:** Our simple centroid object tracking method has associated objects with minimized object distances. What do we do nigh the object in the bottom left though?

The primary supposition of the centroid tracking algorithm is that a given object volition potentially move in between subsequent frames, but the distance between the centroids for frames $F_t$ and $F_{t + 1}$ will be smaller than all other distances between objects.

Therefore, if nosotros choose to associate centroids with minimum distances between subsequent frames we can build our object tracker.

In Figure 3 you can run across how our centroid tracker algorithm chooses to associate centroids that minimize their respective Euclidean distances.

But what almost the alone betoken in the bottom-left?

Information technology didn't become associated with anything — what do we do with it?

Pace #four: Register new objects

**Figure iv:** In our object tracking with Python and OpenCV example, we have a new object that wasn't matched with an existing object, and so it is registered every bit object ID #3.

In the issue that in that location are more input detections than existing objects being tracked, nosotros need to annals the new object. "Registering" just means that nosotros are adding the new object to our list of tracked objects by:

Assigning it a new object ID
Storing the centroid of the bounding box coordinates for that object

We can and then go back to Step #2 and repeat the pipeline of steps for every frame in our video stream.

Figure 4 demonstrates the procedure of using the minimum Euclidean distances to associate existing object IDs and so registering a new object.

Step #5: Deregister former objects

Whatsoever reasonable object tracking algorithm needs to exist able to handle when an object has been lost, disappeared, or left the field of view.

Exactly how you handle these situations is really dependent on where your object tracker is meant to exist deployed, but for this implementation, we will deregister quondam objects when they cannot be matched to whatsoever existing objects for a total of N subsequent frames.

Object tracking project structure

To see today's project construction in your terminal, simply apply the tree command:

$ tree --dirsfirst . ├── pyimagesearch │   ├── __init__.py │   └── centroidtracker.py ├── object_tracker.py ├── deploy.prototxt └── res10_300x300_ssd_iter_140000.caffemodel  one directory, 5 files

Our pyimagesearch module is not pip-installable — it is included with today's "Downloads" (which you'll find at the bottom of this post). Inside you'll find the centroidtracker.py file which contains the CentroidTracker grade.

The CentroidTracker class is an important component used in the object_tracker.py driver script.

The remaining .prototxt and .caffemodel files are part of the OpenCV deep learning face detector. They are necessary for today's face detection + tracking method, simply you could easily utilize another grade of detection (more on that later).

Be sure that you take NumPy, SciPy, and imutils installed before you proceed:

$ pip install numpy scipy imutils

…in addition to having OpenCV 3.3+ installed. If you follow one of my OpenCV install tutorials, be sure to supplant the tail finish of the wget command to take hold of at to the lowest degree OpenCV iii.3 (and update the paths in the CMake command). You'll need 3.iii+ to ensure you accept the DNN module.

Implementing centroid tracking with OpenCV

Before we can utilize object tracking to our input video streams, we first demand to implement the centroid tracking algorithm. While you're digesting this centroid tracker script, just proceed in heed Steps 1-v above and review the steps equally necessary.

Equally you lot'll see, the translation of steps to code requires quite a chip of thought, and while we perform all steps, they aren't linear due to the nature of our various data structures and code constructs.

I would suggest

Reading the steps above
Reading the code explanation for the centroid tracker
And finally reading the steps above over again

This procedure will bring everything full circle and allow you to wrap your head around the algorithm.

Once you're sure you lot understand the steps in the centroid tracking algorithm, open up up the centroidtracker.py inside the pyimagesearch module and allow's review the code:

# import the necessary packages from scipy.spatial import distance as dist from collections import OrderedDict import numpy equally np  class CentroidTracker(): 	def __init__(self, maxDisappeared=fifty): 		# initialize the side by side unique object ID along with ii ordered 		# dictionaries used to keep rail of mapping a given object 		# ID to its centroid and number of consecutive frames information technology has 		# been marked equally "disappeared", respectively 		self.nextObjectID = 0 		self.objects = OrderedDict() 		self.disappeared = OrderedDict()  		# store the number of maximum consecutive frames a given 		# object is allowed to be marked every bit "disappeared" until we 		# need to deregister the object from tracking 		self.maxDisappeared = maxDisappeared

On Lines 2-4 we import our required packages and modules — distance , OrderedDict , and numpy .

Our CentroidTracker form is divers on Line half-dozen. The constructor accepts a unmarried parameter, the maximum number of consecutive frames a given object has to be lost/disappeared for until nosotros remove it from our tracker (Line 7).

Our constructor builds four class variables:

nextObjectID : A counter used to assign unique IDs to each object (Line 12). In the case that an object leaves the frame and does not come back for maxDisappeared frames, a new (next) object ID would exist assigned.
objects : A dictionary that utilizes the object ID every bit the fundamental and the centroid (x, y)-coordinates as the value (Line 13).
disappeared : Maintains number of consecutive frames (value) a particular object ID (key) has been marked equally "lost"for (Line xiv).
maxDisappeared : The number of consecutive frames an object is allowed to be marked every bit "lost/disappeared" until we deregister the object.

Let's define the register method which is responsible for adding new objects to our tracker:

            def register(self, centroid): 		# when registering an object we use the next available object 		# ID to store the centroid 		self.objects[cocky.nextObjectID] = centroid 		cocky.disappeared[self.nextObjectID] = 0 		self.nextObjectID += 1

The register method is defined on Line 21. It accepts a centroid so adds information technology to the objects lexicon using the next available object ID.

The number of times an object has disappeared is initialized to 0 in the disappeared lexicon (Line 25).

Finally, nosotros increase the nextObjectID then that if a new object comes into view, it will be associated with a unique ID (Line 26).

Like to our register method, we too demand a deregister method:

            def deregister(self, objectID): 		# to deregister an object ID we delete the object ID from 		# both of our corresponding dictionaries 		del self.objects[objectID] 		del self.disappeared[objectID]

Just like we can add new objects to our tracker, we besides need the power to remove old ones that take been lost or disappeared from our the input frames themselves.

The deregister method is defined on Line 28. Information technology only deletes the objectID in both the objects and disappeared dictionaries, respectively (Lines 31 and 32).

The centre of our centroid tracker implementation lives inside the update method:

            def update(cocky, rects): 		# bank check to run into if the list of input bounding box rectangles 		# is empty 		if len(rects) == 0: 			# loop over any existing tracked objects and marker them 			# as disappeared 			for objectID in listing(cocky.disappeared.keys()): 				self.disappeared[objectID] += 1  				# if we have reached a maximum number of sequent 				# frames where a given object has been marked as 				# missing, deregister it 				if cocky.disappeared[objectID] > self.maxDisappeared: 					self.deregister(objectID)  			# return early as there are no centroids or tracking info 			# to update 			return self.objects

The update method, defined on Line 34, accepts a list of bounding box rectangles, presumably from an object detector (Haar cascade, Squealer + Linear SVM, SSD, Faster R-CNN, etc.). The format of the rects parameter is assumed to exist a tuple with this construction: (startX, startY, endX, endY) .

If there are no detections, nosotros'll loop over all object IDs and increase their disappeared count (Lines 37-41). We'll also cheque if nosotros have reached the maximum number of consecutive frames a given object has been marked as missing. If that is the case nosotros need to remove it from our tracking systems (Lines 46 and 47). Since there is no tracking info to update, we get ahead and return early on on Line 51.

Otherwise, we have quite a scrap of work to do over the side by side vii code blocks in the update method:

            # initialize an array of input centroids for the current frame 		inputCentroids = np.zeros((len(rects), two), dtype="int")  		# loop over the bounding box rectangles 		for (i, (startX, startY, endX, endY)) in enumerate(rects): 			# utilize the bounding box coordinates to derive the centroid 			cX = int((startX + endX) / 2.0) 			cY = int((startY + endY) / 2.0) 			inputCentroids[i] = (cX, cY)

On Line 54 we'll initialize a NumPy array to store the centroids for each rect .

Then, nosotros loop over bounding box rectangles (Line 57) and compute the centroid and store it in the inputCentroids list (Lines 59-61).

If there are currently no objects we are tracking, we'll register each of the new objects:

            # if nosotros are currently not tracking any objects take the input 		# centroids and annals each of them 		if len(cocky.objects) == 0: 			for i in range(0, len(inputCentroids)): 				self.register(inputCentroids[i])

Otherwise, we need to update any existing object (10, y)-coordinates based on the centroid location that minimizes the Euclidean distance betwixt them:

            # otherwise, are are currently tracking objects so nosotros need to 		# try to match the input centroids to existing object 		# centroids 		else: 			# grab the set of object IDs and corresponding centroids 			objectIDs = list(self.objects.keys()) 			objectCentroids = listing(self.objects.values())  			# compute the distance betwixt each pair of object 			# centroids and input centroids, respectively -- our 			# goal will be to friction match an input centroid to an existing 			# object centroid 			D = dist.cdist(np.array(objectCentroids), inputCentroids)  			# in society to perform this matching we must (i) find the 			# smallest value in each row and then (2) sort the row 			# indexes based on their minimum values and so that the row 			# with the smallest value is at the *front* of the index 			# list 			rows = D.min(centrality=1).argsort()  			# next, we perform a similar process on the columns by 			# finding the smallest value in each column and and so 			# sorting using the previously computed row index listing 			cols = D.argmin(centrality=one)[rows]

The updates to existing tracked objects take place kickoff at the else on Line 72. The goal is to rails the objects and to maintain correct object IDs — this process is achieved by computing the Euclidean distances betwixt all pairs of objectCentroids and inputCentroids , followed by associating object IDs that minimize the Euclidean distance.

Within of the else cake beginning on Line 72, nosotros will:

Grab objectIDs and objectCentroid values (Lines 74 and 75).
Compute the distance betwixt each pair of existing object centroids and new input centroids (Line 81). The output NumPy assortment shape of our altitude map D will be (# of object centroids, # of input centroids) .
To perform the matching we must (1) Observe the smallest value in each row, and (2) Sort the row indexes based on the minimum values (Line 88). We perform a very like process on the columns, finding the smallest value in each cavalcade, and then sorting them based on the ordered rows (Line 93). Our goal is to have the alphabetize values with the smallest corresponding altitude at the front of the lists.

The next step is to utilise the distances to encounter if nosotros can associate object IDs:

            # in order to determine if nosotros need to update, register, 			# or deregister an object we need to go on track of which 			# of the rows and column indexes we accept already examined 			usedRows = set() 			usedCols = set()  			# loop over the combination of the (row, column) index 			# tuples 			for (row, col) in goose egg(rows, cols): 				# if we accept already examined either the row or 				# cavalcade value before, ignore it 				# val 				if row in usedRows or col in usedCols: 					go on  				# otherwise, catch the object ID for the current row, 				# prepare its new centroid, and reset the disappeared 				# counter 				objectID = objectIDs[row] 				cocky.objects[objectID] = inputCentroids[col] 				self.disappeared[objectID] = 0  				# signal that we have examined each of the row and 				# column indexes, respectively 				usedRows.add(row) 				usedCols.add together(col)

Inside the code block above, we:

Initialize ii sets to make up one's mind which row and column indexes we have already used (Lines 98 and 99). Keep in listen that a gear up is similar to a list but it contains only unique values.
Then we loop over the combinations of (row, col) alphabetize tuples (Line 103) in social club to update our object centroids:
- If we've already used either this row or column alphabetize, ignore it and continue to loop (Lines 107 and 108).
- Otherwise, nosotros accept plant an input centroid that:
  - 1. Has the smallest Euclidean distance to an existing centroid
  - 2. And has non been matched with any other object
  - In that case, we update the object centroid (Lines 113-115) and make sure to add the row and col to their respective usedRows and usedCols sets

There are likely indexes in our usedRows + usedCols sets that nosotros have Non examined yet:

            # compute both the row and cavalcade index we take Non even so 			# examined 			unusedRows = set(range(0, D.shape[0])).divergence(usedRows) 			unusedCols = set up(range(0, D.shape[1])).difference(usedCols)

And so we must make up one's mind which centroid indexes we haven't examined nevertheless and store them in two new convenient sets (unusedRows and unusedCols ) on Lines 124 and 125.

Our final cheque handles any objects that have become lost or if they've potentially disappeared:

            # in the event that the number of object centroids is 			# equal or greater than the number of input centroids 			# we need to bank check and see if some of these objects have 			# potentially disappeared 			if D.shape[0] >= D.shape[1]: 				# loop over the unused row indexes 				for row in unusedRows: 					# grab the object ID for the respective row 					# alphabetize and increment the disappeared counter 					objectID = objectIDs[row] 					cocky.disappeared[objectID] += i  					# check to see if the number of consecutive 					# frames the object has been marked "disappeared" 					# for warrants deregistering the object 					if self.disappeared[objectID] > self.maxDisappeared: 						self.deregister(objectID)

To stop upward:

If the number of object centroids is greater than or equal to the number of input centroids (Line 131):
- We need to verify if any of these objects are lost or have disappeared by looping over unused row indexes if any (Line 133).
- In the loop, we will:
  - 1. Increase their disappeared count in the lexicon (Line 137).
  - 2. Check if the disappeared count exceeds the maxDisappeared threshold (Line 142), and, if so nosotros'll deregister the object (Line 143).

Otherwise, the number of input centroids is greater than the number of existing object centroids, so we have new objects to register and rail:

            # otherwise, if the number of input centroids is greater 			# than the number of existing object centroids we need to 			# register each new input centroid as a trackable object 			else: 				for col in unusedCols: 					cocky.register(inputCentroids[col])  		# render the ready of trackable objects 		return self.objects

We loop over the unusedCols indexes (Line 149) and we register each new centroid (Line 150). Finally, we'll return the set of trackable objects to the calling method (Line 153).

Understanding the centroid tracking altitude relationship

Our centroid tracking implementation was quite long, and admittedly, the most confusing aspect of the algorithm is Lines 81-93.

If you're having problem post-obit along with what that lawmaking is doing yous should consider opening a Python shell and performing the following experiment:

>>> from scipy.spatial import distance as dist >>> import numpy as np >>> np.random.seed(42) >>> objectCentroids = np.random.uniform(size=(two, 2)) >>> centroids = np.random.compatible(size=(3, ii)) >>> D = dist.cdist(objectCentroids, centroids) >>> D assortment([[0.82421549, 0.32755369, 0.33198071],        [0.72642889, 0.72506609, 0.17058938]])

One time y'all've started a Python beat in your terminal with the python command, import distance and numpy as shown on Lines 1 and 2).

And then, set a seed for reproducibility (Line 3) and generate 2 (random) existing objectCentroids (Line 4) and iii inputCentroids (Line 5).

From in that location, compute the Euclidean distance between the pairs (Line 6) and display the results (Lines vii-9). The result is a matrix D of distances with two rows (# of existing object centroids) and three columns (# of new input centroids).

Just like nosotros did earlier in the script, let's detect the minimum distance in each row and sort the indexes based on this value:

>>> D.min(axis=ane) array([0.32755369, 0.17058938]) >>> rows = D.min(centrality=1).argsort() >>> rows array([1, 0])

Kickoff, we notice the minimum value for each row, allowing us to figure out which existing object is closest to the new input centroid (Lines 10 and 11). Past and so sorting on these values (Line 12) we can obtain the indexes of these rows (Lines 13 and 14).

In this example, the 2nd row (index one ) has the smallest value and then the first row (index 0 ) has the next smallest value.

Using a similar process for the columns:

>>> D.argmin(axis=1) array([i, two]) >>> cols = D.argmin(centrality=1)[rows] >>> cols array([2, 1])

…we kickoff examine the values in the columns and detect the index of the value with the smallest cavalcade (Lines xv and xvi).

We and so sort these values using our existing rows (Lines 17-19).

Allow's impress the results and analyze them:

>>> impress(list(zip(rows, cols))) [(1, 2), (0, one)]

The final step is to combine them using naught (Lines xx). The resulting list is printed on Line 21.

Analyzing the results, nosotros find that:

D[1, ii] has the smallest Euclidean distance implying that the 2d existing object will exist matched against the third input centroid.
And D[0, ane] has the side by side smallest Euclidean distance which implies that the first existing object will be matched against the 2nd input centroid.

I'd like to reiterate here that now that y'all've reviewed the code, you should go dorsum and review the steps to the algorithm in the previous department. From there you lot'll exist able to associate the code with the more than linear steps outlined hither.

Implementing the object tracking driver script

Now that we have implemented our CentroidTracker course, let'due south put it to piece of work with an object tracking driver script.

The driver script is where you can use your own preferred object detector, provided that it produces a set up of bounding boxes. This could exist a Haar Cascade, Hog + Linear SVM, YOLO, SSD, Faster R-CNN, etc. For this example script, I'thousand making use of OpenCV's deep learning face detector, simply feel free to make your own version of the script which implements a different detector.

Inside this script, we will:

Work with a live VideoStream object to grab frames from your webcam
Load and utilize OpenCV'southward deep learning confront detector
Instantiate our CentroidTracker and use it to track face objects in the video stream
And display our results which includes bounding boxes and object ID annotations overlaid on the frames

When you're ready, open up object_tracker.py from today's "Downloads" and follow along:

# import the necessary packages from pyimagesearch.centroidtracker import CentroidTracker from imutils.video import VideoStream import numpy every bit np import argparse import imutils import fourth dimension import cv2  # construct the statement parse and parse the arguments ap = argparse.ArgumentParser() ap.add_argument("-p", "--prototxt", required=Truthful, 	aid="path to Caffe 'deploy' prototxt file") ap.add_argument("-m", "--model", required=True, 	assistance="path to Caffe pre-trained model") ap.add_argument("-c", "--conviction", blazon=float, default=0.5, 	help="minimum probability to filter weak detections") args = vars(ap.parse_args())

First, we specify our imports. Most notably nosotros're using the CentroidTracker class that we only reviewed. We're also going to use VideoStream from imutils and OpenCV.

Nosotros have 3 command line arguments which are all related to our deep learning face detector:

--prototxt : The path to the Caffe "deploy" prototxt.
--model : The path to the pre-trained model models.
--conviction : Our probability threshold to filter weak detections. I found that a default value of 0.v is sufficient.

The prototxt and model files come from OpenCV'southward repository and I'grand including them in the "Downloads" for your convenience.

Note: In case yous missed it at the commencement of this department, I'll echo that you tin can use any detector you wish. As an case, we're using a deep learning face up detector which produces bounding boxes. Feel free to experiment with other detectors, but be sure that you have capable hardware to keep up with the more than complex ones (some may run best with a GPU, but today'southward confront detector can hands run on a CPU).

Next, let's perform our initializations:

# initialize our centroid tracker and frame dimensions ct = CentroidTracker() (H, W) = (None, None)  # load our serialized model from disk impress("[INFO] loading model...") net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])  # initialize the video stream and allow the camera sensor to warmup print("[INFO] starting video stream...") vs = VideoStream(src=0).first() fourth dimension.slumber(2.0)

In the above cake, we:

Instantiate our CentroidTracker , ct (Line 21). Recall from the explanation in the previous section that this object has 3 methods: (1) annals , (ii) deregister , and (3) update . We're only going to use the update method as it will register and deregister objects automatically. Nosotros also initialize H and W (our frame dimensions) to None (Line 22).
Load our serialized deep learning face detector model from deejay using OpenCV's DNN module (Line 26).
Start our VideoStream , vs (Line 30). With vs handy, we'll be able to capture frames from our camera in our next while loop. Nosotros'll allow our camera 2.0 seconds to warm up (Line 31).

Now let's begin our while loop and start tracking face objects:

# loop over the frames from the video stream while Truthful: 	# read the next frame from the video stream and resize information technology 	frame = vs.read() 	frame = imutils.resize(frame, width=400)  	# if the frame dimensions are None, grab them 	if West is None or H is None: 		(H, W) = frame.shape[:2]  	# construct a blob from the frame, pass it through the network, 	# obtain our output predictions, and initialize the list of 	# bounding box rectangles 	blob = cv2.dnn.blobFromImage(frame, 1.0, (W, H), 		(104.0, 177.0, 123.0)) 	net.setInput(hulk) 	detections = internet.forrad() 	rects = []

We loop over frames and resize them to a fixed width (while preserving attribute ratio) on Lines 34-47. Our frame dimensions are grabbed as needed (Lines forty and 41).

Then we pass the frame through the CNN object detector to obtain predictions and object locations (Lines 46-49).

We initialize a listing of rects , our bounding box rectangles on Line 50.

From there, let's procedure the detections:

            # loop over the detections 	for i in range(0, detections.shape[2]): 		# filter out weak detections by ensuring the predicted 		# probability is greater than a minimum threshold 		if detections[0, 0, i, two] > args["confidence"]: 			# compute the (10, y)-coordinates of the bounding box for 			# the object, then update the bounding box rectangles list 			box = detections[0, 0, i, iii:7] * np.array([West, H, Due west, H]) 			rects.suspend(box.astype("int"))  			# draw a bounding box surrounding the object then we can 			# visualize information technology 			(startX, startY, endX, endY) = box.astype("int") 			cv2.rectangle(frame, (startX, startY), (endX, endY), 				(0, 255, 0), 2)

We loop over the detections beginning on Line 53. If the detection exceeds our confidence threshold, indicating a valid detection, we:

Compute the bounding box coordinates and append them to the rects list (Lines 59 and sixty)
Draw a bounding box around the object (Lines 64-66)

Finally, let'south call update on our centroid tracker object, ct :

            # update our centroid tracker using the computed fix of bounding 	# box rectangles 	objects = ct.update(rects)  	# loop over the tracked objects 	for (objectID, centroid) in objects.items(): 		# depict both the ID of the object and the centroid of the 		# object on the output frame 		text = "ID {}".format(objectID) 		cv2.putText(frame, text, (centroid[0] - 10, centroid[1] - ten), 			cv2.FONT_HERSHEY_SIMPLEX, 0.v, (0, 255, 0), 2) 		cv2.circle(frame, (centroid[0], centroid[1]), 4, (0, 255, 0), -ane)  	# evidence the output frame 	cv2.imshow("Frame", frame) 	fundamental = cv2.waitKey(one) & 0xFF  	# if the `q` key was pressed, intermission from the loop 	if key == ord("q"): 		break  # do a bit of cleanup cv2.destroyAllWindows() vs.stop()

The ct.update call on Line seventy handles the heavy lifting in our unproblematic object tracker with Python and OpenCV script.

We would exist done here and ready to loop back to the top if we didn't care about visualization.

Merely that's no fun!

On Lines 73-79 we brandish the centroid as a filled in circle and the unique object ID number text. Now we'll exist able to visualize the results and check to see if our CentroidTracker properly keeps track of our objects by associating the right IDs with the objects in the video stream.

We'll display the frame on Line 82 until the quit cardinal ("q") has been pressed (Lines 83-87). If the quit central is pressed, we just pause and perform cleanup (Lines 87-91).

Centroid object tracking results

To meet our centroid tracker in action using the "Downloads" section of this blog post to download the source lawmaking and OpenCV confront detector. From at that place, open up up a terminal and execute the following command:

$ python object_tracker.py --prototxt deploy.prototxt \ 	--model res10_300x300_ssd_iter_140000.caffemodel [INFO] loading model... [INFO] starting video stream...

Below you lot can see an example of a single face up (my face) existence detected and tracked:

This second example includes two objects being correctly detected and tracked:

Notice how I even though the second face is "lost" in one case I motility the book cover outside the view of the camera, our object tracking is able to choice the confront back up once more when it comes into view. If the face had existed outside the field of view for more than than 50 frames, the object would accept been deregistered.

The final instance animation here demonstrates tracking three unique objects:

Again, despite object ID #2 beingness unsuccessfully detected between some frames, our object tracking algorithm is able to notice it once more and associate information technology with its original centroid.

For a more than detailed demonstration of our object tracker, including commentary, be certain to refer to the video beneath:

Limitations and drawbacks

While our centroid tracker worked great in this example, there are two primary drawbacks of this object tracking algorithm.

The get-go is that it requires that object detection step to be run on every frame of the input video.

For very fast object detectors (i.e., color thresholding and Haar cascades) having to run the detector on every input frame is probable non an issue.
Only if yous are (i) using a significantly more computationally expensive object detector such as Grunter + Linear SVM or deep learning-based detectors on (ii) a resource-constrained device, your frame processing pipeline will slow downwardly tremendously as yous will be spending the unabridged pipeline running a very irksome detector.

The second drawback is related to the underlying assumptions of the centroid tracking algorithm itself — centroids must lie close together between subsequent frames.

This assumption typically holds, but continue in listen we are representing our 3D world with 2D frames — what happens when an object overlaps with another ane?
The respond is that object ID switching could occur.
If 2 or more than objects overlap each other to the bespeak where their centroids intersect and instead take the minimum distance to the other respective object, the algorithm may (unknowingly) swap the object ID.
It's of import to understand that the overlapping/occluded object problem is non specific to centroid tracking — it happens for many other object trackers as well, including avant-garde ones.
Withal, the problem is more pronounced with centroid tracking as we relying strictly on the Euclidean distances between centroids and no boosted metrics, heuristics, or learned patterns.

As long as you go along these assumptions and limitations in heed when using centroid tracking the algorithm will piece of work wonderfully for y'all.

Alternative object tracking methods

**Figure 5:** More advanced object tracking methods.

The object tracking method we implemented here today is called centroid tracking.

Provided nosotros can apply our object detector on each and every input frame, we can employ centroid tracking to take the results of the object detector and acquaintance each object with a unique ID (and therefore rail the object as it moves throughout the scene).

However, applying a dedicated object detector in each and every frame tin can be computationally expensive, especially if yous are using a resource constrained device (like the Raspberry Pi, for case), and may prevent your computer vision pipeline from running in real-time.

The following tutorial on OpenCV object tracking covers the eight more popular object trackers built into the OpenCV library:

BOOSTING tracker
MIL tracker
KCF tracker
CSRT tracker
MedianFlow tracker
TLD tracker
MOSSE tracker
GOTURN tracker

Alternatively, you lot may want to utilize the dlib library which includes a fantastic implementation of an accurate correlation tracker.

Once you understand how to apply OpenCV and dlib's object trackers, you can movement on to multi-object tracking:

Tracking multiple objects with OpenCV
Multi-object tracking with dlib

And from there, you can start building actual real-world projects that leverage object tracking:

OpenCV Vehicle Detection, Tracking, and Speed Estimation
OpenCV People Counter

What's next? I recommend PyImageSearch University.

Course information:
35+ full classes • 39h 44m video • Last updated: Apr 2022
★★★★★ 4.84 (128 Ratings) • thirteen,800+ Students Enrolled

I strongly believe that if you lot had the right teacher you could master reckoner vision and deep learning.

Do you lot think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in informatics?

That's not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in elementary, intuitive terms. And that's exactly what I do. My mission is to alter education and how circuitous Artificial Intelligence topics are taught.

If you're serious well-nigh learning computer vision, your side by side end should exist PyImageSearch University, the most comprehensive reckoner vision, deep learning, and OpenCV grade online today. Here yous'll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch Academy you lot'll discover:

✓ 35+ courses on essential computer vision, deep learning, and OpenCV topics
✓ 35+ Certificates of Completion
✓ 39+ hours of on-need video
✓ Brand new courses released regularly , ensuring you can go along up with state-of-the-art techniques
✓ Pre-configured Jupyter Notebooks in Google Colab
✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
✓ Admission to centralized code repos for all 450+ tutorials on PyImageSearch
✓ Like shooting fish in a barrel one-click downloads for code, datasets, pre-trained models, etc.
✓ Access on mobile, laptop, desktop, etc.

Click here to bring together PyImageSearch University

Summary

In today's blog post yous learned how to perform simple object tracking with OpenCV using an algorithm chosen centroid tracking.

The centroid tracking algorithm works by:

Accepting bounding box coordinates for each object in every frame (presumably by some object detector).
Computing the Euclidean distance between the centroids of the input bounding boxes and the centroids of existing objects that nosotros already have examined.
Updating the tracked object centroids to their new centroid locations based on the new centroid with the smallest Euclidean distance.
And if necessary, mark objects equally either "disappeared" or deregistering them completely.

Our centroid tracker performed well in this example tutorial but has 2 primary downsides:

It requires that we run an object detector for each frame of the video — if your object detector is computationally expensive to run you would not desire to utilize this method.
It does not handle overlapping objects well and due to the nature of the Euclidean distance between centroids, it'due south actually possible for our centroids to "bandy IDs" which is far from ideal.

Despite its downsides, centroid tracking tin be used in quite a few object tracking applications provided (i) your environment is somewhat controlled and you don't have to worry virtually potentially overlapping objects and (2) your object detector itself can be run in real-time.

If you enjoyed today's blog postal service, be sure to download the code using the class below. I'll exist back next week with some other object tracking tutorial!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resources Guide on Computer Vision, OpenCV, and Deep Learning. Within you'll discover my hand-picked tutorials, books, courses, and libraries to aid you lot primary CV and DL!

tillyardmannion.blogspot.com

Source: https://pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/

Tillyard Mannion