Collection of patches is a better choice if you really want to make it similar to plt.scatter. Suppose you want to plot a scatter of circles with given radius in data unit, you may take a look at this post which provide a well wrapped function named circles.

If you want to use other markers like square or polygon, you may rewrite this function with the correspondent patches as said in answer of dr jimbob.

from pylab import *
figure(figsize=(6,4))
ax = subplot(aspect='equal')

#plot a set of circle
a = arange(11)
circles(a, a, s=a*0.2, c=a, alpha=0.5, ec='none')
colorbar() 

#plot a circle (the lower-right one)
circles(1, 0, s=0.4, c='r', ls='--', lw=5, fc='none', transform=ax.transAxes)

xlim(0,10)
ylim(0,10)

Here is the solution for your Question,

# import the necessary packages
from collections import deque
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--video",
    help="path to the (optional) video file")
ap.add_argument("-b", "--buffer", type=int, default=32,
    help="max buffer size")
args = vars(ap.parse_args())

# define the lower and upper boundaries of the "green"
# ball in the HSV color space
greenLower = (29, 86, 6)
greenUpper = (64, 255, 255)

# initialize the list of tracked points, the frame counter,
# and the coordinate deltas
pts = deque(maxlen=args["buffer"])
counter = 0
(dX, dY) = (0, 0)
direction = ""

# if a video path was not supplied, grab the reference
# to the webcam
if not args.get("video", False):
    camera = cv2.VideoCapture(0)

# otherwise, grab a reference to the video file
else:
    camera = cv2.VideoCapture(args["video"])

# keep looping
while True:
    # grab the current frame
    (grabbed, frame) = camera.read()

    # if we are viewing a video and we did not grab a frame,
    # then we have reached the end of the video
    if args.get("video") and not grabbed:
        break

    # resize the frame, blur it, and convert it to the HSV
    # color space
    frame = imutils.resize(frame, width=600)
    # blurred = cv2.GaussianBlur(frame, (11, 11), 0)
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

    # construct a mask for the color "green", then perform
    # a series of dilations and erosions to remove any small
    # blobs left in the mask
    mask = cv2.inRange(hsv, greenLower, greenUpper)
    mask = cv2.erode(mask, None, iterations=2)
    mask = cv2.dilate(mask, None, iterations=2)

    # find contours in the mask and initialize the current
    # (x, y) center of the ball
    cnts = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL,
        cv2.CHAIN_APPROX_SIMPLE)[-2]
    center = None

    # only proceed if at least one contour was found
    if len(cnts) > 0:
        # find the largest contour in the mask, then use
        # it to compute the minimum enclosing circle and
        # centroid
        c = max(cnts, key=cv2.contourArea)
        ((x, y), radius) = cv2.minEnclosingCircle(c)
        M = cv2.moments(c)
        center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))

        # only proceed if the radius meets a minimum size
        if radius > 10:
            # draw the circle and centroid on the frame,
            # then update the list of tracked points
            cv2.circle(frame, (int(x), int(y)), int(radius),
                (0, 255, 255), 2)
            cv2.circle(frame, center, 5, (0, 0, 255), -1)
            pts.appendleft(center)

    # loop over the set of tracked points
    for i in np.arange(1, len(pts)):
        # if either of the tracked points are None, ignore
        # them
        if pts[i - 1] is None or pts[i] is None:
            continue

        # check to see if enough points have been accumulated in
        # the buffer
        if counter >= 10 and i == 1 and pts[-10] is not None:
            # compute the difference between the x and y
            # coordinates and re-initialize the direction
            # text variables
            dX = pts[-10][0] - pts[i][0]
            dY = pts[-10][1] - pts[i][1]
            (dirX, dirY) = ("", "")

            # ensure there is significant movement in the
            # x-direction
            if np.abs(dX) > 20:
                dirX = "East" if np.sign(dX) == 1 else "West"

            # ensure there is significant movement in the
            # y-direction
            if np.abs(dY) > 20:
                dirY = "North" if np.sign(dY) == 1 else "South"

            # handle when both directions are non-empty
            if dirX != "" and dirY != "":
                direction = "{}-{}".format(dirY, dirX)

            # otherwise, only one direction is non-empty
            else:
                direction = dirX if dirX != "" else dirY

        # otherwise, compute the thickness of the line and
        # draw the connecting lines
        thickness = int(np.sqrt(args["buffer"] / float(i + 1)) * 2.5)
        cv2.line(frame, pts[i - 1], pts[i], (0, 0, 255), thickness)

    # show the movement deltas and the direction of movement on
    # the frame
    cv2.putText(frame, direction, (10, 30), cv2.FONT_HERSHEY_SIMPLEX,
        0.65, (0, 0, 255), 3)
    cv2.putText(frame, "dx: {}, dy: {}".format(dX, dY),
        (10, frame.shape[0] - 10), cv2.FONT_HERSHEY_SIMPLEX,
        0.35, (0, 0, 255), 1)

    # show the frame to our screen and increment the frame counter
    cv2.imshow("Frame", frame)
    key = cv2.waitKey(1) & 0xFF
    counter += 1

    # if the 'q' key is pressed, stop the loop
    if key == ord("q"):
        break

# cleanup the camera and close any open windows
camera.release()
cv2.destroyAllWindows()

It's a bit annoying that it's inconsistent, but to update the position of a circle, set circ.center = new_x, new_y.

As a simple (non-draggable) example:

import matplotlib.pyplot as plt
from matplotlib.patches import Circle

class InteractiveCircle(object):
    def __init__(self):
        self.fig, self.ax = plt.subplots()
        self.ax.axis('equal')

        self.circ = Circle((0.5, 0.5), 0.1)
        self.ax.add_artist(self.circ)
        self.ax.set_title('Click to move the circle')

        self.fig.canvas.mpl_connect('button_press_event', self.on_click)

    def on_click(self, event):
        if event.inaxes is None:
            return
        self.circ.center = event.xdata, event.ydata
        self.fig.canvas.draw()

    def show(self):
        plt.show()


InteractiveCircle().show()

Thank you very much, that did the trick! Since I always advocate the RTFM concept, I'm wondering where I could find this information? I can't see it in patches.Circle, patches.Ellipse, patches.Patch, or artist.Artist.

Unfortunately, it's undocumented. I had to dig through the code to find it (ipython is also a big help). These sort of "corners" of matplotlib definitely need better documentation...

@JoeKington So how do I do this for an arc? I don't want to move its centre rather I want to change its starting and ending theta i.e. the spread of the arc.

An alternative to unutbu's answer, is to use the copy package, which can copy objects. It is very hard to see how things are changing after one calls add_patch, but they are. The axes, figure, extents,clip_box,transform and window_extent properties of the patch are changed. Unfortantely the superficial printing of each of these properties results in the same string, so it looks like they are not changing. But the underlying attributes of some or all of these properties, eg extents is a Bbox, are probably changed.

The copy call will allow you to get a unique patch for each figure you make, without know what kind of patch it is. This still does not answer why this happens, but as I wrote above it's an alternative solution to the problem:

import copy 

def doplot(x,y,patch,count):
    newPatch = copy.copy(patch)
    fig = plt.figure(dpi=50)
    ax = fig.add_subplot(111)
    im = ax.scatter(x,y)
    ax.add_patch(newPatch)
    im.set_clip_path(newPatch)
    plt.savefig(str(count) + '.png')

Also you can use fig.savefig(str(count) + '.png'). This explicitly saves the figure fig where as the plt.savefig call saves the current figure, which happens to be the one you want.