Category: Scripting

Converting .MPG to .AVI with Linux programs

I am trying to convert .mpg files from a sony camera into other, smaller versions for distribution. The files are in the mpeg2video codec, with a resolution of 720×480 at 29.97 fps. Normally, I’d use ffmpeg for this, but apparently “This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.” Thanks Ubuntu, I’m sure that won’t screw up a few thousand people’s video conversion scripts.

Anyway, let’s see what man avconv has to say. I want no sound, and I want to start 55 seconds into the film, to cut off most of a boring first minute. Seeking into the video is -ss 00:00:55. The option -an drops the audio. So, avconv -i blue_ball.mpg -an -ss 00:00:55 blue_ball.avi should do it. And, in fact, it does.

Drinking with Robots

I am building a drink-dispensing robot. It has 5 pumps internally, so I want to find the set of five liquids that will produce the largest variety of mixed drinks. To do this, I’m going to need a huge set of drink recipes. I got a bunch from a cocktail database that esquire maintains, but the biggest list I’m aware of is The Webtender. Unfortunately, that database isn’t in a form that permits me to make queries to find out what is the maximal set of drinks that I can make, given the constraint of 5 liquid ingredients. So instead of the web interface, I want the raw data.

This means I want to download every single drink recipe from The Webtender. The URLs there are of the form http://www.webtender.com/db/drink/6217, so the obvious thing to do would be to write up a little bash oneliner that just wgets each of the files in turn. It would probably look something like this:

for (( i=1; i <= 6217; i++ )); do wget http://www.webtender.com/db/drink/$i; done 

But it may be that The Webtender issues a 403 Forbidden error if you use wget, probably to prevent just this sort of hijinks. Unfortunately for them, wget can be configured to claim to be something else. These instructions provide the config file to cause wget to claim to be Mozilla on Windows NT, which The Webtender should have no problem with.

For my next trick, I'll use BeautifulSoup to turn the HTML files from the webtender and Esquire into a SQL database, and probably perform some form of normalization on the data, such as making all the measurements be in the same units.

Academic Problems in New Media Art

I’m trying to convert Deluze & Guattari’s A Thousand Plateaus into a corpus for use by a program. Among other things, the program will break the text down into its component sentences. The text has a lot of notes, which connect the text to a lot of other sources, but are not always written in complete sentences, and so will result in odd output from the program when considered as sentences.

So I’m faced with a choice: lose the notes, and so lose the cultural context and references to stuff that went before, or keep them and suffer degraded output. Nobody warns you about the odd stuff you’ll have to decide when you start doing new media “art”.

Tags :

Coloring cells based on cell value in OpenOffice

OpenOffice (probably LibreOffice too) has limited functionality for setting the background color of a cell. I wrote a little macro that colors each cell blue, with saturation depending on the range of the value. This is in OOBasic, but it demonstrates the general idea, and the proper way of setting cell background colors.

Sub Main

End Sub

sub ColorCells

dim ii
dim jj
dim r
dim g
dim b
dim doc as object
dim sheet as object
dim cell as object

doc = ThisComponent
sheet = doc.Sheets.getByName("Sheet1")

for ii = 0 to 5
	for jj = 0 to 626
		cell = sheet.getCellByPosition(ii, jj)
		r = 255 - (cell.Value * 255)65536
		g = 255 - (cell.Value * 255)65536
		b = 255
		cell.CellBackColor = RGB(r,g,b)
	next jj
next ii

end sub

Counting Instances of Things

This is something I seem to do a lot. Most recently, I was trying to see how many nodes of each degree there were in a graph, but it could also be used for word counts and such.

#Run through a list and count occurrences of each item,
#store them in a dictionary of items to counts
def bin(list):
    bins = {}
    for item in list:
        if item in bins.keys():
            bins[item] += 1
        else:
            bins[item] = 1
    return bins

The key thing here is not really the little bit of python glue code to do the counting. What is key is to learn to recognize things that you do more than once, and automate them. A folkloric axiom of computer programming is that you need to do something either exactly once, or an unknown number of times. If you have to do it once, then the program should do it once. If it’s the other case, then the program should determine how often to do it, and then do it.

For example, if the program trims the whitespace off the end of a text file, it should either do it once (which allows the user to call it in a shell script loop if they want to process any number of files), or do it to any number of files. The alternative, having it process some arbitrary number of files and then fail, has two problems. First, it adds complexity. The program now has to keep a counter, check it, and fail when it hits a certain value. Second, the program, which previously worked in an infinite number of cases (that is, it would process N files for all N such that N is a positive integer), now has an infinite number of failure cases (fail for all integers greater than M), and has gone from complete success to complete failure, via the most difficult route.

Chatbot is done

I finished writing the chatbot that I was working on. It consists of a set of scripts to prepare the data, and another script that listens for incoming messages and responds. You can get the code and an overview of how it works here.

Obviously, I’m not publishing my chat logs. Use your own. It is designed to work with Pidgin’s HTML-like format for chat logs, but it could be modified to work on almost any corpus. I really should clean up things like the string cleaning routines, but it worked for class, and that’s what actually matters.

Python Pidgin Dbus Monitor

This is a script that registers a callback with Pidgin’s Dbus interface, and then sends a message in response whenever anyone sends an IM. I’m using it for a chatbot, but it could easily be extended to do things like switch X10 light controllers on and off, get the state of hardware connected to the target computer, and so forth. Pidgin has to be running and configured to use Dbus, but it does that by default.

'''
Created on Nov 18, 2011
Watches pidgin over Dbus and responds to incoming messages.
'''

import dbus
from dbus.mainloop.glib import DBusGMainLoop
import gobject

class StupidResponder():
    def getResponse(self, message):
        #This is where you would do something clever to come up with a response
        return "Insect! I cannot bear your words! They are TOO TINY!"

def got_msg_cb(account, sender, message, conversation, flags):
    purple.PurpleConvImSend(purple.PurpleConvIm(conversation), responder.getResponse(message))

if __name__ == '__main__':

    #load a response generator
    responder = StupidResponder()

    #Connect to pidgin on Dbus
    main_loop = DBusGMainLoop()
    session_bus = dbus.SessionBus(mainloop = main_loop)
    obj = session_bus.get_object("im.pidgin.purple.PurpleService", "/im/pidgin/purple/PurpleObject")
    purple = dbus.Interface(obj, "im.pidgin.purple.PurpleInterface")

    #Add the callback
    session_bus.add_signal_receiver(got_msg_cb, dbus_interface="im.pidgin.purple.PurpleInterface", signal_name="ReceivedImMsg")

    #Listen
    loop = gobject.MainLoop()
    loop.run()

Crazy Talk

This is a program that I wrote about ten years ago, to generate nonsense from a given input text. It builds a statistical model of the input text, which allows it to determine, given any two words, what the most likely third word is. Then it can use the last two words of those three to pick a new word, and so on. Since it doesn’t always pick the most likely word, but selects randomly from all options based on likelihood, it tends to create abrupt switches of concepts around short phrases like “it to” or “in the”.

For bonus points, it posts to a Blogger blog, assuming they haven’t changed their API.

It’s written in Perl, and I assume still works. If not, updating it to Python would be fun, and a good excuse for me to practice my Python.

Read more

More Deskewing Rectangles with OpenCV

Someone recently posted to a mailing list that I read, asking for a program that can rotate and crop a lighter-colored rectangle on a black background. Since this fits in with the program I’m working on to locate, rotate, and deskew an image of a card in a photo, I figured I’d give it a shot. I’m planning to do both rotation and deskewing, but for now I’m only doing the rotation.

I’ve installed the latest OpenCV (version 2.3, as of this writing), and started learning the new Python bindings.  There seem to be a lot of cases where functions want a numpy array as an argument, but the functions don’t return numpy arrays, so there is a lot of fiddling around with the results of functions to get something that can be passed to another function.

On top of that, the return values of functions are poorly or rarely documented. Python’s duck-typing can help out a little in these cases, but if you call cv2.minAreaRect, you get something like ((233, 432), (565, 420), -0.2343), which is described in the documentation with the single word “retval”.  It would be helpful to have a way to find out that the first tuple is the center, the second is the width and height, and the third is the tilt (in degrees) of the “first” edge off of horizontal. That tuple makes just as much sense, but is wrong, when interpreted as the top left and bottom right corners and a tilt measured in radians.

Also, the “first” edge is seemingly arbitrary, or at least I can’t find any documentation describing it. This means that the same rectangle could be off by 0.4 degrees or by -89.6 degrees, depending on if the first edge is a horizontal or vertical edge. One thing that may be helpful is this stackoverflow post. Since the rectangle is defined as points, I can reshuffle the points to get them in a consistent order, and then get the angle off of horizontal for a consistent edge. That then goes into producing the transformation matrix for affine transforms (e.g. rotation and deskewing).

The minAreaRect() call gets me the angle I can use to rotate the image, and this trick should get me the four corners of the perspective-skewed image that can be straightened out to get the squared image.

Displaying contours in OpenCV

Suppose you have used FindContours to find the outlines of things in an image, and now want to color each one a different color, perhaps in order to create a seed map for the watershed algorithm. This gets you an image with each area that FindContours found in a different color:

#Showing the contours.
#contour_list is the output of cv.FindContours(), degapped is the image contours were found in
contour_img = cv.CreateImage(cv.GetSize(degapped), IPL_DEPTH_8U, 3)
contour = contour_list
while contour.h_next() != None:
    color = get_rand_rgb(80,255)
    holecolor = get_rand_rgb(80,255)
    cv.DrawContours(contour_img, contour, color, holecolor, -1, CV_FILLED)
    contour = contour.h_next()
show_img(contour_img, "contours " + str(iteration))

The real key here is iterating over the list of contours using h_next(), which took me longer than it should have to find.

The show_img() function is effectively just printf for OpenCV images, and looks like this:

def show_img(img, winName):
    #Debugging printf for images!
    cv.NamedWindow(winName)
    cv.ShowImage(winName, img)
    cv.WaitKey(0)
    cv.DestroyWindow(winName)

The function get_rand_rgb() gets a random RGB color with values for each color set to an integer in the range you pass it, like so:

def get_rand_rgb(min, max):
    return (random.randint(min,max),random.randint(min,max),random.randint(min,max))

I used 80,255 to get bright colors.