MeshMeld

Melding Technology

Vancouver, BC

Python pool.map

During some recent data processing, a nifty built-in library was brought to my attention. Multiprocessing pool.map function, short little snippet below.

from multiprocessing import Pool

def process(data):
    print "{+} Processing: %s" % data
    #do something useful

if __name__ == '__main__':
    data = [1,2,3,4,5,6]
    pool = Pool(processes=6)
    pool.map(process, data)
    pool.close()
    pool.join()

In short, you can with one line almost turn any array that needs to be process into a multiprocess map reduce party.

From a python processing, if you have isolated data, this will allow you to scale in a pretty aggresive way.

MessagePack vs JSON

MessagePack is a binary format that is very similair in terms of feature it supports to JSON. Just like JSON it is supported by a lot of different programming languages.

It brings to the table a few intersting things.

  • It is faster then JSON (Under python, I found it slower under windows, but faster under linux. This had to do with the binary package)
  • Smaller, the data dumps are smaller.
  • If the data is UTF8/Unicode, the size can be 20% smaller if not better.

Currently using it in a few projects, will see if I run into any issues with it. Only time will tell.

pygeoip vs GeoIP

Playing around with two different python geoip lookup libraries thought I would mention some of my findings.

The two libraries in question are pygeoip and GeoIP.

pygeoip, is pure python, making it an easier install, but it is SLOW, just how slow? We will get to that. Other then that, it also deals with odd charaters, extra spaces, tabs and other charaters around the ip address handed to it for lookup gracefully, providing a location.

GeoIP, built ontop of the C library, it is a LOT faster, but it is also less graceful. Have an extra tab at the end of that ip? Yup, let me return None, extra space? None... So be aware if you are moving code from one to the other you end up needing to do more data sanitation.

In terms of speed GeoIP is 3x faster then pygeoip. If you are doing a lot of lookups, in my case I am, that accounts for a massive amount of compute time.

V-Strom 1000 2014 Build Up

I got a new motorcycle a few weeks back, the 2014 V-Strom 1000. I have big plans for it. Here are some of the add-ons/improvements I would like to make to it. And I will keep updating this as I add and build up the bike.

  • Crash Bars [Givi] [Installed]
  • Hand guards
  • Skidplate
  • Centerstand [SW-Motech] [Installed]
  • Soft spot protection,
    • exhaust valve [SW-Motech] [Installed]
    • rear brake reservoir
    • rear brake cylinder
  • Luggage rack
  • Luggage
  • Tank bag

HackRF One and SDR#

I got my HackRF One shortly after defcon, and started playing around with it. I was having a lot of issues getting anything usefull out of it, and then I updated the firmware on the unit with the latest compiled one off the hackrf github, 2014.4.1 And that seems to have solved almost all my issues.

HackRF and SDR# Now picking up all the local FM broadcast at the same time, or almost, 20mhz of bw. Not bad at all.