DEVPYTHON

Post files to S3 with Boto

jc — Fri, 26 Aug 2011 02:38:06 +0000

Part of my job lately has been generating periodic CSVs (maybe I’ll talk more about this in a later post). To share them with my co-workers I decided that the apt thing to do was to pop them on S3 where storage is cheap and secure. To do this I wrote the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from boto.s3.connection import S3Connection
from boto.s3.key import Key

#... bunch of other cool stuff

conn = S3Connection('', '')
bucket = conn.get_bucket('your_bucket')

k = Key(bucket)
# what you want the file to be named on S3
k.key = 'your_data.csv'
# file to be uploaded
csv_file = '/your_data.csv'
k.set_contents_from_filename(csv_file)

Replacing the tedious work with 5 lines of code.

Convert UTC date string to unix timestamp

jc — Thu, 11 Aug 2011 17:38:19 +0000

So you want to convert a UTC date string to a unix timestamp ? e.g. “2011-08-08T12:10:20+0000″ This particular UTC date string was from a Facebook /feed.

1
2
3
4

>>> import dateutil.parser
>>> d = "2011-08-08T12:10:20+0000"
>>> dateutil.parser.parse(d).strftime("%s")
'1312823420'

Enjoy!

Update:

If you’re having issues like the following:

1	TypeError: iter() returned non-iterator of type '_timelex'

Then more than like you’re using Python 2.x and dateutil 2.0 (as listed here) Just downgrade to the compatible Python 2.x version like so:

1	$ easy_install python-dateutil==1.5

How many times does an item occur in a collection?

jc — Sun, 07 Aug 2011 17:24:48 +0000

So you want to count the frequency of an item in a collection? In Python 2.7 and 3.1 there’s a number of new high performance container datatypes, namely the Counter.

Example

1
2
3
4

>>> a = ['a','a','a','b','b','c','c','c','c','c','c']
>>> from collections import Counter
>>> Counter(a)
Counter({'c': 6, 'a': 3, 'b': 2})

Pretty bad ass.