–use_sqlite parameter was not even solving my problem. Inserts on the database were really slow, like 10 per second. A nightmare.

OK, ok, it is an SDK, it emulates the server… it is intended to be used to develop your application, speed is not the most important thing… I agree. But if you have to wait 10 minutes to insert 1000 data entries, then you can desperate. If they were like 1 million in 10 minutes, then I would understand, it would be still slow, but I will understand, but 1000 inserts? 10 minutes? are we all crazy?

Anyway, assuming SQLite seems to be a good database and works usually great, I though I could get that speed improved. Looking at the SQLite FAQ I found following question:

(19) INSERT is really slow – I can only do few dozen INSERTs per second

Basically it seems SQLite should be fine doing 50,000 inserts per second in a normal computer, BUT to guarantee the data integrity it locks inserts until it makes sure they have been written to disk, so if your computer hangs your data remains safe.

These are good news because I don’t care of my local data in the local develoment environment, so in the event of a crash I can just rebuild it in a minute, instead of rebuild it in an hour.

Finally the solution is to use “PRAGMA synchronous=OFF“, and the only way of using this pragma is to edit a line in de Google App Engine SDK, after initializing SQLite database

So you’ll have to edit following file:

google_appengine/google/appengine/datastore/datastore_sqlite_stub.py

Look for the “__init__” constructor and after the connection to the database gets initialized “self.__connection = sqlite3.connect [...]” you can place following statment:

self.__connection.execute(‘PRAGMA synchronous=OFF’)

Changing that line I went from about 10 inserts/second to more than 100 inserts/second. More than a 10x gain for changing a single line. Great!

Now I can continue on the thing I was working on without getting crazy.

With the call above we have 32 hexadecimal characters string (16 random bytes).

Do you know any other approach?

def ampmformat (hhmmss):
  """
    This method converts time in 24h format to 12h format
    Example:   "00:32" is "12:32 AM"
               "13:33" is "01:33 PM"
  """
  ampm = hhmmss.split (":")
  if (len(ampm) == 0) or (len(ampm) > 3):
    return hhmmss

  # is AM? from [00:00, 12:00[
  hour = int(ampm[0]) % 24
  isam = (hour >= 0) and (hour < 12)

  # 00:32 should be 12:32 AM not 00:32
  if isam:
    ampm[0] = ('12' if (hour == 0) else "%02d" % (hour))
  else:
    ampm[0] = ('12' if (hour == 12) else "%02d" % (hour-12))

  return ':'.join (ampm) + (' AM' if isam else ' PM')

Now some examples of the returned values:

ampmformat ("00:00:00") # returns "12:00:00 AM"
ampmformat ("12:00:00") # returns "12:00:00 PM"

ampmformat ("01:23:45") # returns "01:23:45 AM"
ampmformat ("13:23:45") # returns "01:23:45 PM"
ampmformat ("05:43:21") # returns "05:43:21 AM"
ampmformat ("17:43:21") # returns "05:43:21 PM"

ampmformat ("11:59:59") # returns "11:59:59 AM"
ampmformat ("23:59:59") # returns "11:59:59 PM"

This is called geocoding and after spending half a second looking searching on internet I’ve found two simple REST APIs. Yahoo! and Google provide their own APIs and I’m sure a lot of companies out there make use of it, because of their accuracy.

If you want to use Yahoo! API you need to sign up so they give you some sort of appid and you can start making queries.

In the other hand, Google let’s you do this as easily as:

http://maps.googleapis.com/maps/api/geocode/json?address=Emerson+Street,+Palo+Alto,+california&sensor=false

This URL returns a JSON response which can be easily parsed in all languages (you can also ask for the XML, but I sort of hate XML so I prefer JSON).

Apart from returning the latitude and longitude, it also returns the formatted address, which can be great to normalize the address in your database, in case you need such thing. For example, for the query before, it says the formatted address is:

“Emerson St, Palo Alto, CA, USA”

On top of that, you should always read the Terms Of Service to make sure your app stays aligned with the fair use of the API, but that’s up to you.

Although more inacurate, another interesting project is GeoNames (see the URL below).

For more information:

• Google Geocoding API:
  http://code.google.com/apis/maps/documentation/geocoding/index.html
• Yahoo! Place Finder:
  http://developer.yahoo.com/geo/placefinder/
• GeoNames geographical database:
  http://www.geonames.org/

One can get just crazy, sometimes things work, sometimes things crash somewhere unexpectly, or they don’t really crash, they just crash when you want to show some string on the console doing a print…

Anyway, trying to solve my frustrations with unicode strings in Python I found a link that explains lots of things and helped me a lot:

Overcoming unicode frustrations in Python 2

The previous link is a must read for people getting UnicodeEncodeError exceptions, like:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 654: ordinal not in range(128)

Finally, my hope is that Python 3 solves all the evil in the world of the unicode… but I need to see it working to truly believe what I’ve been reading so far…

The thinkgs I was considering were:

• compact representation
• fast serialization/deserialization
• human-readable / human-editable a must

So after thinking for a while, I found 3 possible solutions to serialize arbitrary data:

• pickle/cPickle module for serializing in the internal pickle format which comes with python by default
• simplejson for serializing in JSON format which is human readable
• PyYAML for serializing in YAML format which is human readable

The next step was to get a huge amount of data, and do the tests for both speed and length of the resulting data. Following you can appreciate the speed results for two data sets:

The bars above represent the time in seconds it took to encode/decode the dataset 100 times (although numbers go from 0 to 100, are absolute times, not percentages).

Apart from the speed, I also measured the lenght of the serialized data obtained:

• yaml tiny = 32604 // huge = 912912
• json tiny = 31305 // huge = 876540
• pickle-raw tiny = 34504 // huge = 986531
• pickle-highest tiny = 37541 // huge = 1101480

Conclussion:

According to the results, JSON serialization/deserialization is the best choice. It encodes/decodes slightly slower than pickle module on huge datasets (we are talking about 4% slower, not that much), and the resulting string is not only human readable, but the resulting output is smaller than pickle and yaml encoded strings.

All tests were made on Python 2.5

The code

This is a fragment of the code used to get the speed results:

    start = time.time()
    for i in range (0, 100):
      yaml.safe_load (yaml.safe_dump (data))
    self.write ("YAML TIME = %.2f\n" % (time.time() - start))
 
    start = time.time()
    for i in range (0, 100):
      JSONDecoder().decode (JSONEncoder().encode (data))
    self.write ("JSON TIME = %.2f\n" % (time.time() - start))
 
    start = time.time()
    for i in range (0, 100):
      pickle.loads (pickle.dumps (data, pickle.HIGHEST_PROTOCOL))
    self.write ("PICKLE TIME = %.2f\n" % (time.time() - start))
 
    start = time.time()
    for i in range (0, 100):
      pickle.loads (pickle.dumps (data, 0))
    self.write ("PICKLE-RAW TIME = %.2f\n" % (time.time() - start))

After looking on the Internet I decided to create bbcodeutils a Python module to parse, generate and transform bbcode. I created in a way that I think is really simple to use.

Inside bbcodeutils you will find following classes:

• bbcodeparser: parses bbcode and fixes invalid strings
• bbcode2html: gets a parsed bbcode string and generates HTML from it
• bbcodebuilder: good to generate BBCode programmatically

Here you can find the link to download bbcodeutils v1.0. You can use this module freely (see readme.txt for details).

Following there are some examples with the most common operations.

Examples:

Before you try any example I’m assuming you will do something to load the bbcodeutils module, like:

from bbcodeutils import bbcodebuilder, bbcodeparser

Parsing bbcode strings

Parsing a bbcode string can be achieved using the bbcodeparser constructor, or calling parse method on a bbcodeparser object:

> bbcode = bbcodeparser('[b]bold string[/b]')

or:

> bbcode = bbcodeparser()
> bbcode.parse ('[b]first string[/b]')

This is just to illustrate how to parse a string, but this won’t produce any output at all. What you are probably looking for is to fix an invalid bbcode string or to transform a bbcode string into valid HTML. Here we go…

Fix an invalid BBCode string

To fix an invalid bbcode string you just have to parse a string and stringify the parser object or to call the bbcode method.

> str(bbcode)
"This is [b]bold and [i]this bold and italics[/i][/b]."
 
> bbcode = bbcodeparser("This is [b]bold and [i]this bold and italics[/b].[/color]")
> bbcode.bbcode()
"This is [b]bold and [i]this bold and italics[/i][/b]."

Please note that the [/color] tag is removed and the closing [/i] tag is added properly.

Transform bbcode to HTML

The class bbcodeparser implements html method which uses bbcode2html internally. Calling this function will return valid HTML code.

> bbcode = bbcodeparser('[b]bold string[/b]')
> bbcode.html()
"<b>bold string</b>"

Create BBCode programmatically

> bbcode = bbcodebuilder()
> bbcode.b('string in bold')
"[b]string in bold[/b]"
 
> bbcode.url('http://www.codigomanso.com/')
"[url]http://www.codigomanso.com/[/url]"
 
> bbcode.url('Welcome to Codigo Manso', 'http://www.codigomanso.com/')
"[url=http://www.codigomanso.com/]Welcome to Codigo Manso[/url]"
 
> bbcode.list('item 1', 'item 2')
"[list]
   [*]item 1
   [*]item 2
[/list]"

Common operations in one line

Fix invalid BBCode in one line:

> str(bbcodeparser("[color=red]Fix this [b]string"))
"[color=red]Fix this [b]string[/b][/color]"

Transform BBCode to HTML in one line:

> str(bbcodeparser("[color=red]Fix this [b]string"))
'<span style="color: red;">Fix this <b>string</b></span>'

Thats it!

If you have any doubt feel free to leave a comment on this post

Enlaces de interés:

• bbcodeutils v1.0 for Python
• Referencia de BBCode
• Entrada sobre BBCode en la Wikipedia

Google App Engine rules! The truth is that I’m starting to feel confortable programming in Python, although I still like the curly braces to indentate.

Anyway, the datastore used by Google is superpowerful and supersimple to use, but it has some limitations. With the App Engine SDK you can easily say which attributes you want to index (indexed = True), which ones you want to be required (required = True), which is the default value (default=”whatever”) but you cannot add a unique constraint on an attribute or set of attributes (there is no unique=True).

This is the typical nick or e-mail field in a database. You don’t want two users share the same nick or the same e-mail. You want all users to be identified by their nicks or their e-mails.

¿What’s the problem? As I said, you can’t do that using the Google App Engine SDK… hopefully the solution is really easy

The solution:

The only thing you have to do is to override put inside our model, and launch an exception when a new element that violates our unique constraints is going to be added to the database.

In order to provide a simple example, let’s create a model named User using name, email, language and creation_date as the attributes:

class User (db.Model): 
   creation_date = db.DateTimeProperty (auto_now_add=True)
   name          = db.StringProperty (required = True)
   email         = db.EmailProperty (required = True)
   language      = db.StringProperty (default="en")

Now, to reproduce the problem in GAE, the following code will create two different entries:

User (name="Mark Johnson", email="test@codigomanso.com").put()
User (name="JohnMarkinson", email="test@codigomanso.com", language="es").put()

Now let’s add a constraint to forbid two users to share the same e-mail. With this constraint the second putshould fail or launch an exception. ¿How do we do that? It’s as simple as it seems. We just need to redefine put so when an new element is going to be added to the database, we check if violates our rules and we launch an exception.

The new class with the constraints would be as follows:

class User (db.Model): 
   creation_date = db.DateTimeProperty (auto_now_add=True)
   name          = db.StringProperty (required = True)
   email         = db.EmailProperty (required = True)
   language      = db.StringProperty (default="en")
 
   def put (self):
     # Make sure e-mails are unique for each user
     if (not self.is_saved()) and (User.gql ('WHERE email = :1', self.email).count() > 0):
       raise DuplicatedInstanceError ('User.email', self.email)
 
     # call the parent method
     db.Model.put (self)

We needed only 3 lines of code (4 counting the def line)… it can’t be easier

Some inconveniences

After thinking on it for a while, this implementation is far from being perfect, although it will work on most cases. I see two problems:

• You have to make an extra query before inserting an item (updates behave as always)
• Using this solution you can end up with two users having the same e-mail. How can that be? Have you heard about race conditions? The count-put operation is not atomic, so if two users try to register the same e-mail at almost the same time (you will have two instances of the application running), then it can happen that they do the count at the same time (returning zero) and so none of both will raise an exception.

The first problem is not a problem at all, just something to be aware of.

The second problem it can be a huge problem. There are two solutions that come to my mind right now, one is to run the count-put in a transaction for new instances (transactions are atomic), and the other one would be to create a mutex around the count-put. To me, this is something I’m not worried about, but you need to know that it can happen.

Apendix: DuplicatedInstanceError

If you are smart, you have probably realiced that I launched an exception of the type DuplicatedInstanceError. This exception does not exist, I created to be able to detect duplicated instances. My implementation is below, but you can use whatever you want:

class DuplicatedInstanceError (Exception):
  def __init__(self, attrpath, value = None):
    '''
    Accept a dict of attribute and value pairs, or just one attribute and a value:
 
    Example:
       # user unique constraint has been violated (User.nick to be unique)
       DuplicatedInstanceError ('User.nick', 'jsmith')
 
       # student unique constraint has been violated (student nick's are fin for different schools)
       DuplicatedInstanceError ({
         'Student.nick'   : 'jsmith',
         'Student.school' : 'demo-primary-school'
       })
    '''
    self.attrpath = ''
 
    # accept a dict as the only parameter
    if isinstance (attrpath, dict):
      for (k, v) in attrpath.items():
        self.attrpath += ', ' + str (k) + ' = ' + str(v)
      self.attrpath = self.attrpath[2:]
    # accept a couple of strings
    else:
      self.attrpath = str(attrpath) + ' = ' + str(value) 
 
    self.attrpath = '(' + self.attrpath + ')'
    return
 
  def __str__ (self):
    return str (self.attrpath)

That’s all!

I know the following code might not seem the best code in the world, but keep in mind that I’m just looking for a simple function that does the job.

def stripHTMLTags (html):
  """
    Strip HTML tags from any string and transfrom special entities
  """
  import re
  text = html
 
  # apply rules in given order!
  rules = [
    { r'>\s+' : u'>'},                  # remove spaces after a tag opens or closes
    { r'\s+' : u' '},                   # replace consecutive spaces
    { r'\s*<br\s*/?>\s*' : u'\n'},      # newline after a <br>
    { r'</(div)\s*>\s*' : u'\n'},       # newline after </p> and </div> and <h1/>...
    { r'</(p|h\d)\s*>\s*' : u'\n\n'},   # newline after </p> and </div> and <h1/>...
    { r'<head>.*<\s*(/head|body)[^>]*>' : u'' },     # remove <head> to </head>
    { r'<a\s+href="([^"]+)"[^>]*>.*</a>' : r'\1' },  # show links instead of texts
    { r'[ \t]*<[^<]*?/?>' : u'' },            # remove remaining tags
    { r'^\s+' : u'' }                   # remove spaces at the beginning
  ]
 
  for rule in rules:
    for (k,v) in rule.items():
      regex = re.compile (k)
      text  = regex.sub (v, text)
 
  # replace special strings
  special = {
    '&nbsp;' : ' ', '&amp;' : '&', '&quot;' : '"',
    '&lt;'   : '<', '&gt;'  : '>'
  }
 
  for (k,v) in special.items():
    text = text.replace (k, v)
 
  return text

To see this function in action (and get an idea of what does), following you can find an example of a HTML containing some javascript, styles, classes, and several kind of tags:

print stripHTMLTags ('''
<html>
  <head>
  <title> Whatever </title>
  <script language="javascript">
  var j = 0;
  function asdf() {
    if (j < 0)
      alert ("Whatever");
  }
  </script>
  </head>
  <body style="background: red;">
    <div class="container">
      <h1>This is the title</h1>
      <p>
        This is the first paragraph with little text.
        <br/>
        This is the second line of the first paragraph (note the BR tag).
      </p>
      <p>This is the second paragraph with some extra text.</p>     
    </div>
    <div>
      <p>
        <span>Follow this link:</span>
        &nbsp;&nbsp;<a href="http://www.codigomanso.com/">Codigo Manso</a>
      </p>
 
      <p>Do you like it?</p>
    </div>
  </body>
</html>''')

And this is the output of stripHTMLTags:

This is the title
 
This is the first paragraph with little text.
This is the second line of the first paragraph (note the BR tag). 
 
This is the second paragraph with some extra text.
 
Follow this link:  http://www.codigomanso.com/
 
Do you like it?

My requirements were to do a simple function that transformed HTML into text. I know it won’t handle all cases, not even bad formatted HTML code. It’s OK! I’m fine with that.

If you need a function which handles wrong-formatted HTML for your project I recommend you to look for a good HTML or XML parser (BeautifulSoup is a great HTML parser for Python).

A typical example of leading zeroes is when you want to show the current time and you want the time to be formatted like hh:mm. There is no problem when it’s 12:40, but when is five minutes past one, you get 1:5, which is not what you would expect. You want it formatted like 01:05. The same happens with dates and in several other situations.

Yesterday I found the best solution I’ve seen for this problem. Before that, let’s see possible solutions.

Usually, needing only one zero, I would do something like:

h = (h < 10) ? ("0" + h) : h;

This is not bad at all, but is not elegant. The problem is, what if we want it to be formatted with 3 digits? Then, using the same schema, we can end up with something like:

h = (h < 100) ? ( (h >= 10) ? ("0" + h) : ("00" + h) ) : h;

Man, this is ugly and unreadable!! Maybe it will improve with an if/else, but it will be bad anyway. Probably if you need to do something like that, you will create a function that will handle this problem and then, you will call that function saying you want h padded with 3 zeros. But there is another solution that does not require you to use or create such a funciton…

Have a look for two digit and three digit:

("0" + h).slice (-2);  // devolverá "01" si h=1; "12" si h=12
("00" + h).slice (-3); // "001" si h=1; "012" si h=12;"123" si h = 123

I don’t know what are you thinking after seeing this solution, but I got flashed when I saw it. Just amazing!

Let’s explain what’s going on. The code above only concatenates the “00″ with the number we want to format (e.g: “00″ + 12 = “0012″) and then, we get the last three digits using the slice string function (that’s why you have a -3). Guess what? The last three characters are “012″ which is the string we want.

Of course this solution is not only limited to leading zeroes, you can use it with leading spaces, or whatever character you want.

Sources:

• Padding zero in Javascript
• Little tricks: string padding in Javascript of Eneko Alonso blog (originally seen here)

Update: Milan Adamovsky has created a web page http://jsperf.com/zero-padding where you can check the performance issues of several functions to pad with zeroes, and you can try it directly in your browser