The pulse of Alexander Interactive

I was adding on an app to a Django project at work where I was overriding an existing template but did not have access to the view that called that template. I was left in a scenario where I had the variables that the view was originally set up with, but non of the new models that I had added.

In a filter you can do whatever logic you want, and then pass information back to the view. Please keep in mind, this is probably a horrible practice, but it does have its uses. In this specific scenario I needed to query the new models without modifying the existing view, solution: add a filter and do the querying there.

This is the filter that I used to do the querying:

from django import template
from stager.jira.models import JiraProject, ProjectLink
from stager.staging.models import *
register = template.Library()
def has_jira(value, arg):
client = Client.objects.get(path=value)
project = client.projects.get(path=arg)
try:
jiras = ProjectLink.objects.get(ClientProject=project).JiraProject.exclude(filter_id='')
return True
except:
return False
register.filter('has_jira', has_jira)

Then, in my template:

{% load has_jira %}
{% if client.path|has_jira:project.path %}
Jira

{% endif %}

A more general example if this would be to work around the annoyance of not being able to have multiple tests in an if statement in a template: You can’t do {% if this and that %}
A solution would be:

def if_and(value, arg):
if value and arg:
return True
else:
return False
def if_or(value, arg):
if value or arg:
return True
else:
return False

 {% if True|if_and:False %}
show
{% else %}
don't show
{% endif %}

Let me know your thoughts, pros/cons of this method.

Ai’s stager project is open source and can be found at github.

About a year ago I wrote a webapp called twitter2gtalk, which simply takes your latest Twitter message and sets it as your Google Talk Status. Shortly there after I attended an App Engine Hackathon (GAE) at Google NYC. At the hackathon, I started porting the app over to app engine. At the time, there was no scheduling or cron support in GAE. I created a hybrid app running between GAE and my webfaction server. My webfaction server would handle the cron, proxy out the requests, and do some of the longer running tasks (the limit on an app engine request is 20 seconds).

Now with the release of the Task Scheduler and Task Queues, you can break up a long winded process into the cloud. My batch was taking around 18 minutes to run, with about 800 users being processed each time. For each user the app had to get their Twitter status, then connect to Google’s xmpp server, and after a couple of xmpp calls, update the Google Talk Status. Even being single threaded, this whole process started eating up a lot of juice on the server. The rest of my apache processes suffered and the entire site slowed down

Using the new app engine features, I was able to do two things: 1) Move the scheduling into app engine itself and 2) have app engine do the bulk of the work, by having each user-update be its own task. These worker tasks are then processed anywhere in the cloud, and can be run in parallel, independently of each other.

The original code was something like:

for users in all my users:
get twitter status
connect to gtalk
get current status
update status

My process for converting this to app engine tasks was as follows:

1. Create a process to load all of the users into a Task Queue
2. Set up each Task Queue so that it can independently do the work for the user that it is associated with
3. Schedule Step 1 to run every half hour

Skeleton Code for Step 1:

class TaskLoader(BaseRequestHandler):
def get(self):
logging.info(“Starting to load tasks %s” % datetime.datetime.now())
users = Account.gql(‘WHERE active = :1′, True)
count = 0
for user in users:
send_key = # data to send to the worker
taskqueue.add(url=’/worker/’, params={‘key’: send_key})
count += 1
logging.info(“Ended load tasks (%d users) %s” % (count, datetime.datetime.now()))

The Url call for this method: (‘/taskloader/’, TaskLoader),

The Skeleton code for step 2:

class TaskWorker(BaseRequestHandler):
def post(self):
key = self.request.get(‘key’)
t = ”.join(["http://django.gpowered.net/xmppproxy/", key])
logging.info(“URL !%s!” % t)
result = urlfetch.fetch(t,
None,
urlfetch.GET,
{‘Cache-Control’:'no-cache,max-age=0′, ‘Pragma’:'no-cache’})

Url call for this method: (‘/worker/’, TaskWorker),

So, TaskLoader loads all of the users into the Task queue. App Engine processes these when it has the cpu cycles to do so, and for each of these, TaskWorker is called

I can schedule these using cron.yaml in my project:
cron:
- description: load the task queue
url: /taskloader/

schedule: every 30 minutes

This is a general overview of how to break up your batch in to smaller, easier to manage tasks. Please see the App Engine documentation for more detailed information.