AIAIO: Our Blog

Inevitably, when you get into the weeds with a Magento build, you’ll need to run some big, hairy process — perhaps even on a regular basis. Perhaps it involves processing images, or updating data, or creating reports. Perhaps it runs on cron, but you’d also like to run it from the command line. And if it’s destructive — i.e., it alters data — you definitely want to make sure you’re only running one at a time.

Magento has a nice semaphore locking process handler built in to its indexers. In order for a reindexing process to run, the process needs to obtain a lock. The code lives in the Mage_Index_Model_Process class, and has helpful methods such as isLocked(), lockAndBlock(), and unlock(). These look for and manipulate files in your application’s var/locks directory.

The implementation is a tried and true locking mechanism that you could write yourself, but why bloat an already massive code base? We can just repurpose this relatively self-contained functionality whenever we need to lock a process.

Achievers has announced this year’s 50 Most Engaged Workplaces™– the award aims to inspire enhancements to the workplace by championing the growth of employee-centric organizations. Ai is thrilled to be ranked among the top US organizations.

The judges read, re-read and compare survey questions related to the Eight Elements of Employee Engagement™: Leadership, Communication, Culture, Rewards & Recognition, Professional & Personal Growth, Accountability & Performance and Vision & Values.  Forms are filled out without any company names so submissions can be judged with as much anonymity as possible.

We are delighted to have made the cut!

To Achievers-we love what you stand for. We agree employees are a company’s greatest asset; when companies empower them to succeed and recognize performance, not presence, the employees and the business both reap the benefits. Thank you– we are honored to be among the 2012 50 Most Engaged Workplaces™.

Solr is a very powerful search tool and it is pretty easy to get the basics, such as full text search, facets, and related assets up and running pretty quickly. We will be using haystack to do the communication between Django and Solr. All code for this can be viewed on github.

Install

Assuming you already have Django up and running, the first thing we need to do is install Solr.

curl -O http://mirrors.gigenet.com/apache/lucene/solr/4.0.0-BETA/apache-solr-4.0.0-BETA.zip
unzip apache-solr-4.0.0-BETA.zip
cd apache-solr-4.0.0-BETA
cd example
java -jar start.jar

Next install pysolr and haystack. (At the time of this writing the git checkout of haystack works better with the Solr 4.0 beta then the 1.2.7 that’s in pip.)

pip install pysolr
pip install -e https://github.com/toastdriven/django-haystack.git

Add ‘haystack’ to INSTALLED_APPS in settings.py and add the following haystack connection:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr'
    },
}

Full Text Search

For the example, we’re going to create a simple job database that a recruiter might use. Here is the model:

from django.db import models
from django.contrib.localflavor.us import models as us_models

JOB_TYPES = (
    ('pt', 'Part Time'),
    ('ft', 'Full Time'),
    ('ct', 'Contract')
)

class Company(models.Model):
    name = models.CharField(max_length=64)
    address = models.TextField(blank=True, null=True)
    contact_email = models.EmailField()

    def __unicode__(self):
        return self.name

class Location(models.Model):
    city = models.CharField(max_length=64)
    state = us_models.USStateField()

    def __unicode__(self):
        return "%s, %s" % (self.city, self.state)

class Job(models.Model):
    name = models.CharField(max_length=64)
    description = models.TextField()
    salary = models.CharField(max_length=64, blank=True, null=True)
    type = models.CharField(max_length=2, choices=JOB_TYPES)
    company = models.ForeignKey(Company, related_name='jobs')
    location = models.ForeignKey(Location, related_name='location_jobs')
    contact_email = models.EmailField(blank=True, null=True)
    added_at = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return self.name

    def get_contact_email(self):
        if self.contact_email:
            return self.contact_email
        return self.company.contact_email

The next step is to create the SearchIndex object that will be used to transpose to data to Solr. save this as search_indexes.py in the same folder as your models.py. The text field with its template will be used for full text search on Solr. The other two fields will be used to faceted (drill down) navigation. For more details on this file, check out the haystack tutorial.

class JobIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    type = indexes.CharField(model_attr='type', faceted=True)
    location = indexes.CharField(model_attr='location', faceted=True)

    def get_model(self):
        return Job

    def index_queryset(self):
        return self.get_model().objects.all()

Create the search index template in your template folder with the following naming convention: search/indexes/[app]/[model]_text.txt
For us, this is templates/search/indexes/jobs/job_text.txt

{{ object.name }}
{{ object.description }}
{{ object.salary }}
{{ object.type }}
{{ object.added_at }}

Now, lets get our data into Solr. Run ./manage.py build_solr_schema to generate a schema.xml file. Move this into example\solr\conf in your Solr install. Note: if using Solr 4, edit this file and replace stopwords_en.txt with lang/stopwords_en.txt in all locations. To test everything and load your data, run: manage.py rebuild_index Subsequent updates can be made with: manage.py update_index.

If that all worked we can start working on the front-end to see the data in Django. Add this to your urls.py

(r'^$', include('haystack.urls')),

At this point there are at least two templates we’ll need. One for the search results page, and a sub-template to represent each item we are pulling back. My example uses twitter bootstrap for some layout help and styling, see my base.html here if interested.

Create templates/search/search.html
This gives you a basic search form, the results, and pagination for a number of results

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}
Click around!

{% endblock %}

{% block content %}</pre>
<div class="span12">
<h1>Search</h1>
<form class=".form-search" action="." method="get">{{ form.as_table }}
 <input type="submit" value="Search" /></form></div>
<pre>
{% if query %}</pre>
<div class="span8">
<h3>Results</h3>
<div id="accordion2" class="accordion">{% for result in page.object_list %}
 {% include 'search/_result_object.html' %}
 {% empty %}

No results found.

 {% endfor %}</div>
 {% if page.has_previous or page.has_next %}
<div>{% if page.has_previous %}<a href="?q={{ query }}&page={{ page.previous_page_number }}">{% endif %}« Previous{% if page.has_previous %}</a>{% endif %}
 |
 {% if page.has_next %}<a href="?q={{ query }}&page={{ page.next_page_number }}">{% endif %}Next »{% if page.has_next %}</a>{% endif %}</div>
 {% endif %}</div>
<pre>
{% else %}

{% endif %}
{% endblock %}

And the templates/search/_result_object.txt

{% with obj=result.object %}</pre>
<div class="accordion-group">
<div class="accordion-heading"><a class="accordion-toggle" href="#collapse_{{ obj.id }}" data-toggle="collapse" data-parent="#accordion2">
 {{ obj.name }}
 </a>
<div style="padding: 8px 15px;">
Company: {{ obj.company }}

Type: {{ obj.type }}

 {% if obj.salary %}
Salary: {{ obj.salary }}

{% endif %}

Location: {{ obj.location }}</div>
</div>
<div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
<div class="accordion-inner">
Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a>

 {{ obj.description }}</div>
</div>
</div>
<pre>
{% endwith %}

Start up your dev server for search!

Related Items

Adding Related Items is as simple as using the related_content tag in the haystack more_like_this tag library and tweaking out Solr config. Open up solrconfig.xml and add a MoreLikeThisHandler within the tag:

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler" />

Our full _result_object.html now looks like this:

{% load more_like_this %}

{% with obj=result.object %}
<div class="accordion-group">
    <div class="accordion-heading">
        <a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion2" href="#collapse_{{ obj.id }}">
            {{ obj.name }}
        </a>
        <div style="padding: 8px 15px;">
            <p>Company: {{ obj.company }}</p>
            <p>Type: {{ obj.type }}</p>
            {% if obj.salary %}<p>Salary: {{ obj.salary }}</p>{% endif %}
            <p>Location: {{ obj.location }}</p>
        </div>
    </div>
    <div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
        <div class="accordion-inner">
            <p>Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a></p>
            {{ obj.description }}
            {% more_like_this obj as related_content limit 5  %}
            {% if related_content %}
                <div>
                    <br>
                    <p><strong>Related:</strong></p>
                    <ul>
                        {% for related in related_content %}
                            <li><a>{{ related.object.name }}</a></li>
                        {% endfor %}
                    </ul>
                </div>
            {% endif %}
        </div>
    </div>
</div>
{% endwith %}

Facets

To get our type and location facets, we’ll have to add them to a queryset and pass this to a FacetedSearchView instead of the default one. urls.py now looks like this:

from django.conf.urls import patterns, include, url
from haystack.forms import FacetedSearchForm
from haystack.query import SearchQuerySet
from haystack.views import FacetedSearchView

sqs = SearchQuerySet().facet('type').facet('location')

urlpatterns = patterns('haystack.views',
    url(r'^$', FacetedSearchView(form_class=FacetedSearchForm, searchqueryset=sqs), name='haystack_search'),
)

Then, we can use the generated facets in the search template in the facets variable

{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}<p>Click around!</p>{% endblock %}

{% block content %}
<div class="span12">
    <h1>Search</h1>
    <form method="get" action="." class=".form-search">
        <table>
            {{ form.as_table }}
        </table>
        <input type="submit" value="Search">
    </form>
</div>
        {% if query %}
            <div class="span2">
                <h3>Filter</h3>
                {% if facets.fields.type %}
                    <div>
                        <h4>Type</h4>
                        <ul>
                        {% for type in facets.fields.type %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=type_exact:{{ type.0|urlencode }}">{{ type.0 }}</a> ({{ type.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
                {% if facets.fields.location %}
                    <div>
                        <h4>Location</h4>
                        <ul>
                        {% for location in facets.fields.location %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=location_exact:{{ location.0|urlencode }}">{{ location.0 }}</a> ({{ location.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
            </div>
            <div class="span6">
                <h3>Results</h3>
                <div class="accordion" id="accordion2">
                    {% for result in page.object_list %}
                        {% include 'search/_result_object.html' %}
                    {% empty %}
                        <p>No results found.</p>
                    {% endfor %}
                </div>

                {% if page.has_previous or page.has_next %}
                    <div>
                        {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                        |
                        {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                    </div>
                {% endif %}
            </div>
        {% else %}
            <div class="span6">
                {# Show some example queries to run, maybe query syntax, something else? #}
            </div>
        {% endif %}
{% endblock %}

And we’re done! As I said, check out the haystack documentation for more information. Leave any questions in the comments and I’ll be sure to answer them. Spelling suggestions to come in the next post.

As my time at Alexander Interactive (Ai) comes to a close, I’d like to reflect on what I learned while I was here. I started at Ai with little experience in project management. Ai gave me the opportunity to learn through hands-on experience managing an internal project. This project did not follow the traditional project process, but I was still able to experience some of the major project process milestones.

My first introduction to the project process was the Project Triangle, which represents the three points of a project: Scope, timeline and budget. In any given project one of these points is fixed and cannot change. The fixed point for my project was the timeline, so I began by developing the timeline. I then created a staffing plan, which allowed us to understand how much the project would cost.

Then we began working on the project. Basecamp was crucial to the success of the project. All of the files, weekly key activities, and meeting notes were posted to Basecamp. Having all of the documents in one place kept everything organized and allowed people to quickly access the information they needed. I then used Todoist, a to-do list web app, to track all of the next steps.

While I won’t get to see this project completed while I am at Ai (it launches in early September), I did get to see another project progress almost to completion. This project was already well underway when I began at Ai, but I was able to join the team, experience the design process, learn about client interactions and observe presentation practices. I also attended many meetings, which taught me to share materials beforehand, bring copies of relevant documents to the meetings and to avoid going off topic. I was a mini-PM for the summer, and it was a great opportunity to see the project process in action.

While at Ai I also conducted research on resource management software, Facebook storefronts Open Graph and Shopcade. I developed an Ai dictionary and a site launch dates list on the internal wiki. I also assisted in the development of an Ai Pinterest account and learned how to post to the blog (making this post possible).

At school I manage four organizations and attend four classes each semester. Keeping all of my organizations and schoolwork in order is challenging. While convincing all of the organizations that I manage in college to use Basecamp may prove impossible, Todoist will definitely be coming with me to college. I will also use the system of next steps and task tracking to make sure that people are aware of and actually accomplish the tasks that I assign to them.

Ai is not all work and no play. There were many moments of laughter and fun. For me, Ai was not just a job but a great life experience. I can honestly say that I have learned more than I had anticipated during my time at Ai, and I want to thank everyone at Ai for making my time as a PM Intern so great.

This month’s Careers issue of Time Out New York includes a special feature titled “Cool NYC Companies: Behind the scenes at New York’s best companies”. Ai is pleased to announce that the Immaculate one, Mayor Jack Reynolds, was invited to be included un the TONY gallery of New York City’s cutest and hardest working office dogs.

For our Careers issue, we asked our friends at various digital and media companies to send us photos of their four-legged colleagues, many of whom hold important positions at some of the city’s top publications and agencies. Click through our slide show to meet some of New York hardest-working pups and find out what they do.

Once news of the slide show broke, the Mayor’s public responded immediately.

“This is great,” says Alex Schmelkin, President and co-founder of Ai. “It really is.”

“He’s kind of a big deal,” I said.

David Ow, Director of Project Management, really put it all in perspective:   ”The feature is referred to as “Cute Office Dogs”. Look, this is fantastic and all but lets call it like it is, kids with ice cream on their face are ‘cute’. The Mayor is exquisite… magnificent.”

Much thanks to Time Out for including Jack in the slide show. He is honored to be included in such an accomplished and hard-working group.

"Who the hell is that red headed kid?" commented Brad Borman, VP of New Business Development.

Thousands of children live in homeless and domestic violence shelters throughout New York City. One of the devastating consequences of homelessness is the impact it has on a child’s education. We wanted to help. For most of July we hosted a backpack and school supplies drive in partnership with Volunteers of America’s Operation Backpack.

The drive started slow. But the IT Director stepped it up when he ordered 10 backpacks (!!) for his team to fill with supplies. Once the ball was rolling, we had some contributions from the creative and tech teams. However, it was the account services team that took the lead with only a few days left. These 6 people (including our summer Project Management intern) generously each donated a bag stuffed to its brim with the recommended supplies.

Backpacks were labeled by gender and grade and many of our contributions were coincidentally geared towards elementary school-aged kids (our inner 3rd graders peaked their heads when purchasing Spiderman or neon colored bags). Suggested school supplies to fill the backpack included loose-leaf paper, pencils, pens, markers, pencil bag or box, rulers, glue sticks, folders, three-ring binders, notebooks…and the list went on.

These backpacks and supplies mean a lot to the children, helping them go back to school with all of the material they need and a boost of confidence! We are proud to have made a small contribution to Operation Backpack, especially since we know it will make a big difference to the individual families and children who receive our bags.

GO ACCOUNT SERVICES TEAM!