What is this thing called Personal Search

On a recent visit to a very large storage vendor I had a discussion about a social portal that they had developed. It was struggling to get user engagement and they were puzzled as to why.

My response seemed a surprise to them. Social software is personal. Users think of it as a personal tool. If you look at the most popular web 2.0 platforms like Facebook, Twitter, or even Delicious they are tools that provide a benefit to users on a personal level.

You cannot force user engagement. The tools either help them (so they use it), or they really don't provide much benefit (they don't use them). Facebook is about personal expression, Twitter about having a voice, and Delicious is about sharing your own interests. If tools don't deliver this personal benefit then users will not use them.

In a round about way this gets us back to the companies social portal. Enterprise 2.0 software suffers from this lack of personal, intimate interaction. It has a corporate aura about it, residing on the corporate portal, and workers don't feel the same personal connection with it. Does it really help them get their work done or just create more work for them? One more application they have to use.

As we discussed Jumper and how it might help I told them that if they deployed Jumper on the same corporate portal that it would likely suffer the same fate. The users would not feel any personal connection with it.

Jumper works best when it is deployed directly into a community of users. Smaller deployments that can be customized even personalized to users interests. I asked what groups had heavy information requirements and they mentioned the project managers, research lab teams, product development groups, etc. and I discussed that a customized Jumper deployed onto a small VM with minimal system resource requirements should be deployed for each of these groups. Users feel a more personal connection with a bookmarking engine when it is focused in this way. Search returns only the results relevant to them, not the whole company. Resources have been tagged by colleagues that they know and trust, basically their friends at work who they can holler over the cube wall at.

You have to change the way you think about applications. From the web to mobile phones applications are more specialized, more organic in their user communities. And large organizations need to understand and adapt to this expectation from their users. Search is no different.

Personal search or a point solution approach really means a more customized or tailored approach to search that meets the unique needs of its users. It is precisely because Jumper is an open tool that you can change to meet your own
unique requirements that this works so well. Precisely because it is license free and light-weight that it can easily be deployed in this way. The ability to reflect personal or group interests includes greater flexibility in the terminology or data dictionary to include a hybrid of corporate taxonomy and group based folksonomy. Specialist users have very specific and often highly technical terms that never make it in a formalized corporate taxonomy. Yet these terms matter to these specific users and make it easier to search and find things. Another critical factor is that the tag fields can be customized to meet the unique needs of users. For instance, with structured knowledge tags a materials engineer or biologist will have very different tagging needs than a SAN storage architect or a chemist. One might require a tag to identify the protein the other a tag to identify the compound, etc. A point solution approach allows the local bookmarking engine to be highly customized to meet these unique needs in a way a centralized system never could.

Personal search is not enterprise search. We understand that this tool must be simple. Easy to use, easy to navigate and intuitive. It must also provide a direct and immediate benefit to users. There must be something in it for them, it must not be a generalized tool, it must be very specific, even personal for each user to see and feel the value. It is more like a cube conversation. In this sense it must be localized. Sharing a common skill-set or job description, just as most users with similar skills are sitting together on the same floor and their conversations are based around this shared understanding so the tool must have the same level of intimacy.

This is what we mean by personal search and it requires an entirely new way of thinking about enterprise search. And it is often that thought process that is the hardest thing to change.

Building Social into Solr

We have had a number of customers inquire about customizing specific aspects of Solr search with Jumper.

There are really two approaches: one is to build Jumper tagging into your search engine interface allowing users to tag documents or content when it is stored. The second is to import Jumper tagging fields into solr using the DataImportHandler. This is done using basic JDBC connectivity. Tags stored in the Jumper search engine then are imported into the Solr index and attached to a document and returned when searched. Using faceted_fields you can allow users to filter search based on the knowledge tags applied by other users.

This is perhaps the easiest method. The two services can be bundled in a single web interface. In this way you are removing the Jumper search engine and replacing it with Solr. This gives you the benefit of both worlds – full text searching and user tagging – to deliver better more detailed search results.

If you prefer to embed custom search paths into Solr the primary method is using facet-fields. A Jumper tagging interface can be added when storing documents. The Jumper tag fields are then stored as facet_fields that Solr will search in addition to its full text parsing of the document. This is done on indexed rather than stored values.

This requires that we add a number of Jumper tags to the Solr index separately and add a custom sort to Solr search. Adding a new Jumper tag field to the search results requires two very small hook implementations: hook_apachesolr_update_index() and hook_apachesolr_modify_query(). To start, let’s just add the keyword tag field to the Solr index.

/**
* Implementation of hook_apachesolr_update_index()
*/
function mymodule_apachesolr_update_index(&$document, $node) {
// Index field_keyword_tag as a separate field
if ($node->type == 'profile') {
$user = user_load(array('uid' => $node->uid));
$document->setMultiValue('sm_field_keyword_tag', $user->tags);
}
elseif (count($node->field_keyword_tag)) {
foreach ($node->field_keyword_tag AS $keyword) {
$document->setMultiValue('sm_field_keyword_tag', $keyword['filepath']);
}
}
}

All we do is add the data to the index by adding it to the $document object, which is passed by reference. We used the setMultiValue method since the tag field can have multiple values, but if we were just adding one field, we would just use the addField method. The field name is simply the 'sm_' dynamic field name pattern with field_keyword_tag appended, since the field contains a keyword string, and the sm_ field type represents a small string.

Now that the data has been added to the index, we also need to add it to the query so it can be returned with the search results:
function mymodule_apachesolr_modify_query(&$query, &$params, $caller) {
$params['fl'] .= ',sm_field_keyword_tag';
}

And that's all there is to it… This can be repeated for each of the Jumper knowledge tags that you want to add. All you're doing is some basic PHP string concatenation and appending your newly indexed field to the fields to return array (['fl'])of the $params object. Although, we are simplifying the detail a little bit on the format of $params for the sake of brevity in this post.

In general, adding Jumper social tagging features into your Solr search is pretty easy, and can deliver some very powerful capabilities to your search functionality.

Proposing an Open Index Initiative

Many of the customers who download Jumper are using it for personal search. They are using it themselves, or in a small company, or between a few colleagues. Jumper has many benefits that make it an ideal personal search engine, however, if you are using Jumper in this way you are missing one of its big benefits – collaborative sharing of resources. In a larger community of uses with a shared interest or profession the number of searchable resources grows rapidly and the inherent value of Jumper search increases accordingly.

In an effort to facilitate resource sharing among the many customers who use Jumper for personal search we are proposing an Open Index initiative.

Jumper will make a shared index database available for download on Sourceforge under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license. This Open Index will be built by users like you, who share your tag profiles with other users, who in turn share their tag profiles with you. This collaborative initiative will be aggregated by Jumper in a central Open Index that will be available for download on Sourceforge.

This means that you can share and change the index database as you choose.
• to Share — to copy, distribute and transmit the work
• to Remix — to adapt the work

Under the following conditions:
• Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
• Noncommercial — You may not use this data directly for commercial purposes. You cannot resell or repackage this data for commercial benefit. You cannot commercially benefit by providing access to this data. The intent for use is that any benefit will only be derived indirectly by using this data to search for resources that may provide a commercial benefit as a result.
• Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same license to this one.
It is important to understand that when you contribute your resources to the common index for the general good of all that it can only be used under these license guidelines.

There are two ways you can easily contribute.

You can export your Jumper index database and send the flat file to us via Email at Open Index

Or you can add individual resources to the open index via the Open Index form.