Skip to content

Comment on Show HN: TF-IDF search engine in 30 lines of Scala

Comments

You can do the same in Python (in about the same amount of lines I suspect). The below is the core of the linked implementation (Vector Space) in 15 lines of Python. All you need is something to build a clean concordance on the search terms/documents, then just compare the terms against the documents and sort based on the return value of relation.

Actually the below implementation is usually the first non trivial thing I try to implement in any language I am learning.

  import math

  class VectorCompare:
    def magnitude(self,concordance):
      total = 0
      for word,count in concordance.iteritems():
        total += count ** 2
      return math.sqrt(total)

    def relation(self,concordance1, concordance2):
      relevance = 0
      topvalue = 0
      for word, count in concordance1.iteritems():
        if concordance2.has_key(word):
          topvalue += count * concordance2[word]
      return topvalue / (self.magnitude(concordance1) * self.magnitude(concordance2))

EDIT - Was going to fork this on GitHub and make a quick Python port but corporate firewall got in the way.

Cannot edit anymore, but here is an example I had written some time ago in Python. http://www.wausita.com/2010/08/build-vector-space-search-eng...

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.