Subject: My fixes

From: Stefanie Bruninghaus <[email-masked]>
Date: Mon, 15 Oct 2001 13:31:18 -0400

Diane - attached are my two files. The naming is not exactly appropriate
(but I am sure you know how these things evolve ...). 

You probably want to have quick look at the code - just so that you know
what it looks like. I am confident that it works as expected, but I have
found count discrepancies of +/- 1 in few cases for my lowercase version
of the bigrams (the overall behavior is OK!). The code is not what I
would submit to a Perl contest, either. 

There is a switch for lower case/regular case, and I am ordering things
by counts (and not alphabetically, as in the original code). If you
think that this choice is bad, it's easy to locate and change, I have
added comments to the code. 

Here is what to do: 

- type: telnet unixs.cis.pitt.edu and log in 

Start downloading the files, unpacking, compiling - up the point where
things don't work (which is the call of "Stats"). From there, 

- copy the attached two files into the ngrams directory (Stats will be
replaced by my version)
- like in the instructions on the web, change the first line of
count-unigrams.pl (on unixs.cis, this is /usr/pitt/bin/perl)
- type: chmod +x count-unigrams.pl
- type: ./Stats corpora/GEN.EN genesis/out

... and then proceed as explained on your webpage. 

I hope that works! I have tested this on Solaris 2.6 - and it works
fine, Ilya tried it on the unixs machines, where it worked, and on his
PC version, where something went wrong. 

Best - STeffi. 
-- 
----------------------------------------------------------------------
Stefanie Bruninghaus
Learning Research and Development Center     Mail:    [email-masked]
University of Pittsburgh                     Web: www.pitt.edu/~steffi
3939 O'Hara Street                           Phone:   (412) 624 - 6748
Pittsburgh, PA 15260-5159 -- USA             Fax:     (412) 624 - 9149
----------------------------------------------------------------------

#!/bin/csh
#
#
#  Usage: Stats input-file output-file-prefix
#
#  Code for identifying word-word collocations
#  based on Ken Church's 1995 'NGRAMS' tutorial
#  and extended to use the likelihood ratio as
#  an alternative (and generally better) measure
#  of association than mutual information.
#
#  Author:  Philip Resnik
#
# changes made by Steffi Bruninghaus, 10/2001 to make this code 
# run under Solaris 2.6 

if ($2 == "" || $3 != "") then
  echo "Usage: $0 input-file output-file-prefix"
else

  set INFILE=$1
  set OUTPREFIX=$2

  echo "Processing file $INFILE"

#  echo "Getting unigram counts: see $OUTPREFIX.unigrams"
#  /bin/rm -f $OUTPREFIX.unigrams
#  ./count_words < $INFILE > $OUTPREFIX.unigrams

#  echo "Getting bigram counts: see $OUTPREFIX.bigrams"
#  /bin/rm -f $OUTPREFIX.bigrams
#  ./count_bigrams $INFILE $OUTPREFIX > $OUTPREFIX.bigrams
    
    echo "Steffi is computing bigram and unigram counts" 
    /bin/rm -f $OUTPREFIX.words
    /bin/rm -f $OUTPREFIX.bigrams
    /bin/rm -f $OUTPREFIX.unigrams
    ./count-unigrams.pl $INFILE $OUTPREFIX
    echo "Returning to the original code"
  
    echo "Computing bigram mutual information: see $OUTPREFIX.mi"
    /bin/rm -f $OUTPREFIX.mi
    ./mutual_info $OUTPREFIX > $OUTPREFIX.mi

  echo "Computing likelihood ratio for bigrams: see $OUTPREFIX.lr"
  /bin/rm -f $OUTPREFIX.lr.values $OUTPREFIX.lr 
  cat $OUTPREFIX.mi | ./lr_filter.pl `cat $OUTPREFIX.words | wc -l` \
     | xargs -n 6 ./lr_simple > $OUTPREFIX.lr.values
  paste $OUTPREFIX.lr.values $OUTPREFIX.mi | sort -nr > $OUTPREFIX.lr

endif