Algorithm Analysis

Introduction

Key questions for this section:
- What is algorithm analysis?
- Why is algorithm analysis useful?

So far, we have talked about differences in efficiency between various implementations of the Bag ADT and the Stack ADT, but we have been somewhat vague about it. Now we will look at algorithm efficiencies in a more formal, mathematical way. Why do we care about formalizing this? Consider all of the work involved in implementing a new ADT. It is non-trivial to get all of the operations working correctly. Plus, many special cases and much debugging is required during implementation. If we can pick the best implementation before implementing the ADT, it can save us a lot of time. Inefficient potential implementations could be abandoned before they are even started!

For a simple example, let's consider adding up a sequence of integers starting at zero. There are two algorithms below that both do just that. Which one is faster? Is there an even faster algorithm?

long n = 10000;

//Algorithm A
long sum = 0;
for (long i = 1; i <= n; i++)
    sum = sum + i;
System.out.println("Algorithm A's sum = " + sum);

//Algorithm B
sum = 0;
for (long i = 1; i <= n; i++) {
    for (long j = 1; j <= i; j++) {
        sum = sum + 1;
}
System.out.println("Algorithm B's sum = " + sum);

Let's take a look at another example. Let's try searching through a sorted array for a number. We'll consider two algorithms for searching: sequential search and binary search. What does each algorithm do?

Assume the array contains N items in sorted order. Sequential search can take up to N tests to find the item, but binary search will take at most log₂(N) tests. (How do you think we could figure out the number of tests needed?) Are N and log₂(N) that different from each other? Let's take a look at the number of tests for one search with different values of N:

N	Sequential Search	Binary Search
8
16
32
64
...
1,024
1,048,576
1,073,741,824

What if we were doing 1 million searches instead of one? Well if we had N = 20 million, we could use this formula to figure out the number of tests:

(# of searches) * (# tests / search)

Sequential Search: (1,000,000) * (20,000,000) = 2 * 10¹³
Binary Search: (1,000,000) * log₂(20,000,000) ≅ 2 * 10⁷

If each test takes one nanosecond (10^-9 seconds):

Sequential Search: (2 * 10¹³) * 10^-9 = 2 * 10⁴ seconds = 5.5 hours
Binary Search: (2 * 10⁷) * 10^-9 = 0.02 seconds

The difference is amazing. Just rethinking our algorithm takes us from something that would take hours to something that just takes a fraction of a second. Other examples can have even more extreme differences. CS/COE 1501 will have many examples. By analyzing our algorithm before implementing it, we can thus avoid implementing algorithms that will require too much time to run. A little analysis saves us a lot of programming.

Measuring Execution Time

Key questions for this section:
- What is the technique for measuring execution time (i.e. counting operations)?
- How do you compare execution times of algorithms?
- What are the rankings of simple execution times?

How can you compare execution times of algorithms? Perhaps the most obvious approachis to time them empirically. This will give us actual run-times that we can use to compare. This is very useful for algorithms/ADTs that have already been implemented. However, we said previously that often it is good to get a ballpark on the runtime of an algorithm/ADT before actually implementing it. Perhaps we wouldn't want to go through the effort if the algorithm is not going to be useful. Additionally, measuring implementations depend on things independent of the ADTs/algorithms themselves, such as programming language, computer hardware, and input/data chosen.

The preferred approach is to not actually time a program, even if such a program exists. Instead, we use asymptotic analysis, which follows this procedure:

Determine some key instruction or group of instructions that controls the overall run-time behavior of the algorithm
- For example, with sorting we need to compare items to each other. Even though sorting involves other instructions, we can say that the overall run-time is directly proportional to the number of comparisons done.
Determine a formula / function for how the number of key instructions increase as the problem size increases (typically we use the variable N for the problem/input/data size)
Determine the case you want to deal with. We typically are concerned with two different cases:
- Worst Case Time: What is the formula for the maximum number of key instructions relative to N? We should know what the worst case time can be so that we can plan for it if necessary.
- Average Case Time: What is the formula for the average number of key instructions relative to N? How will the algorithm do normally?
Only worry about the order of magnitude. We use the measure Big-O for this (although you will later learn about Big Omega, Big Theta, and the little versions of all three). Basically, with Big-O, we ignore lower order terms and constant multipliers.
- For example, let's say we determine the formula for the comparisons for a given sorting algorithm in the worst case to be F(N) = (N²/2) - (N/2) We say the Big-O run-time of this sorting algorithm is O(N²)
We ignore...
- lower order terms because they become less significant as the problem size increases
  - See examples on board
  - This is why we call it asymptotic analysis
- constant multipliers because they can depend on programmer, programming language, computer, etc.
  - Program A is written by Alice and runs in time 4N. Program B is written by Bob and runs in time 2N. Maybe Alice is a better programmer. Maybe Alice used a better compiler than Bob. Maybe ...

Let's take a look at some examples:

Name	Big-O	Example(s)
Constant Time	O(1)	y = x; array[i] = array[i-1] + 1;
Linear Time	O(N)	for (int i = 0; i < N; i++) do_some_constant_time_operation
Quadratic Time	O(N²)	for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) do_some_constant_time_operation

There are many (infinitely) others, including some we will see soon.

Search Times

Key questions for this section:
- How do we analyze algorithms?
- How do you analyze a logarithmic runtime algorithm?

Let's take a look at our search example from above. What is the key instruction we want to measure?

For sequential search, what do you think the runtime is? Code for sequential search is:

public static int sequentialSearch(Object[] a, Object key) {
    int i = 0;
    while (i < 0) {
        if (((Comparable)a[i]).compareTo(key))
            return i;

        i++;
    }
    return -1;
}

For binary search, the runtime analysis is a bit trickier. It has a loop like sequential search, but now the number of iterations is very different. Let's look at the code from the standard Java library in the java.util.Arrays class:

public static int binarySearch(Object[] a, Object key) {
    int low = 0;
    int high = a.length-1;
    while (low <= high) {
        int mid =(low + high)/2;
        Object midVal = a[mid];
        int cmp = ((Comparable)midVal).compareTo(key);
        if (cmp < 0)
            low = mid + 1;
        else if (cmp > 0)
            high = mid - 1;
        else
            return mid; // key found
    }
    return -(low + 1); // key not found.

What is the worst case runtime for this? Let's simplify things a bit. First, assume that in each iteration, the array is cut exactly in half. In reality, this won't quite be true, but it's close enough. Second, assume that the initial size of the array is exactly a power of two (i.e. 2^k for some positive integer k). While this will rarely be true, it makes the analysis easier (and has no effect on our results).

Let's combine these simplifying assumptions and apply them to determine the runtime. To determine the problem size at each iteration:

At iteration 0, N₀ = 2^k
At iteration 1, N₁ = N₀ / 2 = 2^{k - 1}
At iteration 2, N₂ = N₁ / 2 = N₀ / 2² = 2^{k - 2}
...
At iteration k, N_k = N₀ / 2^k = 2^{k - k} = 1

So, if we don't find the element in the array (i.e. we're in the worst case), then we have k+1 iterations. At each iteration, we do one comparison, so this yields k+1 comparisons. But we need this in terms of N. From our definition above:

N = 2^k
log₂(N) = k

This makes k+1 = log₂(N) + 1. This makes our final answer O(log(N)). Why did we drop off the "+1"? Why did we drop off the base 2 from the log?

Bag Add Runtimes and Amortized Time

Key questions for this section:
- How do you analyze an operation that can have different runtimes?
- What is amortized time (or amortized analysis)?

We've seen three implementations of the bag ADT. We can now analyze which implementation is better for which operations (if there are any differences). Let's now take a look at the runtime of each implementation.

Fixed Size ArrayBag

//ArrayBag public boolean add(T newEntry) {
    checkInitialization();
    boolean result = true;
    if (isArrayFull()) {
        result = false;
    }
    else {
        // assertion: result is true here
        bag[numberOfEntries] = newEntry;
        numberOfEntries++;
    }
    return result;
} // end add

What is the runtime of the ArrayBag's add operation?

Dynamic Size ResizableArrayBag

//ResizableArrayBag public boolean add(T newEntry) {
    checkInitialization();
    if (isArrayFull()) {
        doubleCapacity();
    }

    bag[numberOfEntries] = newEntry;
    numberOfEntries++;
    return true;
} // end add

Let's take a closer look at ResizableArrayBag. At first glance, it appears to be O(1) because you just go to the last location and insert there (all taking constant time). But what if the array is full? Well, we need to resize it. So, some adds are constant time (O(1)) while others take significantly more time, since we have to first allocate a new array and copy all of the data into it -- taking linear time (O(N)). So, we would have O(1) + O(N) = O(N), right?

Well, we have an operation that sometimes takes O(1) and sometimes takes O(N). What we need to do is figure out the average time required over a sequence of operations. This is called amortized analysis. Although individual operations may vary in their run-time, we can get a consistent time for the overall sequence. Let's stick with the add() method for Resizable Array Bag and consider two different options for resizing:

Increase the array size by 1 each time we resize
Double the array size each time we resize (which is the implementation we chose earlier)

Let's take a look at the first option, where we increase the array size by 1 each time we resize. Note that with this approach, once we resize we will have to do it with every add. Thus rather than O(1) our add() is now O(N) all the time. To see why, assume the initial array is size 1:

On first add, we just add the item (1 assignment)
On second add, we allocate and assign 2 items
On third add, we allocate and assign 3 items
...

Overall, for N add() operations look at the total number of assignments we have to make:

1 + 2 + 3 + ... + N = N(N+1)/2 = O(N²)

Therefore, the amortized add for one add operation is O(N²) / N = O(N).

Now we'll look at the second option, where we double the array size at each resize. Let's again assume that the initial array size is 1:

Add Operation #	# of Assignments	Array Size at End of Operation
1	1	1
2	2 = (copy old array) + (assign new value) = 1 + 1	2
3	3= 2 + 1	4
4	1	4
5	5 = 4 + 1	8
...	1	8
9	9 = 8 + 1	16
...	1	16
17	17 = 16 + 1	32
...	1	32
32	1	32

Note that every row has at least one assignment (for the new value being added, in blue). Some rows have more than one assignment (for copying the old array, in red). For these additional assignments, notice that there is a pattern to their size. Rows that are 2^K + 1 (for some positive integer K) have an additional 2^K assignments to copy data.

So, for N adds, we have:

N assignments (for the actual data being added)
2⁰ + 2¹ + 2² + ... + 2^x assignments (for the copying)

What is that x? Each term in that summation represents a time when the array is doubled. Notice that the array is doubled when N = (array size at start of operation) + 1 Because the array sizes are always powers of 2, we can rewrite that as N = 2^K + 1 Solving for K, we get K = log₂(N - 1) Now, this is for the add that doubles the array. But for determining the total number of assignments, we might not have added exactly enough to double the array (e.g. 13 adds). So we need to round up to get x: x = ceiling[K] = ceiling[log₂(N - 1)]

Now that we know what x is, we now need to figure out the summation of 2⁰ + ... + 2^x:

SUM[i = 0 to x] 2ⁱ = SUM[i = 0 to log₂(N - 1)] 2ⁱ

This summation is the geometric series, so we can apply the summation formula to get the result of the summation:

SUM[i = 0 to log₂(N - 1)] 2ⁱ = 2^{log₂(N - 1)} - 1

Finally, applying some simplifications, we arrive at the summation being O(N):

2^{log₂(N - 1)} - 1 = N - 1 - 1 = O(N)

So, for N adds, we have:

N assignments (for the actual data being added)
2⁰ + 2¹ + 2² + ... + 2^x = O(N) assignments (for the copying)

Thus, the runtime for N adds is N + O(N) = O(N). So our amortized time for one add is O(N) / N = O(1).

Recall that when increasing by 1 we had O(N²) overall for the sequence, which gives us O(N) in amortized time. Note how much better our performance is when we double the array size!

Runtime analysis and amortized analysis can be complicated at times. Often, they'll have a good deal of math in it. That is what algorithm analysis is all about though. If you can do some math you can save yourself some programming.

LinkedBag

//LinkedBag public boolean add (T newEntry)
{
    Node newNode = new Node(newEntry);
    newNode.next = firstNode;
    firstNode = newNode;

    numberofEntries++;

    return true;
}

What about the run-time for the singly linked list? Notice that this implementation always adds one to the size of the bag. How is this implementation's runtime similar/different from the ResizableArrayBag's implementation (where we increased the array size by one) and why?

Other Bag operations

The text discusses other Bag operations. It turns out that for the Bag, the run-times for the array and the linked list are the same for every operation. This will not always be the case, as we'll see later in the semester.

Stack Implementation Runtimes

What about the Stack implementations? What are the runtimes for:

ArrayStack
- push
- pop
- peek
LinkedStack
- push
- pop
- peek

<< Previous Notes

Daily Schedule

Next Notes >>