index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Software and Math on Software and Math</title>
    <link>https://vchernoy.xyz/</link>
    <description>Recent content in Software and Math on Software and Math</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Wed, 20 Apr 2016 00:00:00 +0000</lastBuildDate>
    <atom:link href="/" rel="self" type="application/rss+xml" />
    
    <item>
      <title>Cracking Multivariate Recursive Equations Using Generating Functions</title>
      <link>https://vchernoy.xyz/post/two-var-recursive-func/</link>
      <pubDate>Thu, 06 Jul 2017 07:29:43 +0000</pubDate>
      
      <guid>https://vchernoy.xyz/post/two-var-recursive-func/</guid>
      <description>

&lt;p&gt;In this post, we return back to the combinatorial problem discussed in &lt;a href=&#34;https://vchernoy.xyz/post/intro-to-dp/&#34;&gt;Introduction to Dynamic Programming and Memoization&lt;/a&gt; post.
We will show that generating functions may work great not only for single variable case (see &lt;a href=&#34;https://vchernoy.xyz/post/gen-func-art/&#34;&gt;The Art of Generating Functions&lt;/a&gt;),
but also could be very useful for hacking two-variable relations (and of course, in general for multivariate case too).&lt;/p&gt;

&lt;p&gt;For making the post self-contained, we repeat the problem definition here.&lt;/p&gt;

&lt;h2 id=&#34;the-problem&#34;&gt;The Problem&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Compute the number of ways to choose $m$ elements from $n$ elements such that selected elements in one combination are not adjacent.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For example, for $n=4$ and $m=2$, the answer is $3$, since from the $4$-element set: $\lbrace 1,2,3,4 \rbrace$,
there are three feasible $2$-element combinations: $\lbrace 1,4 \rbrace$, $\lbrace 2,4 \rbrace$, $\lbrace 1,3 \rbrace$.&lt;/p&gt;

&lt;p&gt;Another example: for $n=5$ and $m=3$, there is only one $3$-element combination: $\lbrace 1,3,5 \rbrace$.&lt;/p&gt;

&lt;p&gt;As we discussed in the first post, there is a nice recursive relation for this problem:&lt;/p&gt;

&lt;p&gt;$$ F_{n, m} = F_{n - 1, m} + F_{n - 2, m - 1} + [n=m=0] + [n=m=1] $$&lt;/p&gt;

&lt;p&gt;We assume that for any $n &amp;lt; 0$ or $m &amp;lt; 0$, $F_{n,m} = 0$.
The indicator $[P]$ gives $1$ if the predicate $P$ is true.&lt;/p&gt;

&lt;h2 id=&#34;the-generating-function-for-f-n-m&#34;&gt;The Generating Function for $F_{n,m}$&lt;/h2&gt;

&lt;p&gt;Let&amp;rsquo;s introduce the generating function $\Phi(x,y)$ of two (floating point) variables $x$ and $y$:&lt;/p&gt;

&lt;p&gt;$$\Phi(x,y) = \sum_{n,m} F_{n,m} x^n y^m$$&lt;/p&gt;

&lt;p&gt;Substituting the definition of $F_{n,m}$ and simplifying the sums, we will get:&lt;/p&gt;

&lt;p&gt;$ \Phi(x,y) $
$ = \sum_{n,m} F_{n,m} x^n y^m $
$ = \sum_{n,m} \left(F_{n - 1, m} + F_{n - 2, m - 1} + [n=m=1] + [n=m=0] \right) x^n y^m$
$ = \sum_{n,m} F_{n - 1, m} x^n y^m + \sum_{n,m} F_{n - 2, m - 1} x^n y^m + x \cdot y + 1 $
$ = x \sum_{n,m} F_{n - 1, m} x^{n-1} y^m + x^2 y \sum_{n,m} F_{n - 2, m - 1} x^{n-2} y^{m-1} + x \cdot y + 1 $
$ = x \Phi(x,y) + x^2 y \Phi(x,y) + x \cdot y + 1 $&lt;/p&gt;

&lt;p&gt;We have just found the simple representation for the generating function:&lt;/p&gt;

&lt;p&gt;$$\Phi(x, y) = \frac{1 + x \cdot y}{1 - x - x^2 y}$$&lt;/p&gt;

&lt;h2 id=&#34;the-closed-form-for-f-n-m&#34;&gt;The Closed Form for $F_{n,m}$&lt;/h2&gt;

&lt;p&gt;Using the infinite series: $ \frac{1}{1-z} = \sum_{k\geq 0} z^k $,
we can make the following transforms:&lt;/p&gt;

&lt;p&gt;$\Phi(x, y) $
$ = \frac{1 + x \cdot y}{1 - x - x^2 y}$
$ = (1 + x \cdot y) \sum_{k\geq 0} (x + x^2 y)^k $
$ = (1 + x \cdot y) \sum_{0 \leq i \leq k} {k \choose i} x^{k+i} y^i $
$ = \sum_{0 \leq i \leq k} {k \choose i} x^{k+i} y^i + \sum_{0 \leq i \leq k} {k \choose i} x^{k+i+1} y^{i+1}$&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Introducing the new variables: $n=k+i, m=i$, we cat transform the first expression as follows:
$ \sum_{0 \leq i \leq k} {k \choose i} x^{k+i} y^i $
$ = \sum_{m \geq 0, n \geq 2m} {n-m \choose m} x^{n} y^m $.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;And introducing the new variables: $n=k+i+1, m=i+1$, we can transform the second expression similarly:
$ \sum_{0 \leq i \leq k} {k \choose i} x^{k+i+1} y^{i+1} $
$ = \sum_{m \geq 1, n \geq 2m-1} {n-m \choose m-1} x^n y^m $.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The last two transformations give us the closed form:&lt;/p&gt;

&lt;p&gt;$F_{n, m} $
$ = {n - m \choose m} + {n - m \choose m - 1}$.&lt;/p&gt;

&lt;p&gt;Which actually equals to&lt;/p&gt;

&lt;p&gt;$$ F_{n, m} = {n - m + 1 \choose m} $$&lt;/p&gt;

&lt;h2 id=&#34;fast-solutions-based-on-binomials&#34;&gt;Fast Solutions Based on Binomials&lt;/h2&gt;

&lt;p&gt;Now we can reflect this idea in very trivial Python code:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-Python&#34;&gt;import math

def f_binom(n, m):
    assert n &amp;gt;= 0 and m &amp;gt;= 0

    if n + 1 &amp;lt; 2*m:
        return 0

    return binom(n - m + 1, m)

def binom(n, m):
    assert 0 &amp;lt;= m &amp;lt;= n

    return math.factorial(n) // math.factorial(m) // math.factorial(n - m)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This implementation overperforms significantly the initial DP and memoization solutions.
A naive implementation of &lt;code&gt;math.factorial()&lt;/code&gt; might make $n$ multiplications.
This could still be faster than doing $\Theta(n)$ additions in DP approach.&lt;/p&gt;

&lt;p&gt;The actual implementation of &lt;code&gt;math.factorial()&lt;/code&gt; is written in C
and probably has precomputed results for some range of $n$
and might even cache the results for bigger $n$.&lt;/p&gt;

&lt;h2 id=&#34;implementations-based-on-scipy-and-sympy-libraries&#34;&gt;Implementations Based on &lt;code&gt;scipy&lt;/code&gt; and &lt;code&gt;sympy&lt;/code&gt; Libraries&lt;/h2&gt;

&lt;p&gt;Several third party libraries provide a functionality to compute binomial coefficients.
Let&amp;rsquo;s take a look at &lt;code&gt;scipy&lt;/code&gt; and &lt;code&gt;sympy&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;We can install both of them using &lt;code&gt;pip&lt;/code&gt;-package manager:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-bash&#34;&gt;pip install scipy sympy
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;We can easily write two implementations of  $F_{n,m}$ which will call &lt;code&gt;scipy.special.comb()&lt;/code&gt; or &lt;code&gt;sympy.binomial()&lt;/code&gt; functions:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-Python&#34;&gt;import scipy.special

def f_sci(n, m):
    assert n &amp;gt;= 0 and m &amp;gt;= 0

    if n + 1 &amp;lt; 2*m:
        return 0

    return scipy.special.comb(n - m + 1, m, exact=True)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The second one is very similar to the first one:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-Python&#34;&gt;import sympy

def f_sym(n, m):
    assert n &amp;gt;= 0 and m &amp;gt;= 0

    if n + 1 &amp;lt; 2*m:
        return 0

    return sympy.binomial(n - m + 1, m)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;We can use the same &lt;code&gt;test()&lt;/code&gt; helper function that we defined in &lt;a href=&#34;https://vchernoy.xyz/post/intro-to-dp/&#34;&gt;Introduction to Dynamic Programming and Memoization&lt;/a&gt;.
Let&amp;rsquo;s run it on all the 5 implementatnions:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;funcs = [f_mem, f_dp, f_binom, f_sci, f_sym]
test(6000, 2000, funcs)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;It will print something similar to following output:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;f(6000,2000): 192496093
       f_mem:   6.7195 sec, x 4195.10
        f_dp:   5.3249 sec, x 3324.43
     f_binom:   0.0016 sec, x 1.00
       f_sci:   0.0021 sec, x 1.32
       f_sym:   0.0043 sec, x 2.69
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The first two methods, which are based on memoization and DP, are much slower than the last three,
which are based on the binomial coefficients.&lt;/p&gt;

&lt;h2 id=&#34;the-intuition-for-the-time-complexity-analysis&#34;&gt;The Intuition for the Time Complexity Analysis&lt;/h2&gt;

&lt;p&gt;DP and memoization makes $O(n^2)$ of &amp;ldquo;addition&amp;rdquo; operations over long integers.
The long integers are bounded by $F_{n,m}$ value, which could be bounded by $2^n$.
One &amp;ldquo;addition&amp;rdquo; operation takes $O(N)$ time for $N$-digit integer input.
For our case $N$ could be bounded by $ O(\log 2^n)$ $ = O(n)$.
So the total time complexity is bounded by
$ O(n^2 \cdot N) $
$ = O(n^2 \cdot \log 2^n) $
$ = O(n^2 \cdot n) $
$ = O(n^3)$.&lt;/p&gt;

&lt;p&gt;The binomial based solutions make $O(n)$ &amp;ldquo;multiplication&amp;rdquo; opearations over long integers
The long integers could be bounded by $O(n!)$ $ = O(n^n)$.
The &amp;ldquo;multiplication&amp;rdquo; opertion could be implemented in a naive way which runs $O(N^2)$ in time and is used for a small input.
But it also has a more advanced implementation, which takes $O(N^{\log_2 3})$ $ = O(N^{1.59})$ and is used on big integers.
Note that here $N$ denotes the number of digits in the input long integers for multiplication.
In this case, $N$ is bounded by
$ O(\log n!) $
$ = O(\log (n^n)) $
$ = O(n \log n) $.
The total time complexity of the binomial based implementations is bounded by
$ O\left(n \cdot N^{\log_2 3}\right) $
$ = O\left(n \cdot (\log n!)^{\log_2 3}\right)$
$ = O\left(n \cdot (n \log n)^{1.59}\right)$
$ = O\left(n^{2.59} \cdot (\log n)^{1.59}\right)$.&lt;/p&gt;

&lt;p&gt;This is not really a formal proof, but it gives some intuition why the last approach overperforms the former one.
In practice, the results of factorial computation are cached,
therefor we observe even bigger gap in performance (yeahh again not formal claim, just an intuition).&lt;/p&gt;

&lt;p&gt;You can play with running tests on different $n$ and $m$.
What I saw that actually there is no clear winner between the last 3 implementations.
Probably, the most of the time is spent on the long arithmetic computation.&lt;/p&gt;

&lt;h2 id=&#34;modular-arithmetics&#34;&gt;Modular Arithmetics&lt;/h2&gt;

&lt;p&gt;In questions where it is required to count some objects, not rearly the answer might be very big even on very small input.
In such case, typically it is asked to print the answer modulo some big prime integer, let&amp;rsquo;s say, $M=1000^3+7$.
Since Python has built-in long arithmetics, we can apply modulo on the final result,
but executing the entire agorithm with long arithmetics while knowing that only small part of it is really important is very costly,
and of course, not that efficient.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s look, briefly, at very simple change we can do for &lt;code&gt;f_binom&lt;/code&gt; function that will speed up the computation significantly:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;def f_binom_mod(n, m):
    assert n &amp;gt;= 0 and m &amp;gt;= 0

    if n + 1 &amp;lt; 2*m:
        return 0

    return binom_mod(n - m + 1, m)

def binom_mod(n, m):
    assert 0 &amp;lt;= m &amp;lt;= n

    return ((fact_mod(n) * inv_mod(fact_mod(m))) % M * inv_mod(fact_mod(n - m))) % M

@functools.lru_cache(maxsize=None)
def fact_mod(m):
    if m &amp;lt;= 1:
        return 1

    return (m * fact_mod(m - 1)) % M

def inv_mod(x):
    return pow(x, M - 2, M)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;As we can see, all the operations are computed modulo $M$.
The function &lt;code&gt;fact_mod&lt;/code&gt; is recursive but uses Memoization.
The most tricky part is how to implemenent modular-division.
From &lt;a href=&#34;https://en.wikipedia.org/wiki/Fermat%27s_little_theorem&#34; target=&#34;_blank&#34;&gt;Fermat&amp;rsquo;s little theorem&lt;/a&gt;,
we know that if $M$ is prime and $0 &amp;lt; x &amp;lt; M$, then $x^{-1} \equiv x^{M-2} \pmod M$.
This allows to compute the multiplicative inverse of $x$ using the Python&amp;rsquo;s built-in function
&lt;a href=&#34;https://docs.python.org/3/library/functions.html#pow&#34; target=&#34;_blank&#34;&gt;pow&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s test the new approach against other implementations:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;fact_mod(10000) # for caching factorials

funcs = [f_binom_mod, f_binom, f_sci, f_sym]

test(10000, 1000, funcs)
test(10000, 2000, funcs)
test(10000, 3000, funcs)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;It is not a surprise that taking the benefits of modular computations results in the huge speedup in running-time:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;f(10000,1000): 450169549
  f_binom_mod:   0.0000 sec, x 1.00
      f_binom:   0.0073 sec, x 337.60
        f_sci:   0.0011 sec, x 49.33
        f_sym:   0.0076 sec, x 353.22

f(10000,2000): 75198348
  f_binom_mod:   0.0000 sec, x 1.00
      f_binom:   0.0063 sec, x 368.94
        f_sci:   0.0026 sec, x 153.33
        f_sym:   0.0053 sec, x 308.93

f(10000,3000): 679286557
  f_binom_mod:   0.0000 sec, x 1.00
      f_binom:   0.0060 sec, x 361.12
        f_sci:   0.0056 sec, x 338.13
        f_sym:   0.0053 sec, x 319.02
&lt;/code&gt;&lt;/pre&gt;
</description>
    </item>
    
    <item>
      <title>The Art of Generating Functions</title>
      <link>https://vchernoy.xyz/post/gen-func-art/</link>
      <pubDate>Wed, 05 Jul 2017 07:29:43 +0000</pubDate>
      
      <guid>https://vchernoy.xyz/post/gen-func-art/</guid>
      <description>

&lt;p&gt;The notion of generating functions and its application to solving recursive equations are very well-known.
For reader who did not have a chance to get familiar with the topics,
I recommend to take a look at very good book:
&lt;a href=&#34;https://en.wikipedia.org/wiki/Concrete_Mathematics&#34; target=&#34;_blank&#34;&gt;Concrete Mathematics: A Foundation for Computer Science, by Ronald L. Graham, Donald E. Knuth, Oren Patashnik&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Generating functions are usually applied to single variable recursive equations.
But actually, the technique may be extended to multivariate recursive equations, or even to a system of recursive equations.
Readers who are familiar with one-variable case, may jump directly to the next post:
&lt;a href=&#34;https://vchernoy.xyz/post/two-var-recursive-func/&#34;&gt;Cracking Multivariate Recursive Equations Using Generating Functions&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&#34;the-generating-function-for-fibonacci-sequence&#34;&gt;The Generating Function for Fibonacci Sequence&lt;/h2&gt;

&lt;p&gt;Let&amp;rsquo;s consider the generating function&amp;rsquo;s application to Fibonacci numbers.
The Fibonacci numbers could defined by the recursive expression:&lt;/p&gt;

&lt;p&gt;$$f_n = f_{n-1} + f_{n-2} + [n=0]$$&lt;/p&gt;

&lt;p&gt;We assume that for any $n &amp;lt; 0$, $f_n = 0$.
The indicator $[n=0]$ equals to $1$ only if $n=0$.
This definition produces the following sequence of the Fibonacci numbers: $1$, $1$, $2$, $3$, $5$, $8$, $13$, $\dots$.
Note that sometime, the Fibonacci sequence is defined to start from 0, but it is really not important for the perpose of our discussion.&lt;/p&gt;

&lt;p&gt;The generating function $\Phi(x)$ on a floating point variable $x$ is defined as the infinite sum:&lt;/p&gt;

&lt;p&gt;$$\Phi(x) = \sum_{n\geq 0} f_n x^n$$&lt;/p&gt;

&lt;p&gt;Usually, it is hard to develop an intuition why it is defined that way and why it could be useful.
So let&amp;rsquo;s just focuse on what we can do with this,
and let&amp;rsquo;s start from substituting the defintion of Fibonacci recursion into the formular of generating function:&lt;/p&gt;

&lt;p&gt;$ \Phi(x) $
$ = \sum_{n\geq 0} f_n x^n $
$ = \sum_n f_n x^n $
$ = \sum_n (f_{n-1} + f_{n-2} + [n=0]) x^n $
$ = \sum_n f_{n-1} x^n + \sum_n f_{n-2} x^n + \sum_n [n=0] x^n $
$ = x \sum_n f_{n-1} x^{n-1} + x^2 \sum_n f_{n-2} x^{n-2} + 1 x^0 $
$ = x \sum_n f_n x^n + x^2 \sum_n f_n x^n + 1 $
$ = x \Phi(x) + x^2 \Phi(x) + 1 $&lt;/p&gt;

&lt;p&gt;And we can obtain the generating function for $f_n$:&lt;/p&gt;

&lt;p&gt;$$\Phi(x) = \frac{1}{1 - x - x^2}$$&lt;/p&gt;

&lt;h2 id=&#34;two-closed-forms-for-fibonacci-relation&#34;&gt;Two Closed Forms for Fibonacci Relation&lt;/h2&gt;

&lt;p&gt;Nice expression, but what can we do with this &amp;ldquo;magic&amp;rdquo; formula?
We can hack it in two different ways, and depending on our next step, we can get different closed forms for $f_n$.&lt;/p&gt;

&lt;h3 id=&#34;the-1st-closed-form&#34;&gt;The 1st closed form&lt;/h3&gt;

&lt;p&gt;One of the standard ways is to split the fraction into two simple ones of the form: $\frac{A}{x+B}$.
Note that the roots of the quadratic equation: $1 - x - x^2 = 0$ are $x_0=-\phi$ and $x_1=\phi^{-1}$,
where $\phi$ is the &lt;em&gt;Golden Ratio&lt;/em&gt;:&lt;/p&gt;

&lt;p&gt;$ \phi = \frac{1 + \sqrt{5}}{2} = 1.618\dots $.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s do a quick test:&lt;/p&gt;

&lt;p&gt;$ -(x-x_0) (x-x_1) $
$ = -(x+\phi) \left(x-\phi^{-1}\right) $
$ =-x^2 - x\cdot\left(\phi - \phi^{-1}\right) + 1 $
$ = -x^2 - x + 1 $.&lt;/p&gt;

&lt;p&gt;Now we can transform the generating function $\Phi(x)$ as follows:&lt;/p&gt;

&lt;p&gt;$\Phi(x)$
$ = \frac{1}{1-x-x^2}$
$ = -\frac{1}{(x+\phi) \left(x-\phi^{-1}\right)}$
$ = \frac{A}{x+\phi} - \frac{A}{x-\phi^{-1}}$
$ = A\cdot\phi^{-1} \frac{1}{1+x\cdot\phi^{-1}} + A\cdot\phi \frac{1}{1 - x\cdot\phi}$,&lt;/p&gt;

&lt;p&gt;where $A=\frac{1}{\phi+\phi^{-1}}$.&lt;/p&gt;

&lt;p&gt;The next trick is to apply the infinite series $\frac{1}{1-z} = \sum_n z^n$ to each of two expressions and to simplify the sums:&lt;/p&gt;

&lt;p&gt;$\Phi(x) $
$ = A\cdot\phi^{-1} \sum_n \left(-x\cdot \phi^{-1}\right)^n + A\cdot\phi \sum_n (x\cdot\phi)^n $
$ = A\cdot\phi^{-1} \sum_n (-\phi)^{-n} x^n + A\cdot\phi \sum_n \phi^n x^n $
$ = \sum_n A\cdot\left(\phi^{-1} (-\phi)^{-n} + \phi\cdot \phi^n \right)  x^n $
$ = \sum_n A\cdot\left(\phi^{n+1} - (-\phi)^{-n-1}\right) x^n $
$ = \sum_n \frac{\phi^{n+1} - (-\phi)^{-n-1}}{\phi+\phi^{-1}} x^n $.&lt;/p&gt;

&lt;p&gt;The term before $x^n$ is nothing but $f_n$, which gives the first closed form for the Fibonacci relation:&lt;/p&gt;

&lt;p&gt;$$f_n=\frac{\phi^{n+1} - (-\phi)^{-n-1}}{\phi+\phi^{-1}}$$&lt;/p&gt;

&lt;h3 id=&#34;the-2nd-closed-form&#34;&gt;The 2nd closed form&lt;/h3&gt;

&lt;p&gt;Another way to crack the generating function is to apply the infinite series $\frac{1}{1-z} $ $= \sum_n z^n$ directly to the generating function:&lt;/p&gt;

&lt;p&gt;$ \Phi(x) $
$ = \frac{1}{1 - x - x^2} $
$ = \frac{1}{1 - (x+x^2)}$
$ = \sum_n (x+x^2)^n $
$ = \sum_n (1+x)^n x^n $&lt;/p&gt;

&lt;p&gt;Now we can use the Binomial formula: $(a + b)^n = \sum_{0\leq k\leq n} {n \choose k}  a^{n-k} b^k$ in order to expand $(1+x)^n$:&lt;/p&gt;

&lt;p&gt;$ = \sum_{0\leq k \leq n} {n \choose k} x^{n+k} $.&lt;/p&gt;

&lt;p&gt;By introducing the new variable $t=n+k$, we obtain&lt;/p&gt;

&lt;p&gt;$ = \sum_{t/2 \leq n \leq t} {n \choose t-n} x^t $
$ = \sum_t \sum_{n=t/2}^t {n \choose t-n} x^t $&lt;/p&gt;

&lt;p&gt;Which gives us another closed form for the Fibonacci numbers:&lt;/p&gt;

&lt;p&gt;$$f_t = \sum_{n=t/2}^t {n \choose t-n}$$&lt;/p&gt;

&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;Believe it or not, but the two forms define the same Fibonacci sequence.&lt;/p&gt;

&lt;p&gt;$$ f_n = \frac{\phi^{n+1} - (-\phi)^{-n-1}}{\phi+\phi^{-1}} $$&lt;/p&gt;

&lt;p&gt;$$ f_n = \sum_{i=n/2}^n {i \choose n-i} $$&lt;/p&gt;

&lt;p&gt;Just notice how powerful the generating functions are!
An interested reader can try to prove by induction the correctness of the relations above.
If you liked the topic, you are wellcome to take a look at the next post where we extend the tools to mutivariate recursions.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Introduction to Dynamic Programming and Memoization</title>
      <link>https://vchernoy.xyz/post/intro-to-dp/</link>
      <pubDate>Tue, 04 Jul 2017 07:29:43 +0000</pubDate>
      
      <guid>https://vchernoy.xyz/post/intro-to-dp/</guid>
      <description>

&lt;p&gt;In the post, we discuss the basics of &lt;em&gt;Recursion&lt;/em&gt;, &lt;em&gt;Dynamic Programming&lt;/em&gt; (DP), and &lt;em&gt;Memoization&lt;/em&gt;.
As an example, we take a combinatorial problem, which has very short and clear description.
This allows us to focus on DP and memoization.
Note that the topics are very popular in coding interviews.
Hopefully, this article will help to somebody to prepare for such types of questions.&lt;/p&gt;

&lt;p&gt;In the next posts,
we consider more advanced topics, like
&lt;a href=&#34;https://vchernoy.xyz/post/gen-func-art/&#34;&gt;The Art of Generating Functions&lt;/a&gt; and
&lt;a href=&#34;https://vchernoy.xyz/post/two-var-recursive-func/&#34;&gt;Cracking Multivariate Recursive Equations Using Generating Functions&lt;/a&gt;.
The methods can be applied to the same combinatorial question.
Let&amp;rsquo;s start from presenting the problem.&lt;/p&gt;

&lt;h2 id=&#34;the-problem&#34;&gt;The Problem&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Compute the number of ways to choose $m$ elements from $n$ elements such that selected elements in one combination are not adjacent.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For example, for $n=4$ and $m=2$, the answer is $3$, since from the $4$-element set: $\lbrace 1,2,3,4 \rbrace$,
there are three feasible $2$-element combinations: $\lbrace 1,4 \rbrace$, $\lbrace 2,4 \rbrace$, $\lbrace 1,3 \rbrace$.&lt;/p&gt;

&lt;p&gt;Another example: for $n=5$ and $m=3$, there is only one $3$-element combination: $\lbrace 1,3,5 \rbrace$.&lt;/p&gt;

&lt;p&gt;If you are asked such question during coding interview, interviewer is, probably, expecting to cover with you the following topics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;em&gt;Brute Force&lt;/em&gt; approach that generates and counts all the feasible combinations. It will work slow even for small input.&lt;/li&gt;
&lt;li&gt;Recursive solution which counts the combination without generating them. It will work faster but still has exponential time ecomplexity.&lt;/li&gt;
&lt;li&gt;Use DP or memoization techniques. In both cases, the time complexity becomes linear in $n$ and $m$.&lt;/li&gt;
&lt;li&gt;Corner cases, recursion termination, call stack, testing.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Let&amp;rsquo;s talk about most important topics: building a recursive solution and optimize it using DP or memoization.&lt;/p&gt;

&lt;h2 id=&#34;recursive-relation&#34;&gt;Recursive Relation&lt;/h2&gt;

&lt;p&gt;Let&amp;rsquo;s define $F_{n, m}$ to be the function that computes the answer for given $n$ and $m$.
Let&amp;rsquo;s look at the $n, m$-task. We have two non-overlapping sub-tasks (or cases):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Skip the $n$-th element, then $ F_{n, m} = F_{n-1, m} $.&lt;/li&gt;
&lt;li&gt;Pick the $n$-th element, then $ F_{n, m} = F_{n-2, m-1} $.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;From the above, we can define the solution in the recursive form:&lt;/p&gt;

&lt;p&gt;$ F_{n, m} $ $ = F_{n - 1, m} + F_{n - 2, m - 1} $,&lt;/p&gt;

&lt;p&gt;let&amp;rsquo;s not forget to write down the corner cases:&lt;/p&gt;

&lt;p&gt;$F_{0, 0}$ $= F_{1, 1}$ $= 1$.&lt;/p&gt;

&lt;p&gt;Basically, we can combine the general and the corner cases into one expression:&lt;/p&gt;

&lt;p&gt;$$ F_{n, m} = F_{n - 1, m} + F_{n - 2, m - 1} + [n=m=0] + [n=m=1] $$&lt;/p&gt;

&lt;p&gt;It is very common to define $F_{n,m} = 0$ for any negative $n$ or $m$.
The indicator $[P]$ gives $1$ if the predicate $P$ is true.
But what about special cases, like $n=1$ and $m=0$?
Actually, the relation covers this:&lt;/p&gt;

&lt;p&gt;$F_{1,0}$ $= F_{0,0} + F_{-1,-1} + [1=0=1] + [1=m=1]$ $=F_{0,0} + 0 + 0 + 0$ $=[0=0=0]$ $=1$.&lt;/p&gt;

&lt;p&gt;It is non-intuitive, but there is no need to add extra corner cases!&lt;/p&gt;

&lt;h2 id=&#34;memoization&#34;&gt;Memoization&lt;/h2&gt;

&lt;p&gt;The function $F_{n,m}$ has a straighforward recursive implementation in any programming language.
But the naive recursive solution will have exponential in $n$ time complexity and will be very slow.
Let&amp;rsquo;s look at the following &lt;code&gt;Python 3&lt;/code&gt; code snippet that uses memoization technique:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;import functools
import sys

sys.setrecursionlimit(100000)

@functools.lru_cache(maxsize=None)
def f_mem(n, m):
    if n &amp;lt; 0 or m &amp;lt; 0:
        return 0

    if n + 1 &amp;lt; 2*m:
        return 0

    if n == m == 0:
        return 1

    if n == m == 1:
        return 1

    return f_mem(n - 1, m) + f_mem(n - 2, m - 1)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Nice &lt;a href=&#34;https://docs.python.org/3/library/functools.html#functools.lru_cache&#34; target=&#34;_blank&#34;&gt;@functools.lru_cache&lt;/a&gt;
notation creates a wrapper on the &lt;code&gt;f_mem&lt;/code&gt; function and internally caches the results of all calls.
Such caching (or memoization) significantly improves the speed of the recursion,
and basically reduces the number of calls to something like $O(n \cdot m)$.
The recursion still may fail on the stack overflow even on relatively small values of $n$.
That is why we increase the stack size for the Python interpreter by calling
&lt;a href=&#34;https://docs.python.org/3/library/sys.html#sys.setrecursionlimit&#34; target=&#34;_blank&#34;&gt;sys.setrecursionlimit()-method&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&#34;dynamic-programming-dp&#34;&gt;Dynamic Programming (DP)&lt;/h2&gt;

&lt;p&gt;The next iterative implementation uses DP.
Basically, it fills out the $n \times m$ table starting from the low values of $n$ and $m$.&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;def f_dp(n, m):
    assert n &amp;gt;= 0 and m &amp;gt;= 0

    if n + 1 &amp;lt; 2*m:
        return 0

    table = [[0] * (m + 1) for _ in range(n + 1)]

    table[0][0] = 1
    if n &amp;gt;= 1:
        table[1][0] = 1
        if m &amp;gt;= 1:
            table[1][1] = 1

    for i in range(2, n+1):
        table[i][0] = 1
        for j in range(1, min(m, (i + 1) // 2) + 1):
            table[i][j] = table[i-1][j] + table[i-2][j-1]

    return table[n][m]
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The time and space complexities are $O(n\cdot m) $.
More accurately, time complexity might be bounded by $\Theta(n\cdot \min(n,m))$.
It is not hard to notice that the space consumption could be reduced to $O(\min(n, m))$.&lt;/p&gt;

&lt;h2 id=&#34;testing&#34;&gt;Testing&lt;/h2&gt;

&lt;p&gt;In order to measure the time and to check the correctness, let&amp;rsquo;s write the following helper function:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;import timeit

M = 1000**3 + 7

def test(n, m, funcs, number=1, module=__name__):
    f_mem.cache_clear()
    results = []
    func_times = []
    for func in funcs:
        stmt=&#39;{}({},{})&#39;.format(func.__name__, n, m)
        setup=&#39;from {} import {}&#39;.format(module, func.__name__)
        t = timeit.timeit(stmt=stmt, setup=setup, number=number)
        func_times.append(t)
        results.append(func(n, m) % M)

    assert len(set(results)) &amp;lt;= 1

    if results:
        print(&#39;f({},{}): {}&#39;.format(n, m, results[0]))

    best_time = min(func_times)
    for i, func in enumerate(funcs):
        func_time = func_times[i]
        print(&#39;{:&amp;gt;13}: {:8.4f} sec, x {:.2f}&#39;.format(func.__name__, func_time, func_time/best_time))
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;We can run it as following:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&#34;language-python&#34;&gt;test(n=6000, m=2000, funcs=[f_mem, f_dp])
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Which prints output similar to this one:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;f(6000,2000): 192496093
        f_mem:   6.5852 sec, x 1.28
         f_dp:   5.1507 sec, x 1.00
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The function &lt;code&gt;test()&lt;/code&gt; computes $F_{6000, 2000}$ using two different ways.
It validates that all the solutions are consistent and produce the same result.
It also measures the time it takes to execute each method.
For each run, it also prints the relative factor computed based on the fastest solution.
The function&amp;rsquo;s result is printed modulo $M=1000^3+7$.&lt;/p&gt;

&lt;p&gt;Memoization and DP techniques have $\Theta(n \cdot m)$ time complexity.
Note that we ignore here a lot of interesting details, for example,
Python has built-in long arithmetics for integers, which is used here and definetely not cheap.&lt;/p&gt;

&lt;p&gt;Testing the implementations on different parameters,
we may notice that there is no clear winner between the two algorithms.
Probably, most of the time is spent on the long arithmetic computation.&lt;/p&gt;

&lt;p&gt;Can we do even better?
Definitely, we can!
The table (or cache) could be preprocessed.
Then computing the function $F_{n,m}$ will require just one query from the table (or from the cache, for memoization solution).
As we mentioned before, such approach requires $\Theta(n\cdot m)$ space, which could be huge for $n=10000$.
Actually, there is a way, how we can remain in very low memory consuption and still get very fast solution.
We will discuss this in the next two posts.&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>