Haiku Generation with Causal Language Modeling

M.S. Project — 2018-2020

We examine the theoretical foundations of causal language modeling, where natural language is modeled in terms of the probability distribution of a token in a sequence. This probability distribution is dependent on the values of all of the preceding tokens in the sequence.

Due to time constraints and real-life factors, the final deliverable was changed from a traditional thesis to a project. The project was to be an implementation of an algorithm to generate haiku. This algorithm need not be original.

We tried a number of generative language models, from using Markov chains and naive LSTM networks, to using textgenrnn by Max Woolf, and finally by fine-tuning a pretrained GPT-2 model on haiku scraped from online haiku submission and award websites.

As expected, the results from using a fine-tuned GPT-2 model were by far the best. Some generated haiku were eerily cogent, and many (though not all) generated haiku display many traits of haiku. A sample of some of the generated haiku is given below.

model | seed | haiku |
---|---|---|

Markov | - |
summer / all these extra prayers / at the dead puppies |

GPT-2 commandline | 3191532796 | spoonfuls / of medication / and loneliness |

GPT-2 web | 2607968983 | vodka in the bottle / i go through / the exam |

GPT-2 web | 2913164516 | thunder / the stock market goes up / and down |

GPT-2 commandline | 718585002 | http haiku / some thoughts im afraid of / all alone at the coffee shop |

GPT-2 web | 1691016217 | bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang |

GPT-2 web | 3138988667 | system failure / a distant train / making a sudden silence |

GPT-2 web | 921792121 | may the force of life / reminding me / not to call it quits |

GPT-2 web | 1338062952 | team meeting / everyone is still / talking about joining us |

Understanding *A Fast Contour-Integral Eigensolver for Non-Hermitian Matrices*

B.S. Project — 2017-2018

We examine a new complex contour integral-based eigensolver algorithm using ideas from functional analysis. This eigensolver algorithm extends previous eigensolvers that typically required some special structure, to an eigensolver that works on a more general class of complex-valued matrices.

The algorithm consists of two stages. First, it searches the complex plane for regions dense with eigenvalues. Second, it combs through those regions to find the eigenvalues using a modified form of the FEAST algorithm.

Due to time constraints we were only able to examine the first stage in detail, and for completeness provide a description for the second stage. Our work was exploratory in nature, which in practice consisted of building and testing a Matlab implementation of the first stage.