We examine the theoretical foundations of causal language modeling, where natural language is modeled in terms of the probability distribution of a token in a sequence. This probability distribution is dependent on the values of all of the preceding tokens in the sequence.
Due to time constraints and real-life factors, the final deliverable was changed from a traditional thesis to a project. The project was to be an implementation of an algorithm to generate haiku. This algorithm need not be original.
We tried a number of generative language models, from using Markov chains and naive LSTM networks, to using textgenrnn by Max Woolf, and finally by fine-tuning a pretrained GPT-2 model on haiku scraped from online haiku submission and award websites.
As expected, the results from using a fine-tuned GPT-2 model were by far the best. Some generated haiku were eerily cogent, and many (though not all) generated haiku display many traits of haiku. A sample of some of the generated haiku is given below.
| model | seed | haiku |
|---|---|---|
| Markov | - |
summer / all these extra prayers / at the dead puppies |
| GPT-2 commandline | 3191532796 | spoonfuls / of medication / and loneliness |
| GPT-2 web | 2607968983 | vodka in the bottle / i go through / the exam |
| GPT-2 web | 2913164516 | thunder / the stock market goes up / and down |
| GPT-2 commandline | 718585002 | http haiku / some thoughts im afraid of / all alone at the coffee shop |
| GPT-2 web | 1691016217 | bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang bang |
| GPT-2 web | 3138988667 | system failure / a distant train / making a sudden silence |
| GPT-2 web | 921792121 | may the force of life / reminding me / not to call it quits |
| GPT-2 web | 1338062952 | team meeting / everyone is still / talking about joining us |
| GPT-2 web | 2926333899 | guidance meeting / each path and its source / is a lie |
| GPT-2 web | 80003916 | guidance curves / the plowman chews / his tongue |
| GPT-2 web | 1391183336 | faulty hardware / i remember / why i lost my job |
| GPT-2 web | 3506727018 | faulty hardware / once again on a computer / it never shows |
| GPT-2 web | 2493195556 | http / he says so / haiku has no place |
We examine a new complex contour integral-based eigensolver algorithm using ideas from functional analysis. This eigensolver algorithm extends previous eigensolvers that typically required some special structure, to an eigensolver that works on a more general class of complex-valued matrices.
The algorithm consists of two stages. First, it searches the complex plane for regions dense with eigenvalues. Second, it combs through those regions to find the eigenvalues using a modified form of the FEAST algorithm.
Due to time constraints we were only able to examine the first stage in detail, and for completeness provide a description for the second stage. Our work was exploratory in nature, which in practice consisted of building and testing a Matlab implementation of the first stage.