Ollama Shenanigans

Ollama is a tool that allows one to run LLMs locally. Unfortunately I was unable to reoroduce some guides (probably targetting a different Operating System) and tried to use tools designed to work with ChatGPT. This, of course, involves paying for it and that was outside of my objective.

Looking at the model page finally got me the correct address point: http://localhost:11434/api/generate but it didn’t work with GPT Pilot so I used a python package to try and run it locally

I made a scriptto do just that:

#!/usr/bin/env python
import sys
import signal
import ollama

USER = 'user'
MODEL = 'deepseek-coder'

def signal_handler(sig, frame):
    print('\nYou pressed Ctrl+C!')
    sys.exit(0)

def prompt(prompt, context=''):
    try:
        return ollama.chat(model=MODEL, messages=[
            {
            'role': USER,
            'content': prompt,
            },
        ])
    except ollama.ResponseError as e:
      print('Error:', e.error)
      if e.status_code == 404:
        ollama.pull(model)
        return []

try:
    query = ''
    signal.signal(signal.SIGINT, signal_handler)
    while True:
        query = input('> ')
        if  query.lower() == 'exit' or query.lower() == 'quit':
            break
        if len(query) > 0:
            print(prompt(query)['message']['content'])
except Exception as ex:
    print(ex)