aim is to make everyones life easier to experiment with AI
requirements & deliverables
- create python script to setup a VM in VAST.AI (all automated)
- easy install (integrated with the setup script on VAST.AI)
- bash script for installing all required components (use Ubuntu 22.04)
- requirements script for pip install (required python components)
- a test script to check the cuda is working in right version & the GPU is found with minimal requirements
- a python script which parses pdf (use pdf2text commandline, not the python version), uses wiki txt files, download a website e.g. manual.grid.tf and question/answer files for finetuning and populates the model
- experiment with multiple models and demonstrate results (timing, quality, ...): see below
- bring openai compatible API life on top of Avast GPU machine (ok to do SSH portforwarding)
- all is opensource
- an end2end test
- create a docker build (or use vbuilder even better to make build
- build the docker upload to docker hub (even the test data: pdf, doing a web crawl to manual ... as part of it)
- deploy using vast AI and use the above created docker
- there should now a machine online which has API compatible with openaid
- now do a script which calls the API and queries the content and proves that info from manual, pdf is there
- there should be link back to where the info comes from in the result (can this be done?)
- document all in an mdbook: results of different models, performance in relation to GPU mem, quality, how to start, ...
- idea is that with nothing more than the mdbook a scripter person with some linux expertise can re-do all the tests
gpu usage
models
experiment with following models
- falcon 7B
- falcon 70B (load on 40 GB GPU, there are tricks)
- falcon 70b on DUAL GPU A6000 instance
- llama2 (see if better)
some info which can be used
Fine-tuning Large Language Model (LLM) on a Custom Dataset with QLoRA _ MLExpert - Crush Your Machine Learning interview.pdf
implementation details
- all scripts use 'set -ex' to make sure they stop when error
- can also use vscript (see vlang) as alternative to bash, we have quite some primitives working see crystallib (ask codescalers team for help if needed)
- put some example pdf, and text files in this repo' so its easy for people to experiment
- use info from threefold and see how the results are for questions and answer: https://manual.grid.tf/
aim is to make everyones life easier to experiment with AI
requirements & deliverables
gpu usage
models
experiment with following models
some info which can be used
Fine-tuning Large Language Model (LLM) on a Custom Dataset with QLoRA _ MLExpert - Crush Your Machine Learning interview.pdf
implementation details