-
Notifications
You must be signed in to change notification settings - Fork 10
Add dev quick start guide #16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| --- | ||
| layout: default | ||
| title: Dev Quick Start Guide | ||
| nav_order: 7 | ||
| --- | ||
|
|
||
| # Dev Quick Start Guide | ||
| {: .no_toc } | ||
|
|
||
| This document describes how to quickly get a runtime server built with `sample_lib` library, train a model with gRPC and with that trained model, send an inference call to the server with either HTTP or gRPC call. | ||
|
|
||
| ## Pre-reqs | ||
| To set up development environment and tooling, the [Development Guide]({{ site.baseurl }}{% link docs/developing.md %}) page is a good place to start. | ||
|
|
||
| ## Build and start a runtime server with sample_lib | ||
|
|
||
| Run the `start_runtime_with_sample_lib` python script: | ||
|
|
||
| ```shell | ||
| python3 -m examples.sample_lib.start_runtime_with_sample_lib | ||
| ``` | ||
|
|
||
| This will setup a config with both `grpc` and `http` servers enabled for inference service, as well as training service. The library used with some sample modules is `sample_lib`. The script then starts the caikit runtime server. While the server is running, you can see the generated protobufs in a directory called `protos`. | ||
|
|
||
| We generate 3 services total: | ||
| - A train service. The proto for this service is `protos/samplelibtrainingservice.proto` | ||
| - An inference service. The proto for this service is `protos/samplelibservice.proto` | ||
| - A training management service. The proto for this service is `protos/trainingmanagement.proto` | ||
|
|
||
| You can now leave the server running and open a new terminal to proceed with next steps to train a model, check its training status and send an inference request to your model. | ||
|
|
||
| (To kill the server, press Ctrl + C. This will remove the `protos` directory to clean up.) | ||
|
|
||
| ## To train a model | ||
|
|
||
| As of caikit v`0.18.0`, we do not yet support training model via REST. | ||
| In order to train a model via gRPC, we will use `grpcurl` and point the import-path to `protos` dir, then call one of the Train rpc's available in the `SampleLibTrainingService` (see `protos/samplelibtrainingservice.proto` file generated above for all Train rpcs): | ||
|
|
||
| ```shell | ||
| grpcurl -plaintext -import-path protos/ -proto samplelibtrainingservice.proto -d '{"model_name": "my_model", "training_data": {"file": {"filename": "protos/sample.json"}}}' localhost:8085 caikit.runtime.SampleLib.SampleLibTrainingService/SampleTaskSampleModuleTrain | ||
| ``` | ||
|
|
||
| You should receive a response similar to the below: | ||
|
|
||
| ```shell | ||
| { | ||
| "trainingId": "wTHxlsu:5bdb5949-4efa-4512-bbac-709cbf37c00e", | ||
| "modelName": "my_model" | ||
| } | ||
| ``` | ||
|
|
||
| Copy the `trainingId` to use in next step. | ||
|
|
||
| ## To check on training status of a trainingId | ||
|
|
||
| With a `trainingId`, you can get a training status via gRPC. Replace the command below with your `trainingId`. | ||
|
|
||
| ```shell | ||
| grpcurl -plaintext -import-path protos/ -proto trainingmanagement.proto -d '{"training_id": "<training_id"}' localhost:8085 caikit.runtime.training.TrainingManagement/GetTrainingStatus | ||
| ``` | ||
|
|
||
| You should get a response like this: | ||
|
|
||
| ```shell | ||
| { | ||
| "trainingId": "wTHxlsu:5bdb5949-4efa-4512-bbac-709cbf37c00e", | ||
| "state": "COMPLETED", | ||
| "submissionTimestamp": "2023-08-30T22:19:13.739694Z", | ||
| "completionTimestamp": "2023-08-30T22:19:13.744542Z" | ||
| } | ||
| ``` | ||
| Once your training is completed, you can proceed to call inference on the model. | ||
|
|
||
| ## To call inference on a model with modelId | ||
|
|
||
| You are now ready to call inference via either gRPC or REST. | ||
|
|
||
| ### To use the REST Server for inference | ||
|
|
||
| - In a browser of choice, visit `http://localhost:8080/docs/`. All the available inference rpcs are listed. Expand on the correct task for the model you trained. In this example, we are using `api/v1/{model_id}/task/sample`. | ||
|
||
| - Click "Try It Out" | ||
| - Fill in model_id "my_model" as used in the train a model step. Change the request body to your liking. Then click "Execute". Ex: | ||
|
|
||
| ```shell | ||
| curl -X 'POST' \ | ||
| 'http://localhost:8080/api/v1/my_model/task/sample' \ | ||
| -H 'accept: application/json' \ | ||
| -H 'Content-Type: application/json' \ | ||
| -d '{ | ||
| "inputs": { | ||
| "name": "IBM" | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
| You should receive a 200 response back with a response body: | ||
| ```shell | ||
| { | ||
| "greeting": "Hello IBM" | ||
| } | ||
| ``` | ||
|
|
||
| ### To use the gRPC Server for inference | ||
|
|
||
| You can also use the gRPC Server to call inference on this model by running: | ||
| ```shell | ||
| grpcurl -plaintext -import-path protos/ -proto samplelibservice.proto -d '{"sample_input": {"name": "IBM"}}' -H 'mm-model-id: my_model' localhost:8085 caikit.runtime.SampleLib.SampleLibService/SampleTaskPredict | ||
| ``` | ||
|
|
||
| You should receive a sucessful response back with a response body: | ||
| ```shell | ||
| { | ||
| "greeting": "Hello IBM" | ||
| } | ||
| ``` | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it worth duplicating the information that we have in the Caikit examples README? If we want to add more information, I feel we should add it to that readme instead of duplicating it here? That would make this doc pretty small though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise with the API changes that are incoming, we'll have to make changes to 2 places 😢