Skip to content

Commit ccc47fb

Browse files
zhengxs2018zhengxs2018
authored
feat: support fim special tokens configuration (#62)
* fix: support for chat.models override base configuration * fix: some non-functional bug * feat: support fim special tokens configuration
1 parent e2070d4 commit ccc47fb

11 files changed

+367
-133
lines changed

‎.cspell.json

+4-1
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,16 @@
1111
"useGitignore": true,
1212
"ignorePaths": [],
1313
"words": [
14+
"codeqwen",
15+
"identifer",
1416
"inversify",
17+
"lancedb",
1518
"ollama",
1619
"onnx",
1720
"openai",
1821
"phodal",
19-
"identifer",
2022
"qianfan",
23+
"qwen",
2124
"tolist",
2225
"tongyi",
2326
"xenova"

‎docs/configuration.md

+52-21
Original file line numberDiff line numberDiff line change
@@ -105,42 +105,75 @@ Choose a default model provider to give code generation.
105105

106106
Model for overwrite provider in the provider completion model
107107

108-
### Template
108+
### FIM Special Tokens
109109

110-
Customize your modeling cue template.
110+
Fill-in-the-middle (FIM) is a special prompt format supported by the code completion model can complete code between two already written code blocks.
111111

112-
> [!IMPORTANT]
113-
> Variables use string replacement, please fill in strictly according to instructions.
112+
See [Code Completions](./features/code-completion.md).
114113

115-
The recommended format is FIM ( filling in the middle ), for example:
114+
```json
115+
{
116+
"autodev.completions.fimSpecialTokens": {
117+
"prefix": "<PRE>",
118+
"suffix": "<SUF>",
119+
"middle": "<MID>"
120+
}
121+
}
122+
```
116123

117-
```sh
118-
<fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>
124+
### Parameters
119125

120-
# or
126+
Some model parameters.
121127

122-
<PRE>{prefix}<SUF>{suffix}<MID>
128+
> Please don't modify it unless you know what you're doing.
129+
130+
```json
131+
{
132+
"autodev.completions.parameters": {
133+
"temperature": 0,
134+
"top_p": 0.9,
135+
"max_tokens": 500
136+
}
137+
}
123138
```
124139

140+
### Stops
141+
142+
Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.
143+
144+
### Request Delay
145+
146+
Code auto-completion delay request time. Avoid excessive consumption of API tokens. `requestDelay` only works if `Autodev: Enable Code Completions` is enabled.
147+
148+
### Enable Legacy Mode
149+
150+
Use legacy `/v1/completion` instead of `/v1/chat/completion`. Only working `openai` provider.
151+
152+
> Will be deprecated when infill is universally supported or openai stop "/v1/completions" support.
153+
154+
### Template
155+
156+
Customize your modeling cue template.
157+
158+
> Place use [FIM Special Tokens](#fim-special-tokens) instead.
159+
125160
Available Variables:
126161

127162
- `prefix` Code before the cursor.
128163
- `suffix` Code after the cursor.
129164
- `language` Current editing file language, for example: "javascript".
130165

131-
### Stops
132-
133-
Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.
134-
135-
### Enable Legacy Mode
166+
The recommended format is FIM ( filling in the middle ), for example:
136167

137-
> Only working `openai` provider
168+
```sh
169+
<fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>
138170

139-
Use legacy `/v1/completion` instead of `/v1/chat/completion`
171+
# or
140172

141-
### Request Delay
173+
<PRE>{prefix}<SUF>{suffix}<MID>
174+
```
142175

143-
Code auto-completion delay request time. Avoid excessive consumption of API tokens. `requestDelay` only works if `Autodev: Enable Code Completions` is enabled.
176+
Variables use string replacement, please fill in strictly according to instructions.
144177

145178
## Embeddings
146179

@@ -201,6 +234,4 @@ See [ollama](https://ollama.com/)
201234

202235
### Transformers
203236

204-
local model runtime. For codebase and embedding only.
205-
206-
237+
local model runtime. For codebase and embedding only.

‎docs/features/code-completion.md

+74-19
Original file line numberDiff line numberDiff line change
@@ -7,53 +7,108 @@ nav_order: 1
77

88
Automatically completes your code based on the position of your cursor.
99

10-
## Enable Feature
10+
> We validate the codegemma, codeqwen, and codellama，other models need to be tested on their own.
11+
12+
## Enable Code Completions
1113

1214
Not enabled by default, see [AutoDev: Code Completion](../configuration.md#code-completion)。
1315

1416
```jsonc
1517
{
16-
"autodev.openai.apiKey": "sk-xxxxx", // Your openai api key
17-
"autodev.completions.enable": true // Enabled Inline Completions
18+
"autodev.completions.enable": true, // Enabled or disable
1819
}
1920
```
2021

21-
Now let's try writing some code.
22+
Next step: select a code pre-training variant that specializes in code completion and generation of code prefixes and/or suffixes.
2223

23-
## Select code Model
24+
## Fill-in-the-middle
2425

25-
You can hope that you use specific code models instead of dialog models
26+
Fill-in-the-middle (FIM) is a special prompt format supported by the code completion model can complete code between two already written code blocks.
2627

27-
```jsonc
28+
[codellama](https://ollama.com/blog/how-to-prompt-code-llama) expects a specific format for infilling code:
29+
30+
```json
31+
{
32+
"autodev.experimental.fimSpecialTokens": {
33+
"prefix": "<PRE>",
34+
"suffix": "<SUF>",
35+
"middle": "<MID>"
36+
}
37+
}
38+
```
39+
40+
[codeqwen](https://github.com/QwenLM/CodeQwen1.5/blob/main/examples/CodeQwen1.5-base-fim.py) expects a specific format for infilling code:
41+
42+
```json
43+
{
44+
"autodev.experimental.fimSpecialTokens": {
45+
"prefix": "<fim_prefix>",
46+
"suffix": "<fim_suffix>",
47+
"middle": "<fim_middle>"
48+
}
49+
}
50+
```
51+
52+
[codegemma](https://ai.google.dev/gemma/docs/formatting) expects a specific format for infilling code:
53+
54+
```json
2855
{
29-
"autodev.completions.model": "gpt-4o" // Overriding the default chat model
56+
"autodev.experimental.fimSpecialTokens": {
57+
"prefix": "<|fim_prefix|>",
58+
"suffix": "<|fim_suffix|>",
59+
"middle": "<|fim_middle|>"
60+
}
3061
}
3162
```
3263

33-
Recommended to use a specially trained code model, or a base model that supports fim.
64+
For other models, please select the appropriate special format.
65+
66+
**TIP:** codeqwen and codegemma can be used with the same.
67+
68+
## Best practice
3469

35-
## Connect to local model
70+
Because of the lack of resources, we used "ollama" to verify the reliability of the model.
3671

37-
Here is an example of ollama, see [OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md) for details.
72+
### Use CodeQwen
73+
74+
Support for `codeqwen:7b-code-v1.5-q5_1`.
75+
76+
> The most stable model available.
3877
3978
```jsonc
4079
{
41-
"autodev.openai.baseURL": "http://127.0.0.1:11434/v1/", // Your local service url
42-
"autodev.openai.apiKey": "sk-xxxxx", // Your local service api key
43-
"autodev.completions.model": "codeqwen:7b-code-v1.5-q5_1" // Overriding the default chat model
80+
"autodev.completions.provider": "ollama",
81+
"autodev.completions.model": "codeqwen:7b-code-v1.5-q5_1",
4482
}
4583
```
4684

47-
If your self-built service is deployed in a mode that does not support chat, you may need to enable [legacy mode](#enable-legacy-mode).
85+
### Use CodeLlama
86+
87+
Support for `codellama:7b`, `codellama:7b-code`, `codellama:7b-instruct`
88+
89+
> Unstable generation.
90+
91+
```jsonc
92+
{
93+
"autodev.completions.provider": "ollama",
94+
"autodev.completions.model": "codellama:7b-code",
95+
"autodev.experimental.fimSpecialTokens": {
96+
"prefix": "<PRE>",
97+
"suffix": "<SUF>",
98+
"middle": "<MID>",
99+
},
100+
}
101+
```
48102

49-
## Enable Legacy Mode
103+
### Use CodeGemma
50104

51-
The default is the traditional `/v1/completions` instead of `/v1/chat/completions`, but you can fall back to the old mode.
105+
Support for `codegemma:2b-code`
52106

53-
> Only working on openai provider
107+
> Unstable generation.
54108
55109
```jsonc
56110
{
57-
"autodev.completions.enableLegacyMode": true
111+
"autodev.completions.provider": "ollama",
112+
"autodev.completions.model": "codegemma:2b-code",
58113
}
59114
```

‎package.json

+68-12
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,7 @@
166166
"properties": {
167167
"autodev.completions.enable": {
168168
"type": "boolean",
169-
"default": true,
169+
"default": false,
170170
"description": "%configuration.completions.enable.description%",
171171
"order": 1
172172
},
@@ -180,38 +180,94 @@
180180
"tongyi",
181181
"ollama"
182182
],
183-
"default": "openai",
183+
"default": "ollama",
184184
"order": 1
185185
},
186186
"autodev.completions.model": {
187187
"type": "string",
188188
"description": "%configuration.completions.model.description%",
189+
"default": "codeqwen:7b-code-v1.5-q5_1",
189190
"order": 2
190191
},
191-
"autodev.completions.template": {
192-
"type": "string",
193-
"description": "%configuration.completions.template.description%",
194-
"order": 3
192+
"autodev.completions.parameters": {
193+
"type": "object",
194+
"properties": {
195+
"temperature": {
196+
"type": "number",
197+
"default": 0
198+
},
199+
"top_p": {
200+
"type": "number",
201+
"default": 0.9
202+
},
203+
"max_tokens": {
204+
"type": "integer",
205+
"default": 500
206+
}
207+
},
208+
"additionalProperties": false,
209+
"default": {
210+
"temperature": 0,
211+
"top_p": 0.9,
212+
"max_tokens": 500
213+
},
214+
"description": "%configuration.completions.parameters.description%",
215+
"order": 4
216+
},
217+
"autodev.completions.fimSpecialTokens": {
218+
"type": "object",
219+
"description": "%configuration.completions.fimSpecialTokens.description%",
220+
"properties": {
221+
"prefix": {
222+
"type": "string",
223+
"description": "The prefix of the special token.",
224+
"default": "<|fim_prefix|>"
225+
},
226+
"suffix": {
227+
"type": "string",
228+
"description": "The suffix of the special token.",
229+
"default": "<|fim_suffix|>"
230+
},
231+
"middle": {
232+
"type": "string",
233+
"description": "The middle of the special token.",
234+
"default": "<|fim_middle|>"
235+
}
236+
},
237+
"additionalProperties": false,
238+
"default": {
239+
"prefix": "<|fim_prefix|>",
240+
"suffix": "<|fim_suffix|>",
241+
"middle": "<|fim_middle|>"
242+
},
243+
"order": 5
195244
},
196245
"autodev.completions.stops": {
197246
"type": "array",
198247
"items": {
199248
"type": "string"
200249
},
250+
"additionalProperties": false,
251+
"default": [],
201252
"description": "%configuration.completions.stops.description%",
202253
"order": 4
203254
},
255+
"autodev.completions.requestDelay": {
256+
"type": "integer",
257+
"default": 500,
258+
"markdownDescription": "%configuration.completions.requestDelay.markdownDescription%",
259+
"order": 6
260+
},
204261
"autodev.completions.enableLegacyMode": {
205262
"type": "boolean",
206263
"default": false,
207264
"description": "Use legacy \"/v1/completions\" instead of \"/v1/chat/completions\"",
208265
"markdownDescription": "%configuration.completions.enableLegacyMode.markdownDescription%",
209266
"order": 7
210267
},
211-
"autodev.completions.requestDelay": {
212-
"type": "integer",
213-
"default": 500,
214-
"markdownDescription": "%configuration.completions.requestDelay.markdownDescription%",
268+
"autodev.completions.template": {
269+
"type": "string",
270+
"description": "%configuration.completions.template.description%",
215271
"order": 8
216272
}
217273
},
@@ -545,7 +601,7 @@
545601
"title": "%command.explainCode.title%"
546602
},
547603
{
548-
"command": "autodev.codelens..optimizeCode",
604+
"command": "autodev.codelens.optimizeCode",
549605
"title": "%command.optimizeCode.title%"
550606
},
551607
{
@@ -830,7 +886,7 @@
830886
"when": "false"
831887
},
832888
{
833-
"command": "autodev.codelens..optimizeCode",
889+
"command": "autodev.codelens.optimizeCode",
834890
"when": "false"
835891
},
836892
{

0 commit comments

Comments
 (0)